LLMs Unplugged

Does ‘Unplugged’ still matter in the age of AI?

Everyday life is increasingly regulated by complex technologies that most people neither understand nor believe they can do much to influence.
Albert Bandura, 2001

what we get told about AI

it understands you—it’s basically a mind

it’s just autocomplete—it’s basically a parlour trick

it’s hard to have an opinion
on a machine you’ve never seen
the inside of

what you’ve just been given

a list of token pairs (bigrams) from The Cat in the Hat

(it’s shuffled, so your page isn’t just a book excerpt)

between us, we have the whole book

1,962 words · 240 distinct · 1,961 token pairs dealt round the room

no single sheet has the whole picture

can I have a volunteer?

the algorithm

call out the last word we wrote (w/colour)
hands up if it appears as the boxed word on your sheet
pick a hand at random and you read out the corresponding next (non-boxed) word
write it down—and then repeat the process

let’s run it

5:00

Whiteboard is already up with the volunteer’s two words. Call out the second of them and go. Five minutes on the timer, so keep the rounds moving — no commentary between hands going up and a word landing on the board.

If you need to restart, the loud seeds are the (97 pairs out in the room across 30 different next words), then and (69), I (59), you (34), we (26). Avoid restarting from a full stop; it works, but the first thing the room does is argue about punctuation.

Two things to narrate while it runs, once each, without stopping the flow:

The hands are the distribution. Say the and a forest goes up; say a rare word and you get two. Nobody in this room has been taught weighted sampling and the room is doing it anyway — the number of hands for a continuation is how often it happens in the book. (Pedant’s footnote, only if a pedant asks: for contexts more common than the room is big, somebody holds two matches and raises one hand, which flattens the busiest contexts slightly.)

The choosing is a coin toss. When five hands are up, the four I don’t pick were all real continuations of the book. Different pick, different sentence. This is the same reason the “regenerate” button gives you a different answer.

Hands will go up every round — you can rely on that. Every distinct token in this book appears as a boxed word somewhere, so there is no word you can call that has nothing following it. (True of every corpus in the repo, not just this one.) So don’t plan a “running dry” beat; there isn’t one.

The one way the room goes quiet is an empty seat: that person’s sheet went home with them, and the pairs on it are simply gone. If it happens, say exactly that — it’s the same point as slide 6, made for you — and reseed.

Stop while they still want more, around 15 words. Read the sentence back.

we just ran a language model

what role did you play?

each word on your sheet → a token

your sheets → the parameters/weights of the model

the hands in the air → the probability distribution

the random pick → sampling

the sentence on the board → generation

The jargon that gets used to make this stuff sound unreachable, and you’ve now got a physical referent for every bit of it. Go along the list at a fair clip — the point is the mapping, not a definition of each term.

Token they’ve already met, on the sheet and back on slide 6; this is where it sticks, because everything else is measured in them. One unit of text — for us one word or one punctuation mark, which is why the full stops are out there getting picked like everything else. (A real model chops rarer words into pieces, which is why it can spell out a word it has never seen. Say it only if it comes up.) Parameters and weights are the same thing under two names; you’ll hear both.

Worth saying out loud: those are the real words. When a researcher says a model has 1.5 trillion parameters, they mean this, a very large pile of the thing on your lap.

the sentence we wrote

is not in the book

the same model in grid form

lookatme!lookatme!lookatmenow!

Token	`look`	`at`	`me`	`!`	`now`
`look`		\|\|\|
`at`			\|\|\|
`me`				\|\|	\|
`!`	\|\|
`now`				\|

three things Claude* does that we didn’t

it looks at more than one word of context

it can say what it never saw—cat teaches it about dog

it’s been taught to answer, not just continue

The asterisk is yours to explain, in one line: Claude here means any frontier model — ChatGPT, Gemini, Llama. Nothing on this slide is specific to Anthropic’s; I just need a name to point at.

Say each in a sentence, resist explaining any of them; each one has a full lesson on the website and none of them fits in this talk.

One word of context is why our sentence wandered — when I called the, nobody could tell whether we’d just written “sat in the” or “hit the”. Claude looks back over the whole conversation.

Our model rates any pair it never saw as flatly impossible, however sensible. A real model spreads what it learns across billions of numbers instead of separate cells, so what it learns about cat reaches dog.

And ours only ever continues text. Ask it a question and it continues the question — that’s not it refusing, answering just isn’t a thing it does. Turning that into something that answers you is another round of training, on conversations.

None of these changed the loop. Look at the context, weigh the options, pick one, write it down.

context how much it looks at before each guess

1 token the whole conversation

training text how much the model reads

1,962 tokens 15 trillion tokens

model size how many learned numbers it contains

57,600 grid cells ~1 trillion parameters

three independent scales · different kinds of model

Three independent scales grow. The first is how much the model looks at before each guess. Ours looked at exactly one token — which is why our sentence wandered — and Claude looks back over the whole conversation.

The second is how much text went past the counter. Ours read 1,962 tokens; fifteen trillion is the published figure for a recent open model’s training run. Every one of those tokens went past a counter, the way the book went past yours.

The third is how many learned numbers the model contains. Ours had a 240 by 240 grid: 57,600 places a count could live. A trillion parameters is the right order of magnitude for a frontier model, and the honest answer for the closed ones is “about that, and nobody outside will tell you”.

The bars have separate scales because these quantities can be changed independently. And the endpoints are different kinds of model: grid cells on the left, neural-network parameters on the right. This is a scale comparison, not a claim that one is a compressed version of the other.

much more context

much more training text

many more learned numbers

a different kind of model

still just words in → words out

back to Bandura

most people neither understand nor believe they can do much to influence

a room full of people just did, slowly, the thing you’re told needs a data centre

it doesn’t make you an expert

it makes you harder to bullsh*t

Next sessions

Wednesday 16 September 12:00–14:00
Wednesday 25 November 16:00–18:00

Innovation Space, Birch Building, ANU

No public sessions are scheduled right now — get in touch to arrange one.

llmsunplugged.org

LLMs Unplugged#

Does ‘Unplugged’ still matter in the age of AI?#

what we get told about AI#

what you’ve just been given#

between us, we have the whole book#

the algorithm#

let’s run it#

what role did you play?#

the same model in grid form#

three things Claude* does that we didn’t#

back to Bandura#