LLMs Unplugged

Understand AI by building it yourself

Open with the promise: understand AI by building one yourself, by hand. No computers, no maths background needed.

Before you start — four habits that shape how the room experiences this.

Don’t apologise for explaining things. “Sorry, this bit’s a little dry”, “bear with me” — it tells the room the next few minutes are something to be endured, and they’ll oblige. The explanations are the reason they came. Deliver them like it.

Don’t call anything “the fun part”. It’s the same problem in reverse: naming one section fun says the others weren’t. Everything here is the good bit — the tallying, the dice, the arguing about AGI at the end. Say what’s coming next, not how they should feel about it.

Flip questions back before you answer them. There’s real expertise at the front of this room and it’s tempting to use all of it. But a clarifying question (“what draws you to that?”) or a turn to the room (“has anyone else hit this?”) usually gets somewhere more interesting, and it keeps them participants rather than an audience. Answer second, not first — you’ll still get to.

Let them experience it, then ask what came up. The activities do the teaching. After each one there’s a reflection slide; the temptation is to fill the silence with the point you wanted them to reach. Wait instead. Them saying “the training text really shapes what comes out” is worth far more than you saying it.

Acknowledgement of Country

activity

everyone stand up

sit down if you have

never used ChatGPT/Claude

sit down if you haven’t

used it in the last month

sit down if you haven’t

used it in the last week

sit down if you haven’t

used it in the last day

sit down if you haven’t

used it in the last hour

sit down if you haven’t

used it in the last 5 minutes

What is this about?

you’ll build your own language model—from scratch—with just a kids book, pen & paper, and some dice rolling

you’ll learn how language models work by spotting patterns in text to generate new text

llmsunplugged.org

Training

The recipe

walk through your text and tally up which tokens follow which in a grid

Example

“Hop, Joey, hop. See Joey hop.”

after tidying up:

hop , joey , hop . see joey hop .

The empty grid

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`
`,`
`joey`
`.`
`see`

Training: `hop` → `,`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|
`,`
`joey`
`.`
`see`

Training: `,` → `joey`

hop,joey,hop.seejoeyhop.

Token	`,`	`joey`
`hop`	\|
`,`		\|
`joey`
`.`
`see`

Training: `joey` → `,`

hop,joey,hop.seejoeyhop.

Token	`,`	`joey`
`hop`	\|
`,`		\|
`joey`	\|
`.`
`see`

Training: `,` → `hop`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`
`hop`		\|
`,`	\|		\|
`joey`		\|
`.`
`see`

Training: `hop` → `.`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`
`hop`		\|		\|
`,`	\|		\|
`joey`		\|
`.`
`see`

Complete model

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

A few more tips

you don’t have to start at the beginning—any page will do

the last token in one sentence is followed by the first token in the next (even over the page)

you can ignore quotation/speech marks (")

work in pairs and divide the labour however you like—but don’t neglect the tally marks

Your turn

tidy up your text, then for each pair of tokens:

find the row for the first token — no row yet? it gets a row and a column
find the column for the second token — same again if it’s new
add a tally mark in the cell where they meet
move along by one: the second token starts the next pair

10:00

The language of language models

model
token
weights

Generation

The recipe

use your grid to generate new text, rolling dice to choose each next word

Generation: start with `see`

see joey

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `joey`

seejoey

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

How the die chooses: `joey`

joey → ? roll a d10

01234

hop 1 tally

56789

, 1 tally

equal tallies → equal chances

How the die chooses: `joey`

joey → ? roll a d10

01234

hop 1 tally

56789

, 1 tally

equal tallies → equal chances

rolled 7 → ,

Generation: `joey` → `,`

seejoey ,

rolled 7 → ,

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `,`

seejoey,

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: `,` → `hop`

seejoey, hop

rolled 2 → hop

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `hop`

seejoey,hop

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

How the die chooses: `hop`

hop → ? roll a d10

012

, 1 tally

3456789

. 2 tallies

more tallies → more faces → more likely

How the die chooses: `hop`

hop → ? roll a d10

012

, 1 tally

3456789

. 2 tallies

more tallies → more faces → more likely

rolled 6 → .

Generation: `hop` → `.`

seejoey,hop .

rolled 6 → .

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `.`

seejoey,hop. see

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: back to `see`

seejoey,hop.see

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generated text

“see joey, hop. see”

a new sentence — not in the training data!

Your turn

write down a starting token, then repeat:

find the row for the last token you wrote
share the ten faces of the d10 across that row’s cells, by their tallies
roll — the band it lands in names the next token (one option? no roll)
write it down, and look up its row next

10:00

Shareback

The language of language models

prompt
completion

Scaling up

Now turn everything up

context how much it looks at before each guess

1 word 1 word101001 thousand10 thousand100 thousand1 million tokens

training text how much the model reads

one novel one novel100 thousand1 million10 million100 million1 billion10 billion100 billion1 trillion10 trillion15 trillion tokens

model size how many learned numbers it contains

49 million cells 49 million cells100 million1 billion10 billion100 billion~1 trillion parameters

one axis, in powers of ten · three independent dials

Ask before you explain, because this slide is where their most common complaint lives: hands up if you’ve told one of these things not to do something and it did it anyway. Take two answers. Then the honest account: your instruction didn’t fall off the end of a one-word window — it’s in the context, all of it, every time. It’s one more stretch of text competing with everything else in the window, weighted rather than obeyed: “it read your instruction and something else won”. What the model is looking at right now is all that shapes what comes next — that much their grid taught them straight.

Then the three dials, and let the sweep do the talking — every bar runs at the same pace per zero, so the longer one is the bigger jump. The axis is shared and it’s powers of ten: thousand, million, billion, trillion. Context: their grid looked at one word; Claude looks at up to a million tokens, the whole conversation and then some. Training text: one novel, about 84,000 words, against fifteen trillion tokens, the published figure for a recent open model’s training run — about 180 million copies of the novel, roughly 5.7 million years at the pace they just worked. Model size: their novel-grid has 49 million cells, and a frontier model has about a trillion parameters (the exact figure for the closed ones is unpublished, so “about that, and nobody outside will tell you” is the honest answer).

One sentence on why this isn’t just their grid, inflated, because somebody will ask. Their tool, run on that whole novel, fills 0.08% of the table — one cell in twelve hundred — and reading more makes it emptier, because every new word adds a whole row and column. A trillion-cell grid would be blank. A transformer’s numbers are shared instead: every pattern is smeared across billions of them, so a sentence about a cat helps with a sentence about a dog, and it says things nobody ever wrote. That’s why turning the dials up keeps working — make it bigger, feed it more, let it look at more, and it gets better in a way you can measure, predictably enough to plan a training run around. Nobody fully knows why that holds, and that regularity, more than any single clever idea, is why the last five years of AI happened. Each dial has a lesson on the website.

“Explain the moon landing to a 6 year old in a few sentences.” pre-trained Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old. post-trained People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them.

same model, same next-word loop—post-training is a second round of training on examples of instructions being followed

The dials are done, and this is deliberately not a fourth — it isn’t about how much, it’s about which text. Say that as you arrive: “and one thing that isn’t a matter of size at all”.

No heading — read the prompt out, then the first reply, and let the room get there. That’s GPT-3, straight out of pre-training, and it’s doing exactly what their grid does: continuing the text. Where on the internet does “explain X to a 6 year old” turn up? In lists of homework prompts and writing exercises — so the likeliest thing to come next is another one. It isn’t broken or being unhelpful; nothing in “predict what comes next” has any notion of a question and an answer.

Then the second reply. That’s the same model after post-training (OpenAI published this pair in January 2022, as InstructGPT — the step just before ChatGPT). They trained it a bit more on examples of people writing out good responses to instructions, then had people rank its attempts and nudged it towards the ones they preferred. Same predict-the-next-word loop, same dice — just more text to learn from, of a particular kind, so what tends to follow an instruction is now a response to it. Every chatbot they use has been through this second round; pre-training gives it the language, post-training gives it the habit of answering.

One aside, if the room is with you and you’re not behind. Having people rank its answers is where sycophancy comes from: ask who’s had one of these cave the moment they pushed back, agreeing when they were the one who had it wrong. Raters, faced with two replies, tended to prefer the agreeable one, so agreeing got reinforced — and nowhere in any of this is a fact marked true or false. Agreement from one of these is not evidence you were right. It’s a good moment, but it’s a second point; the slide stands without it.

still just tokens in → tokens out

Questions

is there a term you’ve heard—that we haven’t covered?

what new questions do you have about large language models?

how will this change the way you think about and use LLMs in the future?

Three questions, revealed one at a time, easiest first. Don’t put them all up at once — the third one lands differently after the room has already been talking.

Open on the terms question. It’s the lowest-stakes way into the Q&A: naming a word you’ve heard is easier than forming a question, and it surfaces what they’ve actually been reading. Have prompts ready if the room is slow — data centre, the stack, transformer, agent, AGI — and expect AGI, which is worth making room for rather than batting away.

When a term comes up, put it back to the room before you answer it: “does anyone else know this one, or have a sense of what it means?” You get to hear what they think it means, which is usually the more interesting problem, and it keeps the session from collapsing into expert-at-the-front. Then give your answer.

Same move when a question is really an opinion in disguise (is this taking jobs, is the rollout being handled badly, is any of this worth it). A clarifying question first — what draws you to that, is it a particular industry — gets you a much better conversation than a confident answer to the wrong question. You can still give the technical view; give it second.

Bring up the second question once terms thin out, and the third only when the questions are genuinely running down. That last one turns the session outward, from what the grid does to what they’ll do differently on Monday, and it’s the closest thing this workshop has to a point.

anu.au1.qualtrics.com/jfe/form/SV_cA2ioVFWoQRj0KW

Next sessions

Wednesday 25 November 16:00–18:00

Innovation Space, Birch Building, ANU

No public sessions are scheduled right now — get in touch to arrange one.

events.humanitix.com/host/anu-cecc-school-of-cybernetics

LLMs Unplugged#

Understand AI by building it yourself#

Acknowledgement of Country#

What is this about?#

Training#

The recipe#

Example#

The empty grid#

Training: hop → ,#

Training: , → joey#

Training: joey → ,#

Training: , → hop#

Training: hop → .#

Complete model#

A few more tips#

Your turn#

The language of language models#

Generation#

The recipe#

Generation: start with see#

Generation: from joey#

How the die chooses: joey#

How the die chooses: joey#

Generation: joey → ,#

Generation: from ,#

Generation: , → hop#

Generation: from hop#

How the die chooses: hop#

How the die chooses: hop#

Generation: hop → .#

Generation: from .#

Generation: back to see#

Generated text#

Your turn#

Shareback#

The language of language models#

Scaling up#

Now turn everything up#

Questions#

Next sessions

We’re here to help#

LLMs Unplugged

Understand AI by building it yourself

Acknowledgement of Country

What is this about?

Training

The recipe

Example

The empty grid

Training: `hop` → `,`

Training: `,` → `joey`

Training: `joey` → `,`

Training: `,` → `hop`

Training: `hop` → `.`

Complete model

A few more tips

Your turn

The language of language models

Generation

The recipe

Generation: start with `see`

Generation: from `joey`

How the die chooses: `joey`

How the die chooses: `joey`

Generation: `joey` → `,`

Generation: from `,`

Generation: `,` → `hop`

Generation: from `hop`

How the die chooses: `hop`

How the die chooses: `hop`

Generation: `hop` → `.`

Generation: from `.`

Generation: back to `see`

Generated text

Your turn

Shareback

The language of language models

Scaling up

Now turn everything up

Questions

We’re here to help