LLMs Unplugged

Understand AI by building it yourself

Open with the promise: understand AI by building one yourself, by hand. No computers, no maths background needed.

Before you start — four habits that shape how the room experiences this.

Don’t apologise for explaining things. “Sorry, this bit’s a little dry”, “bear with me” — it tells the room the next few minutes are something to be endured, and they’ll oblige. The explanations are the reason they came. Deliver them like it.

Don’t call anything “the fun part”. It’s the same problem in reverse: naming one section fun says the others weren’t. Everything here is the good bit — the tallying, the dice, the arguing about AGI at the end. Say what’s coming next, not how they should feel about it.

Flip questions back before you answer them. There’s real expertise at the front of this room and it’s tempting to use all of it. But a clarifying question (“what draws you to that?”) or a turn to the room (“has anyone else hit this?”) usually gets somewhere more interesting, and it keeps them participants rather than an audience. Answer second, not first — you’ll still get to.

Let them experience it, then ask what came up. The activities do the teaching. After each one there’s a reflection slide; the temptation is to fill the silence with the point you wanted them to reach. Wait instead. Them saying “the training text really shapes what comes out” is worth far more than you saying it.

Acknowledgement of Country

activity

everyone stand up

sit down if you have

never used ChatGPT/Claude

sit down if you haven’t

used it in the last month

sit down if you haven’t

used it in the last week

sit down if you haven’t

used it in the last day

sit down if you haven’t

used it in the last hour

sit down if you haven’t

used it in the last 5 minutes

What is this about?

you’ll build your own language model—from scratch—with just a kids book, pen & paper, and some dice rolling

you’ll learn how language models work by spotting patterns in text to generate new text

llmsunplugged.org

Training

The recipe

walk through your text and tally up which tokens follow which in a grid

Example

“Hop, Joey, hop. See Joey hop.”

after tidying up:

hop , joey , hop . see joey hop .

The empty grid

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`
`,`
`joey`
`.`
`see`

Training: `hop` → `,`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|
`,`
`joey`
`.`
`see`

Training: `,` → `joey`

hop,joey,hop.seejoeyhop.

Token	`,`	`joey`
`hop`	\|
`,`		\|
`joey`
`.`
`see`

Training: `joey` → `,`

hop,joey,hop.seejoeyhop.

Token	`,`	`joey`
`hop`	\|
`,`		\|
`joey`	\|
`.`
`see`

Training: `,` → `hop`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`
`hop`		\|
`,`	\|		\|
`joey`		\|
`.`
`see`

Training: `hop` → `.`

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`
`hop`		\|		\|
`,`	\|		\|
`joey`		\|
`.`
`see`

Complete model

hop,joey,hop.seejoeyhop.

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

A few more tips

you don’t have to start at the beginning—any page will do

the last token in one sentence is followed by the first token in the next (even over the page)

you can ignore quotation/speech marks (")

work in pairs and divide the labour however you like—but don’t neglect the tally marks

Your turn

tidy up your text, then for each pair of tokens:

find the row for the first token — no row yet? it gets a row and a column
find the column for the second token — same again if it’s new
add a tally mark in the cell where they meet
move along by one: the second token starts the next pair

10:00

The language of language models

model
token
weights

Generation

The recipe

use your grid to generate new text, rolling dice to choose each next word

Generation: start with `see`

see joey

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `joey`

seejoey

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

How the die chooses: `joey`

joey → ? roll a d10

01234

hop 1 tally

56789

, 1 tally

equal tallies → equal chances

How the die chooses: `joey`

joey → ? roll a d10

01234

hop 1 tally

56789

, 1 tally

equal tallies → equal chances

rolled 7 → ,

Generation: `joey` → `,`

seejoey ,

rolled 7 → ,

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `,`

seejoey,

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: `,` → `hop`

seejoey, hop

rolled 2 → hop

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `hop`

seejoey,hop

2 options — roll the die!

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

How the die chooses: `hop`

hop → ? roll a d10

012

, 1 tally

3456789

. 2 tallies

more tallies → more faces → more likely

How the die chooses: `hop`

hop → ? roll a d10

012

, 1 tally

3456789

. 2 tallies

more tallies → more faces → more likely

rolled 6 → .

Generation: `hop` → `.`

seejoey,hop .

rolled 6 → .

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: from `.`

seejoey,hop. see

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generation: back to `see`

seejoey,hop.see

one option — no roll needed

Token	`hop`	`,`	`joey`	`.`	`see`
`hop`		\|		\|\|
`,`	\|		\|
`joey`	\|	\|
`.`					\|
`see`			\|

Generated text

“see joey, hop. see”

a new sentence — not in the training data!

Your turn

write down a starting token, then repeat:

find the row for the last token you wrote
share the ten faces of the d10 across that row’s cells, by their tallies
roll — the band it lands in names the next token (one option? no roll)
write it down, and look up its row next

10:00

Shareback

The language of language models

prompt
completion

Pre-trained generation

Why do it again?

same recipe, more training text: yours learnt from a few sentences, this one learnt from a whole novel

first step on the road to the very big ones

Don’t skip this — from where they’re sitting, the next twenty minutes look like the exercise they just did, a second time, and a section that feels like a repeat gets a room that’s already worked hard for forty minutes checking their phones.

The frame is the journey. Their grid, then a novel, then Frankenstein drawn at true scale in the scaling-up section, then a frontier model. Everything from here is the same loop with the dial turned up, and this is the first turn of it — the point of doing it by hand is that they can feel what changed and what didn’t.

Worth flagging what to watch for, so they read their output rather than just producing it: it’ll sound more like real prose, and it still won’t be about anything. That’s the question to leave hanging at the shareback.

The booklet’s stretch goal is a good hook if the energy needs it: try to guess the training text before you check the title on the cover.

The recipe

use the booklet to generate new text, rolling one or more d10s to choose each next word

Pre-trained: start with `sleep`

sleep again

sleep ♦ 2|,6|.7|again8|as9|with rolled 7

again ♦♦ 45|.61|,68|and76|as84|I'd91|made99|over

I'd ♦ 4|like7|float9|take

float ;

Pre-trained: from `again`

sleepagain I'd

sleep ♦ 2|,6|.7|again8|as9|with

again ♦♦ 45|.61|,68|and76|as84|I'd91|made99|over rolled 80

I'd ♦ 4|like7|float9|take

float ;

Pre-trained: from `I'd`

sleepagainI'd float

sleep ♦ 2|,6|.7|again8|as9|with

again ♦♦ 45|.61|,68|and76|as84|I'd91|made99|over

I'd ♦ 4|like7|float9|take rolled 6

float ;

Pre-trained: from `float`

sleepagainI'dfloat ;

sleep ♦ 2|,6|.7|again8|as9|with

again ♦♦ 45|.61|,68|and76|as84|I'd91|made99|over

I'd ♦ 4|like7|float9|take

float ;

Pre-trained: from `;`

sleepagainI'dfloat;

sleep ♦ 2|,6|.7|again8|as9|with

again ♦♦ 45|.61|,68|and76|as84|I'd91|made99|over

I'd ♦ 4|like7|float9|take

float ;

Your turn

same loop, from the booklet:

look up the last word you wrote — the bold words are the entries
read its numbers: one beside each possible next word
roll a d10 — two if the entry runs past 9, read as tens then units
take the first word your roll doesn’t go past, and look that one up next

8:00

Shareback

The language of language models

pre-trained model
open weight model

Everyday sense first, then ours: the weights are the numbers inside the model — for the booklet, the tallies you’d count to work out what comes next — and open weight means anyone can have a copy of those numbers. The booklet is the clearest version of that we’ll get: take it home, read every number in it, generate from it at the kitchen table, and no-one can switch it off or quietly change it under you. Most of the chatbots they use are the other arrangement — your text goes to someone else’s computer, an answer comes back, and the weights never leave the building. If anyone pushes: open weight isn’t open source (you get the numbers, not the training text or the code that made them), and it isn’t the same as being able to run it — a real open weight model wants a serious machine, where the booklet wants a d10.

Agentic AI

The recipe

an agent is a model that can call tools: pause generation, get information from “outside” the model, and continue

generate from your model as before, but every punctuation token triggers a tool call: a real text message to 3 friends/group chats

Worked example

your text so far is “the cat sat”, and the next dice roll gives you . as the next token—pause, that’s a tool call

text What comes next? "the cat sat..." to 3 friends/group chats; the first reply back might be “out on the verandah”

write out on the verandah, then the . you rolled anyway, and continue generating from .

Your turn

generate as before, but every punctuation token is a tool call:

pause when you roll a . or a ,
text What comes next? "[sentence so far]..." to 3 friends or group chats
write down the whole first reply, then the punctuation token you rolled
carry on generating from that punctuation token

10:00

Shareback

What did the tool calls add that the bare model couldn’t — fresh, outside information? Did the friends’ words fit the text, or derail it? Invite ~5 groups to share back rather than the whole room. ~5 min.

Worth pushing on if the room is up for it: they have just been the harness — they decided when to pause, who to ask, and what to splice back in. Take that away and you have an agent acting on its own, which is most of what “agentic AI” means in the coverage they’re reading. So what would you be comfortable letting it do without asking you first? Answers get specific fast — booking something, sending an email on their behalf, moving money, writing to a system of record — and the interesting part is usually not the task but where they draw the line and why: reversibility, who gets blamed, whether they’d notice a mistake. Don’t resolve it; a tidy conclusion would waste the question.

The language of language models

agent/agentic loop
tool call

Scaling up

Now turn everything up

context how much it looks at before each guess

1 word 1 word101001 thousand10 thousand100 thousand1 million tokens

training text how much the model reads

one novel one novel100 thousand1 million10 million100 million1 billion10 billion100 billion1 trillion10 trillion15 trillion tokens

model size how many learned numbers it contains

49 million cells 49 million cells100 million1 billion10 billion100 billion~1 trillion parameters

one axis, in powers of ten · three independent dials

Ask before you explain, because this slide is where their most common complaint lives: hands up if you’ve told one of these things not to do something and it did it anyway. Take two answers. Then the honest account: your instruction didn’t fall off the end of a one-word window — it’s in the context, all of it, every time. It’s one more stretch of text competing with everything else in the window, weighted rather than obeyed: “it read your instruction and something else won”. What the model is looking at right now is all that shapes what comes next — that much their grid taught them straight.

Then the three dials, and let the sweep do the talking — every bar runs at the same pace per zero, so the longer one is the bigger jump. The axis is shared and it’s powers of ten: thousand, million, billion, trillion. Context: their grid looked at one word; Claude looks at up to a million tokens, the whole conversation and then some. Training text: one novel, about 84,000 words, against fifteen trillion tokens, the published figure for a recent open model’s training run — about 180 million copies of the novel, roughly 5.7 million years at the pace they just worked. Model size: their novel-grid has 49 million cells, and a frontier model has about a trillion parameters (the exact figure for the closed ones is unpublished, so “about that, and nobody outside will tell you” is the honest answer).

One sentence on why this isn’t just their grid, inflated, because somebody will ask. Their tool, run on that whole novel, fills 0.08% of the table — one cell in twelve hundred — and reading more makes it emptier, because every new word adds a whole row and column. A trillion-cell grid would be blank. A transformer’s numbers are shared instead: every pattern is smeared across billions of them, so a sentence about a cat helps with a sentence about a dog, and it says things nobody ever wrote. That’s why turning the dials up keeps working — make it bigger, feed it more, let it look at more, and it gets better in a way you can measure, predictably enough to plan a training run around. Nobody fully knows why that holds, and that regularity, more than any single clever idea, is why the last five years of AI happened. Each dial has a lesson on the website.

“Explain the moon landing to a 6 year old in a few sentences.” pre-trained Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old. post-trained People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them.

same model, same next-word loop—post-training is a second round of training on examples of instructions being followed

The dials are done, and this is deliberately not a fourth — it isn’t about how much, it’s about which text. Say that as you arrive: “and one thing that isn’t a matter of size at all”.

No heading — read the prompt out, then the first reply, and let the room get there. That’s GPT-3, straight out of pre-training, and it’s doing exactly what their grid does: continuing the text. Where on the internet does “explain X to a 6 year old” turn up? In lists of homework prompts and writing exercises — so the likeliest thing to come next is another one. It isn’t broken or being unhelpful; nothing in “predict what comes next” has any notion of a question and an answer.

Then the second reply. That’s the same model after post-training (OpenAI published this pair in January 2022, as InstructGPT — the step just before ChatGPT). They trained it a bit more on examples of people writing out good responses to instructions, then had people rank its attempts and nudged it towards the ones they preferred. Same predict-the-next-word loop, same dice — just more text to learn from, of a particular kind, so what tends to follow an instruction is now a response to it. Every chatbot they use has been through this second round; pre-training gives it the language, post-training gives it the habit of answering.

One aside, if the room is with you and you’re not behind. Having people rank its answers is where sycophancy comes from: ask who’s had one of these cave the moment they pushed back, agreeing when they were the one who had it wrong. Raters, faced with two replies, tended to prefer the agreeable one, so agreeing got reinforced — and nowhere in any of this is a fact marked true or false. Agreement from one of these is not evidence you were right. It’s a good moment, but it’s a second point; the slide stands without it.

still just tokens in → tokens out

Questions

is there a term you’ve heard—that we haven’t covered?

what new questions do you have about large language models?

how will this change the way you think about and use LLMs in the future?

Three questions, revealed one at a time, easiest first. Don’t put them all up at once — the third one lands differently after the room has already been talking.

Open on the terms question. It’s the lowest-stakes way into the Q&A: naming a word you’ve heard is easier than forming a question, and it surfaces what they’ve actually been reading. Have prompts ready if the room is slow — data centre, the stack, transformer, agent, AGI — and expect AGI, which is worth making room for rather than batting away.

When a term comes up, put it back to the room before you answer it: “does anyone else know this one, or have a sense of what it means?” You get to hear what they think it means, which is usually the more interesting problem, and it keeps the session from collapsing into expert-at-the-front. Then give your answer.

Same move when a question is really an opinion in disguise (is this taking jobs, is the rollout being handled badly, is any of this worth it). A clarifying question first — what draws you to that, is it a particular industry — gets you a much better conversation than a confident answer to the wrong question. You can still give the technical view; give it second.

Bring up the second question once terms thin out, and the third only when the questions are genuinely running down. That last one turns the session outward, from what the grid does to what they’ll do differently on Monday, and it’s the closest thing this workshop has to a point.

anu.au1.qualtrics.com/jfe/form/SV_cA2ioVFWoQRj0KW

Next sessions

Wednesday 25 November 16:00–18:00

Innovation Space, Birch Building, ANU

No public sessions are scheduled right now — get in touch to arrange one.

events.humanitix.com/host/anu-cecc-school-of-cybernetics

LLMs Unplugged#

Understand AI by building it yourself#

Acknowledgement of Country#

What is this about?#

Training#

The recipe#

Example#

The empty grid#

Training: hop → ,#

Training: , → joey#

Training: joey → ,#

Training: , → hop#

Training: hop → .#

Complete model#

A few more tips#

Your turn#

The language of language models#

Generation#

The recipe#

Generation: start with see#

Generation: from joey#

How the die chooses: joey#

How the die chooses: joey#

Generation: joey → ,#

Generation: from ,#

Generation: , → hop#

Generation: from hop#

How the die chooses: hop#

How the die chooses: hop#

Generation: hop → .#

Generation: from .#

Generation: back to see#

Generated text#

Your turn#

Shareback#

The language of language models#

Pre-trained generation#

Why do it again?#

The recipe#

Pre-trained: start with sleep#

Pre-trained: from again#

Pre-trained: from I'd#

Pre-trained: from float#

Pre-trained: from ;#

Your turn#

Shareback#

The language of language models#

Agentic AI#

The recipe#

Worked example#

Your turn#

Shareback#

The language of language models#

Scaling up#

Now turn everything up#

Questions#

Next sessions

We’re here to help#

LLMs Unplugged

Understand AI by building it yourself

Acknowledgement of Country

What is this about?

Training

The recipe

Example

The empty grid

Training: `hop` → `,`

Training: `,` → `joey`

Training: `joey` → `,`

Training: `,` → `hop`

Training: `hop` → `.`

Complete model

A few more tips

Your turn

The language of language models

Generation

The recipe

Generation: start with `see`

Generation: from `joey`

How the die chooses: `joey`

How the die chooses: `joey`

Generation: `joey` → `,`

Generation: from `,`

Generation: `,` → `hop`

Generation: from `hop`

How the die chooses: `hop`

How the die chooses: `hop`

Generation: `hop` → `.`

Generation: from `.`

Generation: back to `see`

Generated text

Your turn

Shareback

The language of language models

Pre-trained generation

Why do it again?

The recipe

Pre-trained: start with `sleep`

Pre-trained: from `again`

Pre-trained: from `I'd`

Pre-trained: from `float`

Pre-trained: from `;`

Your turn

Shareback

The language of language models

Agentic AI

The recipe

Worked example

Your turn

Shareback

The language of language models

Scaling up

Now turn everything up

Questions

We’re here to help