Under the hood
Once a room has built a bigram model and rolled dice through it, the question comes up on its own: this thing only ever looks at the word before, so how does ChatGPT hold a conversation? There are three answers. A real model looks at more than one word of context, it can say things it never saw in training, and it has been taught to answer rather than merely continue.
This lesson takes the first two and runs them by hand. Participants build the grid model as usual, then add two rules that read back over the page they’ve already written (in-context learning), and finally turn each word’s row of counts into a vector and measure how close two words are (word embeddings). No new maths beyond counting and subtraction, and no code, but by the end the words attention and embedding mean something concrete rather than something you’ve heard on a podcast.
Run of show
| Section | Time |
|---|---|
| Icebreaker and intro | 10 min |
| Training | 20 min |
| Generation | 20 min |
| Pre-trained Model Generation | 15 min |
| Break | 10 min |
| In-context Learning | 25 min |
| Word Embeddings | 20 min |
| Close: scaling up | 10 min |
That’s two hours and ten minutes. If your slot is a hard two hours, run the pre-trained booklet round at 10 minutes or shorten the break; don’t take the time out of in-context learning.
There’s no deck for this lesson. Run it straight from the module pages: each one has the goal, the algorithm, a worked example and instructor notes, and the two in the back half carry the widgets you’ll want on the projector.
How it goes
The first hour is the fundamentals, and it’s the same first hour as My First Language Model: tally word pairs into the grid, then roll a d10 to generate text from your own counts. If your group has already done that, you can compress it hard; 20 minutes of rebuild is usually enough to get everyone’s hands back in. Don’t skip it, though, because the back half needs a filled-in grid to work on.
The pre-trained booklet round is here for a practical reason as well as a conceptual one. The in-context rules only bite when the model has a decent vocabulary behind it, and a booklet trained on a whole book gives a much better demo than a grid built from two pages. Hand out booklets, let people generate a few sentences, and keep them on the table for the next section.
In-context learning is the centrepiece. Start with the two-minute demonstration: write a short pattern on the board and ask the room what comes next. The model has never seen that pattern, so they find the answer by scanning back up the page. That’s the induction rule. Then add the memory rule, which softly favours words used recently, and generate again with both rules running. The output stays on topic in a way the plain bigram never manages, and nobody has changed a single count.
Word embeddings comes last, and it’s about meaning. Each word’s row in the grid is its vector; two words with similar rows behave similarly. Run the twins activity (find two words whose rows look alike), then the swap test (substitute one for the other in generated text and see whether anyone notices), then put the Embedding widget on the projector and let the room argue with the numbers.
The close is the scaling-up conversation: what changes when the grid has fifty thousand words instead of forty, when the context is thousands of words instead of one, and when the rules you just ran by hand are learned rather than imposed.
You will need
For each pair or small group:
- a printed grid template (or grid paper), plus a second blank one for the embedding distances
- a d10 die
- pens, and plenty of paper—the in-context rules mean reading back over what’s already been generated, so keep it legible
- some training text: a few pages of any book works
- a pre-trained booklet from the Materials page, or generate your own
Plus a laptop and projector for the Embedding widget on the Word Embeddings page.
Before you deliver
- print the grids (two per group) and the booklets, and check the booklets are bound or stapled the right way round—people flip through them a lot in this lesson
- have your induction demo ready: a three-word pattern the room can complete instantly, planted so the current word appears earlier in the generated text
- work the memory rule once yourself on a real booklet before the session. It’s simple, but explaining it cold is harder than it looks
- open the Embedding widget on the room’s computer beforehand and check it renders on the projector; the numbers are small
- got extra time? More Context (30 min) slots in after generation and pairs well with in-context learning—the trigram shows what a bigger context window costs, right before the two rules show how to get context for free
Where to go next
If the mechanism is what your group is into, this is the deeper of the two follow-ons; Shaping a model is the other, and takes the third answer, about models taught to answer rather than continue, into training data, human feedback and the ethics that come with them. Groups that want to see a model act as well as generate should add Agentic AI. And if you’re presenting to a room too big for tables and dice, the talks run the same core idea from a stage.