Under the hood
Once a room has built a Bigram model A model that predicts the next word from one previous word. It's what you build in the fundamental modules: each row of your grid says what can follow a single word. View in glossary and rolled dice through it, the question comes up on its own: this thing only ever looks at the word before, so how does Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary hold a conversation? There are three answers. A real model looks at more than one word of Context window How many previous tokens the model can consider when it predicts the next one. A bigram has a context window of one word and a trigram of two; modern LLMs can consider hundreds of thousands or even millions of tokens. View in glossary , it can say things it never saw in training, and it has been taught to answer rather than merely continue.
This lesson takes the first two and runs them by hand. Participants build the grid model as usual, then add two rules that read back over the page they’ve already written ( In-context learning Picking up a pattern from the prompt and continuing it, without any change to the model's parameters. The "learning" lives in the context, not in the model. It's why few-shot prompting works: give an LLM a handful of examples and it carries on in the same shape. View in glossary ), and finally turn each word’s row of counts into a vector and measure how close two words are (word Embedding A word represented as a list of numbers. A row of your bigram grid is that word's embedding, a fingerprint of the company it keeps. In a real LLM the numbers are learned rather than counted, and computed from the sentence the word appears in, so the two senses of "bank" get different vectors. The principle survives: words used alike get similar vectors. View in glossary ). No new maths beyond counting and subtraction, and no code, but by the end the words Attention mechanism The part of a transformer that decides, for each prediction, which earlier words in the context to draw on and how much. The skip grid in More Context and the memory boost in In-context Learning are fixed, hand-run versions of the same move; real attention is learned and changes with the content. View in glossary and mean something concrete rather than something you’ve heard on a podcast.
Run of show
| Section | Time |
|---|---|
| Icebreaker and intro | 10 min |
| Training | 20 min |
| Generation | 20 min |
| Pre-trained Model Generation | 15 min |
| Break | 10 min |
| In-context Learning | 25 min |
| Word Embeddings | 20 min |
| Close: scaling up | 10 min |
That’s two hours and ten minutes. If your slot is a hard two hours, run the pre-trained booklet round at 10 minutes or shorten the break; don’t take the time out of in-context learning.
There’s no deck for this lesson. Run it straight from the module pages: each one has the goal, the algorithm, a worked example and instructor notes, and the two in the back half carry the widgets you’ll want on the projector.
How it goes
The first hour is the fundamentals, and it’s the same first hour as My First Language Model: tally word pairs into the Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary , then roll a d10 to generate text from your own counts. If your group has already done that, you can compress it hard; 20 minutes of rebuild is usually enough to get everyone’s hands back in. Don’t skip it, though, because the back half needs a filled-in grid to work on.
The pre-trained Booklet A printed dice lookup table for a text: one entry per previous word (or pair of words, in a trigram booklet), listing the possible next words with the d10 range that selects each. A booklet is a pre-trained model you can generate from without having trained it. View in glossary round is here for a practical reason as well as a conceptual one. The in-context rules only bite when the model has a decent vocabulary behind it, and a booklet trained on a whole book gives a much better demo than a grid built from two pages. Hand out booklets, let people generate a few sentences, and keep them on the table for the next section.
In-context learning is the centrepiece. Start with the two-minute demonstration: write a short pattern on the board and ask the room what comes next. The model has never seen that pattern, so they find the answer by scanning back up the page. That’s the induction rule. Then add the memory rule, which softly favours words used recently, and generate again with both rules running. The output stays on topic in a way the plain bigram never manages, and nobody has changed a single count.
Word embeddings comes last, and it’s about meaning. Each word’s row in the grid is its vector; two words with similar rows behave similarly. Run the twins activity (find two words whose rows look alike), then the swap test (substitute one for the other in generated text and see whether anyone notices), then put the Embedding widget on the projector and let the room argue with the numbers.
The close is the scaling-up conversation: what changes when the grid has fifty thousand words instead of forty, when the context is thousands of words instead of one, and when the rules you just ran by hand are learned rather than imposed.
You will need
For each pair or small group:
- a printed grid template (or grid paper), plus a second blank one for the embedding distances
- a d10 die
- pens, and plenty of paper—the in-context rules mean reading back over what’s already been generated, so keep it legible
- some training text: a few pages of any book works
- a pre-trained booklet from the Materials page, or generate your own
Plus a laptop and projector for the Embedding widget on the Word Embeddings page.
Before you deliver
- print the grids (two per group) and the booklets, and check the booklets are bound or stapled the right way round—people flip through them a lot in this lesson
- have your induction demo ready: a three-word pattern the room can complete instantly, planted so the current word appears earlier in the generated text
- work the memory rule once yourself on a real booklet before the session. It’s simple, but explaining it cold is harder than it looks
- open the Embedding widget on the room’s computer beforehand and check it renders on the projector; the numbers are small
- got extra time? More Context (30 min) slots in after generation and pairs well with in-context learning—the trigram shows what a bigger context window costs, right before the two rules show how to get context for free
Where to go next
If the mechanism is what your group is into, this is the deeper of the two follow-ons; Shaping a model is the other, and takes the third answer, about models taught to answer rather than continue, into training data, human feedback and the ethics that come with them. Groups that want to see a model act as well as generate should add Agentic AI. And if you’re presenting to a room too big for tables and dice, the talks run the same core idea from a stage.