More Context

• Audience years 7 and up
• Duration 20-30 min
• Runs on booklet, grid

Key idea: More context sharpens predictions, but the model blows up as you add it, so we also learn a cheaper way to pull in an earlier word.

A bigram model Bigram model A model that predicts the next word based on one previous word. This is what you build in the fundamental lessons---each row of your grid represents what can follow a single word. View in glossary only knows the word immediately before. Give it two words of context Context window How many previous tokens the model considers when making predictions. Bigrams have a context window of 1, trigrams have 2, and modern LLMs can consider hundreds of thousands or even millions of tokens. View in glossary and the text changes character straight away, and so does the size of the model. Part 1 puts that trade-off on the table in ten minutes with two pre-printed booklets. Part 2 builds the same thing by hand, then finds a cheaper way to reach back.

You will need#

• a bigram booklet and the matching trigram booklet for the same short text, one of each per group:
  • The Cat in the Hat: bigram and trigram
  • Green Eggs and Ham: bigram and trigram
• a d10 per group, as in Pre-trained Model Generation
• pen and paper, ruled into two columns—one passage per booklet

Part 2 is optional and needs materials of its own; they’re listed there.

Your goal#

Generate from a bigram and a trigram model of the same book, then say what changed in the text and in the booklet. Stretch goal: build a trigram by hand, watch it run into a wall, then build a skip grid that recovers some of the benefit for far less cost.

Key idea#

More context sharpens predictions, but every extra word multiplies the number of contexts the model has to store. Part 1 shows both halves of that: the trigram predicts better, and the full trigram booklet blows up. Part 2 builds a cheaper way to get some of the benefit, the skip grid. Pulling in earlier context without paying the full cost is the same problem modern models solve with attention.

Part 1: two booklets, one book#

No training and no cutting: hand out both booklets and generate. Ten minutes, and it works with any group that can already roll a booklet.

1. Generate from the bigram booklet. Pick a starting word, look up its entry, roll, write the next word down, repeat until you have about twenty words. This is exactly the Pre-trained Model Generation loop.
2. Swap to the trigram booklet. Each entry is headed by two bold words—the two words that came before—so to start, open it anywhere and copy one entry’s pair.
3. Look up your last two words in order and read off the next word: roll if the entry shows diamonds, and take the single option if it doesn’t.
4. Shift by one word. Your context is now the second of those words plus the word you just wrote. Repeat until this passage is about twenty words too.
5. Compare. Read both passages out, then count how many entries on one page of each booklet actually offer a choice.

Two things usually come out before anyone explains them. The trigram passage is far more coherent, so coherent that groups start recognising lines from the book, because that’s precisely what they are. And the trigram booklet is thicker, while most of its entries show a single next word and no diamonds at all, so there is nothing to roll. The extra pages bought a model that mostly replays the book back at you.

The Library makes the same comparison in hardbound form. Each volume is typeset in bigram, trigram and 4-gram variants, so you can see the trade-off on the shelf: the trigram Frankenstein reads better than the bigram one, and is considerably thicker.

Why the trigram parrots#

Each extra word of context multiplies the number of possible contexts. A bigram of a picture book needs an entry per word; a trigram needs one per word pair, and a picture book has nowhere near enough words to see each pair more than once. So almost every two-word context in a short text occurs exactly once, the model has no choice to make, and it replays the training text until it reaches one of the few contexts that fork.

This is the trade-off in n-gram models N-gram model The general term for models that predict based on the previous n-1 words. Bigrams are 2-grams, trigrams are 3-grams, and so on. View in glossary : more context sharpens predictions, but you need exponentially more data to fill in all those contexts. Push to four or five words and you’d need a library to see each context even once. So instead of extending the window further, part 2 looks for a cheaper way to bring in earlier words.

Part 2: build it by hand#

For groups with more time, or who have already built a bigram in Training. Building the trigram yourself turns the blow-up into something you feel in your wrist instead of something you infer from a page count, and it sets up the skip grid.

Extra materials#

• the same materials as Training
• extra paper for a four-column table (trigram) and a second grid (skip grid)
• pen, paper and dice as per Generation

Training the trigram#

Instead of asking “what follows this word?”, we ask “what follows these two words?”. The two previous words are considered together, as a pair.

1. Draw a four-column table: word1 | word2 | word3 | count.
2. Slide a window over your text, collecting every overlapping triple of words.
3. For each triple, increment its count (or add a new row starting at 1).

After the full text (see spot run . see spot jump .) the model is:

The order of the rows doesn’t matter, so you can group them by word 1 if that helps.

Generating from the trigram#

1. Pick any row and write down word1 and word2 as your starting words.
2. Find all rows where word1 and word2 match your current context; note their counts.
3. Roll weighted by those counts to pick a row; take its word3 as the next word.
4. Shift the window by one word (new context is old word2 + chosen word3) and repeat from step 2.

A cheaper way: the skip grid#

The trigram is expensive because it tracks the two previous words jointly, in one table indexed by the whole pair. The skip grid tracks them separately and adds the evidence together.

You keep two bigram-sized grids:

• the previous-word grid: your ordinary bigram (the word one back → the next word)
• the skip grid: the same shape, but for the word two back (the word two back → the next word)

Two single-word grids cost far less than one word-pair table: each has one row per word, where the trigram table needs one per pair. So you reach back two words without the combinatorial blow-up.

Training the skip grid#

For every triple word1 word2 word3 in your text:

1. tally word2 → word3 in the previous-word grid (this is just your normal bigram)
2. tally word1 → word3 in the skip grid

That’s it—two tally marks per triple, each in a plain two-word grid.

Example#

Text: the cat sat . the dog ran .

The previous-word grid (word one back → next) gets the usual bigram counts: the→cat, cat→sat, the→dog, dog→ran, and so on.

The skip grid (word two back → next) records what tends to appear two words after each word:

Notice the skip grid has learnt that the is often followed two words later by a verb (sat or ran), regardless of which animal came in between.

Generating with the skip grid#

To pick the word after a two-word context word1 word2:

1. read word2’s row in the previous-word grid
2. read word1’s row in the skip grid
3. add the two rows together, cell by cell, to get combined counts
4. roll weighted by the combined counts, write the word down, shift the window, and repeat

Example#

Continuing the text above, suppose your context is the cat and you want the next word.

• previous-word grid, cat’s row: sat 1
• skip grid, the’s row: sat 1, ran 1
• combined: sat 2, ran 1

Roll on those three (1-2 → sat, 3 → ran). The plain bigram would have forced sat every time; the skip grid lets the cat sometimes go to ran, a verb that followed the … elsewhere in the text. The model has generalised beyond the exact pairs it saw, and there is still something to roll for.

What the skip grid can’t do#

The skip grid is cheaper than the trigram, but it’s also weaker. It treats the two earlier positions as if they contribute independently: it adds “what follows cat” to “what tends to come two words after the”. It can never capture the cases where it’s the pair that matters, where new york predicts something that neither new alone nor york alone would.

That gap is exactly the problem modern models solve with the attention mechanism Attention mechanism The ability to focus on relevant previous words when making predictions. In real LLMs, attention is learned, weighted, and dynamic---the model decides what to focus on for each prediction. The in-context learning module illustrates the motivation for attention: reusing more than just the immediately preceding word. View in glossary . The skip grid mixes in earlier context with fixed weights (always the word two back, always added in the same way). Attention learns which earlier words matter for each prediction, and how much, so it can decide when the combination matters.

Instructor notes#

Discussion questions#

• how did the trigram passage differ from the bigram one? Could you find any of it in the book itself?
• which booklet was thicker, and why? What is the extra paper buying you?
• how many entries in each booklet offered a real choice (more than one next word)? What does a booklet full of single-option entries actually do?
• what’s the trade-off between context length and data requirements? What would a 4-gram booklet of the same book look like?
• the skip grid lets the cat sometimes continue like the dog did. When is that helpful? When might it produce nonsense?
• can you think of a two-word phrase where the pair matters more than either word on its own? (this is what the skip grid misses)

Troubleshooting#

• “Our trigram output is just the book.” That’s the finding, not a fault. Ask the group to mark where the passage does depart from the book—those are the handful of contexts the text saw more than once.
• “There’s no entry for our two words.” The pair fell at the very end of the book, so nothing follows it. Back up a word, or start again from another entry’s bold pair.
• “Which word goes first?” In both the booklet and the hand-built table, the two context words are in the order they appear in the text: the leftmost is the older one, the rightmost is the word you wrote most recently.

Connection to current LLMs#

This module bridges simple word-pair models and modern transformers along two threads.

The trigram shows the context-length trade-off: longer context sharpens predictions, but the number of possible contexts—and the data you need to cover them—grows exponentially. Current models use context windows of hundreds of thousands of tokens, which is only possible because they don’t store an explicit count for every possible context the way an n-gram table does.

The skip grid shows the first step beyond counting exact contexts: mixing in earlier positions with fixed weights. Real models generalise this into attention Attention mechanism The ability to focus on relevant previous words when making predictions. In real LLMs, attention is learned, weighted, and dynamic---the model decides what to focus on for each prediction. The in-context learning module illustrates the motivation for attention: reusing more than just the immediately preceding word. View in glossary —instead of “always add the word two back”, the model learns which previous words to pull in for each prediction. Your hand-built skip grid is fixed and additive; attention is learned, weighted, and dynamic. Both share the core insight: you don’t have to see the exact context before to make a good guess.