In-context Learning

LLMs Unplugged (c) Ben Swift, Cybernetic Studio - CC BY-NC-SA 4.0
  • Audience years 9 and up
  • Duration 20 min
  • Runs on grid, cutouts, booklet

Key idea: The text you've already written is context the model can learn from without changing a single count---copy what followed last time (induction), or lean toward recent words (memory).

Used in Under the hood --- ready-to-run lessons and talks that include this module.

A Bigram model A model that predicts the next word from one previous word. It's what you build in the fundamental modules: each row of your grid says what can follow a single word. View in glossary only ever looks at the word before. But the page in front of you holds everything you’ve written so far, and that’s Context window How many previous tokens the model can consider when it predicts the next one. A bigram has a context window of one word and a trigram of two; modern LLMs can consider hundreds of thousands or even millions of tokens. View in glossary the model can use. This module adds two rules that read that page. Neither one changes the model itself, which makes them a hands-on version of In-context learning Picking up a pattern from the prompt and continuing it, without any change to the model's parameters. The "learning" lives in the context, not in the model. It's why few-shot prompting works: give an LLM a handful of examples and it carries on in the same shape. View in glossary .

You will need

There’s nothing new to print, because both rules are procedures you lay over a model you already have.

Your goal

Part 1: make the model complete a made-up pattern it could not possibly have learned in training. Part 2: generate twice from the same model—once plain, once with a short-term memory—and compare how long each stays on topic. Stretch goal: find the memory setting that keeps the text on topic without making it repetitive.

Key idea

Both rules take the text you’ve already written and use it to weigh the next word.

  • Induction: find the last time this exact word appeared and copy whatever followed it. This completes patterns the model has never seen.
  • Memory: give recently-used words a small boost. This keeps a story about a dog being about the dog.

The model never retrains. The “learning” lives in the context, not the counts.

Part 1: finish the pattern

Start here; it takes two minutes. Write down this sequence, using words your model has never seen together:

moon five apple moon five apple moon five

Now generate the next word. Your current word is five. Instead of consulting the model, scan back up the page to the last time five appeared, and copy the word that followed it: apple. Do it again from apple and you get moon, then five, then apple…

You’ve completed the pattern moon five apple, which appears nowhere in the training text and which the Base model A model as it comes out of pre-training, before any post-training shapes it into an assistant, or the unchanged model that an adapter such as a LoRA is laid over. In both senses the base stays as it is and something else supplies the shift in behaviour. View in glossary has no row for. The model picked up your invented rule from a single example, with no retraining at all. Now turn the rule off and generate from the base model alone: the pattern disappears.

The induction rule

  1. Look at your current word.
  2. Scan back through your written output to the most recent earlier time that same word appeared.
  3. If you find one, the word that followed it is your induction candidate.
  4. Blend with the base model: roll a die—mostly copy the candidate, occasionally generate from the model as normal. If there’s no earlier occurrence, there’s no candidate; just generate.
  5. Write the word down and repeat.

The rule only ever reads your page, so it works identically on a grid, a spread or a booklet. Only the fall-back differs: roll on the row, pick from the matching cutouts, or look up the booklet entry.

Part 2: stay on topic

The induction rule is exact: the word matches or it doesn’t. The memory rule is softer. Mention a dog, and two words later a bigram has already forgotten. Real language reuses what came before, so we keep a running list of recent words and tilt the odds toward them.

The memory rule

  1. Keep a memory list of the last eight or so words you’ve written (underline them, or jot them on a sticky note).
  2. Find the model’s candidates for the next word as you normally would—the row in your grid, the matching cutouts, the booklet entry.
  3. Boost any candidate that’s also in memory (see below for how, per base).
  4. Pick from the boosted candidates, write the word down, add it to memory, and drop the oldest word so the list stays short.
  5. Repeat.

Worked example

Your current word is the, and the model offers cat 3, dog 1, sun 1. A moment ago the text mentioned a dog, so dog is in memory. Give it a bonus of, say, +3:

  • cat: 3
  • dog: 1 + 3 = 4
  • sun: 1

Roll on the new totals (1–3 → cat, 4–7 → dog, 8 → sun). dog has gone from unlikely to favourite, so the story is more likely to stay about the dog. cat and sun are still possible, though; the memory only tilted the odds.

Applying the boost to your model

  • On a grid: add the bonus straight to the counts in the current word’s row, then roll on the new totals, as in the worked example.
  • On a cutouts spread: once you’ve gathered the cutouts that match your current word, check their next words against memory. For any match, count that cutout twice (or write a duplicate) before you pick, so the spread does the reweighting for you.
  • On a booklet: the dice thresholds are pre-printed, so blend instead. Roll a “memory die” first (a d10, 1–3 is a hit). On a hit, scan the current entry’s options and, if one is in memory, pick it. On a miss, or if nothing matches, roll the booklet exactly as normal. You only ever pick from this entry’s options, so you never leave what the model allows.

Instructor notes

Discussion questions

  • the model completed moon five apple, which it never trained on. Where was that “knowledge” stored?
  • why does the demo use made-up words rather than an ordinary sentence?
  • what happens if you only ever copy, and never fall back to the base model?
  • with the memory on, does the text hold a topic or a character for longer? Where does “on topic” tip over into “stuck repeating”?
  • why does the memory only boost words the model already offers?
  • induction and memory both read the page. When would each one help? How are they different from the trigram, which also uses more than one word?
  • (stretch) if induction matched on the last two words instead of one, why would the completion be more reliable?

Connection to current LLMs

This module is a hands-on version of three things modern models do.

Induction heads. These are an identified circuit inside Transformer The neural network architecture behind GPT, Claude, Gemini and every other modern LLM. Its defining feature is attention, which lets the model draw on any earlier part of the context when predicting the next token, rather than a fixed number of previous words. View in glossary , and one of the main mechanisms behind in-context learning: an Induction head A circuit found inside transformers that completes patterns by finding an earlier place where the current token appeared and copying what came next. Induction heads are one of the main mechanisms behind in-context learning. View in glossary finds an earlier place where the current token appeared and copies what came after it—exactly your scan-back-and-copy rule. It’s why few-shot prompting works: give an LLM (Large Language Model) A language model trained on a very large amount of text, with billions of parameters. The hand-built models in these lessons are tiny language models; ChatGPT, Claude and Gemini are large ones. The core principles are identical. The difference is scale. View in glossary a few “input → output” examples and it continues the pattern. Researchers detect these circuits by feeding a model a random sequence repeated twice and checking that it predicts the second copy far better than the first, which is the machine version of your moon five apple demo. Real induction heads match softer, richer patterns and blend smoothly with everything else the model knows, but the behaviour is the same idea.

Attention. A bigram looks back one word; your memory looks back over everything you’ve generated and lets it influence the next choice. That’s the heart of the Attention mechanism The part of a transformer that decides, for each prediction, which earlier words in the context to draw on and how much. The skip grid in More Context and the memory boost in In-context Learning are fixed, hand-run versions of the same move; real attention is learned and changes with the content. View in glossary : when predicting the next word, the model reaches back across the whole context and weights what it finds. Your memory is a crude, fixed version (boost recent words); attention is learned and content-sensitive, and can decide on the fly which earlier words matter.

Repetition. Real models get stuck on rails too, which is why generation settings include Repetition penalty A generation setting that lowers the odds of words that have already appeared, to stop a model going round in circles. It's the opposite of the memory boost in the In-context Learning module, and the mechanised version of the Sampling module's no-repeat rule. View in glossary : a deliberate down-weighting of recent words, the opposite of the boost you added.

Edit this page on GitHub