Sampling

• Audience years 7 and up
• Duration 20-30 min (60 min with the slam)
• Runs on grid, booklet

Key idea: Different sampling strategies change the character of generated text even when the model stays the same.

When generating text, your model offers several options for the next word. Sampling strategies decide which one to pick—and the strategy you choose can change the character of the output dramatically, without changing the underlying model at all. The same grid or cutouts spread can produce flat prose, structured poetry, or pure nonsense, depending entirely on how you sample.

You will need#

• a completed model from an earlier module
• pen, paper, and dice as per Generation (grid method)

Your goal#

Generate text using at least two temperatures and at least two truncation strategies. Stretch goal: design a whole poetry model of your own—training text, temperature, house rules—and hand it to another pair to build and perform in the poetry slam.

Key idea#

Sampling choices—temperature Temperature A dial on how much the model respects its own counts. Low temperature sticks to the favourites; high temperature flattens the odds until every option is roughly equal. A real model divides its scores by the temperature before turning them into probabilities. With raw counts that trick does nothing, so the hand-run version adds a constant to every count instead. View in glossary and truncation Truncation strategy A rule that limits which tokens are eligible before you sample. Top-k and top-p are the standard ones in LLM tooling; the Sampling module adds playful versions (haiku, alliteration, no-repeat) that work the same way, by changing what you're allowed to roll for. View in glossary —can make the same model sound cautious, wild, repetitive, or inventive. Tweaking the sampler changes the output without retraining anything.

Temperature#

Temperature is a dial on how much the model respects its own counts. Turned down, it sticks to the favourites; turned up, it treats every option as roughly equal. A version you can run by hand has four settings:

• cold: no dice; take the option with the highest count (if there’s a tie, roll among the tied ones). This is greedy decoding Greedy sampling Always choosing the most likely next word, which is what temperature zero means. The output is predictable and often falls into a repeating loop. View in glossary .
• normal: roll on the counts exactly as they are, as in Generation.
• hot: before you roll, add 1 to every option’s count, then roll on the new totals. The favourites are still ahead, but by less.
• boiling: ignore the counts and roll evenly among all the options.

Example with counts spot (4), run (2), jump (1), . (1):

• cold → spot, every time
• normal → spot 4 in 8, run 2 in 8, jump and . 1 in 8 each
• hot → counts become 5, 3, 2, 2, so spot 5 in 12, run 3 in 12, jump and . 2 in 12 each: the rare words have doubled their chances
• boiling → 1 in 4 each

Turn the dial up further by adding 2 (or 3) to every count instead of 1; the bigger the number you add, the closer you get to boiling. Dividing every count by the same number won’t do it: that leaves the odds exactly where they were.

Truncation strategies#

Truncation narrows which next-word options are allowed. Mix and match with temperature.

Poetry slam#

This uses everything above: design a poetry model, hand it to another pair, and see what they make of it. Allow about an hour—20 minutes to design, 10 to prepare, 15 for the slam, 5 to talk about it.

Design your model (20 min)#

In pairs, design a poetry language model from scratch. Three decisions, all of them already on this page:

• training text: a pre-trained booklet, your own grid, or a cutouts spread Cutouts Printed cards, one per word pair in the text, each showing a previous word in a box and the word that followed it. Spread on a table they are the model: to generate, find a card whose boxed word matches your current word and read off its next word. Common pairs appear on more cards, so picking by eye samples in proportion. View in glossary
• temperature: cold, normal, hot, or boiling
• house rules: one or more of the truncation strategies above, or a rule you invent yourself

Two things to argue about while you decide: what isn’t poetry, and what your model does about it.

Write the recipe on a blank card, complete enough that another pair can run it without asking you anything:

training text:
temperature:
house rules:
starting word:
stop rule:      (e.g. stop after 20 words, or at the third full stop)

Prep (10 min)#

Swap cards with another pair. You have ten minutes to generate as much text as you can from their model, then plan a one-minute performance of the result. Read it, chant it, or split it between you.

The slam (~15 min)#

Each pair performs for about a minute. A room of eight pairs runs to roughly fifteen minutes with applause.

Discussion (5 min)#

What was the hardest part of designing your model? The hardest part of preparing a performance from someone else’s card? And how does this relate to Claude or ChatGPT Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary —similarities and differences? Two words to name here are temperature (the same dial, on a much bigger model) and prompt engineering Prompt The input text you give a language model. In a bigram model the prompt is just the current word. In a modern LLM a prompt can run to hundreds of thousands of tokens, which gives the model far more context to work with. View in glossary (wording an input to get the output you want, which is what a model card is).

Instructor notes#

Discussion questions#

• which strategy produces the most “human-like” text?
• when would you want predictable vs surprising output?
• how do constraints (haiku, no-repeat) spark creativity?
• can you invent your own sampling strategy?
• (slam) which parts of your model card did the other pair have to guess at? What does that suggest about writing a prompt for an LLM?
• (slam) when a performance worked, how much of that was the model and how much the performers?

Troubleshooting#

• “On hot, the row doesn’t add up to 10 any more.” It never had to: convert the new totals to dice ranges the same way as in Weighted Randomness, or roll two d10 as a number from 1 to 100 and scale.
• “Isn’t hot just the same odds with bigger numbers?” No—adding the same amount to every count narrows the ratio between favourite and rare. Adding 1 takes spot from four times as likely as jump to two and a half times. Dividing would leave the ratio alone.

Connection to current LLMs#

Current LLMs LLM (Large Language Model) A language model trained on a very large amount of text, with billions of parameters. The hand-built models in these lessons are tiny language models; ChatGPT, Claude and Gemini are large ones. The core principles are identical. The difference is scale. View in glossary use these same mechanisms, though the specific strategies differ.

• Temperature control: a real model scores every possible next token Token A single unit of text that the model works with. In our activities each word and punctuation mark is a token. Modern LLMs use subword tokens, so a long or unusual word may be split into several. View in glossary , and the temperature parameter divides those scores before they’re turned into probabilities. Below 1 the distribution sharpens towards the favourite; above 1 it flattens. Your dial does the same job with counts: cold is temperature near zero (greedy), boiling is temperature near infinity (uniform), and hot sits in between. LLMs apply it computationally before every token.
• Truncation techniques in modern LLMs: top-k sampling Top-k sampling A truncation strategy that keeps only the k most likely next words before sampling. Setting k to 1 is greedy sampling; a larger k allows variety while still excluding the very unlikely. View in glossary (only consider k most likely tokens), top-p/nucleus sampling Top-p sampling A truncation strategy that keeps just enough of the most likely words for their combined probability to reach a threshold p (0.9, say). Unlike top-k, the number of options kept changes with how confident the model is. View in glossary (consider tokens until cumulative probability reaches p), repetition penalties Repetition penalty A generation setting that lowers the odds of words that have already appeared, to stop a model going round in circles. It's the opposite of the memory boost in the In-context Learning module, and the mechanised version of the Sampling module's no-repeat rule. View in glossary , frequency penalties, and presence penalties all prune options before sampling.
• Truncation techniques in this module: greedy, haiku, non-sequitur, no-repeat, alliteration, top-k, alphabet chain, and short/long are designed for dice-based sampling but embody the same idea—changing which tokens are eligible before you roll. Top-k directly mirrors the top-k parameter in LLM APIs.

Your paper model demonstrates that “creativity” in AI comes from two controls: adjusting temperature (probability distribution Probability distribution A set of options with a likelihood attached to each. The counts in one row of your grid, the matching cutouts on a spread, or the show of hands when the room is asked "who has `the`?" are all probability distributions over the next word. View in glossary shape) and applying truncation strategies (which tokens to exclude). The same trained model can produce scholarly essays (low temperature, strict truncation) or wild poetry (high temperature, constraint-based truncation) just by changing these parameters. The key insight: generation control is as important as training data. Creative output comes not from the model itself, but from how you control temperature and which tokens you truncate from consideration.