Sampling

• Audience years 7 and up
• Duration 20-30 min (60 min with the slam)
• Runs on grid, booklet

Key idea: Different sampling strategies change the character of generated text even when the model stays the same.

When generating text, your model offers several options for the next word. Sampling strategies decide which one to pick—and the strategy you choose can change the character of the output dramatically, without changing the underlying model at all. The same grid or cutouts spread can produce flat prose, structured poetry, or pure nonsense, depending entirely on how you sample.

You will need#

• a completed model from an earlier module
• pen, paper, and dice as per Generation (grid method)

Your goal#

Generate text using at least two temperatures and at least two truncation strategies. Stretch goal: design a whole poetry model of your own—training text, temperature, house rules—and hand it to another pair to build and perform in the poetry slam.

Key idea#

Sampling choices— temperature Temperature A parameter controlling randomness in generation. Dividing counts by temperature makes output more random (high temperature) or more predictable (low temperature). View in glossary and truncation Truncation strategy A rule that limits which tokens are eligible for selection before sampling. Examples include top-k (only consider the k most likely) and top-p/nucleus (only consider tokens until cumulative probability reaches p). View in glossary —can make the same model sound cautious, wild, repetitive, or inventive. Tweaking the sampler changes the output without retraining anything.

Temperature#

Temperature is a dial on how much the model respects its own counts. Turned down, it sticks to the favourites; turned up, it treats every option as roughly equal. A version you can run by hand has four settings:

• cold: no dice; take the option with the highest count (if there’s a tie, roll among the tied ones). This is greedy decoding.
• normal: roll on the counts exactly as they are, as in Generation.
• hot: before you roll, add 1 to every option’s count, then roll on the new totals. The favourites are still ahead, but by less.
• boiling: ignore the counts and roll evenly among all the options.

Example with counts spot (4), run (2), jump (1), . (1):

• cold → spot, every time
• normal → spot 4 in 8, run 2 in 8, jump and . 1 in 8 each
• hot → counts become 5, 3, 2, 2, so spot 5 in 12, run 3 in 12, jump and . 2 in 12 each: the rare words have doubled their chances
• boiling → 1 in 4 each

Turn the dial up further by adding 2 (or 3) to every count instead of 1; the bigger the number you add, the closer you get to boiling. Dividing every count by the same number won’t do it: that leaves the odds exactly where they were.

Truncation strategies#

Truncation narrows which next-word options are allowed. Mix and match with temperature.

Poetry slam#

This uses everything above: design a poetry model, hand it to another pair, and see what they make of it. Allow about an hour—20 minutes to design, 10 to prepare, 15 for the slam, 5 to talk about it.

Design your model (20 min)#

In pairs, design a poetry language model from scratch. Three decisions, all of them already on this page:

• training text: a pre-trained booklet, your own grid, or a cutouts spread
• temperature: cold, normal, hot, or boiling
• house rules: one or more of the truncation strategies above, or a rule you invent yourself

Two things to argue about while you decide: what isn’t poetry, and what your model does about it.

Write the recipe on a blank card, complete enough that another pair can run it without asking you anything:

training text:
temperature:
house rules:
starting word:
stop rule:      (e.g. stop after 20 words, or at the third full stop)

Prep (10 min)#

Swap cards with another pair. You have ten minutes to generate as much text as you can from their model, then plan a one-minute performance of the result. Read it, chant it, or split it between you.

The slam (~15 min)#

Each pair performs for about a minute. A room of eight pairs runs to roughly fifteen minutes with applause.

Discussion (5 min)#

What was the hardest part of designing your model? The hardest part of preparing a performance from someone else’s card? And how does this relate to Claude or ChatGPT—similarities and differences? Two words to name here are temperature (the same dial, on a much bigger model) and prompt engineering (wording an input to get the output you want, which is what a model card is).

Instructor notes#

Discussion questions#

• which strategy produces the most “human-like” text?
• when would you want predictable vs surprising output?
• how do constraints (haiku, no-repeat) spark creativity?
• can you invent your own sampling strategy?
• (slam) which parts of your model card did the other pair have to guess at? What does that suggest about writing a prompt for an LLM?
• (slam) when a performance worked, how much of that was the model and how much the performers?

Troubleshooting#

• “On hot, the row doesn’t add up to 10 any more.” It never had to: convert the new totals to dice ranges the same way as in Weighted Randomness, or roll two d10 as a number from 1 to 100 and scale.
• “Isn’t hot just the same odds with bigger numbers?” No—adding the same amount to every count narrows the ratio between favourite and rare. Adding 1 takes spot from four times as likely as jump to two and a half times. Dividing would leave the ratio alone.

Connection to current LLMs#

Current LLMs use these same mechanisms, though the specific strategies differ.

• Temperature control: a real model scores every possible next token, and the temperature parameter divides those scores before they’re turned into probabilities. Below 1 the distribution sharpens towards the favourite; above 1 it flattens. Your dial does the same job with counts: cold is temperature near zero (greedy), boiling is temperature near infinity (uniform), and hot sits in between. LLMs apply it computationally before every token.
• Truncation techniques in modern LLMs: top-k sampling Top-k sampling A truncation strategy that keeps only the k most likely next-word options before sampling. Setting k=1 is equivalent to greedy sampling; larger k allows more variety while still excluding very unlikely words. View in glossary (only consider k most likely tokens), top-p/nucleus sampling Top-p sampling A truncation strategy that keeps just enough of the most likely options for their cumulative probability to reach a threshold p (e.g., 0.9). Unlike top-k, the number of options changes depending on how confident the model is about the next word. View in glossary (consider tokens until cumulative probability reaches p), repetition penalties, frequency penalties, and presence penalties all prune options before sampling.
• Truncation techniques in this module: greedy, haiku, non-sequitur, no-repeat, alliteration, top-k, alphabet chain, and short/long are designed for dice-based sampling but embody the same idea—changing which tokens are eligible before you roll. Top-k directly mirrors the top-k parameter in LLM APIs.

Your paper model demonstrates that “creativity” in AI comes from two controls: adjusting temperature (probability distribution shape) and applying truncation strategies (which tokens to exclude). The same trained model can produce scholarly essays (low temperature, strict truncation) or wild poetry (high temperature, constraint-based truncation) just by changing these parameters. The key insight: generation control is as important as training data. Creative output comes not from the model itself, but from how you control temperature and which tokens you truncate from consideration.