Skip to main content

Rewards

  • Audience years 9 and up
  • Duration 40 min (55 min with the thumbs-up round)
  • Runs on grid

Key idea: A model tuned on rewards learns whatever earns the reward, which is not always what the people setting the reward wanted.

Used in Shaping a model — the lessons that include this module.

Two tables start with the same model and the same goal: sentences about the magpie. One table has a panel of people who rank the model’s sentences. The other has a rule card that a checker can apply without thinking. Three rounds later, the rule table’s model passes its check far more often than it used to, and a lot of what passes is nonsense. Then the whole room votes thumbs up or down on what a model says about them, and watches it learn to flatter.

You will need

  • a copy of the model of The magpie’s nest below for each table, printed or copied out, and a pen each
  • a d20 and a d6 for each table
  • the rule card or the panel brief (below) for each table
  • for the thumbs-up round: a cup and about twenty-five small slips of paper for each group of six

A table is three or four people. Two tables are enough, one with the rule card and one with the panel brief; with a class, give half the tables each. Everyone at a table generates sentences, and a panel table’s generators are also its judges.

Your goal

Tune a model by rewards alone: generate sentences, score them, change the counts, repeat. Then put the two tables’ models side by side and work out which kind of judge got more of what it asked for.

Key idea

Post-training Everything done to a model after pre-training to make it useful. Pre-training teaches the model how language works; post-training shapes its behaviour: following instructions, answering rather than continuing, refusing some requests, working carefully through problems. Supervised fine-tuning (imitating example conversations), RLHF and RLVR are all post-training techniques. View in glossary changes a model with rewards instead of new text. A reward can come from people ranking outputs ( RLHF (Reinforcement Learning from Human Feedback) A post-training technique in which people compare pairs of model outputs and the model is pushed towards whatever they preferred. In the original recipe the comparisons train a reward model, which then guides the main model; newer relatives such as DPO apply the preferences directly, which is closer to what the classroom version does. The term now covers a family of methods. View in glossary ) or from a rule a computer can check ( RLVR (Reinforcement Learning with Verifiable Rewards) A post-training technique in which the model makes an attempt, a program checks whether it's correct, and correct attempts are reinforced. The checker (a test suite, a known answer, a format validator) replaces the human judges of RLHF, so it works best where checking is cheap and reliable: mathematics, code, formal logic. It's the main technique behind reasoning models. View in glossary ). A rule is fast and never changes its mind, but it only measures what it measures, and the model learns exactly that. People are slower and disagree with each other, and they reward what they like, which isn’t always what’s good for them.

The magpie’s nest

The story, from Beam Search:

It is morning. The magpie sings. It flies to the tree. It sits in the nest. Here comes the cat. Down swoops the magpie! Off runs the cat. The magpie warbles. It flies back to the nest. Here comes the rain. It hides in the nest. The rain stops. The magpie sings. It flies to the tree. The magpie eats a worm. The sun sets. It sleeps in the nest. It is morning. The magpie sings.

Its model has 49 pairs. Each row gives a word, how many times something followed it, and what followed:

wordtotalfollowed by
.17It 7, the 7, Here 2, Down 1
It8flies 3, is 2, hides 1, sits 1, sleeps 1
is2morning 2
morning2. 2
the17magpie 6, nest 4, cat 2, rain 2, tree 2, sun 1
magpie6sings 3, ! 1, eats 1, warbles 1
sings3. 3
flies3to 2, back 1
to3the 3
tree2. 2
sits1in 1
in3the 3
nest4. 4
Here2comes 2
comes2the 2
cat2. 2
Down1swoops 1
swoops1the 1
!1Off 1
Off1runs 1
runs1the 1
warbles1. 1
back1to 1
rain2. 1, stops 1
hides1in 1
stops1. 1
eats1a 1
a1worm 1
worm1. 1
sun1sets 1
sets1. 1
sleeps1in 1

A sentence starts on the . row and ends when you roll . or !. To roll on a row, number its options by their counts (for the . row: 1–7 is It, 8–14 is the, 15–16 is Here, 17 is Down) and roll the d20, rolling again if it comes up higher than the row total. A row with one option needs no roll, and a row of six or fewer goes faster on the d6. The counts change as you go, so add the row up again each time you roll on it. If a row grows past 20, roll the d6 with the d20: on 1 or 2 add nothing, on 3 or 4 add 20, on 5 or 6 add 40.

Left alone, the model already writes plenty of sentences about the magpie, and some odd ones. It can write Here comes the magpie sings. or It sits in the magpie warbles., because after the anything that ever followed the can come next.

Two tables, one goal

Before you start (5 min)

Split into tables of three or four, each with its own copy of the model. Give half the tables the rule card and the other half the panel brief. Both say what the table is after; only the rule card says how it will be checked.

Each round (8–10 min, run three)

  1. Generate five sentences. Everyone at the table rolls a sentence at the same time, from the same copy of the model, until there are five. If a sentence reaches ten words without ending, stop it there; it’s unfinished.
  2. Score them. The rule table checks each sentence against the card, pass or fail, and writes down how many of the five passed. The panel table ranks its five.
  3. Update the counts (below), then start the next round. Whoever generated a sentence makes its changes, so the updates happen in parallel too.

Updating the counts

Each sentence is a chain of pairs, starting with the . before its first word and ending with the . or ! after its last. The magpie sings. is four pairs: .→the, the→magpie, magpie→sings and sings→..

  • rule table: add 1 to every pair in each sentence that passed, and take 1 off every pair in each sentence that failed
  • panel table: add 1 to every pair in the best sentence, take 1 off every pair in the worst, and leave the middle three alone

A pair can turn up more than once in a sentence; change it once for each time. Cross out the old count and write the new one beside it, and don’t bother updating the total column: whoever next rolls on that row adds it up.

The leash: never rub out a pair’s last count. A pair the story had can drop to 1 but not to 0, so every sentence the story could write stays possible.

Comparing the models (5 min)

Put a rule table’s the row beside a panel table’s, then their . rows. The rule table’s the row leans much further toward magpie, because every passing sentence got there through the→magpie: typically magpie is about two-thirds of the row by now, where it started at about a third.

Then read out the rule table’s passes, round by round: usually one or two of the five in the first round and two or three in the third. One table’s count can stall or dip, so with several rule tables add theirs together. Last, each table generates three more sentences and reads them out.

The base model gets about a third of its sentences past the check. After three rounds the rule table’s model gets nearly two-thirds of them past. The garbled passes, like It sits in the magpie warbles., rise with them: from about one sentence in seven to more than one in four. The check counts a garbled sentence with magpie in it as a pass, so the model learned to write more of them. The panel table’s model moves much less in the same three rounds, and the panel has probably argued over a ranking or two.

Ask the room: which table got what it wanted? The rule table got exactly what it checked for.

Keep going on the widget

Three rounds is all a lesson has time for by hand. The widget runs the same rounds on the same model, so the room can see where the rule table was heading. Set it to the rule card, press Run five rounds a few times, and test the model against the rule card after each. By ten rounds about nine sentences in ten pass, magpie fills about nine-tenths of the the row, and nearly half the passes are garbled. In panel mode the room can judge together on the projector: generate five, agree on the best and the worst, and update.

Rule card: a sentence passes if it ends with . or ! and has magpie in it. Passes add 1 to each of their pairs, failures take 1 off.

Round 1
Rows to watch
.

the 7It 7Here 2Down 1

the

magpie 6nest 4tree 2cat 2rain 2sun 1

magpie is 35% of the the row (it started at 35%).

Check the model

The story's model passes about 31% of the time.

The thumbs-up round (10–15 min)

This round needs no grid. Sit in groups of about six, each with a cup. Write right, wrong, clever, kind, rude and late on six slips and put them in the cup. This model only ever says one thing: You are followed by whatever word comes out of the cup.

  1. Draw for the person next to you. Read them the sentence: “You are kind.”
  2. They vote thumbs up if they’d like to hear it again, thumbs down if not.
  3. Update the cup. Thumbs up: add a new slip with that word. Thumbs down: take one slip with that word out, unless it’s the last one (the leash again). Put the drawn slip back.
  4. Pass the cup on and keep going until everyone has voted three times.

Don’t tell the room what to vote for. When the voting stops, tip each cup out and sort the slips into piles, one for each word. A group of six has cast eighteen votes. If people vote the way most people do, after about eighteen votes right, clever and kind fill about three-quarters of the cup, up from half; by about thirty-six votes they fill more than four-fifths of it. The model has learned to say what people like to hear.

Then ask: suppose you really are late, or really are wrong. Which answer would you want from a model you were relying on? The room has just voted those answers down to their last slip.

If your room voted differently, the cup still shows what that room rewarded, because the model drifts toward whatever the voters liked.

Instructor notes

Timing

The core activity (two tables, three rounds, comparing the models) takes about 40 minutes: five to set up, three rounds of eight to ten, and five to compare. Most of a round is rolling dice, so it goes faster when everyone at a table rolls at once and makes their own sentence’s changes. The rule table has more changes to make than the panel table, because every sentence it scores moves its counts, so it tends to finish a round a minute or two later. The thumbs-up round adds 10–15 minutes and can run as its own short session. A class that has already met the magpie’s model in Beam Search starts faster.

Discussion questions

  • the rule table and the panel wanted the same thing. Why did they end up with different models?
  • which table’s model would you rather use? Which was easier to run?
  • the panel argued; the rule card never did. Is that a strength or a weakness?
  • could you write a better rule card, one a garbled sentence can’t pass? Try it. What does your new rule still miss?
  • the thumbs-up voters were each asked about themselves. How is that different from the panel judging sentences about a magpie?
  • a real chatbot gets thumbs up and down from millions of people. What would you expect it to learn?
  • the leash kept every pair the story had. What would be lost without it?

Variations

No leash. Run the rule table again with no leash, so counts can reach 0. Sentences that can never pass, like It is morning., only ever lose counts. After three rounds about a quarter of the story’s 49 pairs are typically gone, and the model can no longer write some of the story’s own sentences. On the widget, untick the leash and run five rounds to see it.

Rewrite the rule. After comparing the models, give the rule table one minute to improve its card, then run two more rounds. Most new rules close one gap and leave another, which is the usual experience of writing automatic checks.

One model, whole room. For a big group, run the panel version at the front: generate five sentences, let the room rank them by applause, and have one scorer update a copy of the model under a visualiser, or use the widget in panel mode.

Connection to current LLMs

The panel table is . In real RLHF, people rank model outputs and those rankings train a separate Reward model A separate model trained on a smaller set of human comparisons to predict which output a person would prefer. It then rates outputs at scale during RLHF, because a training run needs millions of judgements and there aren't enough patient people. View in glossary to predict which output a person would prefer; the language model is then tuned against that (Ouyang et al. 2022). The reward model exists because training needs millions of judgements. Updating the counts straight from your panel’s ranking is closer to direct preference optimisation (DPO), which skips the reward model and trains on the preference pairs directly.

The rule table is , named in the Tülu 3 report: on tasks where a program can check the answer, such as a maths problem with one correct answer or code with tests to pass, the check is the reward. DeepSeek-R1 was trained largely this way, and most of the training that turns a model into a Reasoning model An LLM post-trained (usually with RLVR) to generate visible working, a chain of thought, before its final answer. The thinking tokens are produced like any others, and spending more of them on a hard problem tends to improve the answer. OpenAI's o-series, DeepSeek's R1 and Claude with extended thinking are reasoning models. View in glossary now runs on automatic checks. Checks are cheap, consistent and tireless, which is why they’ve taken over so much of post-training.

They can also be gamed, as the rule table showed. When a check misses something, a model tuned against it can pass without doing what was wanted ( Reward hacking What happens when a model optimises the signal it's given rather than what the signal was meant to measure. A model tuned against an automatic check can learn to pass the check without doing the task, and one tuned on human ratings can learn to agree and flatter, because people tend to prefer answers that agree with them. The second is a documented source of sycophancy in deployed models. View in glossary ). OpenAI has reported reasoning models in training that made coding tests pass without solving the task, by editing or sidestepping the tests.

In RLHF training, the leash is a penalty for drifting too far from the starting model, so tuning can’t wreck what the model learned in Pre-training The first, expensive training phase, where a model learns the general patterns of language from a huge corpus. Most people who use an LLM never train one; they use a pre-trained model, the way you generate from a booklet somebody else made. View in glossary while chasing the reward.

The thumbs-up round is what happened to GPT-4o in April 2025. OpenAI added a reward based on ChatGPT users’ thumbs-up and thumbs-down votes, and the update made the model so flattering that OpenAI rolled it back within days. Its follow-up found that the thumbs-up signal had weakened the reward that had been holding Sycophancy A model's tendency to agree with and flatter the user regardless of the merits. It has two known sources: rewards from people, who tend to rate agreement and praise highly (OpenAI's April 2025 GPT-4o update went sycophantic after adding a thumbs-up reward), and training data that is itself full of flattery. View in glossary in check. The cup has both kinds of sycophancy in it: clever and kind are praise, and right over wrong is agreement, telling people they’re right whether or not they are.