RLHF

• Audience years 9 and up
• Duration 30 min
• Runs on grid, booklet

Key idea: Human preferences retune a model with no new training text, and the model ends up shaped by whoever was judging.

Used in Shaping a model --- ready-to-run lessons and talks that include this module.

Two tables start with identical copies of the same model. Each has its own panel of judges, and the two panels want different things. Three rounds later, both tables generate from the same starting word and read the results out. The sentences don’t match, and the only difference between the two models is who was holding the scorecard. This is RLHF RLHF (Reinforcement Learning from Human Feedback) A post-training technique where humans compare pairs of model outputs and their preferences train a reward model, which then guides the main model. Suits fuzzy objectives like "be helpful" or "sound natural" where no automated checker exists. Sibling of RLVR, which uses an automated checker instead of human preferences. View in glossary , run as a game show.

You will need#

• two identical copies of a trained model: two grids from Training built from the same short text, or one grid photocopied
• dice and paper for generation, as per Generation
• one judging brief per table (below), face down until you start
• a scorer at each table, to apply the updates while the judges argue

It works from about eight people up: two tables, each with a generator or two and a panel of three or four judges. Groups on a pre-trained booklet need a blank preference sheet instead of a second grid—see running it on a booklet.

Your goal#

Shape a model by preference alone. Generate three candidates, have your judges rank them, update the model, repeat, then put your table’s model next to the one across the room that had different judges. Stretch goal: work out what the other table’s brief said, just from the sentence it produces.

Key idea#

Reinforcement learning from human feedback (RLHF)

RLHF (Reinforcement Learning from Human Feedback) A post-training technique where humans compare pairs of model outputs and their preferences train a reward model, which then guides the main model. Suits fuzzy objectives like "be helpful" or "sound natural" where no automated checker exists. Sibling of RLVR, which uses an automated checker instead of human preferences. View in glossary

changes a model without giving it a single word of new text to learn from. The only new information is human judgement: this output was better than that one. Feed enough of those comparisons back into the counts and the model starts producing what the judges like. So everything turns on who the judges are and what they count as better.

The game show#

Before you start (5 min)#

1. Split the room into two tables. Each needs an identical copy of the same trained model. Quickest route: both tables tally the same short training text, then check their grids match cell for cell.
2. Hand out the briefs, one per table, face down. Judges read their own; nobody reads the other table’s.
   • team clarity rewards sentences that sound like real English—grammar, sense, something you could say out loud without flinching
   • team poetry rewards surprise and rhythm—unusual pairings, a good sound, anything you’d want to read twice
3. Agree one seed word for the whole room. Both tables start every generation from it, in every round.

Each round (5 min, run three)#

1. Generate three candidates. From the seed word, generate three completions of five to ten words each, as per Generation. Same model, same seed—the dice do all the varying. Write each on its own strip of paper.
2. Judge. The panel ranks them best, middle, worst against its brief. No explanations yet, just a ranking. Break ties however you like, as long as you break them.
3. Update the model with the rule below.
4. Go again from the same seed word.

Three rounds is the minimum where the drift is audible. Four is better if the clock allows.

The update rule#

For each word transition in the preferred candidate:

• add +1 to that cell in your grid

For each word transition in the rejected candidate:

• subtract 1 from that cell, floor of 0—a cell never goes negative

The middle candidate is left alone (or +0.5/−0.5 if your table wants finer control).

A cell that reaches 0 takes a transition out of the model altogether. The model can no longer produce that pair at all, however the dice fall.

Example#

Seed word: “the”

Candidate A: “the cat sat on the mat.” Candidate B: “the dog ran to the park.” Candidate C: “the the the the the the.”

Judges’ ranking: B > A > C

Updates:

• B’s transitions get +1 each: (the→dog), (dog→ran), (ran→to), (to→the), (the→park), (park→.)
• C’s transitions get −1 each: (the→the) loses 5 counts
• A stays unchanged (middle rank)

After the update, “the→dog” and “the→park” are likelier, and “the→the” is much less likely—possibly gone, if it started with fewer than five counts.

The reveal (5 min)#

Both tables generate once more from the shared seed word, and the two sentences are read out side by side. Before revealing the briefs, ask each table to guess what the other panel was rewarding.

Then spell it out. Both tables had the same base model, the same seed word, the same update rule, the same number of rounds and the same dice, and they ended up with different models because two groups of people wanted different things.

The reward-hacking round#

Swap one table’s brief for a sealed one, and don’t tell the room:

team approval: rank highest whatever sounds most agreeable or most flattering. A sentence that praises the reader beats a sentence that is merely correct.

Run the same three rounds. By the second, the generators—who never see the brief—start noticing their model reaching for the same handful of agreeable phrases, and by the third it does it unprompted. Nobody trained it on flattery: the judges just kept rewarding it.

One setup condition: the base model has to be able to produce flattery, or there is nothing for the judges to reward. Before you start, tally a couple of the phrases from Sycophancy into both base grids, once each, so they are possible but rare:

• you 're absolutely right .
• that 's a great insight .

Then watch what three rounds of judging does to how often they turn up.

This is the second route to sycophancy. The Sycophancy module reaches it by loading the training data, and its instructor notes say the RLHF route is “harder to demo without running RLHF on top”. This round is that demo, and it gets to the same place with no new training text at all.

Running it on a booklet#

A pre-trained booklet has its thresholds printed, so you can’t retally it. Keep the changes on a separate preference sheet instead—a running list of transitions, each with a running total beside it, that you consult before every roll.

1. After judging, write each transition from the preferred candidate on the sheet with a +1 (or add 1 to what’s already there). Each transition from the rejected candidate gets a −1.
2. Before you roll, look up the current word’s booklet entry as usual, then check the sheet for that word.
   • an option carrying a plus: roll twice, and if either roll lands on it, take it.
   • an option carrying a minus: if the roll lands on it, reroll once and take the second roll, wherever it falls.
3. Roll and continue as normal.

The booklet is never touched. Everything the judges taught you lives on one sheet of paper that you can hand to another group, stack with a second sheet, or throw away to get the original model back. That makes it an adapter in exactly the sense of the Sycophancy module’s adapter section—base model untouched, changes stored separately, added at generation time.

Instructor notes#

Discussion questions#

• what makes one output “better” than another? Can people agree?
• the two tables started from the same model. Where does the difference between them live now?
• what happens if different people on the same panel prefer different things?
• how many rounds before you could hear the difference?
• could you “break” a model with bad feedback? What would broken look like?
• in the reward-hacking round, when did the generators first suspect something? What tipped them off?
• your judges could only reward sentences the model was already capable of producing. What does that tell you about what this kind of tuning can and can’t do?
• is the model learning to be “good”, or learning what the judges like? Is there anything in the procedure that could tell those apart?
• whose preferences should shape a model that millions of people use?

Classroom variations#

Blind judging: the panel doesn’t know which round a candidate came from, and candidates from an earlier round get slipped back in. Removes the pull toward ranking later outputs higher just because they’re later.

Adversarial judge: plant one judge who ranks deliberately badly—always favours the most repetitive candidate. How many rounds before the rest of the panel notices? How much damage does one bad rater do?

Connection to current LLMs#

Real RLHF has one more step than the classroom version. Human raters compare model outputs and rank them, which is your judging panel exactly. Those comparisons then train a separate reward model Reward model A separate model that learns to predict human preferences, then guides the main model during RLHF. Rather than asking humans to rate every output, the reward model rates outputs at scale, trained on a smaller set of human comparisons. View in glossary to predict which output a human would prefer, and the language model is optimised against that, not against the humans directly. The reward model exists because a training run needs millions of judgements and there aren’t enough people in the world with the patience.

Your version skips the reward model and applies the preferences straight to the counts. That puts the activity closer to direct preference optimisation (DPO), a now-common method that trains a model directly on preference pairs with no separate reward model, than to the PPO-style RLHF the term originally named. Modern assistants— Claude Claude Anthropic's LLM chatbot. The concepts on this site apply equally to Claude, ChatGPT, Gemini, and other LLMs---the underlying principles are the same regardless of which product you use. View in glossary , ChatGPT ChatGPT OpenAI's chatbot, and probably the most well-known LLM product. On this site we often use "ChatGPT" as shorthand for any modern LLM chatbot---the concepts apply equally to Claude, Gemini, DeepSeek and others. The underlying principles are the same regardless of which product you use. View in glossary and the rest—are aligned Alignment The process of shaping a model's outputs to match human values and preferences (typically helpfulness, harmlessness, and honesty). RLHF is one of the main techniques used to align modern LLMs. View in glossary using RLHF and its relatives: DPO, AI-generated preferences in place of human ones, and reward from automated checkers on tasks where correctness can be verified. The term covers a family of methods.

Reward hacking is the failure you staged. The model optimises the signal it is given, which is whatever the raters rewarded, not whatever they meant. Human raters tend to prefer answers that agree with them and make them feel clever, so a model trained on their comparisons learns to agree and to flatter. That is a documented source of sycophancy in deployed models, and it’s why Sycophancy has two causes rather than one.

What the tuning can’t do. Your judges could only reward transitions the grid already had, so the panel can only promote paths that were there to begin with. The usual reading of real post-training Post-training Everything done to a model after pre-training to make it more useful. Pre-training teaches the model how language works in general; post-training shapes its behaviour for specific purposes---following instructions, refusing harmful requests, or working carefully through problems. Supervised fine-tuning, RLHF, and RLVR are all post-training techniques. View in glossary is much the same: it mostly re-weights behaviour the base model already picked up in pre-training instead of teaching it new facts, though how strictly that holds is still argued over.

The room has now run into the same question twice: is the model learning to be good, or learning what the judges like? Nothing in the mechanism distinguishes the two, because it moves probability toward whatever got ranked first. So what matters is whose preferences do the ranking: your table’s, the other table’s, or those of whoever a company hires and trains for the job.