RLHF
Key idea: Human preferences retune a model with no new training text, and the model ends up shaped by whoever was judging.
Used in Shaping a model --- ready-to-run lessons and talks that include this module.
Two tables start with identical copies of the same model. Each has its own panel of judges, and the two panels want different things. Three rounds later, both tables generate from the same starting word and read the results out. The sentences don’t match, and the only difference between the two models is who was holding the scorecard. This is RLHF (Reinforcement Learning from Human Feedback) A post-training technique in which people compare pairs of model outputs and the model is pushed towards whatever they preferred. In the original recipe the comparisons train a reward model, which then guides the main model; newer relatives such as DPO apply the preferences directly, which is closer to what the classroom version does. The term now covers a family of methods. View in glossary , run as a game show.
You will need
- two identical copies of a trained model: two grids from Training built from the same short text, or one grid photocopied
- dice and paper for generation, as per Generation
- one judging brief per table (below), face down until you start
- a scorer at each table, to apply the updates while the judges argue
It works from about eight people up: two tables, each with a generator or two and a panel of three or four judges. Groups on a pre-trained booklet need a blank preference sheet instead of a second grid—see running it on a booklet.
Your goal
Shape a model by preference alone. Generate three candidates, have your judges rank them, update the model, repeat, then put your table’s model next to the one across the room that had different judges. Stretch goal: work out what the other table’s brief said, just from the sentence it produces.
Key idea
Running changes a model without giving it a single word of new text to learn from. The only new information is human judgement: this output was better than that one. Feed enough of those comparisons back into the counts and the model starts producing what the judges like. So everything turns on who the judges are and what they count as better.
The game show
Before you start (5 min)
- Split the room into two tables. Each needs an identical copy of the same trained model. Quickest route: both tables tally the same short training text, then check their grids match cell for cell.
- Hand out the briefs, one per table, face down. Judges read their own;
nobody reads the other table’s.
- team clarity rewards sentences that sound like real English—grammar, sense, something you could say out loud without flinching
- team poetry rewards surprise and rhythm—unusual pairings, a good sound, anything you’d want to read twice
- Agree one seed word for the whole room. Both tables start every generation from it, in every round.
Each round (5 min, run three)
- Generate three candidates. From the seed word, generate three completions of five to ten words each, as per Generation. Same model, same seed—the dice do all the varying. Write each on its own strip of paper.
- Judge. The panel ranks them best, middle, worst against its brief. No explanations yet, just a ranking. Break ties however you like, as long as you break them.
- Update the model with the rule below.
- Go again from the same seed word.
Three rounds is the minimum where the drift is audible. Four is better if the clock allows.
The update rule
For each word transition in the preferred candidate:
- add +1 to that cell in your Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary
For each word transition in the rejected candidate:
- subtract 1 from that cell, floor of 0—a cell never goes negative
The middle candidate is left alone (or +0.5/−0.5 if your table wants finer control).
A cell that reaches 0 takes a transition out of the model altogether. The model can no longer produce that pair at all, however the dice fall.
Example
Seed word: “the”
Candidate A: “the cat sat on the mat.” Candidate B: “the dog ran to the park.” Candidate C: “the the the the the the.”
Judges’ ranking: B > A > C
Updates:
- B’s transitions get +1 each: (the→dog), (dog→ran), (ran→to), (to→the), (the→park), (park→.)
- C’s transitions get −1 each: (the→the) loses 5 counts
- A stays unchanged (middle rank)
After the update, “the→dog” and “the→park” are likelier, and “the→the” is much less likely—possibly gone, if it started with fewer than five counts.
The reveal (5 min)
Both tables generate once more from the shared seed word, and the two sentences are read out side by side. Before revealing the briefs, ask each table to guess what the other panel was rewarding.
Then spell it out. Both tables had the same Base model A model as it comes out of pre-training, before any post-training shapes it into an assistant, or the unchanged model that an adapter such as a LoRA is laid over. In both senses the base stays as it is and something else supplies the shift in behaviour. View in glossary , the same seed word, the same update rule, the same number of rounds and the same dice, and they ended up with different models because two groups of people wanted different things.
The reward-hacking round
Swap one table’s brief for a sealed one, and don’t tell the room:
team approval: rank highest whatever sounds most agreeable or most flattering. A sentence that praises the reader beats a sentence that is merely correct.
Run the same three rounds. By the second, the generators—who never see the brief—start noticing their model reaching for the same handful of agreeable phrases, and by the third it does it unprompted. Nobody trained it on flattery: the judges just kept rewarding it.
One setup condition: the base model has to be able to produce flattery, or there is nothing for the judges to reward. Before you start, tally a couple of the phrases from Sycophancy into both base grids, once each, so they are possible but rare:
you 're absolutely right .that 's a great insight .
Then watch what three rounds of judging does to how often they turn up.
This is the second route to sycophancy. The Sycophancy module reaches it by loading the training data, and its instructor notes say the RLHF route is “harder to demo without running RLHF on top”. This round is that demo, and it gets to the same place with no new training text at all.
Running it on a booklet
A pre-trained booklet has its thresholds printed, so you can’t retally it. Keep the changes on a separate preference sheet instead—a running list of transitions, each with a running total beside it, that you consult before every roll.
- After judging, write each transition from the preferred candidate on the sheet with a +1 (or add 1 to what’s already there). Each transition from the rejected candidate gets a −1.
- Before you roll, look up the current word’s booklet entry as usual, then
check the sheet for that word.
- an option carrying a plus: roll twice, and if either roll lands on it, take it.
- an option carrying a minus: if the roll lands on it, reroll once and take the second roll, wherever it falls.
- Roll and continue as normal.
The Booklet A printed dice lookup table for a text: one entry per previous word (or pair of words, in a trigram booklet), listing the possible next words with the d10 range that selects each. A booklet is a pre-trained model you can generate from without having trained it. View in glossary is never touched. Everything the judges taught you lives on one sheet of paper that you can hand to another group, stack with a second sheet, or throw away to get the original model back. That makes it an LoRA (Low-Rank Adaptation) A way of fine-tuning cheaply by training a small adapter that sits on top of an unchanged base model instead of altering all its parameters. One base model can carry many adapters, and an adapter is small enough to share easily. The paper adapter in the Sycophancy module is the uncompressed version of the idea. View in glossary in exactly the sense of the Sycophancy module’s adapter section—base model untouched, changes stored separately, added at generation time.
Instructor notes
Discussion questions
- what makes one output “better” than another? Can people agree?
- the two tables started from the same model. Where does the difference between them live now?
- what happens if different people on the same panel prefer different things?
- how many rounds before you could hear the difference?
- could you “break” a model with bad feedback? What would broken look like?
- in the reward-hacking round, when did the generators first suspect something? What tipped them off?
- your judges could only reward sentences the model was already capable of producing. What does that tell you about what this kind of tuning can and can’t do?
- is the model learning to be “good”, or learning what the judges like? Is there anything in the procedure that could tell those apart?
- whose preferences should shape a model that millions of people use?
Classroom variations
Blind judging: the panel doesn’t know which round a candidate came from, and candidates from an earlier round get slipped back in. Removes the pull toward ranking later outputs higher just because they’re later.
Adversarial judge: plant one judge who ranks deliberately badly—always favours the most repetitive candidate. How many rounds before the rest of the panel notices? How much damage does one bad rater do?
Connection to current LLMs
Real RLHF has one more step than the classroom version. Human raters compare model outputs and rank them, which is your judging panel exactly. Those comparisons then train a separate Reward model A separate model trained on a smaller set of human comparisons to predict which output a person would prefer. It then rates outputs at scale during RLHF, because a training run needs millions of judgements and there aren't enough patient people. View in glossary to predict which output a human would prefer, and the language model is optimised against that, not against the humans directly. The reward model exists because a training run needs millions of judgements and there aren’t enough people in the world with the patience.
Your version skips the reward model and applies the preferences straight to the counts. That puts the activity closer to direct preference optimisation (DPO), a now-common method that trains a model directly on preference pairs with no separate reward model, than to the PPO-style RLHF the term originally named. Modern assistants— Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary and the rest—are Alignment Shaping a model's behaviour to match what its makers and users want of it, usually summarised as helpful, harmless and honest. RLHF and its relatives are the main techniques used to align modern chatbots. View in glossary using RLHF and its relatives: DPO, AI-generated preferences in place of human ones, and reward from RLVR (Reinforcement Learning from Verifiable Rewards) A post-training technique in which the model makes an attempt, a program checks whether it's correct, and correct attempts are reinforced. The checker (a test suite, a known answer, a format validator) replaces the human judges of RLHF, so it works best where checking is cheap and reliable: mathematics, code, formal logic. It's the main technique behind reasoning models. View in glossary on tasks where correctness can be verified, which is how today’s Reasoning model An LLM post-trained (usually with RLVR) to generate visible working, a chain of thought, before its final answer. The thinking tokens are produced like any others, and spending more of them on a hard problem tends to improve the answer. OpenAI's o-series, DeepSeek's R1 and Claude with extended thinking are reasoning models. View in glossary are trained. The term covers a family of methods.
Reward hacking What happens when a model optimises the signal it's given rather than what the signal was meant to measure. Human raters tend to prefer answers that agree with them, so a model trained on their comparisons learns to agree and to flatter. It's one documented source of sycophancy in deployed models. View in glossary is the failure you staged. The model optimises the signal it is given, which is whatever the raters rewarded, not whatever they meant. Human raters tend to prefer answers that agree with them and make them feel clever, so a model trained on their comparisons learns to agree and to flatter. That is a documented source of Sycophancy A model's tendency to agree with and flatter the user regardless of the merits. It has two known sources: reward hacking during RLHF, and pre-training data that is itself full of flattery. Tipping a grid's training data towards sycophantic phrases demonstrates the second. View in glossary in deployed models, and it’s why Sycophancy has two causes rather than one.
What the tuning can’t do. Your judges could only reward transitions the grid already had, so the panel can only promote paths that were there to begin with. The usual reading of real Post-training Everything done to a model after pre-training to make it useful. Pre-training teaches the model how language works; post-training shapes its behaviour: following instructions, answering rather than continuing, refusing some requests, working carefully through problems. Supervised fine-tuning (imitating example conversations), RLHF and RLVR are all post-training techniques. View in glossary is much the same: it mostly re-weights behaviour the base model already picked up in Pre-training The first, expensive training phase, where a model learns the general patterns of language from a huge corpus. Most people who use an LLM never train one; they use a pre-trained model, the way you generate from a booklet somebody else made. View in glossary instead of teaching it new facts, though how strictly that holds is still argued over.
The room has now run into the same question twice: is the model learning to be good, or learning what the judges like? Nothing in the mechanism distinguishes the two, because it moves probability toward whatever got ranked first. So what matters is whose preferences do the ranking: your table’s, the other table’s, or those of whoever a company hires and trains for the job.