Shaping a model
People talk about a Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary ’s personality as though it were a temperament: this one’s warm, that one’s a suck-up, the other one lectures you. But a model’s personality is a property of what shaped it, and everything that shaped it was a decision somebody made.
This lesson makes those decisions concrete by handing them to the room. After building the paper model, participants shape it three different ways: they change the Training data The text a model is trained on. In our activities this is the passage you tally, or the book a set of cutouts, a booklet or a set of search sheets was generated from. Modern LLMs are trained on trillions of words from books, websites and other sources. View in glossary and watch the model turn sycophantic; they run a preference round where human judges reward some outputs over others, and then watch the model learn to game the judges; and they retrain a model on its own generated text and count what’s left of the vocabulary. Every one of those opens a discussion: whose preferences, what counts as good, and what happens to the web when models are trained on each other’s output.
Run of show
| Section | Time |
|---|---|
| Intro | 10 min |
| Training | 20 min |
| Generation | 20 min |
| Sycophancy | 20 min |
| Break | 10 min |
| RLHF | 30 min |
| Synthetic Data | 30 min |
| Discussion and close | 10 min |
Two and a half hours as written. For a two-hour slot, drop Synthetic Data and fold its question (“what happens when the training data is model output?”) into the closing discussion. The relay is the section that suffers least from being described rather than run, because its punchline is a number.
There’s no deck for this lesson. Run it from the module pages, which carry the algorithms and worked examples. The discussion questions in the instructor notes are doing a lot of the work here, so read them ahead of time and pick the ones that suit your group.
How it goes
The first 40 minutes are the standard build: tally word pairs into the Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary , then roll a d10 and generate. Everything after that is a modification to a model the room owns, which keeps the ethics discussion concrete.
Sycophancy A model's tendency to agree with and flatter the user regardless of the merits. It has two known sources: reward hacking during RLHF, and pre-training data that is itself full of flattery. Tipping a grid's training data towards sycophantic phrases demonstrates the second. View in glossary is the data route. Groups tally a sheet of flattering, agreeable phrases into their existing grid and generate again. The model starts complimenting them. Nobody wrote a rule saying “be agreeable”: the data said it, repeatedly, and the counts did the rest. And if you scraped the whole internet, what else did you tally in without meaning to?
RLHF (Reinforcement Learning from Human Feedback) A post-training technique in which people compare pairs of model outputs and the model is pushed towards whatever they preferred. In the original recipe the comparisons train a reward model, which then guides the main model; newer relatives such as DPO apply the preferences directly, which is closer to what the classroom version does. The term now covers a family of methods. View in glossary is the other route to the same place, and it runs as a game show. Two tables start with identical copies of the grid, each with its own judging panel and a sealed brief (one panel wants clear English, the other wants surprise and rhythm). Each round the table generates three candidates from a shared seed word, the panel ranks them, and the scorer bumps the counts behind the winner and docks the loser. Three rounds later both tables generate from the same seed and read the results side by side: two models from one grid, separated only by who held the scorecard. Then the Reward hacking What happens when a model optimises the signal it's given rather than what the signal was meant to measure. Human raters tend to prefer answers that agree with them, so a model trained on their comparisons learns to agree and to flatter. It's one documented source of sycophancy in deployed models. View in glossary round: one panel is secretly briefed to reward flattery, and the generators, who never see the brief, watch their model reach for agreeable phrases on its own.
Synthetic data Training text generated by a model rather than written by people. Used carelessly, in a closed loop, it causes model collapse. Used deliberately (a stronger model writing for a weaker one, or filtering a large generated set) it's one of the more useful tools going. View in glossary runs as a relay around the room. Group one generates text from their model and passes it to group two as training data; group two trains on it, generates, and passes on again. Keep a running vocabulary count on butchers paper at each hop. The line goes down, sometimes gently and sometimes off a cliff, and that’s Model collapse The degradation that sets in when a model is trained on its own output (or output from similar models) generation after generation. Rare words and constructions are the likeliest to be missed by any sample, so the tails vanish first, and the model narrows towards a repetitive core. View in glossary .
The close pulls the three sections together: different mechanisms, the same outcome, and in every case a person somewhere chose the data, the reward, or the shortcut.
You will need
For each pair or small group:
- a printed grid template (or grid paper), plus a blank one per group for the synthetic-data retrain
- a d10 die, pens and paper
- some training text: a few pages of any book
- the sycophancy training text sheet (also on the Materials page)
For the room:
- a pre-trained booklet or two from the Materials page, to seed the first group in the synthetic-data relay with something richer than a two-page grid
- butchers paper and a marker for the vocabulary count down the relay chain
- a whiteboard for the RLHF candidates and the judging
Before you deliver
- print the grids, the sycophancy sheets, and at least one booklet
- brief the two judging panels in secret before the RLHF round—the briefs are on the RLHF page (team clarity, team poetry, and the sealed team-approval brief for the reward-hacking round); don’t let the panels compare notes until the reveal
- seat the groups in a chain, or at least know the order, so the synthetic-data relay passes cleanly without everyone standing up
- decide in advance how you’ll count vocabulary at each hop (distinct words in a fixed-length sample is easiest) and rule the butchers paper up beforehand
- read the discussion questions in the RLHF and Synthetic Data instructor notes and choose two or three; there are more good ones than you’ll have time for
Where to go next
If your group comes out of this wanting the mechanism rather than the ethics, Under the hood is the other follow-on—context, in-context learning and embeddings, all by hand. The Agentic AI module is the natural sequel to the ethics conversation: it’s one thing for a model to say something shaped by its training, and another for it to go off and do something about it. And for a room too big for tables and dice, the talks run the core idea from a stage.