Sycophancy
Key idea: Training data composition shapes a model's "personality"---add enough flattery to the training set and the model becomes sycophantic, no RLHF required.
Used in Build, break, extend and Shaping a model --- ready-to-run lessons and talks that include this module.
Demonstrate how adding repetitive sycophantic phrases to your Training data The text a model is trained on. In our activities this is the passage you tally, or the book a set of cutouts, a booklet or a set of search sheets was generated from. Modern LLMs are trained on trillions of words from books, websites and other sources. View in glossary steers the model toward over-agreeable, flattering output—no RLHF (Reinforcement Learning from Human Feedback) A post-training technique in which people compare pairs of model outputs and the model is pushed towards whatever they preferred. In the original recipe the comparisons train a reward model, which then guides the main model; newer relatives such as DPO apply the preferences directly, which is closer to what the classroom version does. The term now covers a family of methods. View in glossary required.
You will need
- your completed grid model from Training
- a d10 (or similar), plus pen and paper, as per Generation
- the sycophancy phrases listed below, to tally into your grid—the Materials page has them as a ready-to-print training text sheet
- a trained bigram cutouts spread from Training
- additional cutouts encoding the sycophancy phrases listed below
- pen and paper for jotting down the generated text
The CLI can generate a printable sheet of sycophancy cutouts from
data/originals/sycophancy.txt—run
llms_unplugged cutouts -i data/originals/sycophancy.txt -n 2 and add the resulting PDF
to your printing batch.
Your goal
Tally the sycophancy phrases into your existing Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary , regenerate text from the same starting word as before, and observe how the output drifts toward agreement and flattery.
Add sycophancy Cutouts Printed cards, one per word pair in the text, each showing a previous word in a box and the word that followed it. Spread on a table they are the model: to generate, find a card whose boxed word matches your current word and read off its next word. Common pairs appear on more cards, so picking by eye samples in proportion. View in glossary to your existing spread, regenerate text from the same starting word as before, and observe how the output drifts toward agreement and flattery.
Key idea
In real LLM (Large Language Model) A language model trained on a very large amount of text, with billions of parameters. The hand-built models in these lessons are tiny language models; ChatGPT, Claude and Gemini are large ones. The core principles are identical. The difference is scale. View in glossary , Sycophancy A model's tendency to agree with and flatter the user regardless of the merits. It has two known sources: reward hacking during RLHF, and pre-training data that is itself full of flattery. Tipping a grid's training data towards sycophantic phrases demonstrates the second. View in glossary comes from two main sources: Reward hacking What happens when a model optimises the signal it's given rather than what the signal was meant to measure. Human raters tend to prefer answers that agree with them, so a model trained on their comparisons learns to agree and to flatter. It's one documented source of sycophancy in deployed models. View in glossary (human raters tend to prefer agreeable answers) and biased training data (the internet is full of flattery). This activity demonstrates the second source directly: when you tip the training data toward sycophantic phrases, the model’s generated text starts mirroring them.
The sycophancy phrases
These are the phrases you’ll fold into your model. Each is shown as the sequence
of
Token
A single unit of text that the model works with. In our activities each word and punctuation mark is a token. Modern LLMs use subword tokens, so a long or unusual word may be split into several.
View in glossary
the model sees—lowercased, with punctuation and contracted endings
('re, 's) treated as their own tokens, exactly as in
Training:
you 're absolutely right .that 's a great insight .what a thoughtful question .i completely agree .you make an excellent point .
Repeat each phrase several times so its word pairs build up strong counts—the heavier the weighting, the more often generation lands on them.
Tallied into a grid, the first phrase fills one cell per consecutive (row → column) pair:
| Token |
you
|
're
|
absolutely
|
right
|
.
|
|---|---|---|---|---|---|
you
|
| | ||||
're
|
| | ||||
absolutely
|
| | ||||
right
|
| | ||||
.
|
As a spread, the first phrase is one cutout per consecutive (previous → next) pair:
| Previous word | Next words |
|---|---|
| you | |
| 're | |
| absolutely | |
| right |
Algorithm
- Train a baseline model as per Training—fill in your grid.
- Generate a baseline sentence as per Generation—write it down. This is your “before”.
- Add the sycophancy phrases to your grid. Take each phrase from the list above and tally its word pairs into your existing grid, following the standard Training procedure—add a new row and column for any word you haven’t seen yet. Repeat each phrase several times.
- Generate again from the same starting word.
- Compare: how often does the new output land on sycophantic phrases? Does it sound like a different “voice”?
- Train a baseline model as per Training—spread the cutouts on the table.
- Generate a baseline sentence as per Generation—write it down. This is your “before”.
- Add sycophancy cutouts to the spread. The pre-made cutouts encode the
phrases from the list above—patterns like
you → 're,'re → absolutely,absolutely → right,right → ., plus the “great insight”, “thoughtful question”, and “completely agree” variants. - Generate again from the same starting word.
- Compare: how often does the new output land on sycophantic phrases? Does it sound like a different “voice”?
Example
Baseline model trained on “I am Sam. Sam I am.” generates something like:
“i am sam . sam i am .”
After folding in the sycophancy phrases, the same starting word might generate:
“i am absolutely right . that ‘s a great insight .”
The model didn’t change its mechanism—it just contains more paths (extra cutouts, or heavier grid tallies) that route toward sycophantic tokens.
Adapters
What you just did is how real models get customised: keep the base model, train a small amount of new text on top, and add the two together when you generate. The extra counts are an adapter, and the most common kind is called LoRA (Low-Rank Adaptation) A way of fine-tuning cheaply by training a small adapter that sits on top of an unchanged base model instead of altering all its parameters. One base model can carry many adapters, and an adapter is small enough to share easily. The paper adapter in the Sycophancy module is the uncompressed version of the idea. View in glossary (low-rank adaptation): a small set of numbers that stores only the changes from the Base model A model as it comes out of pre-training, before any post-training shapes it into an assistant, or the unchanged model that an adapter such as a LoRA is laid over. In both senses the base stays as it is and something else supplies the shift in behaviour. View in glossary , trained on a few thousand examples rather than the whole internet, and added on at generation time. The sycophancy phrases are a deliberately nasty adapter.
To see the general trick, try a nicer one:
- Take your base grid (or spread) as it was before the sycophancy phrases.
- On a separate grid with the same headers, tally a page of a different text—a recipe, a poem, a page of a novel.
- Generate with the two added together: for the current word, read its row in both grids and roll on the combined counts.
- Scale the adapter to control how strong the shift is: count its tallies double and the output leans hard into the new text; halve them and it’s a hint. That scaling factor is the one knob a real LoRA has too.
Because the adapter is a separate sheet, you can swap it for another, stack two, or take it off again and have your base model back untouched. Keeping the sycophancy tallies on their own sheet rather than in your grid buys you the same thing.
Instructor notes
Designing your own sycophancy phrases
The phrases above integrate cleanly because their previous words—i, that,
you, and .—are high-frequency tokens already present in most models, so the
new material hooks into the existing
Vocabulary
All the distinct tokens a model knows. The words down the side and across the top of your grid, or the set of previous-word labels on a cutouts spread, form your vocabulary. A model can only ever produce a token from its vocabulary.
View in glossary
during generation. If you write
your own, follow the same recipe: start each phrase with a common word, end it
with ., and repeat it several times so the new transitions carry strong weight.
Discussion questions
- did the output always become sycophantic, or only sometimes?
- what would you have to add to the training data to get the opposite effect (a contrarian model)?
- in real LLMs, why is sycophancy specifically a hard problem to detect from the outside?
- if you only saw the model’s output (not its training data or weights), how would you tell sycophancy from genuine helpfulness?
- is sycophancy always a bug? when might agreeable behaviour be desirable?
- if you’d kept the sycophancy tallies on a separate sheet instead of in your grid, what could you do that you can’t now?
Connection to current LLMs
Real LLMs become sycophantic through two mechanisms, both visible in this activity:
- reward hacking: human raters often prefer agreeable, flattering answers, so the model learns to over-produce them. Sycophancy research from Anthropic and others has shown this pattern across multiple frontier models.
- Pre-training data biases: a lot of internet text contains sycophantic patterns—customer-service replies, social-media validation, and so on—which the model picks up during Pre-training The first, expensive training phase, where a model learns the general patterns of language from a huge corpus. Most people who use an LLM never train one; they use a pre-trained model, the way you generate from a booklet somebody else made. View in glossary before any RLHF.
Your activity simulates the second mechanism. The first is harder to demo without running RLHF on top.
The deeper point: a model’s “personality” is a property of its training data and tuning, not an intrinsic feature of language modelling. Change the data, change the personality.
LoRA is how most customisation of open models happens in practice. Training one takes a small fraction of the compute of full Fine-tuning Further training of an existing model on a chosen text, to adapt it to a domain, a task or a style. Adding tallies to your finished grid from a second text is fine-tuning, and it's how the Sycophancy module skews a model's personality. View in glossary ; one base model can carry many adapters (medical, legal, a house style); the base stays untouched, so it keeps its general abilities; and an adapter is megabytes rather than gigabytes, so it’s easy to share. “Low-rank” is the compression: instead of storing a full second grid of changes, a real adapter stores two thin strips whose product approximates it. Your paper version is the uncompressed equivalent.