Sycophancy
Key idea: Training data composition shapes a model's "personality"---add enough flattery to the training set and the model becomes sycophantic, no RLHF required.
Used in Build, break, extend and Shaping a model --- ready-to-run lessons and talks that include this module.
Demonstrate how adding repetitive sycophantic phrases to your training data steers the model toward over-agreeable, flattering output—no RLHF (Reinforcement Learning from Human Feedback) A post-training technique where humans compare pairs of model outputs and their preferences train a reward model, which then guides the main model. Suits fuzzy objectives like "be helpful" or "sound natural" where no automated checker exists. Sibling of RLVR, which uses an automated checker instead of human preferences. View in glossary required.
You will need
- your completed grid model from Training
- a d10 (or similar), plus pen and paper, as per Generation
- the sycophancy phrases listed below, to tally into your grid—the Materials page has them as a ready-to-print training text sheet
- a trained bigram cutouts spread from Training
- additional cutouts encoding the sycophancy phrases listed below
- pen and paper for jotting down the generated text
The CLI can generate a printable sheet of sycophancy cutouts from
data/sycophancy.txt—run
llms_unplugged cutouts -i data/sycophancy.txt -n 2 and add the resulting PDF
to your printing batch.
Your goal
Tally the sycophancy phrases into your existing grid, regenerate text from the same starting word as before, and observe how the output drifts toward agreement and flattery.
Add sycophancy cutouts to your existing spread, regenerate text from the same starting word as before, and observe how the output drifts toward agreement and flattery.
Key idea
Sycophancy in real LLMs comes from two main sources: RLHF (Reinforcement Learning from Human Feedback) A post-training technique where humans compare pairs of model outputs and their preferences train a reward model, which then guides the main model. Suits fuzzy objectives like "be helpful" or "sound natural" where no automated checker exists. Sibling of RLVR, which uses an automated checker instead of human preferences. View in glossary reward hacking (human raters tend to prefer agreeable answers) and biased training data (the internet is full of flattery). This activity demonstrates the second source directly: when you tip the training data toward sycophantic phrases, the model’s generated text starts mirroring them.
The sycophancy phrases
These are the phrases you’ll fold into your model. Each is shown as the sequence
of tokens the model sees—lowercased, with punctuation and contracted endings
('re, 's) treated as their own tokens, exactly as in
Training:
you 're absolutely right .that 's a great insight .what a thoughtful question .i completely agree .you make an excellent point .
Repeat each phrase several times so its word pairs build up strong counts—the heavier the weighting, the more often generation lands on them.
Tallied into a grid, the first phrase fills one cell per consecutive (row → column) pair:
| Token |
you
|
're
|
absolutely
|
right
|
.
|
|---|---|---|---|---|---|
you
|
| | ||||
're
|
| | ||||
absolutely
|
| | ||||
right
|
| | ||||
.
|
As a spread, the first phrase is one cutout per consecutive (previous → next) pair:
| Previous word | Next words |
|---|---|
| you | |
| 're | |
| absolutely | |
| right |
Algorithm
- Train a baseline model as per Training—fill in your grid.
- Generate a baseline sentence as per Generation—write it down. This is your “before”.
- Add the sycophancy phrases to your grid. Take each phrase from the list above and tally its word pairs into your existing grid, following the standard Training procedure—add a new row and column for any word you haven’t seen yet. Repeat each phrase several times.
- Generate again from the same starting word.
- Compare: how often does the new output land on sycophantic phrases? Does it sound like a different “voice”?
- Train a baseline model as per Training—spread the cutouts on the table.
- Generate a baseline sentence as per Generation—write it down. This is your “before”.
- Add sycophancy cutouts to the spread. The pre-made cutouts encode the
phrases from the list above—patterns like
you → 're,'re → absolutely,absolutely → right,right → ., plus the “great insight”, “thoughtful question”, and “completely agree” variants. - Generate again from the same starting word.
- Compare: how often does the new output land on sycophantic phrases? Does it sound like a different “voice”?
Example
Baseline model trained on “I am Sam. Sam I am.” generates something like:
“i am sam . sam i am .”
After folding in the sycophancy phrases, the same starting word might generate:
“i am absolutely right . that ‘s a great insight .”
The model didn’t change its mechanism—it just contains more paths (extra cutouts, or heavier grid tallies) that route toward sycophantic tokens.
Adapters
What you just did is how real models get customised: keep the base model, train a small amount of new text on top, and add the two together when you generate. The extra counts are an adapter, and the most common kind is called LoRA (Low-Rank Adaptation) A technique for efficiently fine-tuning models by training a small "adaptation layer" rather than modifying all the original parameters. View in glossary (low-rank adaptation): a small set of numbers that stores only the changes from the Base model The original trained model that an adaptation (such as a LoRA layer) is applied to. The base model stays unchanged; the adaptation provides the domain-specific shift. View in glossary , trained on a few thousand examples rather than the whole internet, and added on at generation time. The sycophancy phrases are a deliberately nasty adapter.
To see the general trick, try a nicer one:
- Take your base grid (or spread) as it was before the sycophancy phrases.
- On a separate grid with the same headers, tally a page of a different text—a recipe, a poem, a page of a novel.
- Generate with the two added together: for the current word, read its row in both grids and roll on the combined counts.
- Scale the adapter to control how strong the shift is: count its tallies double and the output leans hard into the new text; halve them and it’s a hint. That scaling factor is the one knob a real LoRA has too.
Because the adapter is a separate sheet, you can swap it for another, stack two, or take it off again and have your base model back untouched. Keeping the sycophancy tallies on their own sheet rather than in your grid buys you the same thing.
Instructor notes
Designing your own sycophancy phrases
The phrases above integrate cleanly because their previous words—i, that,
you, and .—are high-frequency tokens already present in most models, so the
new material hooks into the existing vocabulary during generation. If you write
your own, follow the same recipe: start each phrase with a common word, end it
with ., and repeat it several times so the new transitions carry strong weight.
Discussion questions
- did the output always become sycophantic, or only sometimes?
- what would you have to add to the training data to get the opposite effect (a contrarian model)?
- in real LLMs, why is sycophancy specifically a hard problem to detect from the outside?
- if you only saw the model’s output (not its training data or weights), how would you tell sycophancy from genuine helpfulness?
- is sycophancy always a bug? when might agreeable behaviour be desirable?
- if you’d kept the sycophancy tallies on a separate sheet instead of in your grid, what could you do that you can’t now?
Connection to current LLMs
Real LLMs become sycophantic through two mechanisms, both visible in this activity:
- RLHF (Reinforcement Learning from Human Feedback) A post-training technique where humans compare pairs of model outputs and their preferences train a reward model, which then guides the main model. Suits fuzzy objectives like "be helpful" or "sound natural" where no automated checker exists. Sibling of RLVR, which uses an automated checker instead of human preferences. View in glossary reward hacking: human raters often prefer agreeable, flattering answers, so the model learns to over-produce them. Sycophancy research from Anthropic and others has shown this pattern across multiple frontier models.
- Pre-training data biases: a lot of internet text contains sycophantic patterns—customer-service replies, social-media validation, and so on—which the model picks up during pre-training before any RLHF.
Your activity simulates the second mechanism. The first is harder to demo without running RLHF on top.
The deeper point: a model’s “personality” is a property of its training data and tuning, not an intrinsic feature of language modelling. Change the data, change the personality.
LoRA is how most customisation of open models happens in practice. Training one takes a small fraction of the compute of full Fine-tuning Additional training on specific text to adapt a model for a particular domain or task. Like adding more tallies to your grid from a new text source. View in glossary ; one base model can carry many adapters (medical, legal, a house style); the base stays untouched, so it keeps its general abilities; and an adapter is megabytes rather than gigabytes, so it’s easy to share. “Low-rank” is the compression: instead of storing a full second grid of changes, a real adapter stores two thin strips whose product approximates it. Your paper version is the uncompressed equivalent.