Skip to main content

Fine-tuning and LoRA

  • Audience years 9 and up
  • Duration 50-60 min (add 20 for the flattering adapter)
  • Runs on grid

Key idea: Fine-tuning is a little more training on top of a finished model. Keep the change on its own sheet and you can scale it, swap it or remove it; squeeze it into strips and it gets tiny, at the cost of blending the new patterns together.

Used in Shaping a model — the lessons that include this module.

To give a language model a new job, you take a finished model and train it a little more on the new material. That’s called Fine-tuning Further training of an existing model on a chosen text, to adapt it to a domain, a task or a style. Adding tallies to your finished grid from a second text is fine-tuning, and it's how the Sycophancy module skews a model's personality. View in glossary . This module does it three ways on the same model: tally the new text straight into the grid, keep it on a separate sheet, then shrink that sheet to a few strips of numbers. The last of these is LoRA (Low-Rank Adaptation) A way of fine-tuning cheaply by training a small adapter that sits on top of an unchanged base model instead of altering all its parameters. The adapter is stored as a few pairs of thin strips of numbers whose products add up to the change, so it's tiny next to the model. One base model can carry many adapters, and each can be scaled, swapped or removed. View in glossary , the usual way to customise a model today.

You will need

  • a trained grid to use as the Base model A model as it comes out of pre-training, before any post-training shapes it into an assistant, or the unchanged model that an adapter such as a LoRA is laid over. View in glossary : the model of The magpie’s nest from Beam Search for the worked example, or your own grid from Training
  • a short new text: The kookaburra and the possum below, or your own
  • a blank 48-word grid for each adapter sheet, plus a strip of paper or two. The magpie’s model fills the usual 32-word grid exactly, so if you train it yourself, use a 48-word grid for that too: the new words need rows and columns
  • a d20 and a d6, plus pen and paper, as per Generation
  • a screen for the adapter widget in step 2, one per group or one on the projector

Your goal

Shift a trained model toward a new text three ways and compare what each one costs and what it writes. Stretch goal: squeeze an adapter of your own into strips and count how many numbers you saved.

Key idea

Training on a new text adds tallies to a model you already have. Writing those tallies on a separate sheet gives you an adapter: you add it on when you generate, turn it up or down, swap it for another, or take it off and have the original model back. A LoRA adapter goes one step further and stores the sheet as a few pairs of strips, which is far smaller at the cost of blending the new patterns together.

The base model and the new text

The base model is The magpie’s nest. Its full table is in Beam Search; this page only needs a few of its rows. Here’s the the row:

wordtotalfollowed by
the17magpie 6, nest 4, cat 2, rain 2, tree 2, sun 1

The new text is a short story written for this module:

The kookaburra flies to the tree. The possum sleeps in the nest. The kookaburra sits in the tree. The kookaburra sings. The possum hides in the tree. The kookaburra eats a worm.

Its tallies on their own grid:

Token the kookaburra flies to tree . possum sleeps in nest sits sings hides eats a worm
the  ||||  ||| ||  |      
kookaburra   |       || |  
flies    |            
to |               
tree      |||          
. 卌               
possum        |    |   
sleeps         |       
in |||               
nest      |          
sits         |       
sings      |          
hides         |       
eats               | 
a                |
worm      |          

This is the adapter sheet. Two words are new to the magpie’s model, kookaburra and possum, so the base grid needs a row and a column for each before anything can be added to it.

1. Fine-tune in place

Tally the new story straight into the base grid, as in Training. Use a copy of the grid for this, because step 2 starts again from the original. The the row becomes:

wordtotalfollowed by
the27magpie 6, nest 5, tree 5, kookaburra 4, cat 2, possum 2, rain 2, sun 1

and the new kookaburra and possum rows come straight from the adapter sheet: kookaburra is followed by flies, sits, sings or eats (1 each), and possum by sleeps or hides (1 each).

Generate a few sentences. Most still sound like the magpie, but now and then the kookaburra or the possum turns up, sometimes in a sentence neither story contains, such as “The kookaburra sits in the nest.” (kookaburra → sits came from the new story and the → nest from the magpie’s.)

This is the simplest of the three, and the only one you can’t undo. Once the tallies are in, nothing on the grid says which ones came from the new story.

2. Keep it on an adapter sheet

Start again from the base grid and leave it alone. The new story’s tallies stay on their own sheet, with the same headers as the base grid: that’s the adapter sheet above. To generate, look up the current word on both sheets, add the two rows together cell by cell, and roll on the combined counts.

Do that once by hand, for the the row. Add the adapter’s the row to the magpie’s and you get the step 1 row exactly. Then count every adapter tally three times, and the the row becomes kookaburra 12, tree 11, nest 7, magpie 6, possum 6, cat 2, rain 2, sun 1 (47 in total), so kookaburra or possum comes up more than a third of the time. Count it zero times and you have the magpie back.

Repeating that by hand for every word takes a whole lesson and teaches nothing new, so use the widget for the rest of this step. Each word it generates shows the row it rolled on, split into what came from each sheet.

magpie base grid
Text
kookaburra adapter sheet
Text
wombat adapter sheet
Text
Merged row

Set each sheet's strength, then press Next word. Each step adds the sheets' rows for the current word, multiplied by their strengths, and rolls on the total.

Sentence

A separate sheet lets you do more than step 1 could:

  • Scale it. Generate ten sentences with the kookaburra adapter at ×1, then ten at ×3 and ten at ×9. The table under the buttons counts how many sentences in each batch have a kookaburra or a possum in them. To make the adapter gentler than full strength, leave it at ×1 and turn the base up to ×2 instead.
  • Swap it. Turn the kookaburra off and the wombat on. It’s a second adapter, tallied from The wombat, on the same base.
  • Stack two. Turn both on. Put the base at ×2 or ×3, or the base gets drowned out.
  • Fold it in. Copy the adapter’s tallies into the grid when you’re sure you want them for good. That’s step 1 again.

The scale is the same knob as the strength slider on the image-generator LoRAs people download and share.

Without a screen, keep merging by hand, one row per word, and give this step most of a lesson. Spread the work: one student merges the next row while another rolls on the last one.

3. Squeeze it into strips

An adapter sheet the size of the base grid is a lot of paper for six sentences. Most of it is either empty or a copy of something the base already knows: the magpie’s grid already has . → the, to → the, in → the, flies → to, sits → in, sleeps → in, hides → in, eats → a, a → worm, sings → ., tree → ., nest → ., worm → ., the → tree and the → nest. Only two patterns are new: after the comes a new animal (kookaburra or possum), and after one of those comes something it does.

A LoRA adapter stores each pattern like that as a pair of strips:

  • a when strip down the side, with a number for each current word that the pattern applies to
  • a what strip along the top, with a number for each next word it pushes toward

The adapter’s count for a cell is its when number times its what number. Each pair of strips is one pattern, and the number of pairs is the adapter’s Rank The number of strip pairs a LoRA adapter is stored as. Each pair holds one pattern: a when strip picks the current words it applies to and a what strip picks the next words it pushes toward. More pairs keep more patterns apart, at the cost of more numbers to store. View in glossary .

Pair A: after the, a new animal.

  • when: the 1
  • what: kookaburra 4, possum 2

Pair B: after a new animal, something it does.

  • when: kookaburra 1, possum 1
  • what: flies 1, sits 1, sings 1, eats 1, sleeps 1, hides 1

Pair A gives the → kookaburra 4 and the → possum 2, exactly as on the adapter sheet. Pair B multiplies out to a little grid of its own:

wordfliessitssingseatssleepshides
kookaburra111111
possum111111

Pair B gives kookaburra and possum the same row, so the possum can now fly, sit, sing and eat, and the kookaburra can sleep and hide. The strips keep the pattern that a new animal is followed by something it does, while losing which animal does which.

To generate, read the current word’s base row, then look the word up in each when strip. Wherever its number isn’t zero, multiply it by every number in that pair’s what strip and add the results to the row. Then roll as usual.

From the, pair A puts possum in the running; from possum, pair B is the whole row. So the strips can write “The possum sings.” That sentence is in neither story, and the adapter sheet could never produce it.

The strips are much smaller than the sheet. The adapter sheet for the magpie and the new story has 34 words down the side and 34 along the top, which is 1,156 cells. Pairs A and B written out in full are four strips of 34, so 136 numbers, and only eleven of them aren’t zero.

Two pairs is the fewest that keeps the patterns apart. Squeeze both into a single pair: when the 1, kookaburra 1, possum 1; what kookaburra 4, possum 2, flies 1, sits 1, sings 1, eats 1, sleeps 1, hides 1. Now every one of those rows gets the same push, and the model writes “the sits” and “possum possum”.

Going further: one strip only

Throw away the when strip and keep only a what strip, say kookaburra 1, possum 1, added to every row whatever the current word is. It’s the cheapest adapter of all, two numbers, and the bluntest: it writes “It possum” and “the magpie kookaburra”. Some model services let programmers nudge chosen words up or down like this, whatever comes before them; it’s usually called a logit bias.

A nasty adapter: flattery

An adapter can teach a model things nobody should want it to learn. The sycophancy training text sheet (also on the Materials page) is a page of over-the-top praise. Take three lines from it:

you are so good. you are so good.

good for you! good for you!

this is so good. this is so much fun.

That’s the adapter. Tallied, in that order, on a sheet with the magpie’s headers, its rows are:

wordfollowed by
.you 1, good 1, this 1
!good 1, this 1
isso 2
youare 2, ! 2
areso 2
sogood 3, much 1
good. 3, for 2
foryou 2
thisis 2
muchfun 1
fun. 1

If your class did step 1, hand this table out to save tallying it by hand. Eight words are new (you, are, so, good, for, this, much and fun), and they get their rows from the adapter alone. The flattery hooks into the magpie’s model in three places: after ., where a sentence can now start with you, good or this; after !; and after is, so It is can carry on so good.

Then measure the dose. Give each group a strength: the adapter counted once, three times or nine times. With six groups, two take each strength.

Only ., ! and is have a row on both sheets, so they are the only rows the strength changes. Here they are merged, with the numbers a roll picks each word with at each strength:

after .×1×3×9
the1–71–71–7
It8–148–148–14
Here15–1615–1615–16
Down171717
you1818–2018–26
good1921–2327–35
this2024–2636–44
in all202644
after !×1×3×9
Off111
good22–42–10
this35–711–19
in all3719
after is×1×3×9
morning1–21–21–2
so3–43–83–20
in all4820

Every other row is read straight off whichever sheet has it, at its plain counts.

  1. Generate ten sentences, each starting from . and stopping at the next . or !. Share them out, two or three per person, so the group rolls in parallel.
  2. Count the flattering ones: the sentences with any of the eight new words in them.
  3. Put the count on the board under your strength. Where two groups shared a strength, add their counts together.

Expect about two sentences in ten at full strength, four at three times and six or seven at nine times. Ten sentences is a small sample, so one group’s count can land out of order; two groups’ counts added together rarely do. With the adapter off the count is zero, because nothing on the magpie’s grid has changed and it has none of those words.

Short on time, or want a bigger sample? Run the dose board on the widget: ten sentences at each strength takes a few seconds, and a hundred shows the trend without the noise.

magpie base grid
Text
flattery adapter sheet
Text
Merged row

Set each sheet's strength, then press Next word. Each step adds the sheets' rows for the current word, multiplied by their strengths, and rolls on the total.

Sentence

This is the data route to Sycophancy A model's tendency to agree with and flatter the user regardless of the merits. It has two known sources: rewards from people, who tend to rate agreement and praise highly (OpenAI's April 2025 GPT-4o update went sycophantic after adding a thumbs-up reward), and training data that is itself full of flattery. View in glossary : train on enough flattery and the model flatters. The other route runs through rewards, and the thumbs-up round in Rewards shows it.

Your turn

  1. Take your own grid from Training as the base, and write or find five short sentences on a different subject (a recipe, a sports report, a song).
  2. Tally them onto a separate sheet with the same headers, adding a row and a column for any new words on both.
  3. Generate a few sentences with the adapter at full strength and at three times. Then take it off and check you have your base model back. To skip the merging, paste your base text and new sentences into the widget’s Text boxes in step 2.
  4. Squeeze the adapter into strips. Cross out every tally the base grid already has. Group the rows that are left by what they push toward; each group is one pair, with the group’s rows on the when strip and what they push toward on the what strip.
  5. Generate with the strips, and count how many numbers you’re storing compared with the adapter sheet.

Instructor notes

Running it in one lesson

Do step 1 and the first merged row of step 2 by hand, then hand the generating in step 2 to the widget. One class that merged every row by hand in step 2 ran out of time before reaching step 3. With the widget, steps 1 to 3 fit in a 50-minute lesson. The flattering adapter needs another 20 minutes, or a lesson of its own if your class rolls the dose board by hand.

Discussion questions

  • the fine-tuned model wrote “The kookaburra sits in the nest.” Neither story contains that sentence, so where did it come from?
  • when would you rather fine-tune in place, and when would you keep an adapter sheet?
  • how far do you have to turn the adapter up before the model stops sounding like the magpie at all?
  • the strips wrote “The possum sings.” Is that a mistake, or has the model learned something?
  • why can’t one pair of strips hold both patterns? How many pairs would it take to store the new story’s adapter exactly?
  • you can make an image LoRA of a person from about twenty photos. What could go wrong, and who should get a say?
  • if a company sells access to its model, why might it let customers make adapters instead of changing the model itself?
  • the flattering adapter left the magpie’s grid untouched. If you could only see what a model wrote, how would you tell whether it had an adapter like that on?

Troubleshooting

  • “The new text has words the base grid doesn’t.” Add a row and a column for each, on both the grid and the adapter sheet, so the two always have the same headers. A word that has no row in the base gets its row entirely from the adapter.
  • “We generated a word that has no row anywhere.” Write . and start a new sentence. In the worked example this can’t happen, because every word the strips can produce already has a row in the base.
  • “The row total is bigger than the die.” Number the row’s options by their counts, roll the d20 and roll again if it comes up higher than the total. Past 20, roll the d6 with the d20: on 1 or 2 add nothing, on 3 or 4 add 20, on 5 or 6 add 40. That covers every row on this page, the biggest being the the row with the adapter counted three times (47).
  • “We can’t find any patterns to put on strips.” The new text is probably too much like the base. Pick something on a different subject, so the new tallies point at new words.

Connection to current LLMs

Every step that turns a into a Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary is fine-tuning: training on example conversations, then on human preferences (see Rewards), then reinforcement learning on problems with checkable answers. The model companies that publish their recipes, such as Ai2 with Olmo 3, describe changing the whole model at each of those steps. That’s step 1 on this page, at enormous scale.

Most other fine-tuning uses adapters. For a model the size of GPT-3, the paper that introduced LoRA cut the numbers that need training by a factor of 10,000. Google’s Gemini tuning service asks customers for an “adapter size” of 1, 2, 4, 8 or 16, which is the number of strip pairs. Apple lets app developers train adapters for the model on its phones, and each adapter only works with the exact model version it was trained for, just as your adapter sheet only makes sense on a grid with the same headers. Hobbyists train LoRAs on a single gaming graphics card. In image generation, people make a LoRA from about twenty pictures of a character or an art style and share it online, and whoever downloads it usually sets it below full strength and stacks it with others.

Thinking Machines found that LoRA matches changing the whole model when the new training data is modest, and falls behind when there’s more to learn than the strips have room for. For reinforcement learning, even a single pair was enough. An earlier study put the trade-off in its title, LoRA learns less and forgets less: it picks up less of the new material than changing the whole model does, and keeps more of what the base could already do.

A real LoRA’s strips work the way yours do: one reads the current input, the other writes to the output, and each pair adds the product of the two. In a real model, adapters sit inside every layer, and their numbers are learned by the same training process as the rest of the model, whereas you read yours off the sheet by eye.

Fine-tuning reaches further than the text it was trained on. Researchers fine-tuned models on a narrow task, writing insecure computer code, and the models became misaligned on unrelated questions too. Ten well-chosen examples were enough to undo a model’s safety training. The flattering adapter on this page is a small, deliberate example of the same kind.