LLMs Unplugged (c) Ben Swift, Cybernetic Studio - CC BY-NC-SA 4.0

Try it yourself

Twenty minutes, one person, a pen, some paper and something random: a ten-sided die if you have one, otherwise a coin, or the random-number function on your phone. By the end you’ll have built a Language model A system that predicts what text comes next, using patterns learned from training data. Your hand-built grid, a cutouts spread, a booklet, and a room holding search sheets are all language models. View in glossary , used it to write a sentence nobody has written before, and seen what Chatbot (ChatGPT, Claude, Gemini) A product that wraps an LLM in a conversation. ChatGPT is OpenAI's, Claude is Anthropic's, Gemini is Google's, and there are many others. On this site we often name ChatGPT or Claude as shorthand for any of them: the concepts apply equally to every modern LLM chatbot, whichever one you use. View in glossary is doing when it writes one.

1. Pick a text

Something short that you know well. A verse of a song, the opening of a picture book, a nursery rhyme. Shorter is better for a first go: three or four sentences. We’ll use this one:

See Spot run. See Spot jump. Run, Spot, run. Jump, Spot, jump.

2. Train it

Write every word in lowercase, and treat each punctuation mark as a word of its own:

see spot run . see spot jump . run , spot , run . jump , spot , jump .

Now draw a Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary : one row and one column for each distinct word (here that’s see, spot, run, jump, . and ,, so six rows and six columns). Read through the text one pair of neighbouring words at a time—see→spot, spot→run, run→., and so on—and for each pair put a tally mark in the cell at that row and column. When you reach the end, the grid is your model: each row says which words followed that word, and how often.

Here it is being built, one pair at a time:

Training text
Training text (tokenised)
thecatsatonthemat.
Model
Tokenthecatsatonmat.
the​​​​​​
cat​​​​​​
sat​​​​​​
on​​​​​​
mat​​​​​​
.​​​​​​

That’s the whole of training. A real language model does exactly this over trillions of words, with the tallies stored as numbers instead of pencil marks. (If you’d rather not draw the grid, there’s a printable one, and the Training module has the long version with worked examples.)

3. Generate

Pick a starting word and write it down. Look along its row: the tallies are the candidates for the next word, and the more tallies, the more likely. Choose one at random in proportion to the tallies, write it down, then do the same from the word you just wrote. Keep going until you hit a full stop, or you’ve had enough.

How to choose in proportion:

  • two candidates with equal tallies: flip a coin
  • a d10: share the numbers 1–10 among the candidates in proportion (three tallies against one becomes 1–7 for the first, 8–10 for the second), roll, and read off the winner
  • your phone: ask it for a random number between 1 and the total number of tallies in the row, then count along the row to see which word it lands in

Watch the same thing happen with dice:

Training text
Training text (tokenised)
thecatsatonthemat.
Model
Tokenthecatsatonmat.
​|​​|​
​​|​​​
​​​|​​
|​​​​​
​​​​​|
.​​​​​​
Output

4. Read it out

Read your sentence aloud. It will be half sensible and half nonsense, and it appears nowhere in the text you started from: the model made it, by looking one word back at a time and rolling. Now do it again from the same starting word and get a different sentence.

What you just did

  • the grid is the model; its tallies are the Parameters The numbers a model stores to encode what it has learned. Each cell of your grid, holding the count for one word pair, is a parameter. A modern LLM has billions of them, and a frontier model on the order of a trillion. View in glossary (a frontier model has about a trillion of them, same idea)
  • reading through the text and tallying was Training Building a model by counting patterns in text. When you read through a passage and tally which word follows which, you are training a model. The cutouts, booklets and search sheets are models that the generator has already trained for you from a text. View in glossary
  • the tallies in a row are a Probability distribution A set of options with a likelihood attached to each. The counts in one row of your grid, the matching cutouts on a spread, or the show of hands when the room is asked "who has `the`?" are all probability distributions over the next word. View in glossary over the next word
  • the roll was Weighted random sampling Choosing the next token with probability proportional to its count. A d10 roll against the counts does this, and so does letting your eye land on one of the matching cutouts, because common words appear on more of them. View in glossary , and the sentence is Generation Using a trained model to produce new text by repeatedly predicting and choosing the next token. Look up the current word, pick a next word in proportion to the counts, write it down, repeat. View in glossary

Every reply from Claude or ChatGPT is produced by that loop: look at the context, weigh the options, pick one, write it down. What’s different is the scale, how far back the model looks, and machinery that can spread what it learns about one word onto related ones. It’s still words in, words out.