Pre-trained Model Generation

• Audience years 5 and up
• Duration 20 min
• Runs on booklet

Key idea: You can generate from a model you didn't train---just follow its lookup rules and sample next words.

Used in My First Language Model and Under the hood --- ready-to-run lessons and talks that include this module.

Use a (slightly larger) pre-trained model booklet to generate new text through weighted random sampling Weighted random sampling Choosing the next token with probability proportional to its count. A d10 roll against the counts does this, and so does letting your eye land on one of the matching cutouts, because common words appear on more of them. View in glossary .

You will need#

• a pre-trained model booklet Booklet A printed dice lookup table for a text: one entry per previous word (or pair of words, in a trigram booklet), listing the possible next words with the d10 range that selects each. A booklet is a pre-trained model you can generate from without having trained it. View in glossary
• a d10 for weighted sampling
• pen and paper for your generated text

Your goal#

Generate text from a pre-trained language model Language model A system that predicts what text comes next, using patterns learned from training data. Your hand-built grid, a cutouts spread, a booklet, and a room holding search sheets are all language models. View in glossary without training it yourself. Stretch goal: try to guess the training text before checking the booklet title.

Key idea#

Pre-trained models capture patterns from larger datasets. You can use them just like your hand-built bigram model Bigram model A model that predicts the next word from one previous word. It's what you build in the fundamental modules: each row of your grid says what can follow a single word. View in glossary : look up the current word, roll dice, choose the next word, and repeat.

Algorithm (quick recap)#

1. Choose a starting word—any bold word in the booklet—and write it down.
2. Look up that word’s entry to see possible next words and their thresholds.
3. Roll your d10(s):
   • If the entry shows diamonds (e.g. ♦♦), roll one d10 per diamond and read the digits as one number.
   • Otherwise, roll one d10.
4. Scan down the options; the first threshold greater than or equal to your roll gives the next word. Write it down.
5. Repeat from step 2 using the new word until you reach a natural stopping point or your desired length.

Example#

Before you try generating text yourself with a booklet, work through these examples to see the algorithm in action.

• Single d10: Current word cat with options 4|sat, 7|ran, 10|slept. Roll once; a 6 lands you on ran.
• Multiple d10s: Current word the with two diamonds (♦♦) and options 33|cat, 66|dog, 99|end. Roll twice; a 5 and 8 combine to 58, so you choose dog.

Instructor notes#

Discussion questions#

• can you guess what text the model was trained on from the generated output?
• how does using a pre-trained model differ from training your own?
• what vocabulary Vocabulary All the distinct tokens a model knows. The words down the side and across the top of your grid, or the set of previous-word labels on a cutouts spread, form your vocabulary. A model can only ever produce a token from its vocabulary. View in glossary size does the booklet model have compared to your hand-built model?
• why might some word combinations feel more natural than others?
• without looking at the title: can you identify the training text’s genre or style?

Troubleshooting#

• “Which die is the tens digit?” When rolling multiple d10s (i.e. in the “multiple diamond” case), it doesn’t really matter which die you read first—as long as you decide before you roll and stay consistent. One easy convention: use different-coloured dice and always read them in the same order (e.g. red first, then blue). But really, don’t stress too much about it. The activity will still work fine if you mix them up.

Connection to current LLMs#

This module demonstrates the foundation of how people interact with modern AI:

• pre-training Pre-training The first, expensive training phase, where a model learns the general patterns of language from a huge corpus. Most people who use an LLM never train one; they use a pre-trained model, the way you generate from a booklet somebody else made. View in glossary : companies train massive models on huge text corpora Training data The text a model is trained on. In our activities this is the passage you tally, or the book a set of cutouts, a booklet or a set of search sheets was generated from. Modern LLMs are trained on trillions of words from books, websites and other sources. View in glossary (like your booklet model, but with trillions of words)
• LLM-as-a-service: users generate text without seeing or modifying the underlying model (just like using the booklet)
• model distribution: the booklet format shows how models can be packaged and shared—current LLMs are distributed as parameter files
• deterministic generation: given the same starting word and dice rolls, you get the same output (though randomness creates variety between runs)

The key insight: training and text generation are separate processes. Most AI users never train models—they use pre-trained ones through APIs or interfaces. Your hand-built model from Training gives you insight into what’s inside the booklet, but you don’t need that knowledge to generate text. This separation is why companies like OpenAI can provide AI services: they do the expensive training once, then millions of users perform text generation. The booklet captures thousands of training examples in a portable format, just as neural networks Neural network A computational system, loosely inspired by biological neurons, that learns patterns from data by adjusting numerical weights. Modern LLMs are very large neural networks. Your hand-built model is not one, but it stores the same kind of information: which words tend to follow which. View in glossary compress training data into billions of parameters Parameters The numbers a model stores to encode what it has learned. Each cell of your grid, holding the count for one word pair, is a parameter. A modern LLM has billions of them, and a frontier model on the order of a trillion. View in glossary .

Interactive widget#

Step through the booklet lookup process at your own pace. The model is displayed in the same format as the printed booklets: each entry shows a bold previous word, diamond symbols indicating how many d10s to roll, and the possible next words with their thresholds.