Pre-trained Model Generation
Use a (slightly larger) pre-trained model booklet to generate new text through weighted random sampling.
You will need
- a pre-trained model booklet
- a d10 for weighted sampling
- pen and paper for your generated text
Download pre-trained model booklets
While it’s fun to upload your own documents to create your very own language model using the online model booklet generator here are a few examples to get you started.
- Green Eggs and Ham (Dr Seuss, 5 pages)
- The Cat in the Hat (Dr Seuss, 7 pages)
- Beatles Lyrics (35 pages)
- A Christmas Carol (Dickens, 52 pages)
- Frankenstein (Shelley, 101 pages)
- Collected Hemingway (379 pages)
Your goal
Generate text from a pre-trained language model without training it yourself. Stretch goal: try to guess the training text before checking the booklet title.
Key idea
Pre-trained models capture patterns from larger datasets. You can use them just like your hand-built bigram model: look up the current word, roll dice, choose the next word, and repeat.
Algorithm (quick recap)
- Choose a starting word—any bold word in the booklet—and write it down.
- Look up that word’s entry to see possible next words and their thresholds.
- Roll your d10(s):
- If there’s an indicator (e.g., a small box with
2), roll that many d10s and treat the digits as one number. - Otherwise, roll one d10.
- If there’s an indicator (e.g., a small box with
- Scan down the options; the first threshold greater than or equal to your roll gives the next word. Write it down.
- Repeat from step 2 using the new word until you reach a natural stopping point or your desired length.
Example
Before you try generating text yourself with a booklet, work through these examples to see the algorithm in action.
- Single d10: Current word
catwith options4|sat,7|ran,10|slept. Roll once; a 6 lands you onran. - Multiple d10s: Current word
thewith a2indicator and options33|cat,66|dog,99|end. Roll twice; a 5 and 8 combine to 58, so you choosedog.
Instructor notes
Discussion questions
- can you guess what text the model was trained on from the generated output?
- how does using a pre-trained model differ from training your own?
- what vocabulary size does the booklet model have compared to your hand-built model?
- why might some word combinations feel more natural than others?
- without looking at the title: can you identify the training text’s genre or style?
Connection to current LLMs
This lesson demonstrates the foundation of how people interact with modern AI:
- pre-training: companies train massive models on huge text corpora (like your booklet model, but with trillions of words)
- LLM-as-a-service: users generate text without seeing or modifying the underlying model (just like using the booklet)
- model distribution: the booklet format shows how models can be packaged and shared—current LLMs are distributed as parameter files
- deterministic generation: given the same starting word and dice rolls, you get the same output (though randomness creates variety between runs)
The key insight: training and text generation are separate processes. Most AI users never train models—they use pre-trained ones through APIs or interfaces. Your hand-built model from Training gives you insight into what’s inside the booklet, but you don’t need that knowledge to generate text. This separation is why companies like OpenAI can provide AI services: they do the expensive training once, then millions of users perform text generation. The booklet captures thousands of training examples in a portable format, just as neural networks compress training data into billions of parameters.
Interactive widget
Step through the booklet lookup process at your own pace. The model is displayed in the same format as the printed booklets: each entry shows a bold prefix word, diamond symbols indicating how many d10s to roll, and the possible next words with their thresholds.