Workshop formats

Format 1: Fundamentals

time: 90mins
for ages: 12+

This core workshop covers the essential training-to-generation pipeline. Start with a brief introduction to set the scene, then move through training a model, generating text from it, and finally exploring what happens when you use a larger pre-trained model. Each step builds on the last, giving participants a complete picture of how language models work.

Training

Build a bigram language model that tracks which words follow which other words in text.

Generation

Use your hand-built bigram model to generate new text through weighted random sampling.

Pre-trained Model Generation

Use a provided pre-trained booklet to generate text without training your own model.

Suggested timing

00:00 kick off — who in the room has used ChatGPT, Claude, or similar tools? what for? how do you think it works? today we’re going to build our own version—small and simple, but using the exact same approach.
00:20 Training
00:40 Generation
01:00 Pre-trained model generation
01:20 close — how has this workshop changed how you think about language models? how has this workshop changed how you will use language models?

Notes for Fundamentals

this outline doesn’t include the Weighted randomness lesson, but if participants aren’t so familiar with that stuff then you could add it in before the Training lesson (add another 30mins)
once you get to the Generation lesson and beyond, get participants to do “dramatic readings” as they share back the text their new language models have generated...
if you have a bit longer, then adding the Sampling lesson at the end is a fun option—it builds on either the Generation or the Pre-trained model generation work and shows how different parts of the “LLM process” can have different effects on the output

Format 2: Going deeper

time: 2–3 hours (or split across sessions)
for: senior high school onwards, or any group ready to tackle more abstract material

For groups ready to go further, this extended trajectory adds the “how models understand” topic. After covering the fundamentals, you explore how models reuse earlier context, complete patterns from a prompt, and how words get represented as numerical vectors. This path suits later-year high school students, computing electives, professional-development cohorts, or anyone who wants to understand what “ Attention mechanism The ability to focus on relevant previous words when making predictions. In real LLMs, attention is learned, weighted, and dynamic---the model decides what to focus on for each prediction. The in-context memory and induction-head lessons illustrate the motivation for attention: reusing more than just the immediately preceding word. View in glossary ” and “ Embedding A numerical representation of a word. Each row in your bigram grid is that word's embedding vector---a fingerprint of its usage context. In real LLMs, embeddings are learned separately rather than derived from raw counts, but the principle is the same: words used in similar ways get similar vectors. View in glossary ” actually mean.

Training

Build a bigram language model that tracks which words follow which other words in text.

Generation

Use your hand-built bigram model to generate new text through weighted random sampling.

Pre-trained Model Generation

Use a provided pre-trained booklet to generate text without training your own model.

In-context Memory

Add a short-term memory that nudges generation toward recently-used words, keeping text on topic---a hands-on model of attention and in-context learning.

Induction Heads

Add a rule that completes patterns from the text itself---find the last time the current word appeared and copy what followed it---the circuit behind in-context learning.

Word Embeddings

Turn each word's row into a vector and measure similarities between words in your model.

What these additions cover

In-context memory adds a short-term memory that nudges generation toward recently-used words, so the text stays on topic. It’s a hand-run version of what transformer “attention” does—reusing earlier context, not just the previous word.
Induction heads goes a step further: find the last time the current word appeared and copy what followed. This completes patterns the model never trained on—the circuit behind in-context learning and few-shot prompting.
Word embeddings turns each word’s row in the model into a numerical vector and measures similarities between words. Words that behave similarly in the training text end up close together. This is the foundation of how modern LLMs represent meaning—and you can calculate it by hand.

Why split the trajectory?

The fundamentals work for any audience and require only 90 minutes. The “understanding” lessons require more time and comfort with abstraction, but they connect directly to concepts anyone will encounter in deeper study of AI: attention mechanisms, embeddings, vector similarity. Running them as a second session (or a follow-up for the interested) keeps the core workshop accessible while offering a clear path forward.

Format 3: Controlling output

time: 30mins (as an add-on)
for ages: 14+

Once participants can generate text, a natural question is: “How do you make it more or less creative?” The sampling lesson shows how temperature and truncation strategies change the character of output without changing the model itself. This is a quick add-on to either the fundamentals or the deeper trajectory.

Sampling

Same model, different output---experiment with temperature and truncation strategies to shape how your model picks the next word.

This lesson explains:

temperature: how dividing counts by a temperature value flattens or sharpens the distribution, making surprising words more or less likely
truncation: strategies like greedy selection, no-repeat, or even haiku constraints that narrow which words are eligible before sampling

Participants discover that “creativity” in AI comes from two controls: adjusting probability distributions and filtering which tokens to consider. The same model can produce cautious prose or wild poetry just by tweaking these parameters.

Adaptation and data

For groups interested in data science, ethics, or media literacy, the “Adaptation and data” topic explores what happens when models train on their own output.

LoRA

Add a lightweight adaptation layer to retarget a trained model without retraining everything.

Synthetic Data

Generate synthetic text with your model, retrain on it, and see how patterns drift or collapse.

RLHF

Use human preferences to adjust your model's weights, making it generate text people prefer.

Sycophancy

Skew your model toward over-agreeable, flattering responses by piling in sycophantic training data.

The Synthetic data lesson is particularly effective for discussions about:

AI-generated content flooding the internet
model collapse and why training data quality matters
the difference between human-written and AI-generated text

This works well as a standalone activity after participants have done basic training and generation, or as part of a broader unit on AI ethics and media literacy.