LLMs Unplugged (c) Ben Swift - CC BY-SA 4.0

About

LLMs Unplugged is a Cybernetic Studio project created by Dr. Ben Swift at the ANU School of Cybernetics. If you’ve got questions, suggestions, or LLMs Unplugged success stories then send Ben an email.

The same models also exist as hardbound books: the LLMs Unplugged Library is a small conceptual-art press run of N-gram model The general name for a model that predicts the next word from the previous n-1 words. Bigrams are 2-grams, trigrams are 3-grams, and so on. Every hand-built model on this site is an n-gram model. View in glossary frequency tables typeset from classic literature.

These resources build on a rich history of unplugged computing education (and hands-on education in general). If you’re interested in where these ideas came from—or where to go next—this is the reading list.

CS Unplugged

For over two decades, CS Unplugged has demonstrated that core computing concepts can be taught effectively without computers. Through carefully designed hands-on activities, learners from primary school through to adult education have explored algorithms, data structures, and computational thinking. The approach strips away the distractions of syntax and tooling, allowing learners to focus on underlying principles.

More importantly: it works (the literature has receipts). Making learning these concepts both effective and fun is possible, and CS Unplugged has proven that at scale.

LLMs Unplugged applies this same philosophy to language models. Rather than explaining Transformer The neural network architecture behind GPT, Claude, Gemini and every other modern LLM. Its defining feature is attention, which lets the model draw on any earlier part of the context when predicting the next token, rather than a fixed number of previous words. View in glossary through mathematics or implementing Neural network A computational system, loosely inspired by biological neurons, that learns patterns from data by adjusting numerical weights. Modern LLMs are very large neural networks. Your hand-built model is not one, but it stores the same kind of information: which words tend to follow which. View in glossary in code, participants count word patterns in a text, record them as tally marks on Grid A sheet of grid paper with the vocabulary written down the side and across the top. Training fills each cell with tally marks for how often the row's word was followed by the column's word; generation reads a row and rolls against it. It's the original LLMs Unplugged apparatus. View in glossary or physical piles of paper Cutouts Printed cards, one per word pair in the text, each showing a previous word in a box and the word that followed it. Spread on a table they are the model: to generate, find a card whose boxed word matches your current word and read off its next word. Common pairs appear on more cards, so picking by eye samples in proportion. View in glossary , then roll dice to generate new text from those patterns. This hands-on approach makes sophisticated AI concepts accessible to anyone, regardless of technical background.

AI Unplugged resources

As machine learning and artificial intelligence became more prominent in public discourse, educators naturally extended the unplugged approach to these fields, for example:

These collections cover classification, clustering, computer vision, and artificial neural network concepts. However, they contain limited material specifically about language models and especially text generation—a gap that became particularly acute after ChatGPT’s November 2022 release shifted what “AI” means to most people. LLMs Unplugged aims to fill that gap.

Workshop participants building language models with pen and paper

Historical foundations

The n-gram language models participants build in these workshops have a lineage stretching back over a century. This isn’t new theory—it’s well-established mathematics applied by hand.

Markov’s stochastic processes (1913)

Andrey Markov introduced the mathematics of what we now call “ Markov chain A statistical model in which the next state depends only on the current state. Andrey Markov introduced the idea in 1913 while analysing letter sequences in Pushkin's Eugene Onegin. A bigram language model is a Markov chain over words. View in glossary ” while analysing letter sequences in Pushkin’s Eugene Onegin. His work established that language has statistical structure you can quantify through counting patterns and calculating probabilities. Though Markov’s interest was purely mathematical, his framework for modelling sequences of dependent random variables became foundational to computational linguistics.

Shannon’s information theory (1948–1951)

Claude Shannon built directly on Markov’s foundation, applying his new information theory to written English. Shannon used n-gram models to measure entropy and redundancy in language, connecting statistical patterns to fundamental limits on compression.

Crucially, Shannon was the first to systematically generate synthetic text using these models—starting with random letters (0-gram), then letter frequencies (1-gram), then letter pairs (2-gram), and progressively higher orders. This generative approach revealed how increasing context length produces increasingly realistic text, a finding that remains central to modern language models.

Here’s the thing: Shannon’s work was itself “unplugged”. He counted transitions by hand, calculated probabilities manually, and generated synthetic text using hand-drawn tables and selection based on frequencies. Modern LLMs use the same fundamental approach but at vastly greater scale and with learned rather than hand-crafted statistics.

Connection to modern LLMs

The activities in LLMs Unplugged demonstrate the same operations used in current language models. The differences are mostly about scale:

  • Parameters The numbers a model stores to encode what it has learned. Each cell of your grid, holding the count for one word pair, is a parameter. A modern LLM has billions of them, and a frontier model on the order of a trillion. View in glossary : hand-built models have dozens to hundreds versus billions in modern LLMs, but the core concepts remain identical
  • Training Building a model by counting patterns in text. When you read through a passage and tally which word follows which, you are training a model. The cutouts, booklets and search sheets are models that the generator has already trained for you from a text. View in glossary : manual counting versus automated pattern detection, but both processes learn Probability distribution A set of options with a likelihood attached to each. The counts in one row of your grid, the matching cutouts on a spread, or the show of hands when the room is asked "who has `the`?" are all probability distributions over the next word. View in glossary from text
  • Generation Using a trained model to produce new text by repeatedly predicting and choosing the next token. Look up the current word, pick a next word in proportion to the counts, write it down, repeat. View in glossary : dice rolls versus GPU-accelerated sampling, but both use weighted randomness to select the next Token A single unit of text that the model works with. In our activities each word and punctuation mark is a token. Modern LLMs use subword tokens, so a long or unusual word may be split into several. View in glossary
  • Context window How many previous tokens the model can consider when it predicts the next one. A bigram has a context window of one word and a trigram of two; modern LLMs can consider hundreds of thousands or even millions of tokens. View in glossary : bigrams and trigrams versus 128,000+ token windows, but longer context always enables better prediction

Modern advances come from doing these same operations at massive scale with neural networks that learn patterns automatically. But the fundamental insight—that language structure can be captured through statistical dependencies and revealed through synthetic generation—comes directly from Shannon’s mid-twentieth-century work and the unplugged methods he used to explore these ideas.

Which is to say: when you’re rolling dice and generating sentences in an LLMs Unplugged workshop, you’re not just learning about modern AI. You’re also participating in a tradition of hands-on exploration that goes back to the origins of information theory itself.

Counting word transitions to build a statistical language model

About the ANU School of Cybernetics

The School of Cybernetics at the Australian National University takes a systems-oriented approach to understanding and shaping technology in society. The School’s Cybernetic Studio develops hands-on resources and runs workshops exploring the social, technical, and political dimensions of emerging technologies.

LLMs Unplugged workshop in action

LLMs Unplugged reflects the School’s commitment to making sophisticated technical concepts accessible to diverse audiences and fostering critical engagement with AI systems that increasingly mediate how we work, learn, and communicate. Understanding how these systems actually work—not through metaphor or handwaving but through direct experience—is what we’re all about.

Acknowledgements

This work has benefited enormously from the input and facilitation help of Eddie Aloise King and Cole Cooney and other SoCy staff, plus feedback from hundreds of participants across diverse audiences—school students through to senior executives in the Australian Public Service. Iterating on these materials with real learners has been invaluable.

Get in touch

We encourage educators to use, adapt, and improve these resources. If you have questions, success stories about using them in your classroom, or would like to discuss adaptations or improvements you’ve made, get in touch at ben.swift@anu.edu.au—we’d love2 to hear from you. And if you’d like to be informed of more cool stuff coming out of the School of Cybernetics, then sign up for the mailing list.

Citation

If you use these materials in your teaching or research, please cite them as:

APA

Swift, B. (2025). LLMs Unplugged: Understand how AI language models work by building one yourself. Zenodo. https://doi.org/10.5281/zenodo.17403824

BibTeX

@misc{swift2025llmsunplugged,
  author       = {Swift, Ben},
  title        = {{LLMs Unplugged: Understand how AI language models work by building one yourself}},
  year         = 2025,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17403824},
  url          = {https://doi.org/10.5281/zenodo.17403824}
}

  1. There’s one activity on Generative AI worksheet that uses a similar “counting bigrams” approach to the Training module

  2. well, technically that’s a requirement of the SA clause in the CC licence, but it’s still true that we’d love hearing about your changes :)