LLMs Unplugged
How AI writes stories
Acknowledgement of Country
Some words we’ll use
artificial intelligence (AI) — technology that mimics some human abilities, like writing stories or answering questions
language model — a system that predicts what word comes next, using patterns it learned from text
training — building a language model by finding patterns in text (its training data)
generation — using a language model to write new text, one word at a time
Generation
This is a language model
One row at a time
Your group holds the whole model
The colours
Worked example
“It sits on the fence. Here comes the magpie.”
the book said “Here comes the postie.” and “Back goes the magpie.”
Your turn
write down a starting word, then repeat:
- find the row for the last word you wrote
- fill the cup: one counter for every tally mark, each in its word’s colour
- draw one without looking — its colour tells you the next word
- write it down, tip the cup back into the tub, and pass the job to whoever has that word’s row
Shareback
No two stories the same
two groups with the same book: why aren’t your stories the same?
could any group’s model write a story about a crocodile?
Where did the marks come from?
Listen for the word “the”
It is spring. It sits on the fence. It watches. Here comes the postie. Down swoops the magpie! Off runs the postie. Back goes the magpie. It sits on the fence. Here comes the dog. Down swoops the magpie! Off runs the dog. Back goes the magpie. It sits on the fence. Here comes the postie and the dog. Down swoops the magpie! Off run the dog and the postie. Back goes the magpie. It sits on the fence. It watches.
Check the sheet
It is spring. It sits on the fence. It watches. Here comes the postie. Down swoops the magpie! Off runs the postie. Back goes the magpie. It sits on the fence. Here comes the dog. Down swoops the magpie! Off runs the dog. Back goes the magpie. It sits on the fence. Here comes the postie and the dog. Down swoops the magpie! Off run the dog and the postie. Back goes the magpie. It sits on the fence. It watches.
About your stories
if you started with the same starting word, would you generate the same story again?
what would help our paper language models write better stories?
what is different about the way you write a story and the way a language model does?