LLMs Unplugged
How AI writes stories
Acknowledgement of Country
What is AI?
Think → share
when you hear the words artificial intelligence, what comes to mind?
think on your own, then share with the person next to you
Let’s ask one to write us a story
we need three words or ideas — hands up!
That was a language model
What just happened?
no person on the other side wrote that story for us
the language model learned patterns from huge amounts of text, then copied those patterns to make something new
today we’ll build our own language models — without a computer
Predict the next word
we humans are good at spotting patterns
I’ll start a sentence — hands up if you know the next word
So how do we teach a model?
humans pick up patterns from listening, reading, watching
a language model has to learn them from text
Some words we’ll use
artificial intelligence (AI) — technology that mimics some human abilities, like writing stories or answering questions
language model — a system that predicts what word comes next, using patterns it learned from text
training — building a model by finding patterns in text
training data — the text used to train a language model
Generation
This is a language model
one sheet of it: a word down the left, and beside it every word that came next
One row at a time
after the word the , this book said magpie five times, fence six times, and postie and dog four times each
Your group holds the whole model
every sheet holds a run of words in alphabetical order — the header says which
to find a word, read the headers: who has it ?
The colours
every word in a row sits in a coloured box, and your counters come in those same four colours
a red counter means the word in the red box
Start after a full stop
Fill the cup
Fill the cup
Fill the cup
Fill the cup
The cup is the row
Draw one
Fill the cup, draw one
Only one place to go
Fill the cup, draw one
Only one place to go
Unlikely isn’t impossible
Only one place to go
The same row, a different word
Fill the cup, draw one
“It sits on the fence. Here comes the magpie.”
a hint of the original, not a copy
Your turn
write down a starting word, then repeat:
- find the row for the last word you wrote
- fill the cup: one counter for every tally mark, each in its word’s colour
- draw one without looking — its colour tells you the next word
- write it down, tip the cup back into the tub, and pass the job to whoever has that word’s row
Shareback
No two stories the same
two groups with the same book: why aren’t your stories the same?
could any group’s model write a story about a crocodile?
Where did the marks come from?
Start with a text
Read it two words at a time
the then magpie
one mark beside magpie in the row for the
Move along by one
magpie then is
one mark beside is in the row for magpie
Move along by one
is then back
one mark beside back in the row for is
Move along by one
back then .
one mark beside . in the row for back
A new word after the same word
the then fence
one mark beside fence in the row for the
The same pair comes round again
. then it
one mark beside it in the row for .
Things that catch people out
. is a word: the row for . says how sentences start
a new word goes in the next empty box — don’t skip boxes, the colours go in order
one row per word, and say it out loud when you start one, or two people will keep counts of the same word
Your turn
read your new text two words at a time, and for each pair:
- find the row for the first word — no row yet? start one, and say the word out loud
- find the box for the second word — not there yet? write it in the next empty box
- add a mark in that box
- move along by one: the second word starts the next pair
One story, all together
Five texts, five models
each group trained on a different text, so each group’s row for the is different
together the class has a bigger model than any one group
The same word, five different rows
same word, five models, five different answers
How we’ll do it
I’ll call out the word we’re on
hands up if your sheets have a row for it — I’ll pick a group
that group reads out its colours and tallies, we fill the cup up here and draw: their word is the class’s next word
They call it out, we draw it
the class story grows by one word — and the new word is the next one we hunt for
One story, all together
About your stories
if you started with the same starting word, would you generate the same story again?
what would help our paper language models write better stories?
what is different about the way you write a story and the way a language model does?