Demystifying large language models#
it’s hard to have an opinion
on a machine you’ve never seen inside
a page of token pairs — a word, and the word before it
the algorithm#
- I call out the last token we wrote (and its colour)
- hands up if it’s the boxed token on your page
- I pick a hand at random — you read out the token beside it
- I write it down, and we go again
we just ran a language model
what role did you play?#
the pairs on your page → the parameters of the model
the hands in the air → the probability distribution over next tokens
my random pick → sampling
the sentence I wrote → generation
the sentence we wrote is in no book
three books#
“He was an old man who fished alone in a skiff in the Gulf Stream.”
— Ernest Hemingway, The Old Man and the Sea (1952)
“TRUE! nervous, very, very dreadfully nervous I had been and am.”
— Edgar Allan Poe, The Tell-Tale Heart (1843)
“The sun did not shine. It was too wet to play.”
— Dr. Seuss, The Cat in the Hat (1957)
ANU was founded in 1946
for a country facing a transformed world
a hall full of people just did
the thing you’re told needs a data centre
it doesn’t make you an expert
it makes you harder to bullsh*t