Word Embeddings

• Audience years 9 and up
• Duration 20 min
• Runs on grid

Key idea: A word's row in the grid is its embedding. Words that keep the same company have similar rows, which is why they can stand in for each other in the text.

Used in Under the hood --- ready-to-run lessons and talks that include this module.

Your bigram grid already holds an opinion about every word in it: the row says what tends to follow that word, the column says what tends to come before it. Two words with much the same row and much the same column are, as far as the model is concerned, the same kind of word. This module finds a pair like that and checks it three ways—by eye, by swapping the two words in the text, and by one piece of arithmetic.

You will need#

• your completed bigram grid, built from a few sentences or more
• the original text you trained on
• pen and paper
• a screen for the last step (one between a pair is plenty)

Your goal#

Predict which two words in your grid behave most alike (call them the twins), then test the prediction. Stretch goal: find a close pair in the widget that nobody in the room guessed, and work out what the two words have in common.

Key idea#

A word’s row is a list of numbers: how often each other word followed it. That list is the word’s embedding Embedding A numerical representation of a word. Each row in your bigram grid is that word's embedding vector---a fingerprint of its usage context. In real LLMs, embeddings are learned separately rather than derived from raw counts, but the principle is the same: words used in similar ways get similar vectors. View in glossary —a fingerprint of the company it keeps. Two words with similar fingerprints get used in the same places, which is why you can swap them in the text and it still reads. What the list captures is where a word gets used, and words used in the same places usually turn out to mean similar things.

Find the twins (5 min)#

Pen and paper, no arithmetic. Look down your grid and pick the two words you think behave most alike:

• compare rows: which two words are followed by the same words?
• compare columns: which two words follow the same words?

A twin pair matches on both. Write your guess down before you go any further. Committing to a prediction is what makes the next two steps worth doing; an unwritten guess has a way of becoming whatever the answer turns out to be.

The swap test (5 min)#

Take the original text, swap your two candidate words everywhere they appear, and read it aloud. If it still reads as English—odd perhaps, or untrue, but grammatical—they’re twins. If it collapses, they’re not.

For the text the cat sat . the dog sat . the cat ran . the dog slept .:

• swap cat and dog: “the dog sat. the cat sat. the dog ran. the cat slept.” Still reads.
• swap cat and sat: “the sat cat. the dog cat. the sat ran…” Gone.

That is the distributional idea in a single move: words that keep the same company can stand in for each other. “The cat slept” may be false about your particular cat, but the swap still passes, because the test asks whether the sentence holds together, not whether it’s true.

One distance by hand (5 min)#

Now put a number on it. The distance between two words is the sum of the absolute differences between their rows, cell by cell, with blanks counting as zero. Here is the grid for the same short text:

Reading the rows off as vectors, with the columns in the order the cat sat . dog ran slept:

• cat → 0 0 1 0 0 1 0
• dog → 0 0 1 0 0 0 1
• sat → 0 0 0 2 0 0 0
• the → 0 2 0 0 2 0 0

And the arithmetic:

• d(cat, dog) = 0 + 0 + 0 + 0 + 0 + 1 + 1 = 2
• d(cat, sat) = 0 + 0 + 1 + 2 + 0 + 1 + 0 = 4
• d(cat, the) = 0 + 2 + 1 + 0 + 2 + 1 + 0 = 6

The twins score lowest. Do the same for your own twin pair and for one pair you’re confident isn’t, and check the numbers agree with your ears.

Interactive widget#

Click any row to see its vector, click a second row to get the distance between the two, and read the matrix underneath as every pair at once—that’s the similarity matrix Similarity matrix A grid showing how similar or different each pair of words is, calculated by comparing their embedding vectors. Words used in similar contexts have similar embeddings. View in glossary . The text box is shared with the Training widget, so you can paste your own text in and compare it with the grid on your desk.

This is the one module where the screen is the better apparatus, and it’s worth saying so to the class. Seven words means 21 pairs; ten words means 45. Working through all of them is an afternoon of arithmetic that teaches nothing new after the third pair. The one distance you did on paper is there so you know what the numbers mean, and the machine can do the rest.

Notice what the widget turns up in the worked example that most rooms don’t guess: ran and slept sit at distance 0, because each one only ever appears before a full stop.

Instructor notes#

Discussion questions#

• did the swap test agree with your written guess? Which of the two would you trust on a longer text?
• ran and slept came out identical, and so, nearly, did cat and dog. What would you have to add to the text to pull each pair apart?
• the distance only compares rows. Which pair in your grid would it get wrong?
• the method measures position, not meaning. Which pairs of opposites would it call twins—and why do hot and cold keep almost exactly the same company?
• if you trained on ten times as much text, would your twins end up closer together or further apart?
• your grid gives a word one row and one fingerprint. Is bank one word?

Connection to current LLMs#

The idea you just used by hand has a name and a date. The linguist J.R. Firth put it in 1957 as “you shall know a word by the company it keeps”, and it’s still the idea behind every embedding in every modern model: words used in the same contexts tend to mean similar things.

• counts versus learned vectors: your row is a count vector—one number per word in your vocabulary, nearly all of them zero, every number a tally you wrote down yourself. A real embedding is learned: a few hundred to several thousand numbers, none of them a count of anything, nudged during training until words used alike end up near each other.
• learning is what makes it generalise: this is the point Search Sheets sets up. Your grid rates any pair it never saw as flatly impossible: an empty cell is a zero chance. Because a learned model spreads what it knows across shared numbers instead of separate cells, what it learns about cat reaches dog, and it can rate a pair it has never seen as perfectly plausible. Two words can even end up close having never shared a single exact context, each being close to words the other kept company with.
• the famous analogy: king − man + woman ≈ queen, from the word2vec work of Mikolov and colleagues in 2013. The vectors come out arranged so that arithmetic on them lands somewhere sensible. Worth showing, with the caveat that it works for some analogies and not others, and that the usual evaluation excludes the three input words from the candidate answers, which flatters the result.
• modern embeddings are contextual: in an LLM a word’s vector is computed from the sentence it appears in, so the bank of a river and the bank that holds your money get different vectors.

What survives all of that is the move you made on paper: turn a word into a list of numbers, and “these two behave alike” becomes something you can measure.