Embeddings

Picture a giant room. Now imagine every word you know is a dot somewhere in that room.

The word "dog" has its own spot. So does "cat", and "cat" is placed right next to "dog" because they're both pets. Meanwhile, "airplane" is way across the room because it has nothing to do with pets.

This is the big idea behind embeddings: every word gets a position in space. Words that mean similar things end up near each other. Words that mean different things end up far apart.

The AI doesn't read words the way you do. It converts each word into a list of numbers, like GPS coordinates, that describe where that word lives in this imaginary space.

On the right, you can see words plotted as dots in space. We've squished hundreds of dimensions down to just two so you can see the pattern.

Notice how words cluster together:
- Animals hang out near other animals
- Colors group with colors
- Numbers cluster with numbers

This isn't something anyone programmed. The AI figured out these groupings on its own, just by reading billions of sentences and learning which words appear in similar situations.

Try hovering over the dots to see which words are which. Can you spot the clusters?

How does the AI know that "king" and "queen" are related? It's not because someone told it, it's because they're close together in embedding space.

"King" and "queen" appear in similar sentences. "The king ruled the land." "The queen ruled the land." Because they can be swapped in so many sentences, the AI learns to put them near each other.

Meanwhile, "king" and "banana" almost never appear in the same contexts, so they end up far apart.

This is the core insight: meaning is captured by position. You don't need a dictionary. You just need to know which words hang out together.

Here's where things get wild. You can do math with words.

Take the embedding for "king." Subtract the embedding for "man." Add the embedding for "woman." What do you get?

Queen.

It's like the AI learned that "king" minus "man" equals "royalty", and when you add "woman" back, you get the female version of royalty.

This works for other things too:
- Paris - France + Italy ≈ Rome
- Walking - walk + swim ≈ Swimming

The AI isn't doing word problems. It's doing math with positions in space, and the relationships between positions carry meaning.

When a model first starts training, every word's position is completely random. "King" might be next to "pizza." "Cat" might be across the room from "kitten."

But as the model reads millions of sentences, it starts adjusting. Every time "cat" and "kitten" appear in similar sentences, the model nudges them a little closer together. Every time "cat" and "algebra" appear in different contexts, they drift apart.

After reading billions of sentences, the random dots have rearranged themselves into a meaningful map, animals near animals, verbs near verbs, countries near countries.

Nobody designed this map. It emerged from patterns in language. The training data, billions of sentences from books, websites, and articles, is where all these patterns came from. By seeing millions of sentences where "cat" and "dog" appear in similar contexts ("the ___ sat on the mat," "I fed the ___"), the model learned to place them near each other.

You now know that AI turns every word into a list of numbers, a position in space, and that similar words end up in similar positions.

But here's a problem. When the model reads a sentence like "The bank of the river," it needs to know that "bank" means the edge of the water, not a place for money. The same word can mean different things depending on the words around it.

How does the model figure out which other words to pay attention to? That's exactly what the next lesson is about: attention, the mechanism that lets the model look at all the words in a sentence and decide which ones matter most.