Attention

Read this sentence: "The cat sat on the mat because it was tired."

What does "it" refer to? The cat, obviously, not the mat. Your brain figured that out instantly.

But how? You looked back at the earlier words and decided that "cat" was more important than "mat" for understanding "it." You paid attention to "cat" and mostly ignored "the," "on," and "mat."

AI does the same thing. When predicting or understanding each word, the model looks at all the other words and decides which ones matter most. It learned which words to pay attention to from its training data, by seeing millions of sentences with pronouns, it picked up on the patterns of what "it," "she," and "they" usually refer to.

This process is called attention, and it's the single most important idea in modern AI.

Let's test your instincts. On the right, you'll see a sentence with one word highlighted. Your job: which other words in the sentence are most important for understanding that highlighted word?

Click the words you think matter most. Don't overthink it, your gut feeling is usually right.

For example, if the highlighted word is "tired" in "The cat sat on the mat because it was tired," you'd probably click "cat" (who is tired?) and maybe "it" (connects back to what's tired).

After you make your picks, we'll show you what the AI actually focuses on. You might be surprised how similar your intuition is to the model's attention.

Now let's reveal the model's attention. The brighter or thicker the connection between two words, the more attention the model is paying.

Some things you might notice:

It makes sense. The model connects "it" to "cat" in our example, just like you did.

Some surprises. The model sometimes focuses on "boring" words like "the" or "." more than you'd expect. These words carry structural information that matters to the math even if they don't seem meaningful to us.

It changes per layer. The model has many layers of attention, and different layers focus on different things. Early layers might focus on nearby words. Later layers pick up on meaning and relationships.

So how does the model decide which words to pay attention to? It uses a clever system with three parts, each with an everyday analogy.

Imagine you're at a library:
- You have a question in mind (this is the Query): "I need a book about cats."
- Each book on the shelf has a label (this is the Key): "Animals," "Cooking," "History."
- Each book contains actual information (this is the Value): the content inside.

You compare your question to every label. When you find a good match ("Animals" matches "cats"), you grab the information from that book.

The model does this for every word: it asks "what am I looking for?", checks all the labels, and collects the information from the best matches.

Here's the thing, one attention pattern isn't enough. When you read a sentence, you're simultaneously tracking many different relationships: Who did what? Where did it happen? When? What does "it" refer to?

So the model doesn't just use one set of attention. It uses many, running in parallel. These are called attention heads.

Each head learns to focus on a different pattern:
- One head might track pronouns and the nouns they refer to
- Another head might connect verbs to their subjects
- Another might focus on nearby words for local grammar

A typical model might have 12 to 96 heads, all running at the same time, each seeing the sentence differently. Their results get combined into a single, rich understanding of each word.

You've now learned the most important mechanism in modern AI: attention, how the model decides which words to focus on when processing each word.

Let's take a step back and put it all together. You know that AI predicts the next word. You know it breaks text into tokens. You know it learns from mistakes. You know words become positions in space. And now you know the model can focus on what matters.

In the next lesson, we'll see how all of these pieces fit together into one complete system, the transformer, and understand the full journey from your typed text to the AI's response.