---
title: "What does attention do inside an AI model?"
description: "Attention lets each word of a text pick up what earlier words mean: it scores them, turns the scores into weights, and takes a weighted average."
topic: "AI systems"
chapter: "how-ai-answers"
order: 6
minutes: { plain: 6, deep: 18 }
prerequisites: ["Reading short Python functions", "Adding and multiplying lists of numbers"]
seo_title: "What does attention do in an AI model?"
date: 2026-10-19
---
# What does attention do inside an AI model?
Read "The cat sat because it was tired." You know "it" means the cat. A language model, the kind of program behind a chatbot, has to work that out too, and attention is the step that lets a word pick that up from the words before it. In a tiny model with numbers picked by hand, the word "it" puts 68% of its weight (its share of attention) on "cat", because "cat" is the word whose label best answers what "it" is looking for.
## In one minute
Attention is the step where each token, a small piece of text and often a whole word, collects information from the tokens before it. Picture a room where every earlier word holds up a label and carries a note. The word that is reading asks a question, compares it with every label, and takes a blend of the notes, leaning toward the labels that answer it best. The question is called a query, the label a key, and the note a value. "It" asks "what do I point to?", the label on "cat" says "an animal", so "cat" gets the most weight, and "it" leaves with a note that is mostly "animal".
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/what-attention-does/widget.html). It opens on "it" and shows the weight, or share of attention, "it" gives each earlier word. Tap another word to see its weights, then change the kind of look and watch the bars move. Words after the chosen one are not looked at. Every number is picked by hand to show the mechanism; a real model learns its numbers.
## How it works
### Looking back at every word
[Why do answers appear word by word?](https://spicyshark.co/learn/why-answers-appear-word-by-word/) uses a toy that looks back only a few words. A real model looks back at every earlier token, and attention decides which ones matter for the word being read. In the toy here, each word is six numbers: four say what the word is like (an animal, pointing to something, an action or state, a small function word), and two give its place in the sentence.
### Query, key and value
A head is one way of looking. It turns each word's six numbers into three short lists, by multiplying and adding with fixed tables of numbers. The query is what the word looks for, the key is the label it holds up, and the value is the note it carries. In the head called "Refers to", the query of "it" is its "pointing to something" number times 3, and the key of "cat" is its "an animal" number. Every other word has a key of 0.
### Scores, then weights
For each earlier word, the score is the query times the key (multiply matching numbers, then add), divided by the square root of how many numbers the query has. For "it" and "cat" that is 3 × 1 ÷ √2 = 2.12. For every other word it is 0. A word looks only at itself and the words before it, so "was" and "tired" get a score of −∞ and are hidden. That hiding is called a causal mask. Softmax then turns the scores into weights that are positive and add up to 1; [lesson 21](https://spicyshark.co/learn/online-softmax/) explains it. These attention weights are not the learned numbers inside a model, which share the name. "Cat" gets 68% and each other visible word 8%.
*Figure 1. The weights of "it" in the head that looks for what a word points to. Green marks the biggest weight.*
### The blend
The word then takes the weighted average of the values: 68% of the note on "cat", 8% of each other note. Before, "it" held 0 on "an animal". After, it holds 0.68. It has picked up what it points to, and nothing told it so.
### Every word at once
Every word does the same at once. The grey half of Figure 2 is the mask. The first word, "The", has only itself to look at, so it puts 100% there.
*Figure 2. Every word's weights over the words before it. Green outlines the biggest weight in each row.*
### Several heads
One head finds one kind of link. A model runs many heads side by side, each with its own query table (key and value tables can be shared by several heads, as in grouped-query attention), and joins what they return. Figure 3 adds a head that looks for the word just before. Under that head, "it" puts 71% on "because", and the joined result carries both links.
*Figure 3. Two heads read the same word. Green, "Refers to": what the word points to. Orange, "Word before": the word just before.*
### What the toy leaves out
The toy's numbers are chosen, not trained. A real model learns its numbers from data, has thousands per token, and stacks many layers, each with many heads, with more steps after every attention step. Its position information also differs. Attention alone does not know the order of words, so a model is given the place of each word; one published way adds sine and cosine waves.
## What this means when you build
- **Cost grows with the square.** Each token scores itself and the tokens before it, so n tokens make about n²/2 scores per head per layer: 500,500 for 1,000 tokens, 32,004,000 for 8,000. Code that builds the whole n by n table and then hides the later half does about twice that work; fast kernels such as FlashAttention-2 skip the blocks that are fully hidden. [Lesson 21](https://spicyshark.co/learn/online-softmax/) shows how real code avoids storing the scores, and [lesson 23](https://spicyshark.co/learn/why-ai-waits-before-the-first-word/) what long prompts cost before the first word.
- **Servers keep keys and values.** A new token needs every earlier token's key and value, so a server saves them instead of recomputing: the attention cache. [Lesson 22](https://spicyshark.co/learn/paged-kv-cache/) shows how it is stored, and [lesson 3](https://spicyshark.co/learn/why-chatbots-forget/) its size.
- **Read weight pictures as hints.** A big weight shows which note a head blended in, not why the model answered as it did. Many heads and layers follow.
- **You rarely write it yourself.** Libraries ship this step, and [lesson 21](https://spicyshark.co/learn/online-softmax/) names where. Writing it once, as in Go deeper, is how you learn what to check.
## Go deeper
*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. From inside that folder, `python3 show_numbers.py` prints the numbers below, and `python3 -B -m pytest -q -p no:cacheprovider` runs the tests.*
### The formula
Put the queries of all words in a table Q with one row per word, and the keys and values in K and V. Vaswani et al. write scaled dot-product attention as softmax(QKᵀ / √d_k) V, where d_k is the length of a query and of a key. For a model that writes text, a word must not look ahead, so the scores of later words are set to −∞ before the softmax, which gives them weight 0. The paper does this in the decoder's self-attention. In steps:
```text
scores = Q K^T / sqrt(d_k) row i, column j: how well word j's key answers word i's query
scores = -inf where j > i the causal mask
weights = softmax of each row positive, adds up to 1
output = weights V row i: the weighted average of the values
```
### The toy's numbers for "it"
In the head "Refers to", the query of "it" is [3, 0], and the only key that is not [0, 0] belongs to "cat", [1, 0]. The length of a query is d_k = 2, so each score is divided by √2 = 1.4142.
| Word | Score | Weight |
|---|---|---|
| The | 0.000 | 0.0810 |
| cat | 2.121 | 0.6759 |
| sat | 0.000 | 0.0810 |
| because | 0.000 | 0.0810 |
| it | 0.000 | 0.0810 |
| was | −∞ | 0 |
| tired | −∞ | 0 |
Softmax turns each score into a size, e to the power of the score, and divides by the total. The size for "cat" is e^2.121 = 8.34, and each other visible word has size 1. The total is 12.34, so "cat" gets 8.34 ÷ 12.34 = 0.6759 and each other word 1 ÷ 12.34 = 0.0810. The values are four numbers per word: how much it is an animal, points to something, is an action or state, is a small function word. The weighted average is:
```text
0.6759 x [1, 0, 0, 0] (cat)
+ 0.0810 x ([0, 0, 0, 1] + [0, 0.6, 1, 0]
+ [0, 0, 0, 1] + [0, 1, 0, 0]) (The, sat, because, it)
= [0.676, 0.130, 0.081, 0.162]
```
The first number, 0.676, is how much "an animal" the blend carries. The word "it" alone had 0.
Other words use the same head and the same keys, but their queries differ, because the query is 3 times the word's own "pointing to something" number. For "sat" that number is 0.6, so the score on "cat" is 0.6 × 3 ÷ 1.4142 = 1.273, its size is 3.57, and with two other visible words of size 1 its weight is 3.57 ÷ 5.57 = 0.64. For "tired" it is 0.8, the score is 1.697, the size is 5.46, and with six other words the weight is 5.46 ÷ 11.46 = 0.48. For "because" the number is 0, every score is 0, and its four weights are 0.25 each. So the weight on "cat" depends on the word's own query and on how many words compete with it.
### The code
These are the real functions from `code/attention.py` with their comments left out. Plain Python lists, no libraries:
```python
def scores_for(q, keys, i, scale=True):
divisor = math.sqrt(len(q)) if scale else 1.0
return [dot(q, keys[j]) / divisor if j <= i else -math.inf for j in range(len(keys))]
def softmax(scores):
top = max(scores)
sizes = [math.exp(s - top) for s in scores]
total = sum(sizes)
return [size / total for size in sizes]
def weighted_average(weights, values):
return [sum(w * v[c] for w, v in zip(weights, values)) for c in range(len(values[0]))]
def attend(vectors, head, scale=True):
queries = [times(x, head["wq"]) for x in vectors]
keys = [times(x, head["wk"]) for x in vectors]
values = [times(x, head["wv"]) for x in vectors]
scores = [scores_for(queries[i], keys, i, scale) for i in range(len(vectors))]
weights = [softmax(row) for row in scores]
outputs = [weighted_average(w, values) for w in weights]
return {"queries": queries, "keys": keys, "values": values,
"scores": scores, "weights": weights, "outputs": outputs}
```
Subtracting the biggest score inside `softmax` changes no weight and avoids overflow, as [lesson 21](https://spicyshark.co/learn/online-softmax/) shows; `math.exp(-inf)` is 0, which is how the mask works. The three tables of each head are `wq`, `wk` and `wv`. They are the toy's chosen numbers, and a real model learns them.
### Why divide by the square root
Vaswani et al. suspect that for large d_k the dot products grow large, which pushes the softmax into regions where it has very small gradients, so they divide by √d_k. If the numbers of a query and a key are independent with mean 0 and variance 1, a dot product has mean 0 and variance d_k. The paper also reports, citing another study, that without the division dot-product attention does worse than another scoring method, additive attention, at larger d_k.
`scaling_experiment` checks the first part. It draws random queries and keys with numbers of mean 0 and spread 1 (seed 6), scores one query against 8 keys, and repeats 100 times at three lengths. Figure 4 shows how far the scores spread. Without the division the spread is 2.16, 8.11 and 31.56, close to √d = 2, 8 and 32. With it, the spread is 1.08, 1.01 and 0.99. The table shows the biggest weight in a row of 8, on average. Plain is without the division, Divided is with it.
*Figure 4. The spread of 800 scores at each length d, without and with the division. Orange grows with √d. Green stays near 1.*
| d | Plain | Divided |
|---|---|---|
| 4 | 0.53 | 0.34 |
| 64 | 0.90 | 0.36 |
| 1,024 | 0.96 | 0.36 |
At d = 1,024 the biggest score takes 96% of the weight without the division, so softmax almost ignores the other seven words. With it, the biggest takes 36%, the same as at d = 64. This tests only random inputs; it shows the spread, not what happens in training.
### Two heads, joined
`multi_head` runs each head and writes each word's outputs side by side. For "it", the first head gives [0.676, 0.130, 0.081, 0.162] and the second, the word just before, gives [0.004, 0.231, 0.145, 0.707], so the joined list has eight numbers. In the paper's base model there are 8 heads of 64 numbers each; the paper joins them and applies one more table of learned numbers, which the toy leaves out. The paper's stated reason is that several heads let the model look at different kinds of information at once. The toy's second head builds its query by turning the word's place back one step, with strength 6, so the key of the previous word matches best.
### What the code costs
A word scores itself and each word before it, so word 1 makes 1 score, word 2 makes 2, and word n makes n. A prompt of n words makes n(n+1)/2 scores in one head.
| Words | Scores per head |
|---|---|
| 10 | 55 |
| 1,000 | 500,500 |
| 8,000 | 32,004,000 |
[Lesson 23](https://spicyshark.co/learn/why-ai-waits-before-the-first-word/) uses a model with 32 layers of 32 query heads (they share 8 key-value heads, see [lesson 22](https://spicyshark.co/learn/paged-kv-cache/#how-big-it-gets)), which is 1,024 query heads in all. Across them, 10 words make 56,320 scores, 1,000 words make 512,512,000, and 8,000 words make 32,772,096,000. Doubling the words from 1,000 to 2,000 takes the count from 500,500 to 2,001,000, a little under four times. This counts scores only, not the multiplications that make each score or the work around attention. [Lesson 21](https://spicyshark.co/learn/online-softmax/) shows how FlashAttention avoids storing the table, and [lesson 22](https://spicyshark.co/learn/paged-kv-cache/#how-big-it-gets) how big the attention cache gets.
### What the toy leaves out, in detail
The six numbers of each word are chosen: four about the word and two for its place. The place numbers are the cosine and sine of an angle of 0.9 radians per step. Vaswani et al. add sine and cosine position signals to the input embeddings, the learned vector for each token, so the toy's version is a simpler cousin and not the paper's. The toy has one layer, no learned tables, no table to join heads, and none of the steps that follow attention in a real layer.
### Proof it works
The independent check is `reference` in `test_attention.py`. It follows the formula with NumPy matrix operations, with `x` the table of word vectors and `wq`, `wk`, `wv` a head's three tables:
```python
q, k, v = x @ wq, x @ wk, x @ wv
s = q @ k.T / math.sqrt(k.shape[1])
s = s + np.triu(np.full(s.shape, -np.inf), k=1)
e = np.exp(s - s.max(axis=1, keepdims=True))
w = e / e.sum(axis=1, keepdims=True)
output = w @ v
```
`np.triu(..., k=1)` is the mask: it puts −∞ above the diagonal and 0 elsewhere. The loops in `attention.py` and these matrix lines share no code, so a slip in one shows up as a difference from the other.
`test_attention.py` pins the numbers above. The weights of "it" must match the hand-worked 0.0810 and 0.6759 and the sentence "it" puts 68% of its weight on "cat". For every head, with and without the division, weights must be positive and add up to 1, and rewriting any later word must leave every earlier row unchanged, so no word looks ahead. An independent version in NumPy matrix operations, with `np.triu` for the mask, must agree with the Python to 1e-12 on the toy and on seeded random inputs. The scaling experiment must print the numbers in the text. The score count is checked against a brute-force count and the layer and head counts against [lesson 23](https://spicyshark.co/learn/why-ai-waits-before-the-first-word/)'s code. `test_widget.py` runs the widget's JavaScript in Node against the Python for every head and every word, and drives the page through a small stand-in for the browser.
### Where this is used in practice
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin (2017) built a model from attention alone, with no recurrence or convolution. Its section 3.2 gives the scaled dot-product attention, the division by √d_k and the multiple heads described above, and it uses the mask to keep a decoder from looking ahead. Bahdanau, Cho and Bengio (ICLR 2015) had earlier used a related idea for translation. Instead of squeezing a source sentence into one fixed-length vector, their model learned to look back at parts of the source for each word it wrote. Their abstract reports English-to-French results about as good as the best phrase-based system of the time, and says the learned alignments agree with intuition. Both were read on 2026-10-19: the first in full, the second its abstract only. Neither claims the toy's numbers, which are made for this lesson.
## Skills this lesson builds
- Explain attention as a query matched against keys, then a weighted average of values.
- Compute a word's weights by hand from its scores: divide by the square root, mask, softmax.
- Write scaled dot-product attention with a causal mask in plain Python and check it against NumPy.
- Say why scores are divided by √d, and show it with random vectors.
- Count the scores a prompt makes and say how that grows with length.
**Portfolio project:** pick a sentence of 6 to 8 words that has a pronoun in it. Choose a small vector for each word and one head whose query and key make the pronoun look at the right word. Write the attention, test it against NumPy, and publish the weight grid as a picture. Add a note on what happens to the weights when you remove the mask or the division.
## Check yourself
1. In your own words, what do the query, the key and the value of a word do?
Answer
The query is what the word looks for. Each earlier word's key is a label that is compared with the query to give a score. The value is the note a word hands over. The scores become weights, and the word takes the weighted average of the values.Answer
The weight is the size of a score over the total of all sizes. The size for "cat" is e^2.121 = 8.34 and the others have size 1 each, so the total is 12.34 and "cat" gets 8.34 ÷ 12.34 = 0.676. A score of 2.12 is a clear lead but not a huge one.Answer
They come after "it". The causal mask sets their scores to −∞ before the softmax, and e to the power of −∞ is 0, so a word never looks at a later one.Answer
One head makes 500,500 scores for 1,000 tokens and 2,001,000 for 2,000. That is 1,500,500 more, a little under four times as many.Answer
The scores spread out in proportion to √d: the spread is 2.16 at d = 4 and 31.56 at d = 1,024. The biggest score then takes almost all the weight (96% at d = 1,024), so the other words are ignored. Dividing by √d keeps the spread near 1 and the biggest weight near 36% for 8 words.