---
title: "Why does an AI wait a moment before the first word?"
description: "Before its first word, an AI reads your whole prompt in one pass. Long prompts wait longer, short ones do not, and later words arrive at a steady pace."
topic: "AI systems"
chapter: "serving-many-chats"
order: 23
minutes: { plain: 6, deep: 18 }
prerequisites: ["What a token is", "Division and ratios", "Reading short Python functions"]
seo_title: "Why does an AI wait before its first word?"
date: 2026-10-19
---
# Why does an AI wait a moment before the first word?
Paste a long report into a chatbot and it sits silent before the first word appears; ask a short question and the first word comes almost at once. Take an AI model with 8 billion numbers inside, called weights, on one A100 chip built for AI work. Its best case is 7.88 milliseconds (ms, thousandths of a second) for a 20-token question and 412 ms for an 8,000-token report, 52 times as long. A token is a small piece of text, often a whole word. After the first word, both replies come at nearly the same pace. Real services are slower than these best cases, and this lesson says why.
## In one minute
An AI waits before its first word because it must read your whole prompt, the text you send, before it can write anything, and that reading is one big pass, a single run of the model, called prefill. Picture reading a whole letter before you reply: a long letter takes longer to read than a short one. After the first word the reply comes one token at a time, each from one small pass called decode. Reading is limited by arithmetic, which is how fast the chip can calculate, and writing is limited by memory speed, which is how fast the chip can fetch the model.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/why-ai-waits-before-the-first-word/widget.html) and drag the prompt length from 10 to 8,000 tokens. It shows best-case times from a model at the chip's peak speed, not measurements. Up to about 150 tokens the prefill block stays one decode step wide; after that it grows, while every decode step keeps its size.
## How it works
### Reading first, then writing
In prefill, one pass handles every token of the prompt together, and the first token of the reply comes out of it. Decode then makes the reply one token per pass, as [Why do answers appear word by word?](https://spicyshark.co/learn/why-answers-appear-word-by-word/) shows. [What is a token?](https://spicyshark.co/learn/what-is-a-token/) covers tokens.
*Figure 1. The two phases. Green marks prefill, which is limited by arithmetic. Orange marks decode, which is limited by memory speed.*
### Two numbers to watch
Time to first token is the time from sending the prompt to the first token of the reply. Tokens per second is how many tokens the model then writes each second. Prefill sets the first number, and decode sets the second.
### Why a longer prompt waits longer
The model is a pile of numbers called weights. Llama 3.1 8B, the model used here, has about 8 billion, and [at 2 bytes each](https://spicyshark.co/learn/quantization/) (a byte is the unit memory is counted in) they [fill 16 GB](https://spicyshark.co/learn/model-memory/). The A100's data sheet allows 312 trillion operations a second, where an operation is one addition or one multiplication, and 2,039 GB a second of reading from memory, called bandwidth. The pass for one token needs about 16 billion operations, two for each weight, and it must fetch every weight. [Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/#why-a-chatbot-waits) explains these two limits. A pass cannot end before both jobs are done, so its time is the slower one.
Fetching the weights takes 7.88 ms, however short the prompt. Each fetched byte serves every token of the prompt, so a 100-token prompt gives the chip about 100 operations per byte, and the A100 can do 153 operations in the time it fetches one byte. Up to 153 tokens, then, the chip waits for memory, and the extra tokens ride along free: the pass takes 7.88 ms. From 154 tokens up, arithmetic is the slower job, and every added token adds time: 1,000 tokens take 51.5 ms and 8,000 take 412 ms.
*Figure 2. Best-case time to first token against prompt length, on a straight scale from 0 to 8,000 tokens. The flat part, up to 153 tokens, is the short stretch at the left. Orange waits on memory; green is limited by arithmetic.*
### Why each later token takes the same time
A decode step handles one token, yet it must fetch all 16 GB of weights again. The fetch takes 7.88 ms, and the arithmetic needs only 0.05 ms. The chip is busy 0.7% of the time, and in this model the step takes 7.88 ms whatever the prompt length. That caps one reply at 127 tokens a second.
*Figure 3. A short and a long prompt on one time axis, to scale. Orange blocks wait on memory; the green block is limited by arithmetic.*
### What this model leaves out
These are best cases at the chip's data-sheet speed. A real service is slower: your message crosses a network, your request may wait in line behind other users, and real chips fall short of their peak. Two costs of attention, where each token looks back at the earlier ones, are also left out. Its arithmetic grows with the square of the prompt: at 8,000 tokens it adds about 13% to the first word. Each decode step also reads the attention cache, the notes kept for earlier tokens, which adds about 6.5% per step at 8,000 tokens. The widget counts both under More settings.
## What this means when you build
- **Measure the two speeds apart.** A slow first word points at prefill, and slow words after it point at decode. They need different fixes.
- **Trim long prompts, not short ones.** In this model, cutting a prompt below about 150 tokens saves nothing; cutting 8,000 tokens to 1,000 takes the best case from 412 ms to 51.5 ms.
- **A shorter reply does not speed up the first word.** The first word comes out of prefill. Reply length only adds later steps, 7.88 ms each here.
- **Show progress during a long wait.** A silent page looks frozen. Show a status, then stream the reply, which means showing each token as it arrives.
- **Reuse a repeated opening.** If many requests start with the same long text, prompt caching lets a server reuse the kept result of reading it, instead of reading it again.
## Go deeper
*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. From inside that folder, `python3 show_numbers.py` prints every number below, and `python3 -B -m pytest -q -p no:cacheprovider` runs the tests.*
### The two floors
Take P weights, a prompt of n tokens, and the chip's peak speed and bandwidth. A forward pass does 2 operations per weight per token (one multiply, one add) and reads every weight once, at 2 bytes each. Its time is at least the larger of two floors, the math floor and the memory floor:
```text
prefill time = max(2 P n / peak, 2 P / bandwidth)
decode step = max(2 P / peak, 2 P / bandwidth) = 2 P / bandwidth
operations per byte fetched, in prefill = (2 P n) / (2 P) = n
```
For Llama 3.1 8B, P is 8,030,261,248, counted from the layer sizes in the model's configuration; Meta's model card rounds it to 8B. Peak is 312e12 operations a second and bandwidth 2,039e9 bytes a second, the figures [lesson 19](https://spicyshark.co/learn/tiled-matmul-roofline/#decode-versus-prefill) takes from NVIDIA's A100 data sheet. Then 2 P is 16,060,522,496 operations per token, the weights are 16.06 GB, and the decode step is 16.06 GB over 2,039 GB a second: 7.877 ms, or at most 126.96 tokens a second. The ridge, peak over bandwidth, is 153.02 operations per byte. A prompt of n tokens gives n operations per byte, so 153 tokens are still limited by memory and 154 are the first limited by arithmetic. A tie counts as limited by arithmetic, as in lesson 19.
### A worked example at two prompt lengths
Take one prompt below the ridge and one above it; each pass fetches the same 16.06 GB. In the table, Per byte is operations per byte fetched, Math and Memory are the two floors in milliseconds, Time is the larger floor, Limit names which floor that is, and Busy is the share of the chip's peak arithmetic reached.
| Tokens | 100 | 1,000 |
|---|---|---|
| Per byte | 100 | 1,000 |
| Math, ms | 5.15 | 51.48 |
| Memory, ms | 7.88 | 7.88 |
| Time, ms | 7.88 | 51.48 |
| Limit | memory | math |
| Busy | 65.4% | 100% |
At 100 tokens the prefill pass costs one decode step, and the chip is two-thirds busy: 5.15 over 7.877. At 1,000 tokens it costs 6.5 decode steps (51.476 over 7.877), and the chip is as busy as it can be. At 8,000 tokens the math floor is 411.8 ms, 52 steps, and the reader waits that long for the first word, in this model.
### The code
The whole model is a pass with a count of operations and a count of bytes:
```python
@dataclass(frozen=True)
class Pass:
flops: int
nbytes: int
@property
def math_s(self) -> float:
return self.flops / A100_PEAK_FLOPS
@property
def memory_s(self) -> float:
return self.nbytes / A100_BANDWIDTH
@property
def seconds(self) -> float:
return max(self.math_s, self.memory_s)
def prefill(n: int, extras: bool = False) -> Pass:
flops, nbytes = 2 * PARAMS * n, FP16_BYTES * PARAMS
if extras:
flops += attention_flops_prefill(n)
nbytes += n * KV_BYTES_PER_TOKEN
return Pass(flops, nbytes)
def decode_step(context: int, extras: bool = False) -> Pass:
flops, nbytes = 2 * PARAMS, FP16_BYTES * PARAMS
if extras:
flops += 4 * LAYERS * HIDDEN * context
nbytes += context * KV_BYTES_PER_TOKEN
return Pass(flops, nbytes)
```
These are the real functions with their comments left out. `prefill_sim.py` has the full file, with the `intensity`, `bound` and `busy` properties and the widget's view, and `show_numbers.py` prints every figure in this lesson.
### Why 154 here and 166 in lesson 19
[Lesson 19](https://spicyshark.co/learn/tiled-matmul-roofline/) reaches the ridge at 166 tokens for one 4096 by 4096 grid of weights, because it also counts the numbers going into the grid and the results coming out. At 166 tokens those add 8.1% to the 33.6 MB of weights. This lesson counts weight bytes only, which is the right choice for a whole model at prompt lengths of a few hundred tokens, where the weights dominate the traffic. That gives 154. Both are the same ridge seen with different bookkeeping.
### What the short formulas leave out
*Figure 4. What a person waits for. Green marks the one step this lesson counts, at the chip's peak speed; dashed grey steps depend on the service.*
Under More settings the widget adds the two costs of attention. For the scores between prompt tokens, each token is counted against itself and the ones before it: 4 x layers x hidden size x n(n+1)/2 operations, which is 1.678e13 at 8,000 tokens, 13.1% of the weight operations. It reaches 5% at 3,063 tokens and 10% at 6,126. Counting every pair would double it. The cache that prefill writes adds bytes to the fetch, and each decode step reads the cache of the whole prompt. With both counted, an 8,000-token prefill takes 465.6 ms instead of 411.8, and a decode step 8.39 ms instead of 7.88.
The short formulas also overstate some things. The input table of 525,336,576 weights, 6.5% of all weights, is a lookup: a pass reads one row per token, not the whole table. Counting it as fetched and multiplied overstates both floors by about 6.5%, which offsets the cache read at long prompts. So read the times as model results near the chip's data-sheet speed, not as proven lower bounds. The largest gap is likely elsewhere: real kernels do not reach peak, and the two floors do not overlap perfectly. Nothing here was measured on a server. Network time and queueing, the first two boxes in the figure, depend on the service.
### The cache that prefill writes
Prefill also writes the attention cache: for each prompt token, each layer stores a key and a value for each of the 8 key-value heads. That is 2 x 32 x 8 x 128 x 2 = 131,072 bytes (128 KiB) per token, so 8,000 tokens take 1.049 GB and 8,192 tokens exactly 1 GiB. That is 6.5% of the 16.06 GB of weights. [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/#how-much-memory-the-cache-takes) shows the sum for another model, and [Why AI servers waste memory, and how paging fixes it](https://spicyshark.co/learn/paged-kv-cache/#how-big-it-gets) shows how a server stores the cache.
### Why one user leaves the chip idle
In decode the chip is busy 0.65% of the time, which the lesson rounds to 0.7%: the same figure [lesson 19](https://spicyshark.co/learn/tiled-matmul-roofline/) finds for one token. One user cannot use the rest, because the next token needs the last one. A server that decodes many users' next tokens in one pass fetches the weights once for all of them, so each user's step costs about the same 7.88 ms while the chip does more work. [Lesson 24](https://spicyshark.co/learn/how-an-ai-server-answers-many-people-at-once/) looks at that sharing and at the waiting in line it brings.
### Proof it works
`test_prefill_sim.py` pins the numbers above. The parameter count must come to 8,030,261,248, which fails if the config values or the layer formula are wrong. The chip constants must equal [lesson 19](https://spicyshark.co/learn/tiled-matmul-roofline/)'s. Prefill takes exactly one decode step through 153 tokens and then grows in proportion to the prompt. A decode step stays limited by memory for every context, with the extras on and off. The attention pair count is checked against a brute-force count. `test_widget.py` runs the widget's JavaScript in Node against the Python for every slider stop, with the extras on and off, and drives the page through a small stand-in for the browser.
### Where this is used in practice
Papers on serving language models use this split. Only their abstracts were read for this lesson, on 2026-10-19, so the summaries stay general. Splitwise (Patel et al., 2023) describes a compute-hungry prompt phase and a memory-hungry token phase, and proposes running them on separate machines. DistServe (Zhong et al., 2024) treats time to first token as the prefill target and time per output token as the decode target, and reports that running both phases on the same GPUs makes them interfere. Sarathi-Serve (Agrawal et al., 2024) notes that decode uses little of the chip's compute, which makes batching effective for decode, and it targets the trade-off between throughput and latency. These papers study whole serving systems; this lesson models one request. [Lesson 33](https://spicyshark.co/learn/why-split-reading-the-prompt-from-writing-the-reply/) runs a toy of the prefill-decode split and measures its goodput.
## Skills this lesson builds
- Explain the wait before the first word as prefill, and the pace after it as decode.
- Work out a pass time as the larger of a math floor and a memory floor, from a model's parameter count and a chip's data sheet.
- Find the prompt length where prefill stops being limited by memory.
- Say what a best-case model leaves out, and keep it apart from a measurement.
- Measure time to first token and tokens per second separately for a model you call.
**Portfolio project:** write a small script that sends the same question after padding it to 10 different prompt lengths, and records time to first token and tokens per second for each. Plot the first against prompt length on a straight scale, as in this lesson's chart, next to the formula from this lesson for the model and chip you think you are using. Publish the script, the plot and a note on where the two disagree.
## Check yourself
1. A friend says the chatbot is slow to start because it is thinking hard about the answer. What do you tell them?
Answer
Before the first word the model reads the whole prompt in one pass, called prefill. For a short prompt that pass takes about as long as one later step. For a long prompt it takes longer, because the chip has more arithmetic to do.Answer
No for 60 to 30: both are below about 153 tokens, so the pass takes 7.88 ms either way, set by fetching the weights. Yes for 8,000 to 1,000: the best case falls from 412 ms to 51.5 ms, because above 153 tokens the time grows with the prompt.Answer
The math floor is 5.15 ms (100 x 16.06 billion operations over 312 trillion a second) and the memory floor is 7.88 ms (16.06 GB over 2,039 GB a second). The larger one wins, so prefill takes 7.88 ms and memory sets the time. The chip is 65.4% busy.Answer
Above the ridge, the math floor wins: 2,000 x 16,060,522,496 operations over 312e12 a second is 102.95 ms. A decode step is 7.877 ms, so that is about 13 steps.Answer
The short formula counts only the weights, which every step fetches in full. The cache is extra traffic that the short formula leaves out; counting it adds about 6.5% to a step at 8,000 tokens (8.39 ms instead of 7.88 ms), which is small next to the weights but not zero.