Left of the line: prefill, the whole prompt in one pass. Right of it: decode, one token per step.
A model at the chip’s peak speed, not a measurement.
Adds the attention scores between prompt tokens, which grow with the square of the prompt, and the cache of earlier tokens that every decode step reads. At 8,000 tokens the first word comes about 13% later and each later token about 6.5% later.
Network time, waiting in line behind other users, and the gap between the chip’s data-sheet peak and real speed. Real servers are slower than these times. The model is Llama 3.1 8B with 16-bit weights on one A100 chip.