--- title: "Why do answers appear word by word?" description: "A chatbot makes one token per pass and adds it to the text, then runs again, so a reply arrives a piece at a time and the app shows each piece as it comes." topic: "AI systems" chapter: "how-ai-answers" order: 4 minutes: { plain: 6, deep: 12 } prerequisites: ["Reading short Python functions", "Percentages and simple fractions"] seo_title: "Why do AI answers appear word by word?" date: 2026-10-19 --- # Why do answers appear word by word? Ask a chatbot how to make tea, and the recipe fills in a piece at a time. That is not only a screen effect: the model really makes the reply one token at a time. This lesson uses a toy, a tiny program that learned from 17 short recipe lines, 5 of them for tea. Its tokens are whole words and marks like the period. Asked "How do I make tea?", it writes 30 tokens and then an end mark, a special token that means the reply is finished. That takes 31 passes, each one run of the model. In the first it picks Boil, which had 4 chances out of 5. Real models are far bigger; the lesson says where the toy misleads. ## In one minute AI answers appear word by word because the model makes only one token at a time, and the app shows each token as soon as it arrives. A token is a small piece of text, often a whole word; picture a person writing a reply on a pad one word at a time, rereading everything on the pad before choosing the next word. The model does the same: each run reads the question and the reply so far, and gives chances for the next token. One is picked and added to the text, and the model runs again. Showing each token as it arrives is called streaming the reply. ## Explore the numbers [Open the widget](https://spicyshark.co/learn/why-answers-appear-word-by-word/widget.html). It is the toy from the opening, starting on the tea question. The list shows the chances for the next pass. Press Next token to run that pass: the likeliest token joins the reply, and the count goes up by one. Press Play to stream the rest, or pick another question. Under More settings, a length cap of 10 stops the reply mid-sentence after 10 passes. ## How it works ### The model makes one token at a time A language model is a program that learned, from a very large amount of text, which tokens tend to follow which ([What is a token?](https://spicyshark.co/learn/what-is-a-token/) explains them). To answer, it runs a loop. A pass, also called a generation step, is one run of the model: it reads the text so far, the question plus the reply already made, and gives each possible next token a chance. One token is picked, here the likeliest, and added to the text. The next pass starts from the longer text. The loop stops when the model picks the end mark, or when the reply reaches a length limit. ![Four boxes joined by arrows: the text so far, one pass of the model giving chances for the next token, pick one token, add it to the text. An arrow returns to the first box, labelled repeat until the end mark.](https://spicyshark.co/learn/why-answers-appear-word-by-word/figs/generation-loop.svg) ### Each pass starts from what the last one made In the toy, pass 1 starts from "How do I make tea?" It looks at the last three tokens, "make tea ?", and counts what followed them in the 5 tea lines it read: Boil 4 times, Heat once. So Boil has 4 chances out of 5 (80%) and Heat 1 out of 5 (20%). Boil is added, and pass 2 starts from the text with Boil on the end. Pass 11 is less sure: after "bag in a" the toy read cup 3 times and mug twice, so cup has 60% and mug 40%, and the demo takes the likeliest, cup. A real model also picks from chances, not from a script; [Why does AI make things up?](https://spicyshark.co/learn/why-ai-makes-things-up/) covers how adventurous the pick is. The toy is much simpler than a real model. It looks back only three tokens and counts what followed them in the 17 lines it read, while a real model reads all the text so far, up to its limit, and has learned from far more text. ![Two small bar charts of the chances for the next token. Pass 1, text ends make tea ?: Boil 80%, Heat 20%. Pass 11, text ends bag in a: cup 60%, mug 40%. The token added is orange.](https://spicyshark.co/learn/why-answers-appear-word-by-word/figs/chances-by-pass.svg) ### The reply is not written ahead The model does not write the reply ahead and then reveal it. Inside, the server (the computer running the model) keeps notes about earlier tokens, called the attention cache, so a pass need not redo that work. But no later token exists until its own pass has run. A reply that is still arriving is a reply that is still being made. ### Each token costs a pass One token per pass means the tea reply's 30 tokens and its end mark need 31 passes, and a longer reply needs more, so the time to finish grows with the length. How long a pass takes depends on the model and the machine, so this lesson gives no speeds. [Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/#why-a-chatbot-waits) says what makes each pass take time. ### Why apps show each token as it arrives An app could wait for the last pass and then show the whole reply, leaving the reader with a blank screen until then. Many chat apps instead show each token as it arrives. Streaming does not make the model faster: the last token appears at about the same time either way. What changes is the wait before anything shows, which builders call time to first token. Behind the screen, the server sends the reply as a series of small messages over one open connection, and the page adds each message to the text. The widget's Play button imitates this at a steady pace; a real stream arrives as fast as passes finish. ![Two bars of equal length on a time arrow. Wait for the whole reply: 31 passes with nothing on screen, then the whole reply. Show each token as it is made: a piece after every pass, the first after pass 1.](https://spicyshark.co/learn/why-answers-appear-word-by-word/figs/wait-or-stream.svg) ## What this means when you build - **Stream replies that people read.** Turn on your provider's streaming and add each piece to the page as it arrives. Anthropic's streaming page (read 2026-10-19) says that turning it on sends the reply incrementally as server-sent events, a standard way for a server to push small messages to a page. - **Plan for a stream that breaks.** The same page says errors can arrive inside the stream, so part of a reply may already be on screen when one does. Keep what arrived, tell the reader, and offer a retry. - **Do not use half a reply as a whole one.** Text that must be complete to be valid, such as JSON (a strict data format), cannot be read until the last piece has arrived. - **Measure two times.** Time to first token is the wait before the first piece shows; tokens per second is how fast pieces then arrive. A slow reply can come from either, so record both. - **Set a length cap and show when it was hit.** A reply that reaches the cap stops mid-sentence, as the widget shows at a cap of 10. ## Go deeper *Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. `python3 show_numbers.py` prints every number below, and `python3 -B -m pytest -q -p no:cacheprovider code/` runs the tests.* ### The generation loop Every system that writes a reply one token at a time runs this loop; only `model` differs. ```text text = tokens of the prompt repeat up to the length cap: chances = model(text) # one pass: a chance for every possible next token token = pick(chances) # the likeliest, or a draw by chance if token is the end mark: stop add token to text show token # streaming: the app displays it now ``` In the toy, `generate` is that loop. It returns one record per pass: the text before it, what it looked at, the chances and the token it added. (Docstrings are left out here.) ```python def generate(model, prompt, cap=30, seed=None): tokens = tokenize(prompt) state = seed_state(seed) if seed is not None else None passes = [] for _ in range(cap): looked_at, ranked, total = chances(model, tokens) if not ranked: break u = None if state is not None: state, u = seed_step(state) token = pick(ranked, total, u) passes.append({"before": list(tokens), "looked_at": list(looked_at), "ranked": ranked, "total": total, "token": token}) if token == END: break tokens.append(token) return passes ``` A reply that ends with the end mark has one more pass than it has tokens: 30 tokens and the end mark took 31 passes. A reply stopped by the cap has as many passes as tokens, and it ends mid-sentence. ### The toy: count what followed what The toy has no neural network. `train` reads the 17 lines of `corpus.txt` (656 tokens, counting one end mark per line, and 82 different tokens besides the end mark). It pads every line with three `` marks, so the first word of a line also follows something, and ends it with ``. For every token it then counts what followed the previous 3, the previous 2, the previous 1 and nothing at all: four tables. The first has 186 different 3-token runs, the second 155, the third 83. ```python def train(lines): tables = [{} for _ in range(ORDER + 1)] for line in lines: tokens = [START] * ORDER + tokenize(line) + [END] for i in range(ORDER, len(tokens)): for k in range(ORDER + 1): followers = tables[k].setdefault(tuple(tokens[i - k:i]), {}) followers[tokens[i]] = followers.get(tokens[i], 0) + 1 return tables ``` `chances` looks up the last 3 tokens of the text. If that run never occurred in the corpus it falls back to the last 2, then the last 1, then to all tokens together. The chance of a follower is its count divided by the total, and followers are ranked by count, with ties broken by the token that sorts first, so the demo repeats exactly. ```python def chances(model, tokens): text = [START] * ORDER + list(tokens) for k in range(ORDER, -1, -1): looked_at = tuple(text[len(text) - k:]) followers = model[k].get(looked_at) if followers: ranked = sorted(followers.items(), key=lambda item: (-item[1], item[0])) return looked_at, ranked, sum(followers.values()) return (), [], 0 ``` `pick` takes the first token of the ranking, or, with a seed, draws by chance: with `u` between 0 and 1 it computes `r = floor(u * total)` and walks down the ranking, subtracting each count, until `r` is smaller than the count it reaches. The seeded draw uses the Park-Miller generator (factor 48271, modulus 2^31 - 1), which needs only whole-number arithmetic, so Python and JavaScript make the same picks. ### Three passes by hand The tea question has 6 tokens: How, do, I, make, tea and ?. The table lists, for the first three passes, what the model looked at (the last three tokens of the text) and what followed them in the corpus. | pass | looked at | followed by | |---|---|---| | 1 | make tea ? | Boil 4, Heat 1 | | 2 | tea ? Boil | some 4 | | 3 | ? Boil some | water 4 | Pass 1: "make tea ?" occurs in the 5 tea lines (the other questions read differently), so the total is 5. Boil follows it 4 times and Heat once, giving 4/5 = 0.8 and 1/5 = 0.2. The likeliest is Boil, so the text grows to 7 tokens. Pass 2: "tea ? Boil" occurs in 4 lines, and each goes on with "some": 4/4 = 1. The text grows to 8 tokens. Pass 3: "? Boil some" occurs in 4 lines, each followed by "water": 4/4 = 1. After three passes the text is "How do I make tea? Boil some water", 9 tokens, and `python3 show_numbers.py` prints the same kind of line for every pass, up to pass 31. ### Why each pass needs a full run of the model The model's knowledge sits in weights, billions of numbers kept in memory. To write one token, a standard model's chip must fetch every weight and use it, and each weight does little work per fetch. [Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/#why-a-chatbot-waits) works this out, and calls writing a reply token by token decode. The first pass reads the whole prompt together (prefill); every pass after that adds one token. The passes cannot run side by side, because pass 2 reads the token that pass 1 made. Leviathan, Kalman and Matias (2022 preprint, ICML 2023) note that writing K tokens needs K runs of the model, one after another. ### The attention cache saves repeated work Without any saved work, every pass would run the network over the whole text so far: 6 positions, meaning places in the text, for pass 1, 7 for pass 2, 8 for pass 3. That is 6 + 7 + 8 = 21 positions after three passes. A server instead keeps, for every earlier token, notes that later tokens look back at: the attention cache, also called the KV cache. A pass then runs the network over the newest token only (it still reads the cached notes), and the first pass reads the prompt. After three passes that is 6 + 1 + 1 = 8 positions. ![Two pairs of horizontal bars of token positions the network runs over for the tea reply. After 10 passes: 105 without a cache, 15 with one. After all 31 passes: 651 without, 36 with.](https://spicyshark.co/learn/why-answers-appear-word-by-word/figs/cache-positions.svg) `positions_read` gives the counts: 651 against 36 for the full tea reply, and a test confirms the formula by running the loop on paper. "Positions" counts places in the text, not seconds or bytes: a real server also reads the cached notes with each pass, and they take memory (see [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/) and [Why AI servers waste memory, and how paging fixes it](https://spicyshark.co/learn/paged-kv-cache/)). The cache does not change the tokens, only the work behind them. ### More than one token per pass The plain loop makes exactly one token per pass, and a cache does not change that. Some systems make more. In speculative decoding, a cheaper model's guesses are checked by the large model, so a pass of the large model can produce several tokens. Leviathan, Kalman and Matias (2022 preprint, ICML 2023) describe it as computing several tokens in parallel without any change to the outputs, and report a 2X-3X acceleration on T5-XXL over the standard T5X implementation, in their own tests. Chen and colleagues (2023) describe speculative sampling, which generates multiple tokens from each transformer call. A [later lesson](https://spicyshark.co/learn/how-a-small-model-speeds-up-a-big-one/) covers the method. ### Why apps stream, and what a stream looks like A reply of a few hundred tokens needs a few hundred passes, so waiting for all of them can leave the screen blank. Provider documentation describes streaming as many small events. The HTML standard defines server-sent events: a server pushes messages over one HTTP response with the content type `text/event-stream`; a message is one or more `data:` lines, and a blank line ends it. An invented example in that shape, carrying the toy's first two tokens: ```text data: {"token": "Boil"} data: {"token": " some"} ``` The page adds each message's token to the screen, which is what the widget's Play button imitates with a timer. Two numbers describe the result: time to first token, the wait before the first message, and tokens per second, how many arrive each second after that. Both depend on the model, the server and the network, so no figure from this lesson transfers to a real service. ### Proof it works ```bash python3 -B -m pytest -q -p no:cacheprovider code/ python3 -B code/embed_widget.py --check ``` The tests check these claims. Each pass adds exactly one token, so the passes equal the tokens plus the end mark, and every pass starts from the text the last one left. The chances add up to 1 and are ranked. The first pass on the tea question is Boil 4, Heat 1; a pass looks at the last 3 tokens only, and text before them changes nothing; unseen text falls back to shorter runs. For each of the four questions the greedy reply is the line the corpus repeats most. A cap of N allows exactly N passes. The seeded generator gives the published check value (from state 1, the 10000th state is 399268537), a draw gives each token its share, and 300 seeds per question stay on the question and end. The cache counts match a pass-by-pass count and give 651 and 36. `widget.html` runs the same rules in JavaScript, so `test_widget.py` runs the widget's own code in Node and compares it with the Python: the tokenizer, the tables, every pass's chances, picks and reply, on five corpora, ten questions, eleven ways of picking (the likeliest token and ten seeds) and four caps (more than a thousand comparisons). A second test drives the page with a stand-in document and a fake clock through 28 runs, and checks Play, Pause, Next, Reset and the closing messages. `embed_widget.py --check` fails if the corpus or the questions inside the widget differ from `corpus.txt` and `prompts.py`. ### Where this is used in practice The public record is documentation. Anthropic's streaming page (read 2026-10-19) says that setting `"stream": true` streams the response incrementally with server-sent events. Each event has a named type and JSON data: a message starts, content blocks arrive as a start, one or more deltas and a stop, and the message ends; text comes in `text_delta` events. The page also says errors can arrive inside the stream. OpenAI's streaming guide (read 2026-10-19) lists events such as `response.created`, `response.output_text.delta`, `response.completed` and `error`. Google's text generation page for the Gemini API (read 2026-10-19) includes a streaming example. None of the three pages promises a speed, and this lesson gives none. ## Skills this lesson builds - Explain why a reply arrives piece by piece, and what streaming changes (the wait before the first piece) and what it does not (the model's speed). - Trace a generation loop by hand: text so far, chances, pick, add, and the two ways it stops (the end mark and the length cap). - Count passes for a reply, and the positions the network runs over with and without a cache. - Read a server-sent events stream, add each piece to a page, and handle an error that arrives mid-stream. - Measure time to first token and tokens per second separately. **Portfolio project:** build a small command-line tool that calls any chat model with streaming on, prints each piece as it arrives, and records when each piece arrived. Run it on 10 prompts of different lengths and print a table of time to first token, total time and tokens per second for each. Publish the prompts, the script and the table. ## Check yourself 1. A friend says the chatbot already wrote the whole answer and is only revealing it slowly. What do you tell them?
AnswerThe model makes one token per pass, from the text so far, and the app shows each token as it arrives. No later token exists until its pass has run, so a reply that is still arriving is still being made.
2. A reply is 120 tokens long, and the model ends it by picking the end mark. How many passes did that take?
Answer121: one pass for each of the 120 tokens, and one more for the end mark.
3. Streaming does not make the model faster. What does it improve?
AnswerThe wait before anything shows. The last token arrives at about the same time either way, but with streaming the reader sees the first piece after the first pass, which is the time to first token.
4. A prompt has 8 tokens and the reply takes 5 passes. How many positions does the network run over without a cache, and with one?
AnswerWithout a cache: 8 + 9 + 10 + 11 + 12 = 50. With a cache: 8 for the prompt, then 1 for each of the other 4 passes, so 12.
5. Why can the 31 passes of the tea reply not run at the same time?
AnswerEach pass reads the token the pass before it made, so it cannot start until that token exists. This is why writing K tokens needs K runs of the model, one after another; speculative decoding is a way to get several tokens from one pass of the large model.
## Further reading **Gentle** - [Streaming messages](https://platform.claude.com/docs/en/build-with-claude/streaming) (Anthropic documentation; read 2026-10-19) - [Streaming API responses](https://developers.openai.com/api/docs/guides/streaming-responses) (OpenAI documentation; read 2026-10-19) - [Text generation](https://ai.google.dev/gemini-api/docs/text-generation) (Google Gemini API documentation, with a streaming example; read 2026-10-19) - [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) (WHATWG HTML Standard; read 2026-10-19) **Deep** - [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) (Leviathan, Kalman and Matias, 2022 preprint, ICML 2023; abstract read 2026-10-19) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://arxiv.org/abs/2302.01318) (Chen, Borgeaud, Irving, Lespiau, Sifre and Jumper, 2023; abstract read 2026-10-19)