---
title: "Why split reading the prompt from writing the reply?"
description: "One long prompt stops every reply on a shared GPU. A toy shows what splitting reading from writing fixes, what the hand-over costs and when it stops paying."
topic: "AI systems"
chapter: "serving-many-chats"
order: 33
minutes: { plain: 6, deep: 17 }
prerequisites: ["Prefill and decode (lesson 23)", "Continuous batching (lesson 24)", "Reading short Python functions"]
seo_title: "Why split reading the prompt from writing the reply? A toy you can run"
date: 2026-10-20
---
# Why split reading the prompt from writing the reply?
An AI chat server does two jobs for every request: it reads your prompt, then writes its reply a word at a time. When one GPU, the chip that does the arithmetic, does both, a long prompt from someone else stops it writing. In this lesson's toy, a prompt of 2000 tokens, which are short pieces of text, takes 103.0 ms to read, and every reply in progress waits: a word due 7.9 ms after the last comes 110.8 ms after it. At 4 requests a second only 13.5% of requests then get a first word within 0.5 s and no later gap over 0.1 s. Give the reading to one GPU and the writing to another and 98.7% do. The price is that the reading GPU's notes on the prompt (the cache, 262 MB for 2000 tokens) must cross the link between the two GPUs.
## In one minute
Picture a shop with two clerks who answer letters by hand. A letter is read in full before its answer starts, and the answer is written one word at a time, so a clerk is usually in the middle of several answers. When a long letter arrives, a clerk who both reads and writes puts every answer down to read it, and her customers see a long pause mid-sentence. The shop can put one clerk on reading and one on writing. The writer never stops, but the notes the reader takes must be carried across the room, and a slow corridor can cost more than the pause it removed.
The reader does the prefill, one big pass over the prompt, and the writer does the decode, one small step per word. The notes are the KV cache of lesson 22 and the corridor is the link between the GPUs. The first word comes later (a longer time to first token), because the prompt waits its turn at one reader and then crosses the link. So the split pays only for long prompts on a fast link, and goodput, the share of requests that meet every target, tells you when.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/why-split-reading-the-prompt-from-writing-the-reply/widget.html). It opens on the toy's default: 4 requests a second, prompts of 2000 tokens, replies of 200 words and a 25 GB/s link. The sentence on top says that shared GPUs get 13.5% of requests right and the split 98.7%. Drag Requests a second to 10: shared falls to 1.7% and the split to 32.2%. Drag Link speed to 0.25 GB/s: the split falls to 0.0%, because no prompt's cache crosses in time. Drag Prompt tokens to 200: the two tie at 100.0%.
## How it works
### The picture, step by step
A toy with two GPUs serves 600 requests, laid out two ways. In the shared layout both GPUs read prompts and write replies, and requests take turns. In the other, one GPU reads and the other writes, which is called a prefill-decode split, or disaggregated serving. A request is good if its first word comes within 0.5 s and no gap between its words is over 0.1 s (invented targets). Figure 1 shows one stall.
*Figure 1. One long prompt stops a shared GPU's replies and does not stop a writer GPU.*
1. **Requests arrive.** 600 requests come at 4 a second, a random time apart, with prompts of 1000 to 3000 tokens and replies of 100 to 300 words.
2. **A shared GPU reads whole.** A 2000-token prompt takes 103.0 ms to read, and no reply on that GPU moves meanwhile. A word due 7.9 ms after the last comes 110.8 ms after it.
3. **Measure the shared pair.** Half of all gaps between words are 7.9 ms (the p50), but the slowest 1 in 100 (the p99, the 99th percentile) is 110.8 ms. Only 13.5% of requests are good.
4. **Give the reading to one GPU.** The reader reads prompts in order, one at a time, with 0.40 s of reading to do each second. The writer only writes, 7.9 ms a step.
5. **Carry the cache across.** A 2000-token prompt's notes are 2000 x 131,072 bytes = 262.1 MB. At 25 GB/s that takes 10.5 ms, against 103.0 ms to read.
6. **Count good requests.** On the split no gap passes 7.9 ms and 98.7% of requests are good, against 13.5%. The first word comes later: p50 143.0 ms against 106.6 ms.
### Each step in code
One real line of the program does each step. The files are in `code/`, and the line numbers are those of this lesson's files.
| Step | The code line | What it does |
|---|---|---|
| 1. Space the arrivals | `pd_sim.py:41`
`t += rng.expovariate(rps)` | Each request comes a random time after the last, 1 / rps seconds on average. |
| 2. Read the whole prompt | `pd_sim.py:57`
`t += read` | The GPU's clock jumps by the whole read before any reply gets another word. |
| 3. Record the gap | `pd_sim.py:72`
`gaps[a[0]].append(t - last[a[0]])` | The gap before a word is the time now minus the time of that reply's last word, so a read in between shows up here. |
| 4. Read in order | `pd_sim.py:98`
`free = max(free, r.arrival) + r.prompt * READ_S_PER_TOKEN` | The reader starts a prompt when it is free and the request has arrived. `free` is when it finishes. |
| 5. Price the hand-over | `pd_sim.py:80`
`return prompt_tokens * CACHE_BYTES_PER_TOKEN / (link_gbps * 1e9)` | The cache's bytes divided by the link's bytes per second. |
| 6. Count good requests | `pd_sim.py:112`
`good = sum(1 for i, r in enumerate(requests) if first[i] - r.arrival <= first_target and max(gaps[i], default=0.0) <= gap_target)` | A request counts when its first word came within the target and its largest gap is within the target. |
### A real run
`trace.py` runs the six steps with `pd_sim.py`'s own functions and prints one line for each, then one line for the limit. Run it from the repository root:
```bash
cd lessons/why-split-reading-the-prompt-from-writing-the-reply/code && python3 trace.py
```
Phone readers: slide the box sideways to see every column.
```text
step 1 600 requests in 148.9 s, 4 a second; prompts of 1000, 2000, 3000 tokens, replies of 100, 200, 300 words
step 2 shared: a 2000-token prompt takes 103.0 ms to read and no reply on that GPU moves meanwhile; a word due 7.9 ms after the last comes 110.8 ms after it
step 3 shared: first word p50 106.6 ms, p99 267.0 ms; gap between words p50 7.9 ms, p99 110.8 ms
step 4 split: the reader GPU has 0.40 s of prompts to read each second; the writer GPU only writes, 7.9 ms a step
step 5 hand-over: 2000 tokens x 131072 bytes = 262.1 MB; at 25 GB/s that takes 10.5 ms, against 103.0 ms to read (equal at 2.55 GB/s)
step 6 split: first word p50 143.0 ms, p99 616.3 ms; gap p50 7.9 ms, p99 7.9 ms; goodput 98.7% against 13.5% shared
limit goodput of the split by link speed, GB/s: 100 98.7%, 25 98.7%, 12.5 98.7%, 1.25 86.3%, 0.125 0.0%
```
Read it against the steps.
- **Steps 2 and 3.** The word waits 110.8 ms, 14 times the usual 7.9. A 200-word reply lasts about 1.6 s, long enough for a prompt to land on its GPU, so 86.5% of replies have a gap over 0.1 s and 13.5% are good. Figure 2 shows the tail grow with load: the shared p99 gap passes 100 ms at 4 requests a second and is 213.8 ms at 10, while the split stays at 7.9 ms.
*Figure 2. The shared p99 gap passes the limit at 4 requests a second. The split's does not move.*
- **Step 4.** One reader can read about 9.7 prompts a second (1 / 0.103 s). Past that a queue builds: the split's first-word p99 climbs from 0.27 s at 1 request a second to 3.72 s at 10.
- **Step 5.** The hand-over equals the read at 2.55 GB/s for any prompt length, because the cache and the read both grow with the prompt.
- **Step 6.** The first word is slower, p99 616.3 ms against 267.0 ms, because a prompt waits for one reader and crosses the link, while the shared pair reads on two GPUs. Goodput counts both targets, so it shows which layout keeps requests inside both.
- **The limit.** Slow the link and the split fails: 86.3% at 1.25 GB/s and 0.0% at 0.125. It beats the shared layout on links faster than about 0.3 GB/s (Figure 3). Below that even the shortest prompt, 1000 tokens, has a 131 MB cache that cannot be read and cross within 0.5 s, so no request is good.
*Figure 3. The split beats shared GPUs on goodput on links faster than about 0.3 GB/s, and loses below it.*
## Your turn
The link was 25 GB/s, fast enough to carry the cache in a tenth of the read. See what a link 100 times slower does.
1. In `code/trace.py`, change line 8 to `LINK_GBPS = 0.25`.
2. Run `python3 trace.py` again in the `code` folder.
Look at steps 5 and 6. How long does the hand-over of a 2000-token prompt take, and what share of requests is good?
What you should see
Steps 5 and 6 change and the other lines stay as they were:
```text
step 1 600 requests in 148.9 s, 4 a second; prompts of 1000, 2000, 3000 tokens, replies of 100, 200, 300 words
step 2 shared: a 2000-token prompt takes 103.0 ms to read and no reply on that GPU moves meanwhile; a word due 7.9 ms after the last comes 110.8 ms after it
step 3 shared: first word p50 106.6 ms, p99 267.0 ms; gap between words p50 7.9 ms, p99 110.8 ms
step 4 split: the reader GPU has 0.40 s of prompts to read each second; the writer GPU only writes, 7.9 ms a step
step 5 hand-over: 2000 tokens x 131072 bytes = 262.1 MB; at 0.25 GB/s that takes 1048.6 ms, against 103.0 ms to read (equal at 2.55 GB/s)
step 6 split: first word p50 1151.5 ms, p99 1976.0 ms; gap p50 7.9 ms, p99 7.9 ms; goodput 0.0% against 13.5% shared
limit goodput of the split by link speed, GB/s: 100 98.7%, 25 98.7%, 12.5 98.7%, 1.25 86.3%, 0.125 0.0%
```
The hand-over takes 1048.6 ms, about ten times the read, and no request is good. Even the shortest prompt, 1000 tokens, needs 524 ms to cross, which is over the 0.5 s allowed for the first word. The gaps stay at 7.9 ms, because the writer is never held up. The split fails on the corridor, not on the writing. The limit line does not move, because it tries its own link speeds.
## What this means when you build
- **Watch the p99 gap, not the middle.** In the toy the middle gap is 7.9 ms on both layouts. The slowest 1 in 100 is 110.8 ms shared and 7.9 ms split ([lesson 27](https://spicyshark.co/learn/why-average-speed-is-misleading/)).
- **Know your traffic before you split.** In the toy, prompts of 600 tokens or fewer nearly tie, and at 2000 tokens the split leads 98.7% to 13.5%. A pause between words matters to a person who watches the words appear, not to a program that waits for the whole reply.
- **Price the hand-over first.** The cache is the prompt's tokens times the bytes per token, and the link must carry it inside your first-word target. In the toy the split stops paying below about 0.3 GB/s. Real figures depend on the model and the hardware.
- **Time each leg separately.** In pull request 3201 of vllm-project/aibrix (merged 2026-10-20; read 2026-10-20), a request set to `pd:1,least-request:1` was routed as `pd`, but two places compared the raw setting with `pd`. Routing time then ran past the prefill leg, and the response side recorded none of its prefill, KV-transfer or decode counters. The merge resolves the setting before both checks, at `gateway_req_body.go:214` and `gateway_rsp_body.go:698` (line numbers at the merge commit `3469dad`). A metric that mixes two legs, or skips one, hides where the delay is.
- **On one GPU, try chunked prefill first.** If you cannot add a second pool or a fast link, mixing pieces of a prompt into each writing step shortens the stall with no hand-over (Go deeper).
## Go deeper
*Optional. This part is for readers who write Python. The code is in the `code/` folder next to this lesson. From inside that folder, `python3 trace.py` prints the run above, `python3 make_figs.py` redraws the figures and `python3 -B -m pytest -q -p no:cacheprovider` runs the tests.*
`pd_sim.py` is 127 lines, with its comments and blank lines. It uses only the standard library, reads no file, opens no connection and loads no model. Two of its costs echo lesson 23's best case for one chip, the chip of lesson 19: 51.5 microseconds to read a prompt token and 7.877 ms for a writing step. The 131,072 cache bytes per token echo lesson 22. The link speeds, the 32 slots a GPU writes at once and the two targets are invented.
### The simulation
```python
waiting, active, first, gaps, last, t = sorted(jobs), [], {}, {}, {}, 0.0
while waiting or active:
if not active and waiting[0][0] > t:
t = waiting[0][0] # an idle GPU waits for the next job
ready = list(takewhile(lambda j: j[0] <= t, waiting)) # `waiting` is sorted, so the jobs that are ready are its first few
if ready and len(active) < SLOTS and ready[0][1] > 0: # step 2: a prompt waits and a slot is free, so read the whole prompt now
ready_t, read, left, i = ready[0]
waiting.remove(ready[0])
t += read
first[i], gaps[i], last[i] = t, [], t
if left > 0:
active.append([i, left])
continue
```
`serve` is one GPU. A job is its ready time, the seconds of reading it needs, the words left to write and an id. When a prompt is waiting and a slot is free, the whole prompt is read at once: `t` jumps, the first word is stamped at the end, and `continue` returns to the top without running a writing step. There is no chunking. That is the stall.
```python
t += STEP_S # one writing step: every reply in progress gets one word
for a in active:
gaps[a[0]].append(t - last[a[0]])
last[a[0]], a[1] = t, a[1] - 1
active = [a for a in active if a[1] > 0]
return first, gaps
```
A writing step costs `STEP_S` for any number of replies in progress, up to 32, and gives each one a word. The gap recorded is the time since that reply's last word, so a read that ran between two steps is inside it. `shared` sends request 0 to the first GPU, request 1 to the second, and so on, each with its own `serve`.
```python
free, jobs = 0.0, []
for i, r in enumerate(requests):
free = max(free, r.arrival) + r.prompt * READ_S_PER_TOKEN
jobs.append((free + handover_seconds(r.prompt, link_gbps), 0.0, r.reply - 1, i))
return serve(jobs)
```
`split` makes the reader a single queue. `free` is when the reader finishes each prompt, and the job reaches the writer at `free` plus the hand-over, with 0.0 seconds of reading to do. `serve` treats such a job as one that arrives with its first word made, so the writer GPU never reads and its steps are never interrupted. The link has no limit on how many caches cross at once.
### Why the two jobs differ
```python
READ_S_PER_TOKEN = 2 * PARAMS / A100_PEAK_FLOPS # reading the prompt: 2 FLOPs per weight per token, limited by arithmetic (lesson 23)
STEP_S = 2 * PARAMS / A100_BANDWIDTH # one writing step: every weight is read once, limited by memory, the same for any number of replies (lesson 23)
```
Reading a prompt does about 2 operations per weight for every token, so a long prompt keeps the arithmetic busy and a token costs `READ_S_PER_TOKEN`. A writing step has to fetch every weight from memory once, whatever the number of replies, so it costs `STEP_S` however full the slots are (lessons 19 and 23). Splitwise states the same pair, prompt computation compute-intensive and token generation memory-intensive. One GPU doing both serves each badly: the prompt holds the writers back, and the writers hold the arithmetic back.
### The cost of the hand-over
The cache is `prompt tokens x 131,072 bytes`, so it grows with the prompt just as the read does. For a 2000-token prompt, 262.1 MB crosses, and the time depends only on the link:
| Link | Hand-over of 2000 tokens | Against the 103.0 ms read |
|---|---|---|
| 100 GB/s (800 Gbit/s) | 2.6 ms | 2.5% |
| 25 GB/s (200 Gbit/s) | 10.5 ms | 10.2% |
| 12.5 GB/s (100 Gbit/s) | 21.0 ms | 20.4% |
| 1.25 GB/s (10 Gbit/s) | 209.7 ms | 2.0 times |
| 0.125 GB/s (1 Gbit/s) | 2097.2 ms | 20 times |
The hand-over is slower than the read below 2.55 GB/s, but the split still wins goodput long after that: at 2 GB/s the split is good for 94.8% of requests, because the first-word target is 0.5 s and the hand-over only has to fit inside what is left after the read. The split stops paying at about 0.3 GB/s (0.2922 in the code: at 0.29 no request is good and at 0.293, 22.3% are), where the 1000-token prompts run out of time: 51.5 ms to read plus 448.5 ms to cross fills the 0.5 s (Figure 3). The sweep is in `make_figs.py`, which finds the crossover by bisection on the same `compare` function.
### Where the split does not pay
Figure 4 holds the link at 25 GB/s and the load at 4 requests a second, and varies the prompt.
*Figure 4. The two layouts nearly tie up to 600 tokens, and the split pulls ahead after that. At 6000 tokens the single reader is overloaded and both collapse.*
With short prompts, a read is short, so the stall is short: at 200 tokens the read takes 10.3 ms, a stall is about 18 ms and the 0.1 s gap target is never missed. The split then pays for a link it does not need. At the other end, 6000-token prompts need 0.31 s of reading each, so one reader falls behind (the split's first-word p99 is 32.5 s) and the shared pair, which reads on two GPUs, falls behind too. The split has a window between these: at this load the two nearly tie up to 600 tokens, and the split leads from 800 to 4000.
### What the papers report
DistServe (Zhong et al., OSDI 2024, arXiv 2401.09670; read 2026-10-20) finds that putting prefill and decoding on the same GPUs causes "strong prefill-decoding interferences", and assigns them to different GPUs. It reports that it can serve 7.4 times more requests or meet a 12.6 times tighter latency target than the systems it compares with, "while staying within latency constraints for > 90% of requests", and on one chat workload a 2.0 to 4.6 times higher request rate than vLLM. Its goodput is a rate, the most requests per second per GPU that meet the latency targets for over 90% of requests. On the transfer, it puts the cache of one 512-token request to a 66-billion-parameter model at about 1.13 GB, which at 10 requests a second is 11.3 GB/s (90 Gbit/s), and reports the transfer under 0.1% of total latency for a 175-billion-parameter model and under 30 ms for over 95% of requests. Its discussion says that a throughput-only offline job may favour chunked prefill, that one or a few GPUs may suit a non-split system, and that disaggregation "remains promising in long-context serving": the cache, and with it the transfer, grows "linearly with the prompt length" but "the prefill computation grows quadratically, so the relative duration of transmission and prefill job decreases".
Splitwise (Patel et al., ISCA 2024, arXiv 2311.18677, version 2; read 2026-10-20) reports up to 1.4 times higher throughput at 20% lower cost than current designs, or 2.35 times more throughput under the same power and cost budgets. It sends the cache layer by layer while the prompt is still being read, which leaves a constant transfer time that is not hidden of about 8 ms on A100 GPUs with a 200 Gbit/s link and about 5 ms on H100 GPUs with 400 Gbit/s. For small prompts it sends the cache in one piece, below 512 tokens on the H100 setup.
Sarathi-Serve (Agrawal et al., OSDI 2024, arXiv 2403.02310; read 2026-10-20) keeps reading and writing on the same GPUs and "splits a prefill request into near equal sized chunks", mixing a chunk into each step so that writing is not stalled. It reports 2.6 times higher serving capacity for Mistral-7B on one A100 and up to 3.7 times for Yi-34B on two A100s, both against vLLM, and up to 5.6 times for Falcon-180B with pipeline parallelism. It notes that a split needs the cache moved, which is hard without a fast link.
### What the toy leaves out
- **One reader and one writer.** Real systems run several of each and choose the ratio. The toy's single reader saturates near 9.7 requests a second.
- **No chunking and no batching of prompts.** The reader takes one prompt at a time, whole, and the shared GPU does the same.
- **An uncontended link.** Any number of caches cross at once. A real link is shared, and the cache can cross in layers as the prompt is read (Splitwise).
- **A fixed split of traffic.** The shared layout alternates requests between two GPUs. A real scheduler can send a request to the less busy one.
- **A share, not a rate.** The toy's goodput is the share of requests that meet both targets. DistServe's goodput is a rate per GPU.
- **Toy costs.** The read and write times are best-case figures for one chip, not measurements, and every request is served to its last word with no failures.
### Proof it works
`test_pd_sim.py` checks that the copied costs equal the ones in lesson 23's code, that a read stalls a shared GPU's replies for the read plus one step, that a hand-over is bytes over link speed, and that the split removes the stall and reproduces the crossover. `test_lesson.py` checks that every number this lesson, the trace and the figures quote comes from the code, that each code block and each `file:line` in the table matches the real file, that the figures are the ones `make_figs.py` draws, and that `trace.py` prints the run above and the Your turn change prints the lines shown. `test_widget.py` checks that the numbers inside the widget are the ones `show.py` computes from `pd_sim.py`, that the page's JavaScript gives the same words as `show.py` for every setting of its six controls, that each state named in "Explore the numbers" is reachable and matches a fresh run of the simulation, and that the chart's key stays above the plot and its text stays large enough to read on a phone.
### Where this is used in practice
The papers above describe systems that serve with the reading and the writing apart. The open-source AIBrix gateway has a routing strategy named `pd` for it, and the pull request in the section above is a fix to how that strategy is timed (read 2026-10-20). This lesson shows the idea on a toy and measures no real server.
## Skills this lesson builds
- Say what a prefill-decode split changes and what it costs.
- Read goodput and the p99 gap instead of an average.
- Work out a cache's size and its hand-over time from bytes per token and link speed.
- Find the link speed or prompt length at which a split stops paying.
- Time the legs of a split request separately.
**Portfolio project:** extend the simulation with a second reader GPU and with a shared layout that reads prompts in chunks. Chart goodput against requests a second for all three layouts, and add a short note on which one your own traffic would need and why. Publish the code, the chart and the note.
## Check yourself
1. On shared GPUs the middle gap between words is 7.9 ms, the same as on the split. Why do they differ so much in goodput?
Answer
The middle gap hides the stalls. A reply lasts about 1.6 s, long enough for a prompt to land on its GPU, and one gap over 0.1 s makes a request fail: 86.5% of the shared requests have one. The split's gaps never pass 7.9 ms.Answer
No. The split is good for 94.8% of requests at 2 GB/s, against 13.5% shared. Goodput is set by the targets, not by comparing the hand-over with the read: the first word has 0.5 s, and a stall of 110.8 ms on shared GPUs breaks the gap target while a hand-over delays only the first word.Answer
A 200-token prompt takes 10.3 ms to read, so a shared GPU stalls its replies for about 18 ms, well inside the 0.1 s gap target. There is no stall for the split to remove, and it still adds a hand-over to the first word.Answer
The single reader. An average 2000-token prompt takes 0.103 s to read, so one reader handles about 9.7 prompts a second. Above that, prompts queue and each waits for those ahead of it. The writer GPU is not the limit, and a second reader would be the fix.Answer
Routing time should end where the prefill leg starts. For the `pd` strategy the router sends the prefill leg from inside routing, so a routing time that runs to the end of routing holds the whole prefill: a slow prefill looks like slow routing, and the operator looks in the wrong place. Pull request 3201 of vllm-project/aibrix made a weighted setting that resolves to `pd` end routing time at the prefill start too.