On this page
A service with 1,000 users does not need a GPU for each one. It needs enough for the busiest hour. Take a toy service with 1,000 users, 600 of them active in that hour and each receiving 120 replies: 72,000 replies, or 20 new replies a second. The chip is lesson 28’s A100 chip with 80 GB. One such GPU carries about 38 chats at once before the slowest 1 reply in 100 has words more than 50 ms (thousandths of a second) apart (lesson 27). The toy plan is 4 GPUs plus a spare.
In one minute
A town of 1,000 people does not need a restaurant with 1,000 seats. It needs seats for the busiest hour: the parties arriving each minute times the minutes each one stays. That rule is Little’s law. For an AI service a party is a chat in progress, and a stay is the time from a question to its last word, so chats at once equal new replies a second times the seconds a chat stays.
Each GPU, like a restaurant, has two limits, and either can run out first. Seats are memory, the pool of notes about chats in progress (lesson 28). Cooks are speed: every added chat makes each user wait longer between words (lesson 24). Divide the demand by what one restaurant carries, keep some room in hand and add a spare for a breakdown. Then test at the planned peak: a paper plan is a guess.
Explore the numbers
Open the widget. Set the users, the share active in the busiest hour, the replies each receives and the gap between words you accept. It shows the chats at once, the GPUs speed needs against those memory needs, and a plan with a spare. At the opening setting speed needs 4 and memory 1, so speed sets the limit. Under More settings, pick Long answers: the replies each drop to 30, and memory, now needing 6 GPUs to speed’s 4, sets the limit.
How it works
Size for the busiest hour
Only some users are present at once. Here 600 of 1,000 are active and each receives one reply every 30 seconds: 20 new replies a second. While a reply is being written its chat is in progress. Little’s law: the average number in progress is the arrival rate times the average time each stays, from question to last word. That time depends on how full a GPU runs, so the next sections find it: about 110 chats, not 600 or 1,000. The toy frees a chat’s notes at its last word, so a user’s next question reads the whole conversation again (lesson 8 covers keeping them).
Two limits on one GPU
Memory: at a 90% share, lesson 28’s pool holds 200 chats of the usual length, about 2,000 tokens. Speed: each added chat makes every step read more notes, so words arrive further apart. The toy serves the same chats at ever higher numbers at once and records the p99 time between words. It reaches the 50 ms target at 38.6 chats (38 in the figure), the speed limit, long before memory runs out at 200.
Figure 1. The p99 time between words against chats in progress on one GPU.
Count the GPUs
At its speed limit, 38.6 chats, one GPU finishes 7.0 replies a second, so each chat stays 38.6 / 7.0 = 5.5 s. With 20 new replies a second, about 20 x 5.5 = 110 chats are in progress. The load test on 3 GPUs, each near its limit, counts 109 (Figure 2).
Figure 2. A 10-minute load test on 3 GPUs. The chats in progress stay near 109; the plan said 110.
Speed alone gives 110 / 38.6 = 2.85, so 3 GPUs. That leaves no headroom: room kept unused so random bursts do not push users past the target. The plan fills each GPU to 80% of its limit. Speed then needs 110 / (0.8 x 38.6) = 3.56, so 4 GPUs. Memory needs 110 / (0.8 x 200) = 0.69, so 1. The larger, 4, is the plan, and speed binds. One spare makes 5.
Which limit binds depends on the chats
Long prompts make every reply read a lot, so speed runs out first: with documents of about 27,000 tokens one GPU holds 14 chats but meets the target with only 2, and the crowd needs 55 GPUs. Long replies keep chats open, so memory fills first at this target: with replies of 1,500 to 4,000 tokens memory holds 85 chats and speed 140. A chat on a GPU at its limit then stays 81 s, so there a user gets about 44 replies an hour, not 120. At 30 each, 405 chats are in progress and need 6 GPUs.
Figure 3. Memory and speed limits for three kinds of chat, on one scale. The shorter bar binds.
Test before you sign
Real arrivals are random and bunch up. A load test sends random arrivals at the planned rate through the planned GPUs and measures what users see. With 3 GPUs the p99 time between words is 52.6 ms, a narrow miss of 50 ms, and the p99 wait for the first word (lesson 23) is 4.23 s, a clear miss of 2 s. With 4 it is 36.9 ms and 0.51 s: it passes. With 5, 25.8 ms and 0.38 s. The spare matters when a GPU fails: plan 4 and one failure leaves 3, which misses; plan 5 leaves 4, which passes.
Figure 4. The load test by number of GPUs, against the targets of 50 ms and 2 s.
What this means when you build
- Ask for the busiest hour, not the user count. Here 1,000 users became 20 replies a second.
- Take chat lengths from your own logs. The same 1,000 users need 4 GPUs for usual chats and 55 for long documents; long answers, at 30 replies each, need 6.
- Size for a percentile, not the average. Choose the p99 gap and wait you accept.
- Find which limit binds before buying. More memory does not help when speed binds, nor more speed when memory does.
- Load-test at your planned peak before you sign. Three GPUs looked enough on paper and missed.
- Keep headroom and a spare. The first covers bursts, the second a failure.
- Plan again when use or chat length grows. The next lesson compares renting an API with running your own.
Optionalabout 17 min
Go deeper
Optional. This part is for readers who write Python. The code is in the code/ folder next to this lesson. From inside that folder, python3 show_numbers.py prints every number below, and python3 -B -m pytest -q -p no:cacheprovider runs the tests.
capacity_plan.py does the plan in three steps: demand, the limits of one GPU, the count. It loads lesson 28’s server (settings_calc.serve), lesson 27’s load generator and percentile, and lesson 26’s least_n by path.
Little’s law
In symbols L = λW: L is the average number in the system, λ the arrival rate and W the average time one item spends there. Little’s 2011 paper (read 2026-10-20) says the law holds whatever the order of service and without assuming steady arrivals, and that over a finite period it is exact, after the fact: it measures, and does not forecast. The reason is one area counted twice. Each reply is in progress from its arrival to its last word, so the area under the in-progress line equals the sum of the reply times. load_test computes both sides:
length = max(done for _, done in spans) # the system starts and ends empty
times = [done - arrival for arrival, done in spans]
rate, in_progress = len(spans) / length, sum(times) / lengthOn 3 GPUs it gives 19.99 a second times 5.45 s, 109.1 chats, against the plan’s 110.5. The two count differently: the test times a reply from its arrival, queue included, and the plan reads a chat’s stay from the moment it joins a batch. Their closeness is partly luck. The 3 GPUs hold 36 chats each, fewer than 38.6, which shortens a stay, and a queue lengthens it. On 4 GPUs each has room, replies come faster and it gives 20.01 times 2.94 s, 58.7 chats. Planning needs W before any test, and W grows with the load, so the plan reads it from a table, at the design point where each GPU is as full as its tighter limit allows.
The speed table
step_table serves a backlog of 600 chats (lesson 28’s draw) at ever higher numbers at once, with a pool that never runs out so that only speed shows. Each row holds the chats in progress, the replies a second and the p99 time between words.
| Chats | Replies/s | p99 gap |
|---|---|---|
| 1.0 | 0.744 | 8.3 ms |
| 7.9 | 3.761 | 18.5 ms |
| 15.8 | 5.304 | 30.1 ms |
| 31.4 | 6.687 | 43.4 ms |
| 46.8 | 7.324 | 57.5 ms |
| 62.1 | 7.664 | 70.1 ms |
| 123.0 | 8.290 | 108.5 ms |
Divide a row’s chats by its replies to get the time a chat stays, Little’s law again: at 31.4 chats, 4.69 s. stay reads the table at 38.6 chats: 38.6 / 6.98 = 5.52 s. speed_limit interpolates between the two rows around the target. For 50 ms that is between 31.4 and 46.8, at 38.6 chats, which the lesson shows as 38. Lesson 28 chose a cap of 64, with 47.3 ms; that was a mean wait, and at 62.1 chats the p99 is 70.1 ms. The target moves the limit a great deal: 30 ms gives 15 chats and 5 GPUs, 75 ms gives 72 and the same 4.
The memory limit
memory_limit is lesson 28’s pool divided by the blocks of an average chat. The usual chats average 1,882 tokens of prompt and 157 of reply, 2,039 in all, which is 128 blocks of 16. The pool at a 90% share is 25,720 blocks, and 25,720 // 128 is 200. Lesson 28’s 205 used a chat of 2,000 tokens. The count takes every chat at its average full length.
The count
def gpus(chats, limit):
return max(1, math.ceil(chats / (HEADROOM * limit) - 1e-9))HEADROOM is 0.8, a choice and not a law; the 1e-9 stops a division that is whole in exact arithmetic from rounding up. plan takes the smaller limit as the design point, reads W there with stay, multiplies by the replies a second and runs gpus once for each limit, keeping the larger. Without headroom the arithmetic gives 3, and the load test shows that 3 misses. The three kinds of chat at 50 ms, with the memory limit and then the speed limit in chats:
| Chats | Limits | Replies each | GPUs, spare not counted |
|---|---|---|---|
| Usual | 200 and 38 | 120 | 4 |
| Documents | 14 and 2 | 120 | 55 |
| Answers | 85 and 140 | 30 | 6 |
Speed binds for the usual chats and the documents, memory for the answers. For the documents W is 4.8 s, so 96 chats, and the speed limit is 2.2: 96 / (0.8 x 2.2) = 54.5, so 55 GPUs. For the answers W is 81 s, so 5 new replies a second keep 405 chats in progress: 405 / (0.8 x 85) = 5.96, so 6 GPUs, against 405 / (0.8 x 140.6) = 3.60, so 4 for speed. At 120 replies each that mix would need more chats than there are active users, so it uses 30. The plan overshoots a load test here too, as for the usual chats: on 6 GPUs the test averages 231 chats and 47.5 s a reply, against the plan’s 405 and 81 s. The documents have prompts of 24,000 to 31,000 tokens, 27,386 on average, and the answers have replies of 1,500 to 4,000. Length alone does not decide. Speed runs out when each reply makes the GPU read a lot, as with long prompts. Memory runs out when long replies keep many chats open at once.
The load test
arrivals sends 12,000 requests at 20 a second: random gaps from lesson 27’s generator (seed 77) and chats drawn with seed 2029. load_test sends request k to GPU k mod N, which spreads them evenly, and runs a lesson 28 server on each, with a cap of 38 chats (the speed limit rounded down) and the real pool. A pass is a p99 gap of 50 ms or less and a p99 wait of 2 s or less, both over all requests. Both must hold: with 2 GPUs the cap keeps the gap at 49.3 ms, yet the wait is 249 s.
| GPUs | p99 gap | p99 wait |
|---|---|---|
| 3 | 52.6 ms | 4.23 s |
| 4 | 36.9 ms | 0.51 s |
| 5 | 25.8 ms | 0.38 s |
The chats in progress, averaged over the run, came to 109.1 on 3 GPUs, 58.7 on 4 and 46.6 on 5. Three GPUs finish at most about 21 replies a second against 20 arriving, so each is busy 95% of the time and any burst builds a queue. The busiest minute is 5.8% above the average; random gaps bunch only mildly at this size, and real traffic bunches more.
How much headroom
The 80% is a choice. Measured against the 7.0 replies a second one GPU finishes at its speed limit, 3 GPUs run at 95%, 4 at 72% and 5 at 57%. The test missed at 95% and passed at 72%, and 5 GPUs at 57% brought the p99 wait down from 0.51 s to 0.38 s. A busier plan buys fewer GPUs and leaves less room for bursts; an idler one costs more and leaves more. Your own traffic and a load test on it tell you where the edge is.
The spare
Planning for the demand alone leaves no margin for a failed GPU, a restart or an upgrade. With 5 GPUs planned, one failure leaves 4 and the test passes at 36.9 ms; with 4 planned, one failure leaves 3, which misses. In normal hours the spare is not idle: 5 GPUs give 25.8 ms where 4 give 36.9 ms.
Run the plan backwards
Growth asks the question the other way round: how many users can these GPUs carry? Four GPUs at 80% finish about 22 replies a second against the 20 needed, which is room for about 1,100 users with the same activity and replies each. With one GPU down, 3 GPUs finish about 17 replies a second: about 840 users, short of 1,000, the same reason that test missed. Run the load test again at the new rate before relying on it, because the table was measured on one kind of chat.
A model too big for one GPU
The 38 and 200 are for the 8-billion-number model on one A100 chip with 80 GB. A model that needs several GPUs for one copy has its own table and its own pool. Lesson 26’s least_n gives the GPUs for one copy: 1 for the 8B model, 2 for the 70B and 11 for the 405B. gpus_to_hold multiplies by the copies: 4 copies take 4, 8 and 44 GPUs.
What the toy leaves out
- The hour is steady. Real days and weeks rise and fall, and a launch or a campaign adds a burst. Plan from the busiest hour measured, plus growth.
- Reading prompts competes with writing replies. Chunked prefill (lesson 24) changes the speed table.
- A chat’s notes are freed at its last word, so each turn reads the whole conversation again. Prefix caching (lesson 8) keeps and shares cached notes, so memory per chat is smaller.
- Quotas (lesson 11) cap the replies one user can ask for, and so cap the demand.
- The toy balances perfectly by taking turns. A real balancer sees uneven chats.
- The chats are invented and the chip runs at its peak speed.
Proof it works
test_capacity_plan.py guards what a reader sees. The 600 usual chats are lesson 28’s. Every speed table rises and reaches every target. The chats in progress equal the rate times the mean time, and the sweep that draws figure 2 agrees with a plain count. The limits, W, the chats in progress, the GPU counts with and without headroom and the limit that binds are pinned for the three kinds of chat, with the rounding edge of gpus; dividing the rounded numbers the page prints gives the page’s counts. The load test misses at 3 GPUs and passes at 4 and 5, and the same holds with one GPU down. With 2 GPUs the gap passes and the wait fails, which guards the rule that both must hold. A model too big for one GPU takes several for each copy. One test checks that every number quoted in this lesson and in the four figures comes from the code, that the figures on disk are the ones make_figs.py draws, and that the code holds no web address. test_widget.py runs the widget’s JavaScript in Node against the Python on a grid of settings, and drives the page with a stand-in for the browser.
Where this is used in practice
Little’s 2011 paper (INFORMS, doi 10.1287/opre.1110.0940; read 2026-10-20) reviews the law’s first 50 years. Among its uses it quotes a hospital example of rough sizing: the average in the system from the law, divided by a target utilization such as 90%, suggests the number of beds. That is close to this lesson’s headroom. The Google SRE book’s introduction (read 2026-10-20) lists regular load testing “to correlate raw capacity … to service capacity” among the steps of capacity planning, and says that SREs “provision to meet a capacity target at a specific response speed”. The vLLM Benchmark CLI page (stable version, read 2026-10-20) says a finite request rate uses a Poisson process by default, with --burstiness to make traffic bunchier or more even. For capacity planning it suggests a concurrency limit of 80 to 90% of the reported maximum, where that maximum is the cache size divided by max_model_len: for lesson 28’s pool, 411,520 / 8,000 = 51. That is a setting for a test, a different quantity from this lesson’s headroom. Documentation changes between versions, so check the one you run.
Skills this lesson builds
- Turn users into new replies a second for the busiest hour, and into chats at once with Little’s law.
- Find a GPU’s speed limit from a measured table and its memory limit from a pool, and say which binds.
- Turn the limits into a GPU count with headroom and a spare.
- Run a load test with random arrivals and read the p99 gap and wait to pass or fail a plan.
- Say what a plan leaves out: bursts, prompts that compete with replies, shared prefixes and uneven balancing.
Portfolio project: write a planner that takes a request log (arrival time, prompt tokens, reply tokens) and a measured table of replies a second and p99 gap against chats in progress, then reports the busiest hour, the chats at once, both limits, the GPU count and a spare. Run a load test on a real server at the planned rate through N-1, N and N+1 copies. Publish the code, one chart of the p99s by count, and a note on where the measured chats in progress differ from the table’s.
Check yourself
-
A service has 1,000 users, 600 of them active in the busiest hour, each receiving 120 replies in it. How many new replies arrive a second?
Answer
600 x 120 = 72,000 replies in 3,600 seconds, which is 20 a second.
-
Replies arrive at 20 a second and each chat stays 3 s. How many chats are in progress, and why does that not depend on the number of users?
Answer
20 x 3 = 60 chats, by Little’s law. Only the users who are active at the peak create arrivals, and the law counts the chats in progress, not the users.
-
One GPU keeps the target at 40 chats, and a chat stays 5 s at that load. With 80% headroom, how many GPUs do 20 replies a second need, and how many with a spare?
Answer
20 x 5 = 100 chats in progress; 100 / (0.8 x 40) = 3.1, which rounds up to 4. With a spare, 5.
-
The same 1,000 users need 4 GPUs for usual chats. For long answers at 30 replies each they need 6. Which limit binds for long answers, and why?
Answer
Memory, at the 50 ms target. Replies of 1,500 to 4,000 tokens keep each chat open about 81 s, so the pool fills at 85 chats, before the speed limit of 140.
-
The arithmetic says 3 GPUs are enough, yet the load test with 3 fails. Why?
Answer
3 GPUs finish at most about 21 replies a second against 20 arriving, so they are busy 95% of the time. Random arrivals bunch up and a queue builds: the p99 wait is 4.23 s against the 2 s target, a clear miss, and the p99 gap 52.6 ms against 50 ms, a narrow one. With 4 GPUs the test passes.
Further reading
Gentle
- Why is the average speed of an AI service a misleading number? (lesson 27, percentiles and the load generator)
- What do a model server’s settings change? (lesson 28, the pool and the cap)
- How does an AI server answer many people at once? (lesson 24, the server this lesson runs)
- Why does a big model need several GPUs? (lesson 26, the GPUs for one copy of a model)
- Introduction, Site Reliability Engineering (Benjamin Treynor Sloss, edited by Betsy Beyer, Google; demand forecasting, capacity planning and load testing; read 2026-10-20)
Deep
- Little’s Law as Viewed on Its 50th Anniversary (John D. C. Little, Operations Research 59(3), INFORMS, 2011; what the law needs and what it cannot do; read 2026-10-20)
- Benchmark CLI (vLLM documentation, the “stable” version; request rate, burstiness and the concurrency limit; read 2026-10-20)
Optional
Code
Each file runs on its own. Download it, or read it here.
Show code
"""How many GPUs a crowd needs: lesson 29 (how-many-gpus-for-1000-users).
Everything runs inside this one Python process. Nothing here talks to a network or to a real service.
The server is lesson 28's (settings_calc.py), which is lesson 24's continuous-batching server with a pool of cache blocks. The load generator is lesson
27's: random gaps between requests, a fixed seed. The plan has three steps.
1. Demand. `peak_rate` turns users into new replies a second in the busiest hour.
2. Limits of one GPU. Speed: the most chats at once for which the p99 time between words stays under a target (`speed_limit`, read off a table
`step_table` made by running the server with ever more chats at once). Memory: the chats the cache pool holds (`memory_limit`).
3. Count. One time W serves the whole plan: the seconds a chat holds a batch slot and its cache when a GPU is as full as its tighter limit allows, the
design point (`stay`, read off the same table; with speed binding, every user sees the p99 target). Little's law gives the chats in progress, rate
times W. `plan` divides them by what one GPU may carry of each kind, never more than HEADROOM of its limit, and takes the larger count.
`load_test` then sends random arrivals at that rate through that many GPUs and reports what a user sees.
Each request is one reply of one chat: its prompt is the whole conversation so far, read in full, and lesson 28's `serve` frees the chat's cache blocks at
its last word. The toy keeps no cache between the turns of a conversation.
The widget (../widget.html) repeats `x_at`, `speed_limit`, `stay`, `peak_rate`, `gpus` and `plan` in JavaScript; test_widget.py runs both and compares them.
Python 3 standard library only.
"""
from __future__ import annotations
import importlib.util
import math
import random
import statistics
import sys
from pathlib import Path
sys.dont_write_bytecode = True # loading the sibling lessons' files below must not leave a __pycache__ in their folders
LESSONS = Path(__file__).resolve().parents[2]
def _load(name: str, folder: str, filename: str):
"""A sibling lesson folder is not a package, so its file is loaded by path."""
spec = importlib.util.spec_from_file_location(name, LESSONS / folder / "code" / filename)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module # dataclasses looks the module up by name
spec.loader.exec_module(module)
return module
L26 = _load("gpu_split_lesson_26", "why-a-big-model-needs-several-gpus", "gpu_split.py")
L27 = _load("tail_sim_lesson_27", "why-average-speed-is-misleading", "tail_sim.py")
L28 = _load("settings_calc_lesson_28", "what-model-server-settings-change", "settings_calc.py")
Request = L28.Request
# --- the invented crowd and its targets -----------------------------------------------------------------------------------------------------
USERS = 1000
ACTIVE = 60 # percent of the users who send anything in the busiest hour
REPLIES = 120 # replies each of those users receives in that hour: one every 30 seconds
TARGET_MS = 50 # a user sees words arrive at most this far apart, for 99 replies in 100 (p99)
TARGETS = [30, 50, 75] # the targets the widget offers, in ms
SLIDERS = {"users": (100, 5000, 100), "active": (10, 100, 5), "replies": (10, 300, 10)} # (lowest, highest, step) of the widget's three sliders
WAIT_TARGET = 2.0 # and the first word comes within this many seconds, for 99 replies in 100 (p99)
HEADROOM = 0.8 # plan each GPU at no more than this share of what it can carry: 20% in hand for random bursts
SHARE = 90 # lesson 28's opening memory share, in percent
UNLIMITED = 10**9 # blocks: a pool that never runs out, so the speed table shows speed alone
LOAD_SECONDS = 600 # the load test lasts 10 minutes of the busiest hour, shrunk to keep the tests quick
# --- three kinds of chat. Each has the ranges prompts and replies are drawn from, on the log scale of lesson 28, and "each": the replies one active user
# receives in the busiest hour in the lesson's example. A long answer takes over a minute at the design point, so 120 an hour would be a user who asks again
# before the last answer ends; the answers get 30 ---------------------------------------------------------------------------------------------------------
MIXES = {
"usual": {"label": "Usual chats", "prompts": (300, 6000), "replies": (40, 400), "longest": 8000, "n": 600, "seed": 2026, "each": 120,
"caps": [1, 2, 4, 8, 12, 16, 24, 32, 48, 64, 96, 128, 192, 256]},
"documents": {"label": "Long documents, short answers", "prompts": (24000, 31000), "replies": (40, 400), "longest": 32000, "n": 200, "seed": 1, "each": 120,
"caps": [1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64]},
"answers": {"label": "Long answers", "prompts": (1000, 4000), "replies": (1500, 4000), "longest": 8000, "n": 500, "seed": 1, "each": 30,
"caps": [8, 16, 32, 64, 96, 128, 192, 256, 384]},
}
OPENING_MIX = "usual"
OPENING = {"users": USERS, "active": ACTIVE, "replies": REPLIES, "target": TARGET_MS, "mix": OPENING_MIX}
LOAD_SEED = 77 # gaps between arrivals
CHAT_SEED = 2029 # which chat each arrival is
def draw(seed: int, n: int, prompts: tuple, replies: tuple) -> dict:
"""Lesson 28's draw, with the ranges as arguments: with (300, 6000) and (40, 400) it gives lesson 28's 600 chats."""
rng = random.Random(seed)
out_p, out_r = [], []
for _ in range(n):
out_p.append(round(prompts[0] * (prompts[1] / prompts[0]) ** rng.random()))
out_r.append(round(replies[0] * (replies[1] / replies[0]) ** rng.random()))
return {"prompts": out_p, "replies": out_r}
def chats_of(mix: str) -> dict:
m = MIXES[mix]
return draw(m["seed"], m["n"], m["prompts"], m["replies"])
# --- step 1: demand -------------------------------------------------------------------------------------------------------------------------
def peak_rate(users: int, active: int, replies: int) -> float:
"""New replies a second in the busiest hour, taking the hour as steady."""
return users * active * replies / 360000
# --- step 2: what one GPU can carry ---------------------------------------------------------------------------------------------------------
def per_chat_gaps(chats: dict, run: dict) -> list[float]:
"""Seconds between words for each reply: the time from its first word to its last, over the words after the first."""
return [(done - first) / (reply - 1) for done, first, reply in zip(run["finished"], run["first_token"], chats["replies"])]
_TABLES: dict = {}
def step_table(mix: str) -> list[list[float]]:
"""One row per cap: [chats in progress on average, replies a second, tokens a second, p99 ms between words], from a backlog served with a pool that never
runs out. A row whose chats in progress did not rise above the last one's (the backlog was too short to fill the cap) is dropped."""
if mix not in _TABLES:
m, chats = MIXES[mix], chats_of(mix)
backlog = L28.backlog(chats)
rows = []
for cap in m["caps"]:
run = L28.serve(backlog, cap, UNLIMITED, m["longest"])
row = [run["running"], run["served"] / run["seconds"], run["tokens_per_second"], L27.percentile(per_chat_gaps(chats, run), 99) * 1000]
if not rows or row[0] > rows[-1][0] + 0.5:
rows.append(row)
_TABLES[mix] = rows
return _TABLES[mix]
def x_at(rows: list, chats: float) -> float:
"""Replies a second one GPU finishes with `chats` in progress. Below the first row the line runs from zero; past the last it stays flat."""
if chats <= rows[0][0]:
return rows[0][1] * chats / rows[0][0]
for a, b in zip(rows, rows[1:]):
if chats <= b[0]:
return a[1] + (b[1] - a[1]) * (chats - a[0]) / (b[0] - a[0])
return rows[-1][1]
def speed_limit(rows: list, target_ms: float) -> float:
"""The most chats in progress at which the p99 time between words is still at most `target_ms`, between two rows of the table."""
for a, b in zip(rows, rows[1:]):
if b[3] > target_ms:
return a[0] + (b[0] - a[0]) * (target_ms - a[3]) / (b[3] - a[3])
raise ValueError("the table never reaches the target")
def stay(rows: list, chats: float) -> float:
"""Seconds one chat holds its batch slot and its cache with `chats` in progress on a GPU: Little's law read off the table, chats divided by replies a second.
A chat is in a slot from the step it joins to its last word, so the wait for the first word after joining is in it."""
return chats / x_at(rows, chats)
def memory_limit(mix: str, share: int = SHARE) -> int:
"""Chats of the mix's average length (prompt plus reply) that lesson 28's cache pool holds."""
m, chats = MIXES[mix], chats_of(mix)
average = round(statistics.fmean(chats["prompts"]) + statistics.fmean(chats["replies"]))
return L28.room(share, m["longest"])["blocks"] // L28.blocks_for(average)
# --- step 3: how many GPUs ------------------------------------------------------------------------------------------------------------------
def gpus(chats: float, limit: float) -> int:
"""GPUs so that none carries more than HEADROOM of `limit` chats. The 1e-9 keeps a division that is whole in exact arithmetic from rounding up."""
return max(1, math.ceil(chats / (HEADROOM * limit) - 1e-9))
def plan(users: int, active: int, replies: int, mix: str, target_ms: float) -> dict:
"""The count for a crowd. `full` is the design point, the chats one GPU holds when it is as full as its tighter limit allows; `stay` is W there; `chats` is
rate times W, the chats in progress by Little's law. Each limit divides the same `chats`."""
rows, rate = step_table(mix), peak_rate(users, active, replies)
memory, speed = memory_limit(mix), speed_limit(rows, target_ms)
full = min(speed, memory)
seconds = stay(rows, full)
chats = rate * seconds
n_speed, n_memory = gpus(chats, speed), gpus(chats, memory)
return {"rate": rate, "memory": memory, "speed": speed, "binding": "speed" if speed < memory else "memory", "full": full, "stay": seconds, "chats": chats,
"n_speed": n_speed, "n_memory": n_memory, "n": max(n_speed, n_memory),
"without": max(1, math.ceil(chats / full - 1e-9))} # no headroom: every GPU full to its tighter limit
def commas(n: int) -> str:
return f"{n:,}"
def one_decimal(x: float) -> str:
return f"{math.floor(x * 10 + 0.5) / 10:.1f}"
def rate_text(rate: float) -> str:
"""Replies a second to one decimal, or to two below 1 so that a small crowd does not read 0.0."""
return one_decimal(rate) if rate >= 1 else f"{math.floor(rate * 100 + 0.5) / 100:.2f}"
def gpu_word(n: int) -> str:
return f"{n} GPU" if n == 1 else f"{commas(n)} GPUs"
def chats_words(chats: float) -> str:
return "less than 1 chat" if chats < 0.5 else f"about {commas(math.floor(chats + 0.5))} chats"
def describe(users: int, active: int, replies: int, mix: str, target_ms: float) -> dict:
"""Everything the widget shows for one setting of its controls: the two bars and four lines of words."""
p = plan(users, active, replies, mix, target_ms)
people = users * active // 100
m = MIXES[mix]
crowded = " At the plan's speed each user would need a new reply before the last one ends: lower the replies per user." if p["chats"] > people else ""
return {"bars": [p["n_memory"], p["n_speed"]],
"line": f"{commas(users)} users: {chats_words(p['chats'])} at once in the busiest hour. Speed needs {gpu_word(p['n_speed'])}, memory {gpu_word(p['n_memory'])}. "
f"Plan {gpu_word(p['n'] + 1)}, with a spare.",
"rate": f"The busiest hour brings {rate_text(p['rate'])} new replies a second from {commas(people)} active users.{crowded}",
"note": f"Prompts of {commas(m['prompts'][0])} to {commas(m['prompts'][1])} tokens, replies of {commas(m['replies'][0])} to {commas(m['replies'][1])}.",
"limits": f"One GPU holds {commas(p['memory'])} chats in memory, and keeps the p99 under {target_ms} ms up to {commas(math.floor(p['speed']))}: {p['binding']} sets the limit."}
# --- the load test --------------------------------------------------------------------------------------------------------------------------
def arrivals(rate: float, seconds: float, mix: str) -> list:
"""Random arrivals at `rate` a second for `seconds`: lesson 27's generator, the gaps from a fixed seed, and a fixed draw of which chat arrives."""
m, count = MIXES[mix], int(rate * seconds)
rng = random.Random(LOAD_SEED)
times = L27.intended_times([round(rng.expovariate(1.0), 4) for _ in range(count)], rate)
chats = draw(CHAT_SEED, count, m["prompts"], m["replies"])
return [Request(t, p, r) for t, p, r in zip(times, chats["prompts"], chats["replies"])]
def load_test(count: int, requests: list, mix: str, cap: int, share: int = SHARE) -> dict:
"""Spread the requests over `count` GPUs, request k to GPU k mod count, each a lesson-28 server with `cap` chats at most and its real pool.
Reports over all requests the p99 time between words (ms), the p99 wait before the first word (s), the mean time per reply, and the chats in progress on
average: the sum of every reply's time (arrival to last word) over the length of the run, which is Little's law written as a sum."""
m = MIXES[mix]
blocks = L28.room(share, m["longest"])["blocks"]
gaps, waits, spans = [], [], []
for g in range(count):
mine = requests[g::count]
run = L28.serve(mine, cap, blocks, m["longest"])
gaps += [(done - first) / (r.reply - 1) for done, first, r in zip(run["finished"], run["first_token"], mine)]
waits += [first - r.arrival for first, r in zip(run["first_token"], mine)]
spans += [(r.arrival, done) for r, done in zip(mine, run["finished"])]
length = max(done for _, done in spans)
times = [done - arrival for arrival, done in spans]
gap, wait = L27.percentile(gaps, 99) * 1000, L27.percentile(waits, 99)
return {"gpus": count, "gap": gap, "wait": wait, "passes": gap <= TARGET_MS and wait <= WAIT_TARGET, "mean_time": statistics.fmean(times), "rate": len(spans) / length,
"in_progress": sum(times) / length, "spans": spans}
_LOADS: dict = {}
def opening_load_test() -> dict:
"""The lesson's load test, run once per process: the opening crowd's random arrivals through the GPUs the arithmetic alone gives, the plan, and one more."""
if not _LOADS:
p = plan(USERS, ACTIVE, REPLIES, OPENING_MIX, TARGET_MS)
cap, requests = cap_for(OPENING_MIX), arrivals(p["rate"], LOAD_SECONDS, OPENING_MIX)
_LOADS.update({"plan": p, "cap": cap, "requests": requests, "busiest": busiest_minute(requests),
"runs": {count: load_test(count, requests, OPENING_MIX, cap) for count in (p["without"], p["n"], p["n"] + 1)}})
return _LOADS
def in_progress_curve(spans: list, seconds: int) -> list[int]:
"""Chats in progress at the start of each second, counted from the arrival and last-word times by sweeping through them in order."""
events = sorted([(a, 1) for a, _ in spans] + [(d, -1) for _, d in spans])
out, now, k = [], 0, 0
for t in range(seconds):
while k < len(events) and events[k][0] <= t:
now += events[k][1]
k += 1
out.append(now)
return out
def busiest_minute(requests: list) -> dict:
"""Arrivals in each whole minute: the busiest against the average."""
minutes = int(max(r.arrival for r in requests) // 60)
counts = [0] * minutes
for r in requests:
if r.arrival // 60 < minutes:
counts[int(r.arrival // 60)] += 1
return {"busiest": max(counts), "average": statistics.fmean(counts), "ratio": max(counts) / statistics.fmean(counts)}
def cap_for(mix: str, target_ms: float = TARGET_MS) -> int:
"""The cap on chats at once per GPU for a load test: the smaller limit, rounded down."""
return math.floor(min(speed_limit(step_table(mix), target_ms), memory_limit(mix)))
def gpus_to_hold(model: str, replicas: int) -> int:
"""Lesson 26: a model too big for one GPU takes `least_n` of them for every copy."""
return replicas * L26.least_n(model)Show code
"""Keep the tables, limits and settings inside ../widget.html equal to the ones in capacity_plan.py.
python3 embed_widget.py rewrite the data block in widget.html
python3 embed_widget.py --check exit 1 if widget.html differs (test_widget.py runs the same check)
"""
import json
import re
import sys
from pathlib import Path
import capacity_plan as cp
WIDGET = Path(__file__).resolve().parent.parent / "widget.html"
CONFIG_BLOCK = re.compile(r'(<script type="application/json" id="cp-config">)(.*?)(</script>)', re.S)
def config():
"""What the page needs to repeat the plan: for each chat mix its table of one GPU, its memory limit, its replies per user and its words, the targets the slider offers and the opening settings."""
return {
"headroom": cp.HEADROOM, "targets": cp.TARGETS, "opening": cp.OPENING, "mixOrder": list(cp.MIXES),
"mixes": {mix: {"label": m["label"], "prompts": list(m["prompts"]), "replies": list(m["replies"]), "each": m["each"], "memory": cp.memory_limit(mix),
"table": cp.step_table(mix)}
for mix, m in cp.MIXES.items()},
}
def embed(html):
"""html with the data block rewritten from capacity_plan.py."""
settings = json.dumps(config(), ensure_ascii=False, separators=(", ", ": "))
return CONFIG_BLOCK.sub(lambda m: m.group(1) + "\n" + settings + "\n" + m.group(3), html, count=1)
if __name__ == "__main__":
current = WIDGET.read_text(encoding="utf-8")
updated = embed(current)
if "--check" in sys.argv:
sys.exit(0 if updated == current else 1)
WIDGET.write_text(updated, encoding="utf-8")
print("widget.html data block now matches capacity_plan.py")Show code
"""Draw the four figures of lesson 29 from capacity_plan.py, so that no number in a figure is typed by hand.
python3 make_figs.py rewrite ../figs/*.svg
python3 make_figs.py --check exit 1 if a figure on disk differs (test_capacity_plan.py runs the same check)
All four figures are 600 units wide. Text is at least 22 units (20 for axis ticks) in ink or grey on white or a light fill; orange and green are used on marks
only. The two limits of a GPU differ by pattern and by label, not by colour alone.
"""
import math
import sys
from pathlib import Path
from xml.sax.saxutils import escape
import capacity_plan as cp
FIGS = Path(__file__).resolve().parent.parent / "figs"
INK, GREY, ORANGE, GREEN, FILL, PANEL = "#0a1626", "#44566b", "#b36200", "#00855F", "#d2dae4", "#ebf0f6"
DASH = "10 8"
def num(x: float) -> str:
return f"{x:.2f}".rstrip("0").rstrip(".")
# The SVG namespace is a name that every standalone .svg file must carry, not an address this program visits. It is written in two pieces so that a plain
# search of code/ for the start of a web address finds nothing (test_capacity_plan.py runs that search).
NAMESPACE = "ht" + "tp://www.w3.org/2000/svg"
def frame(width: int, height: int, title: str, desc: str, body: list[str]) -> str:
head = (f'<svg xmlns="{NAMESPACE}" viewBox="0 0 {width} {height}" width="{width}" height="{height}" role="img" '
f'font-family="system-ui, sans-serif" font-size="22" fill="{GREY}">')
return "\n".join([head, f" <title>{escape(title)}</title>", f" <desc>{escape(desc)}</desc>", ' <rect width="100%" height="100%" fill="#ffffff"/>',
*(" " + line for line in body), "</svg>", ""])
def text(x: float, y: float, content: str, anchor: str = "start", color: str = INK, bold: bool = False, size: int = 22) -> str:
extra = (f' text-anchor="{anchor}"' if anchor != "start" else "") + (' font-weight="bold"' if bold else "") + (f' font-size="{size}"' if size != 22 else "")
return f'<text x="{num(x)}" y="{num(y)}"{extra} fill="{color}">{escape(content)}</text>'
def seconds_text(x: float) -> str:
return f"{math.floor(x * 100 + 0.5) / 100:.2f}".rstrip("0").rstrip(".")
# --- figure 2: chats in progress through the load test ------------------------------------------------------------------------
def fig_in_progress() -> str:
load = cp.opening_load_test()
n = load["plan"]["without"] # the GPUs the arithmetic alone gives run about as full as the plan assumes
run = load["runs"][n]
curve = cp.in_progress_curve(run["spans"], cp.LOAD_SECONDS)
people = cp.USERS * cp.ACTIVE // 100
x0, x1, top, base, ymax = 80, 580, 76, 330, 1000
px = lambda t: x0 + t / cp.LOAD_SECONDS * (x1 - x0)
py = lambda chats: base - chats / ymax * (base - top)
body = [text(20, 40, "chats in progress", color=GREY)]
for tick in (0, 500, 1000):
body.append(f'<path d="M{x0} {num(py(tick))}H{x1}" stroke="{FILL}" stroke-width="2"/>')
body.append(text(x0 - 8, py(tick) + 7, cp.commas(tick), "end", GREY, size=20))
body.append(f'<path d="M{x0} {num(py(cp.USERS))}H{x1}" stroke="{ORANGE}" stroke-width="4" stroke-dasharray="{DASH}"/>')
body.append(text(x0 + 8, py(cp.USERS) - 10, f"{cp.commas(cp.USERS)} users", bold=True))
body.append(f'<path d="M{x0} {num(py(people))}H{x1}" stroke="{GREY}" stroke-width="3" stroke-dasharray="{DASH}"/>')
body.append(text(x0 + 8, py(people) - 10, f"{cp.commas(people)} active in the hour"))
points = " L".join(f"{num(px(t))} {num(py(curve[t]))}" for t in range(0, cp.LOAD_SECONDS, 3))
body.append(f'<path d="M{points}" fill="none" stroke="{INK}" stroke-width="3"/>')
body.append(text(x0 + 8, py(run["in_progress"]) - 28, f"about {round(run['in_progress'])} chats in progress", bold=True))
body.append(f'<path d="M{x0} {base}H{x1}" stroke="{GREY}" stroke-width="2"/>')
for minutes in (0, 5, 10):
body.append(text(px(minutes * 60), base + 32, str(minutes), "middle", GREY, size=20))
body.append(text((x0 + x1) / 2, base + 62, "minutes of the load test", "middle", GREY))
desc = (f"Chats in progress through {cp.LOAD_SECONDS // 60} minutes of the load test on {n} GPUs, from 0 to {cp.commas(ymax)}. Two dashed lines mark {cp.commas(cp.USERS)} users and "
f"{cp.commas(people)} users active in the hour. The solid line stays low, near {round(run['in_progress'])} chats in progress on average.")
return frame(600, 400, f"Chats in progress: about {round(run['in_progress'])}, for {cp.commas(cp.USERS)} users", desc, body)
# --- figure 1: the two limits of one GPU ------------------------------------------------------------------------------------
def fig_limits() -> str:
rows = cp.step_table("usual")
speed, memory = cp.speed_limit(rows, cp.TARGET_MS), cp.memory_limit("usual")
x0, x1, top, base, xmax, ymax = 84, 580, 70, 320, 250, 200
px = lambda chats: x0 + chats / xmax * (x1 - x0)
py = lambda ms: base - ms / ymax * (base - top)
body = [text(20, 40, "p99 time between words, ms", color=GREY)]
for tick in (0, 50, 100, 150, 200):
body.append(f'<path d="M{x0} {num(py(tick))}H{x1}" stroke="{FILL}" stroke-width="2"/>')
body.append(text(x0 - 8, py(tick) + 7, str(tick), "end", GREY, size=20))
points = [(0, 0)] + [(r[0], r[3]) for r in rows]
curve = "M" + " L".join(f"{num(px(c))} {num(py(ms))}" for c, ms in points)
body.append(f'<path d="{curve}" fill="none" stroke="{INK}" stroke-width="3"/>')
body.extend(f'<circle cx="{num(px(r[0]))}" cy="{num(py(r[3]))}" r="4" fill="{INK}"/>' for r in rows)
body.append(f'<path d="M{x0} {num(py(cp.TARGET_MS))}H{x1}" stroke="{ORANGE}" stroke-width="4" stroke-dasharray="{DASH}"/>')
body.append(text(px(60), py(cp.TARGET_MS) + 30, f"target {cp.TARGET_MS} ms", bold=True))
body.append(f'<path d="M{num(px(speed))} {base}V{top}" stroke="{GREEN}" stroke-width="4"/>')
body.append(f'<circle cx="{num(px(speed))}" cy="{num(py(cp.TARGET_MS))}" r="9" fill="{GREEN}" stroke="{INK}" stroke-width="2.5"/>')
body.append(text(px(speed) + 12, py(190), f"speed limit {math.floor(speed)} chats", bold=True))
body.append(f'<path d="M{num(px(memory))} {base}V{top}" stroke="{GREY}" stroke-width="4" stroke-dasharray="{DASH}"/>')
body.append(text(px(memory) - 12, py(160), f"memory holds {memory}", "end", bold=True))
body.append(f'<path d="M{x0} {base}H{x1}" stroke="{GREY}" stroke-width="2"/>')
for tick in range(0, xmax + 1, 50):
body.append(text(px(tick), base + 30, str(tick), "middle", GREY, size=20))
body.append(text((x0 + x1) / 2, base + 62, "chats in progress on one GPU", "middle", GREY))
desc = (f"The p99 time between words on one GPU against the chats in progress on it, for usual chats. It rises from {rows[0][3]:.0f} ms with {rows[0][0]:.0f} chat to "
f"{rows[-1][3]:.0f} ms with {rows[-1][0]:.0f}. A dashed orange line marks the {cp.TARGET_MS} ms target, which the curve crosses at {math.floor(speed)} chats: the speed limit, "
f"a solid green line. A dashed grey line at {memory} chats is the memory limit, where the cache pool is full.")
return frame(600, 400, f"One GPU: speed limit {math.floor(speed)} chats, memory limit {memory}", desc, body)
# --- figure 3: which limit comes first --------------------------------------------------------------------------------------
def fig_flip() -> str:
x0, longest = 112, 330
rows = []
for mix, m in cp.MIXES.items():
speed, memory = math.floor(cp.speed_limit(cp.step_table(mix), cp.TARGET_MS)), cp.memory_limit(mix)
rows.append((m["label"], memory, speed, "speed" if speed < memory else "memory"))
scale = longest / max(max(memory, speed) for _, memory, speed, _ in rows) # one scale for every bar: the longest limit of any kind of chat
body = ['<defs><pattern id="hatch" width="10" height="10" patternUnits="userSpaceOnUse" patternTransform="rotate(45)">'
f'<rect width="10" height="10" fill="#ffffff"/><path d="M0 0V10" stroke="{INK}" stroke-width="4"/></pattern></defs>']
y, facts = 30, []
for label, memory, speed, binds in rows:
body.append(text(20, y, label, bold=True))
for k, (name, value) in enumerate((("memory", memory), ("speed", speed))):
top = y + 14 + k * 46
width = max(value * scale, 3)
fill = PANEL if name == "memory" else "url(#hatch)"
border = ORANGE if name == binds else INK
body.append(text(20, top + 26, name, color=GREY))
body.append(f'<rect x="{x0}" y="{top}" width="{num(width)}" height="36" fill="{fill}" stroke="{border}" stroke-width="{4 if name == binds else 2}"/>')
body.append(text(x0 + width + 10, top + 26, f"{value}" + (" binds" if name == binds else ""), bold=name == binds))
facts.append(f"{label}: memory {memory}, speed {speed}, {binds} first")
y += 162
desc = ("Chats in progress one GPU can carry at a p99 of " + f"{cp.TARGET_MS} ms, by chat mix. Each mix has two bars, all on one scale: memory (plain) and speed (striped). The smaller one, "
"outlined in orange, binds. " + "; ".join(facts) + ".")
return frame(600, y - 24, "Which limit comes first depends on the chats", desc, body)
# --- figure 4: the load test ------------------------------------------------------------------------------------------------
def fig_load_test() -> str:
runs = cp.opening_load_test()["runs"]
x0, longest = 100, 300
panels = (("p99 time between words, ms", "gap", cp.TARGET_MS, 60, lambda v: f"{v:.1f} ms", f"target {cp.TARGET_MS} ms"),
("p99 wait before the first word, s", "wait", cp.WAIT_TARGET, 5, lambda v: f"{seconds_text(v)} s", f"target {seconds_text(cp.WAIT_TARGET)} s"))
body, y, facts = [], 30, []
for title, key, target, top_value, show, target_text in panels:
scale = longest / top_value
body.append(text(20, y, title, bold=True))
tx = x0 + target * scale
for k, (count, run) in enumerate(runs.items()):
top = y + 16 + k * 46
ok = run[key] <= target
body.append(text(20, top + 26, f"{count} GPUs", color=GREY))
body.append(f'<rect x="{x0}" y="{top}" width="{num(run[key] * scale)}" height="36" fill="{PANEL}" stroke="{GREEN if ok else ORANGE}" stroke-width="4"/>')
body.append(text(max(x0 + run[key] * scale, tx) + 10, top + 26, show(run[key]) + (", passes" if ok else ", misses"), bold=not ok))
body.append(f'<path d="M{num(tx)} {y + 8}V{y + 16 + 3 * 46 - 8}" stroke="{INK}" stroke-width="2" stroke-dasharray="6 5"/>')
body.append(text(tx, y + 16 + 3 * 46 + 18, target_text, "middle", GREY))
y += 16 + 3 * 46 + 52
for count, run in runs.items():
facts.append(f"{count} GPUs: {run['gap']:.1f} ms and {seconds_text(run['wait'])} s, {'passes' if run['passes'] else 'misses'}")
desc = (f"A {cp.LOAD_SECONDS // 60}-minute load test of {len(cp.opening_load_test()['requests']):,} random arrivals, {cp.peak_rate(cp.USERS, cp.ACTIVE, cp.REPLIES):.1f} a second, through "
f"{' and '.join(str(c) for c in runs)} GPUs. Two bar charts, with a dashed line at each target. " + "; ".join(facts) + ".")
return frame(600, y - 14, "The load test by number of GPUs", desc, body)
FIGURES = {
"chats-in-progress.svg": fig_in_progress,
"two-limits.svg": fig_limits,
"which-limit-binds.svg": fig_flip,
"load-test.svg": fig_load_test,
}
def main() -> int:
stale = []
for name, draw in FIGURES.items():
path, svg = FIGS / name, draw()
if "--check" in sys.argv:
if not path.exists() or path.read_text(encoding="utf-8") != svg:
stale.append(name)
else:
FIGS.mkdir(exist_ok=True)
path.write_text(svg, encoding="utf-8")
if "--check" in sys.argv:
print("figures differ from capacity_plan.py: " + ", ".join(stale) if stale else "figures match capacity_plan.py")
return 1 if stale else 0
print("wrote " + ", ".join(FIGURES))
return 0
if __name__ == "__main__":
sys.exit(main())Show code
"""Print every number the lesson quotes. Run: python3 show_numbers.py"""
import statistics
import capacity_plan as cp
def main() -> None:
rate = cp.peak_rate(cp.USERS, cp.ACTIVE, cp.REPLIES)
print(f"crowd: {cp.USERS:,} users, {cp.ACTIVE}% active in the busiest hour = {cp.USERS * cp.ACTIVE // 100}, {cp.REPLIES} replies each = "
f"{cp.USERS * cp.ACTIVE * cp.REPLIES // 100:,} replies an hour = {rate:.2f} a second")
print(f"targets: p99 time between words <= {cp.TARGET_MS} ms, p99 wait before the first word <= {cp.WAIT_TARGET} s; headroom {cp.HEADROOM:.0%}")
print()
print("the table for usual chats (server run with a pool that never runs out): chats in progress | replies/s | tokens/s | p99 ms | mean time per reply s")
rows = cp.step_table("usual")
for chats, x, tokens, p99 in rows:
print(f" {chats:6.1f} | {x:6.3f} | {tokens:6,.0f} | {p99:6.1f} | {chats / x:6.2f}")
print()
for mix, m in cp.MIXES.items():
chats = cp.chats_of(mix)
mean_p, mean_r = statistics.fmean(chats["prompts"]), statistics.fmean(chats["replies"])
print(f"{mix}: {m['n']} chats, prompts {m['prompts'][0]:,} to {m['prompts'][1]:,} (mean {mean_p:,.0f}), replies {m['replies'][0]:,} to {m['replies'][1]:,} (mean {mean_r:,.0f}), "
f"longest {m['longest']:,}; memory holds {cp.memory_limit(mix)} chats of {round(mean_p + mean_r):,} tokens")
print(f" the crowd: {m['each']} replies each")
for target in cp.TARGETS:
p = cp.plan(cp.USERS, cp.ACTIVE, m["each"], mix, target)
t = cp.step_table(mix)
print(f" p99 <= {target:2} ms: speed limit {p['speed']:6.1f} chats ({cp.x_at(t, p['speed']):5.2f} replies/s), memory limit {p['memory']:3} chats ({cp.x_at(t, p['memory']):5.2f} replies/s); "
f"binds: {p['binding']:6}; W {p['stay']:6.3f} s; chats in progress {p['chats']:7.1f}; GPUs: speed {p['n_speed']:3}, memory {p['n_memory']:3}, arithmetic only {p['without']:3}, "
f"with headroom {p['n']:3}")
print()
print("the opening line:", cp.describe(cp.USERS, cp.ACTIVE, cp.REPLIES, cp.OPENING_MIX, cp.TARGET_MS)["line"])
print()
for mix, m in cp.MIXES.items():
q = cp.plan(cp.USERS, cp.ACTIVE, m["each"], mix, cp.TARGET_MS)
seconds = round(q["stay"], 1) if q["stay"] < 10 else round(q["stay"]) # the page writes W to one decimal, whole above 10 s, and the speed limit to one
chats, speed = q["rate"] * seconds, round(q["speed"], 1)
print(f"the plain layer, {mix} ({m['each']} replies each), as the page rounds it: {q['rate']:.2f} replies a second x {seconds} s = {chats:.0f} chats in progress (unrounded {q['chats']:.1f}); "
f"speed {chats:.0f} / ({cp.HEADROOM} x {speed}) = {chats / (cp.HEADROOM * speed):.2f} -> {q['n_speed']}; "
f"memory {chats:.0f} / ({cp.HEADROOM} x {q['memory']}) = {chats / (cp.HEADROOM * q['memory']):.2f} -> {q['n_memory']}; "
f"no headroom {chats:.0f} / {min(speed, q['memory'])} = {chats / min(speed, q['memory']):.2f} -> {q['without']}")
print()
print("the answers mix at other crowds (replies each | active users mid-reply | replies/s | chats in progress | speed | memory | no headroom | plan):")
for each in (10, 20, 30, 40):
q = cp.plan(cp.USERS, cp.ACTIVE, each, "answers", cp.TARGET_MS)
print(f" {each:3} | {q['chats'] / (cp.USERS * cp.ACTIVE // 100):4.0%} | {q['rate']:5.2f} | {q['chats']:7.1f} | {q['n_speed']:3} | {q['n_memory']:3} | {q['without']:3} | {q['n']:3}")
print()
load = cp.opening_load_test()
p, cap, requests, busy = load["plan"], load["cap"], load["requests"], load["busiest"]
print(f"load test: {len(requests):,} random arrivals at {rate:.1f} a second over {cp.LOAD_SECONDS // 60} minutes, cap {cap} chats a GPU; busiest minute {busy['busiest']:,} against "
f"an average minute of {busy['average']:,.0f}: {busy['ratio'] - 1:+.1%}")
for count, r in load["runs"].items():
print(f" {cp.gpu_word(count)}: p99 between words {r['gap']:5.1f} ms, p99 wait {r['wait']:5.2f} s -> {'passes' if r['passes'] else 'MISSES'}; chats in progress {r['in_progress']:5.1f} "
f"= {r['rate']:.2f} a second x {r['mean_time']:.2f} s")
print(f" the plan said {p['chats']:.1f} chats in progress ({p['rate']:.2f} a second x {p['stay']:.2f} s, each GPU as full as the speed limit {p['speed']:.1f} allows)")
print(f" planned {p['n']} (no spare): one down leaves {p['n'] - 1}; planned {p['n'] + 1} (with a spare): one down leaves {p['n']}")
print()
for mix in ("documents", "answers"):
m = cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES[mix]["each"], mix, cp.TARGET_MS)
print(f"{mix}: {m['n']} GPUs ({m['binding']} binds), {m['chats']:,.0f} chats in progress against {cp.USERS * cp.ACTIVE // 100} active users")
a = cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES["answers"]["each"], "answers", cp.TARGET_MS)
t = cp.load_test(a["n"], cp.arrivals(a["rate"], cp.LOAD_SECONDS, "answers"), "answers", cp.cap_for("answers"))
print(f"answers load test on {cp.gpu_word(a['n'])}: p99 between words {t['gap']:.1f} ms, p99 wait {t['wait']:.2f} s -> {'passes' if t['passes'] else 'MISSES'}; "
f"chats in progress {t['in_progress']:.1f} = {t['rate']:.2f} a second x {t['mean_time']:.1f} s, against the plan's {a['chats']:.1f} and W {a['stay']:.1f} s")
print()
for model in ("8b", "70b", "405b"):
print(f"model {model}: {cp.gpu_word(cp.L26.least_n(model))} for one copy; {p['n']} copies = {cp.gpu_word(cp.gpus_to_hold(model, p['n']))}")
if __name__ == "__main__":
main()Show code
"""Run: PYTHONDONTWRITEBYTECODE=1 python3 -m pytest -q -p no:cacheprovider code/ -- each test guards a number the lesson, a figure or the widget shows a reader."""
import functools
import math
import re
import statistics
from pathlib import Path
import capacity_plan as cp
import make_figs
LESSON = Path(__file__).resolve().parent.parent
PEAK = (cp.USERS, cp.ACTIVE, cp.REPLIES)
WEB_ADDRESS = re.compile("ht" + r"tps?://") # written in two pieces so that this file holds none either
@functools.cache
def two_gpus():
"""The one load test the lesson quotes that is not in `opening_load_test`: 2 GPUs, where the gap passes and the wait fails."""
load = cp.opening_load_test()
return cp.load_test(2, load["requests"], "usual", load["cap"])
@functools.cache
def loose_cap():
"""3 GPUs with the cap raised from 38 to 64: chats stop queueing, so the wait passes, and the batches grow past the speed target, so the gap fails."""
load = cp.opening_load_test()
return cp.load_test(3, load["requests"], "usual", 64)
@functools.cache
def answers_load():
"""The load test for the answers mix on the GPUs the plan gives it, which the lesson quotes beside the plan's chats in progress."""
p = cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES["answers"]["each"], "answers", cp.TARGET_MS)
return cp.load_test(p["n"], cp.arrivals(p["rate"], cp.LOAD_SECONDS, "answers"), "answers", cp.cap_for("answers"))
def test_the_usual_chats_are_lesson_28s_chats():
"""The memory limit of 200 and the speed table carry lesson 28's numbers on only if the 600 chats are the same ones."""
assert cp.chats_of("usual") == cp.L28.CHATS
def test_every_speed_table_rises_and_reaches_every_target():
"""A table that fell, or never reached a target, would make the speed limit jump or fail for one setting of the widget."""
for mix in cp.MIXES:
rows = cp.step_table(mix)
for a, b in zip(rows, rows[1:]):
assert b[0] > a[0] and b[1] > a[1] and b[3] >= a[3], mix
for target in cp.TARGETS:
assert rows[0][3] < target < rows[-1][3] and rows[0][0] < cp.speed_limit(rows, target) < rows[-1][0], (mix, target)
stays = [cp.stay(rows, row[0]) for row in rows]
assert stays == sorted(stays), mix # a fuller GPU keeps each chat longer: the page's W grows with the load
def test_chats_in_progress_is_the_arrival_rate_times_the_mean_time():
"""Little's law as the lesson states it, one area counted twice. If the two sides came apart, the 109 of figure 2 would not be the 109.1 of the test."""
load = cp.opening_load_test()
runs, plan = load["runs"], load["plan"]
for run in runs.values():
assert math.isclose(run["in_progress"], run["rate"] * run["mean_time"], rel_tol=1e-9)
spans = runs[plan["without"]]["spans"] # figure 2 draws the run whose GPUs are as full as the plan assumes
curve = cp.in_progress_curve(spans, 600)
for t in (0, 1, 60, 299, 599): # the sweep through the sorted events counts like a plain count
assert curve[t] == sum(a <= t < d for a, d in spans)
assert math.isclose(statistics.fmean(curve), runs[3]["in_progress"], rel_tol=0.03) # and its average is the 109.1, give or take the ramp at both ends
assert math.isclose(runs[3]["in_progress"], plan["chats"], rel_tol=0.03) # and the plan's 110.5 is that same count read off the table
def test_limits_gpu_counts_and_the_limit_that_binds_for_three_kinds_of_chat():
"""The lesson's table: 1,000 users need 4, 55 or 6 GPUs, and which limit binds changes with the chats. `without` is the count with no headroom, which the load test refutes."""
want = {"usual": (200, 38, "speed", 1, 4, 4, 3), "documents": (14, 2, "speed", 9, 55, 55, 44), "answers": (85, 140, "memory", 6, 4, 6, 5)}
assert [m["each"] for m in cp.MIXES.values()] == [120, 120, 30]
for mix, (memory, speed, binding, n_memory, n_speed, n, without) in want.items():
p = cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES[mix]["each"], mix, cp.TARGET_MS)
assert (p["memory"], math.floor(p["speed"]), p["binding"], p["n_memory"], p["n_speed"], p["n"], p["without"]) == (memory, speed, binding, n_memory, n_speed, n, without), mix
assert p["without"] < p["n"], mix # headroom costs a GPU in every kind of chat
assert p["chats"] < cp.USERS * cp.ACTIVE // 100 and cp.MIXES[mix]["each"] * p["stay"] <= 3600, mix # a crowd that can exist: fewer chats than active users, each user mid-reply no more than all hour
usual, ans = cp.plan(*PEAK, "usual", cp.TARGET_MS), cp.plan(*PEAK, "answers", cp.TARGET_MS)
assert (usual["without"], usual["n"]) == (3, 4) and cp.peak_rate(*PEAK) == 20.0
assert ans["chats"] > cp.USERS * cp.ACTIVE // 100 and cp.REPLIES * ans["stay"] > 3600 # 120 replies each of 81 s would be impossible: the page uses 30
own = cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES["answers"]["each"], "answers", cp.TARGET_MS)
test = answers_load()
assert test["passes"] and test["in_progress"] < 0.6 * own["chats"] and test["mean_time"] < 0.6 * own["stay"] # the plan's 405 chats at 81 s are design-point figures: the test on 6 GPUs sees about 231 at 47.5 s
assert [cp.plan(*PEAK, "usual", t)["n"] for t in (30, 75)] == [5, 4] and [math.floor(cp.plan(*PEAK, "usual", t)["speed"]) for t in (30, 75)] == [15, 72]
assert cp.gpus(0.56, 0.7) == 1 and cp.gpus(0.0, 5.0) == 1 and cp.gpus(110, 38) == 4 # 0.56 / (0.8 * 0.7) is 1.0000000000000002 in floating point and must not give 2
def test_dividing_the_rounded_numbers_the_page_prints_gives_the_pages_gpu_counts():
"""A reader divides the numbers on the page: W to one decimal (whole above 10 s), the speed limit to one decimal, the memory limit as is. They must get the page's counts for every kind of chat."""
for mix, m in cp.MIXES.items():
p = cp.plan(cp.USERS, cp.ACTIVE, m["each"], mix, cp.TARGET_MS)
seconds = round(p["stay"], 1) if p["stay"] < 10 else round(p["stay"])
chats, speed = p["rate"] * seconds, round(p["speed"], 1)
assert (cp.gpus(chats, speed), cp.gpus(chats, p["memory"]), math.ceil(chats / min(speed, p["memory"]) - 1e-9)) == (p["n_speed"], p["n_memory"], p["without"]), mix
def test_the_load_test_misses_at_3_gpus_and_passes_at_4_and_5():
"""Three GPUs pass the arithmetic and miss under random arrivals. If the toy stopped missing, 'test before you sign' would have no example."""
load = cp.opening_load_test()
runs, plan = load["runs"], load["plan"]
assert (plan["without"], plan["n"], load["cap"], len(load["requests"])) == (3, 4, 38, 12000)
assert [runs[n]["passes"] for n in (3, 4, 5)] == [False, True, True]
assert runs[3]["gap"] > cp.TARGET_MS and runs[3]["wait"] > cp.WAIT_TARGET and runs[5]["gap"] < runs[4]["gap"] < cp.TARGET_MS
assert not runs[plan["n"] - 1]["passes"] and runs[plan["n"]]["passes"] # one GPU down: 4 planned leaves 3 and misses, 5 planned leaves 4 and passes
assert 3 * cp.x_at(cp.step_table("usual"), plan["speed"]) > plan["rate"] > 0.9 * 3 * cp.x_at(cp.step_table("usual"), plan["speed"]) # 3 GPUs are about 95% busy
assert runs[3]["gap"] < 1.1 * cp.TARGET_MS and runs[3]["wait"] > 2 * cp.WAIT_TARGET # the lesson calls the gap a narrow miss and the wait a clear one
two = two_gpus() # 2 GPUs: the gap passes and the wait fails, so only both conditions together reject it
assert two["gap"] <= cp.TARGET_MS and two["wait"] > cp.WAIT_TARGET and not two["passes"]
loose = loose_cap() # the other way round: the gap fails and the wait passes, so neither condition alone is the rule
assert loose["gap"] > cp.TARGET_MS and loose["wait"] <= cp.WAIT_TARGET and not loose["passes"]
assert (load["busiest"]["busiest"], round(load["busiest"]["average"])) == (1283, 1213)
def test_a_model_too_big_for_one_gpu_takes_several_for_each_copy():
assert [cp.gpus_to_hold(m, 4) for m in ("8b", "70b", "405b")] == [4, 8, 44]
def test_every_number_in_the_lesson_and_figures_comes_from_the_code():
"""The prose and the four figures quote numbers by hand; a change to the constants or the server must show up here, not in a reader's check."""
prose = (LESSON / "lesson.md").read_text(encoding="utf-8")
figures = {path.name: path.read_text(encoding="utf-8") for path in sorted((LESSON / "figs").glob("*.svg"))}
assert set(figures) == set(re.findall(r"\]\(figs/([\w-]+\.svg)\)", prose)) == set(make_figs.FIGURES) and len(figures) == 4
for name, draw in make_figs.FIGURES.items(): # the files on disk are what make_figs.py draws now
assert figures[name] == draw(), name
rows, p = cp.step_table("usual"), cp.plan(*PEAK, "usual", cp.TARGET_MS)
docs, ans, p30, p75 = (cp.plan(cp.USERS, cp.ACTIVE, cp.MIXES[mix]["each"], mix, t) for mix, t in (("documents", 50), ("answers", 50), ("usual", 30), ("usual", 75)))
load = cp.opening_load_test()
runs, busiest = load["runs"], load["busiest"]
chats = cp.chats_of("usual")
mean_prompt, mean_reply = statistics.fmean(chats["prompts"]), statistics.fmean(chats["replies"])
room = cp.L28.room(cp.SHARE, 8000)
sweep = {row["cap"]: row for row in cp.L28.sweep(cp.SHARE, 8000)}
speed, x_speed = math.floor(p["speed"]), cp.x_at(rows, p["speed"])
rate, floor, people = p["rate"], math.floor, cp.USERS * cp.ACTIVE // 100
s1, w1 = round(p["speed"], 1), round(p["stay"], 1) # the page writes the speed limit and W to one decimal
page_chats = round(rate * w1) # 20 x 5.5 = 110, the number every division below starts from
ans_w, ans_chats = round(ans["stay"]), round(ans["rate"] * round(ans["stay"])) # 81 s, and 5 x 81 = 405
documents = cp.MIXES["documents"]
mean_document = statistics.fmean(cp.chats_of("documents")["prompts"])
answer_replies = cp.MIXES["answers"]["replies"]
row = lambda r: f"| {r[0]:.1f} | {r[1]:.3f} | {r[3]:.1f} ms |"
run_row = lambda n: f"| {n} | {runs[n]['gap']:.1f} ms | {runs[n]['wait']:.2f} s |"
busy = rate / (3 * x_speed) * 100
q3 = 20 * 5 / (0.8 * 40)
wanted = {"lesson.md": [
f"{cp.USERS:,} users, {people} of them active", f"each receiving {cp.REPLIES} replies", f"{round(rate * 3600):,} replies, or {rate:.0f} new replies a second",
f"about {speed} chats at once", f"{cp.TARGET_MS} ms", f"{p['n']} GPUs plus a spare", "A100 chip with 80 GB",
f"speed needs {p['n_speed']} and memory {p['n_memory']}", f"the replies each drop to {cp.MIXES['answers']['each']}, and memory, now needing {ans['n_memory']} GPUs to speed's {ans['n_speed']}, sets the limit",
f"each receives one reply every 30 seconds: {rate:.0f} new replies a second", f"about {page_chats} chats, not {people} or {cp.USERS:,}",
"reads the whole conversation again ([lesson 8]",
f"holds {p['memory']} chats", f"at {p['speed']:.1f} chats ({speed} in the figure), the speed limit, long before memory runs out at {p['memory']}",
f"At its speed limit, {p['speed']:.1f} chats, one GPU finishes {x_speed:.1f} replies a second, so each chat stays {p['speed']:.1f} / {x_speed:.1f} = {w1} s",
f"about {rate:.0f} x {w1} = {page_chats} chats are in progress", f"on 3 GPUs, each near its limit, counts {round(runs[3]['in_progress'])} (Figure 2)",
f"The solid line stays near {round(runs[3]['in_progress'])}.", f"the plan said {page_chats}.",
f"Speed alone gives {page_chats} / {s1} = {page_chats / s1:.2f}, so {p['without']} GPUs", f"Speed then needs {page_chats} / (0.8 x {s1}) = {page_chats / (0.8 * s1):.2f}, so {p['n_speed']} GPUs",
f"Memory needs {page_chats} / (0.8 x {p['memory']}) = {page_chats / (0.8 * p['memory']):.2f}, so {p['n_memory']}", f"One spare makes {p['n'] + 1}",
f"{docs['memory']} chats but meets the target with only {floor(docs['speed'])}", f"the crowd needs {docs['n']} GPUs", f"{round(mean_document, -3):,.0f} tokens",
f"replies of {answer_replies[0]:,} to {answer_replies[1]:,} tokens memory holds {ans['memory']} chats and speed {floor(ans['speed'])}",
f"A chat on a GPU at its limit then stays {ans_w} s, so there a user gets about {floor(3600 / ans['stay'])} replies an hour, not {cp.REPLIES}",
f"At {cp.MIXES['answers']['each']} each, {ans_chats} chats are in progress and need {ans['n']} GPUs", "so memory fills first at this target: with",
f"With 3 GPUs the p99 time between words is {runs[3]['gap']:.1f} ms, a narrow miss of 50 ms, and the p99 wait for the first word",
f"is {runs[3]['wait']:.2f} s, a clear miss of 2 s", f"With 4 it is {runs[4]['gap']:.1f} ms and {runs[4]['wait']:.2f} s", f"With 5, {runs[5]['gap']:.1f} ms and {runs[5]['wait']:.2f} s",
f"Here {cp.USERS:,} users became {rate:.0f} replies", f"The same {cp.USERS:,} users need {p['n']} GPUs for usual chats and {docs['n']} for long documents; long answers, at {cp.MIXES['answers']['each']} replies each, need {ans['n']}",
f"On 3 GPUs it gives {runs[3]['rate']:.2f} a second times {runs[3]['mean_time']:.2f} s, {runs[3]['in_progress']:.1f} chats, against the plan's {p['chats']:.1f}",
f"The 3 GPUs hold {runs[3]['in_progress'] / 3:.0f} chats each, fewer than {p['speed']:.1f}", f"it gives {runs[4]['rate']:.2f} times {runs[4]['mean_time']:.2f} s, {runs[4]['in_progress']:.1f} chats",
*(row(rows[i]) for i in (0, 3, 5, 7, 8, 9, 11)),
f"at {rows[7][0]:.1f} chats, {rows[7][0] / rows[7][1]:.2f} s", f"`stay` reads the table at {p['speed']:.1f} chats: {p['speed']:.1f} / {x_speed:.2f} = {p['stay']:.2f} s",
f"between {rows[7][0]:.1f} and {rows[8][0]:.1f}, at {p['speed']:.1f} chats",
f"a cap of 64, with {sweep[64]['seconds_per_word'] * 1000:.1f} ms", f"at {rows[9][0]:.1f} chats the p99 is {rows[9][3]:.1f} ms",
f"{cp.TARGETS[0]} ms gives {floor(p30['speed'])} chats and {p30['n']} GPUs, {cp.TARGETS[2]} ms gives {floor(p75['speed'])} and the same {p75['n']}",
f"average {mean_prompt:,.0f} tokens of prompt and {mean_reply:.0f} of reply, {round(mean_prompt + mean_reply):,} in all, which is {cp.L28.blocks_for(round(mean_prompt + mean_reply))} blocks",
f"{room['blocks']:,} blocks, and {room['blocks']:,} // {cp.L28.blocks_for(round(mean_prompt + mean_reply))} is {p['memory']}", f"`HEADROOM` is {cp.HEADROOM}",
"def gpus(chats, limit):", "math.ceil(chats / (HEADROOM * limit) - 1e-9)", f"Without headroom the arithmetic gives {p['without']}",
"| Chats | Limits | Replies each | GPUs, spare not counted |",
f"| Usual | {p['memory']} and {speed} | {cp.MIXES['usual']['each']} | {p['n']} |", f"| Documents | {docs['memory']} and {floor(docs['speed'])} | {cp.MIXES['documents']['each']} | {docs['n']} |",
f"| Answers | {ans['memory']} and {floor(ans['speed'])} | {cp.MIXES['answers']['each']} | {ans['n']} |", "Speed binds for the usual chats and the documents, memory for the answers.",
f"For the answers W is {ans_w} s, so {ans['rate']:.0f} new replies a second keep {ans_chats} chats in progress: {ans_chats} / (0.8 x {ans['memory']}) = {ans_chats / (0.8 * ans['memory']):.2f}, so {ans['n_memory']} GPUs, "
f"against {ans_chats} / (0.8 x {ans['speed']:.1f}) = {ans_chats / (0.8 * ans['speed']):.2f}, so {ans['n_speed']} for speed",
f"For the documents W is {round(docs['stay'], 1)} s, so {round(docs['rate'] * round(docs['stay'], 1))} chats, and the speed limit is {round(docs['speed'], 1)}: "
f"{round(docs['rate'] * round(docs['stay'], 1))} / (0.8 x {round(docs['speed'], 1)}) = {round(docs['rate'] * round(docs['stay'], 1)) / (0.8 * round(docs['speed'], 1)):.1f}, so {docs['n']} GPUs",
f"on {ans['n']} GPUs the test averages {answers_load()['in_progress']:.0f} chats and {answers_load()['mean_time']:.1f} s a reply, against the plan's {ans_chats} and {ans_w} s",
f"came to {runs[3]['in_progress']:.1f} on 3 GPUs, {runs[4]['in_progress']:.1f} on 4 and {runs[5]['in_progress']:.1f} on 5",
f"{documents['prompts'][0]:,} to {documents['prompts'][1]:,} tokens, {mean_document:,.0f} on average", f"(seed {cp.LOAD_SEED})", f"seed {cp.CHAT_SEED}", f"a cap of {load['cap']} chats",
*(run_row(n) for n in (3, 4, 5)), f"at most about {3 * x_speed:.0f} replies a second against {rate:.0f} arriving, so each is busy {busy:.0f}% of the time",
f"with 2 GPUs the cap keeps the gap at {two_gpus()['gap']:.1f} ms, yet the wait is {two_gpus()['wait']:.0f} s",
f"The busiest minute is {(busiest['ratio'] - 1) * 100:.1f}% above the average", f"the test passes at {runs[4]['gap']:.1f} ms", f"5 GPUs give {runs[5]['gap']:.1f} ms where 4 give {runs[4]['gap']:.1f} ms",
f"{cp.L26.least_n('8b')} for the 8B model, {cp.L26.least_n('70b')} for the 70B and {cp.L26.least_n('405b')} for the 405B", f"4 copies take {cp.gpus_to_hold('8b', 4)}, {cp.gpus_to_hold('70b', 4)} and {cp.gpus_to_hold('405b', 4)} GPUs",
f"3 GPUs run at {rate / (3 * x_speed) * 100:.0f}%, 4 at {rate / (4 * x_speed) * 100:.0f}% and 5 at {rate / (5 * x_speed) * 100:.0f}%", f"from {runs[4]['wait']:.2f} s to {runs[5]['wait']:.2f} s",
f"Four GPUs at 80% finish about {4 * cp.HEADROOM * x_speed:.0f} replies a second against the {rate:.0f} needed", f"about {round(4 * cp.HEADROOM * x_speed / rate * cp.USERS, -2):,.0f} users",
f"3 GPUs finish about {3 * cp.HEADROOM * x_speed:.0f} replies a second: about {round(3 * cp.HEADROOM * x_speed / rate * cp.USERS, -1):,.0f} users, short of {cp.USERS:,}, the same reason that test missed",
f"{room['pool_tokens']:,} / 8,000 = {room['pool_tokens'] // 8000}", f"{people} x {cp.REPLIES} = {round(rate * 3600):,} replies in 3,600 seconds, which is {rate:.0f} a second",
f"20 x 5 = 100 chats in progress; 100 / (0.8 x 40) = {q3:.1f}, which rounds up to {math.ceil(q3)}. With a spare, {math.ceil(q3) + 1}.",
f"Memory, at the {cp.TARGET_MS} ms target. Replies of {answer_replies[0]:,} to {answer_replies[1]:,} tokens keep each chat open about {ans_w} s, so the pool fills at {ans['memory']} chats, before the speed limit of {floor(ans['speed'])}",
f"the p99 wait is {runs[3]['wait']:.2f} s against the 2 s target, a clear miss, and the p99 gap {runs[3]['gap']:.1f} ms against 50 ms, a narrow one",
"the \"stable\" version", "edited by Betsy Beyer, Google",
],
"chats-in-progress.svg": [f"about {round(runs[3]['in_progress'])}", f"{cp.USERS:,} users", f"{people} active", "on 3 GPUs"],
"two-limits.svg": [f"speed limit {speed} chats", f"memory holds {p['memory']}", f"target {cp.TARGET_MS} ms", f"{rows[0][3]:.0f} ms with 1 chat to {rows[-1][3]:.0f} ms with {rows[-1][0]:.0f}"],
"which-limit-binds.svg": [f"{label}: memory {q['memory']}, speed {floor(q['speed'])}, {q['binding']} first" for label, q in
(("Usual chats", p), ("Long documents, short answers", docs), ("Long answers", ans))] + [f"{q['memory'] if q['binding'] == 'memory' else floor(q['speed'])} binds" for q in (p, docs, ans)]
+ ["all on one scale"],
"load-test.svg": [f"{runs[n]['gap']:.1f} ms" for n in (3, 4, 5)] + [f"{runs[n]['wait']:.2f} s" for n in (3, 4, 5)] + [f"12,000 random arrivals, {rate:.1f} a second", "target 50 ms", "target 2 s"]}
widths = [float(w) for w in re.findall(r'<rect x="112" y="[\d.]+" width="([\d.]+)"', figures["which-limit-binds.svg"])] # the six bars: memory and speed for each kind of chat
values = [q[k] if k == "memory" else floor(q[k]) for q in (p, docs, ans) for k in ("memory", "speed")]
assert len(widths) == 6 and all(math.isclose(w / v, widths[0] / values[0], rel_tol=0.02) for w, v in zip(widths, values)), "figure 3 bars are not on one scale"
missing = [f"{name}: {fragment!r}" for name, fragments in wanted.items() for fragment in fragments if fragment not in (prose if name == "lesson.md" else figures[name])]
assert not missing, "not found: " + "; ".join(missing)
for name, text in figures.items(): # one palette, no blue or vermillion, a white page, a title and a description, no address
assert not re.search(r"0072B2|56B4E9|D55E00", text, re.I) and "<title>" in text and "<desc>" in text and 'fill="#ffffff"' in text, name
assert "<script" not in text and not WEB_ADDRESS.search(text.replace("ht" + "tp://www.w3.org/2000/svg", "")), name
for path in (LESSON / "code").glob("*.py"):
assert not WEB_ADDRESS.search(path.read_text(encoding="utf-8")), path.nameShow code
"""Guards for ../widget.html: its data block, that its plan gives the same numbers and words as capacity_plan.py, and what a reader sees and can press."""
import json
import re
import subprocess
import capacity_plan as cp
from embed_widget import CONFIG_BLOCK, WIDGET, config, embed
HTML = WIDGET.read_text(encoding="utf-8")
GRID = [(u, a, r, mix, t) for u in (100, 1000, 2500, 5000) for a in (10, 60, 100) for r in (10, 120, 300) for mix in cp.MIXES for t in cp.TARGETS]
# Reads the plan out of the widget between its marker comments, runs every setting of the grid with Node and compares the two bars and the four lines of words with
# the ones Python wrote into the fixture, character for character.
PARITY = r"""
import fs from 'node:fs';
const [fixture, widget] = process.argv.slice(2);
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const html = fs.readFileSync(widget, 'utf8');
const cfg = JSON.parse(html.match(/<script type="application\/json" id="cp-config">([\s\S]*?)<\/script>/)[1]);
const js = html.slice(html.indexOf('/* ---------- the model ('), html.indexOf('/* ---------- display'));
const api = new Function('cfg', "'use strict';" + js + '; return {describe};')(cfg);
const bad = [];
fx.cases.forEach((c) => {
const got = api.describe(...c.args);
if (JSON.stringify(got) !== JSON.stringify(c.want)) bad.push([c.args, JSON.stringify(got).slice(0, 300), JSON.stringify(c.want).slice(0, 300)]);
});
console.log(JSON.stringify({ checks: fx.cases.length, bad: bad.slice(0, 3) }));
process.exit(bad.length ? 1 : 0);
"""
# Runs the page script against a small stand-in for the DOM, moves the sliders, picks another chat mix and reads back what a reader sees.
PAGE = r"""
import fs from 'node:fs';
const [widget, fixture] = process.argv.slice(2);
const html = fs.readFileSync(widget, 'utf8');
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const config = html.match(/<script type="application\/json" id="cp-config">([\s\S]*?)<\/script>/)[1];
const code = html.match(/<script>([\s\S]*?)<\/script>/)[1];
function el(tag) {
const e = { tag, children: [], listeners: {}, value: '', checked: false, style: {}, attrs: {}, textContent: '' };
e.setAttribute = (k, v) => { e.attrs[k] = v; };
e.getAttribute = (k) => e.attrs[k];
e.appendChild = (c) => { e.children.push(c); };
e.addEventListener = (type, f) => (e.listeners[type] = e.listeners[type] || []).push(f);
return e;
}
const page = { root: el('html'), messages: [], parent: { other: true } };
[...html.matchAll(/id="(cp-[^"]+)"/g)].forEach((m) => { page[m[1]] = el('x'); });
page['cp-config'].textContent = config;
const doc = { getElementById: (id) => page[id], documentElement: page.root, createElement: el };
const win = { addEventListener: (type, f) => page.messages.push(f) };
new Function('document', 'window', 'parent', code)(doc, win, page.parent);
const fire = (e, type) => (e.listeners[type] || []).forEach((f) => f({ target: e }));
const bad = [];
const want = (what, got, expected) => { if (JSON.stringify(got) !== JSON.stringify(expected)) bad.push([what, JSON.stringify(got).slice(0, 300), JSON.stringify(expected).slice(0, 300)]); };
const shown = () => ({ names: [0, 1].map((k) => page['cp-name-' + k].textContent), widths: [0, 1].map((k) => parseFloat(page['cp-bar-' + k].style.width)),
line: page['cp-line'].textContent, rate: page['cp-rate'].textContent, limits: page['cp-limits'].textContent, note: page['cp-note'].textContent });
const cfg = JSON.parse(config);
want('opening controls', [page['cp-users'].value, page['cp-active'].value, page['cp-replies'].value, page['cp-target'].value, page['cp-target'].max, page['cp-mix'].value],
[String(cfg.opening.users), String(cfg.opening.active), String(cfg.opening.replies), String(cfg.targets.indexOf(cfg.opening.target)), String(cfg.targets.length - 1), cfg.opening.mix]);
want('the chat mixes offered', page['cp-mix'].children.map((o) => [o.value, o.textContent]), fx.options);
want('opening state', shown(), fx.opening);
want('nothing spoken yet', page['cp-live'].textContent, '');
page['cp-target'].value = '0';
fire(page['cp-target'], 'input');
want('30 ms, dragging', shown(), fx.thirty);
want('dragging stays silent', page['cp-live'].textContent, '');
fire(page['cp-target'], 'change');
want('release speaks', page['cp-live'].textContent, fx.thirty.line + ' ' + fx.thirty.limits);
want('words on the controls', [page['cp-users-out'].textContent, page['cp-active-out'].textContent, String(page['cp-replies-out'].textContent), page['cp-target-out'].textContent,
page['cp-target'].attrs['aria-valuetext'], page['cp-users'].attrs['aria-valuetext']], ['1,000', '60%', '120', '30 ms', '30 milliseconds', '1,000 users']);
fx.others.forEach((c) => {
page['cp-mix'].value = c.args[3];
fire(page['cp-mix'], 'change');
want('picking ' + c.args[3] + ' sets its replies per user', page['cp-replies'].value, String(cfg.mixes[c.args[3]].each));
page['cp-users'].value = String(c.args[0]); page['cp-active'].value = String(c.args[1]); page['cp-replies'].value = String(c.args[2]);
page['cp-target'].value = String(cfg.targets.indexOf(c.args[4]));
fire(page['cp-replies'], 'input');
want('state ' + c.args.join('/'), shown(), c.want);
});
const theme = (source, name) => page.messages.forEach((f) => f({ data: { type: 'spicyshark:theme', theme: name }, source }));
theme({ other: 'frame' }, 'dark');
want('theme from another sender is ignored', page.root.attrs['data-theme'], undefined);
theme(page.parent, 'dark');
want('theme from the host page', page.root.attrs['data-theme'], 'dark');
console.log(JSON.stringify({ bad: bad.slice(0, 3) }));
process.exit(bad.length ? 1 : 0);
"""
def run_node(tmp_path, script, *args):
path = tmp_path / "check.mjs"
path.write_text(script, encoding="utf-8")
done = subprocess.run(["node", str(path), *args], capture_output=True, text=True, timeout=120)
assert done.returncode == 0, done.stdout + done.stderr
return json.loads(done.stdout)
def seen(users, active, replies, mix, target):
"""What the page shows for one setting, in the form the PAGE script reads back."""
d = cp.describe(users, active, replies, mix, target)
widest = max(d["bars"])
return {"names": [f"Memory needs {cp.gpu_word(d['bars'][0])}", f"Speed needs {cp.gpu_word(d['bars'][1])}"], "widths": [max(2, n / widest * 100) for n in d["bars"]],
"line": d["line"], "rate": d["rate"], "limits": d["limits"], "note": d["note"]}
def test_widget_data_block_matches_the_python_module():
assert embed(HTML) == HTML
block = json.loads(CONFIG_BLOCK.search(HTML).group(2))
assert block == json.loads(json.dumps(config()))
assert block["opening"] == {"users": 1000, "active": 60, "replies": 120, "target": 50, "mix": "usual"}
for name, (low, high, step) in cp.SLIDERS.items(): # each slider covers the settings the grid below covers
assert re.search(r'id="cp-%s" min="%d" max="%d" step="%d"' % (name, low, high, step), HTML), name
def test_widget_gives_the_same_numbers_and_words_as_python(tmp_path):
cases = [{"args": list(g), "want": cp.describe(*g)} for g in GRID]
lines = " ".join(case["want"]["line"] + case["want"]["rate"] for case in cases)
assert "less than 1 chat" in lines and "lower the replies per user" in lines and "memory sets the limit" in " ".join(c["want"]["limits"] for c in cases) # the grid reaches the odd cases
(tmp_path / "fx.json").write_text(json.dumps({"cases": cases}), encoding="utf-8")
assert run_node(tmp_path, PARITY, str(tmp_path / "fx.json"), str(WIDGET))["checks"] == len(GRID)
def test_widget_page_shows_the_opening_state_and_follows_the_controls(tmp_path):
opening = seen(1000, 60, 120, "usual", 50)
assert opening["line"] == "1,000 users: about 110 chats at once in the busiest hour. Speed needs 4 GPUs, memory 1 GPU. Plan 5 GPUs, with a spare."
others = [(5000, 100, 300, "answers", 75), (100, 10, 10, "usual", 50), (1000, 60, 120, "documents", 30), (1000, 60, 30, "answers", 50)]
fixture = {"options": [[mix, m["label"]] for mix, m in cp.MIXES.items()], "opening": opening, "thirty": seen(1000, 60, 120, "usual", 30),
"others": [{"args": list(o), "want": seen(*o)} for o in others]}
(tmp_path / "fx.json").write_text(json.dumps(fixture), encoding="utf-8")
run_node(tmp_path, PAGE, str(WIDGET), str(tmp_path / "fx.json"))
def test_widget_is_small_themed_and_uses_only_the_site_palette():
body = HTML[HTML.index('<div class="cp-root">'):HTML.index("<script")]
visible = re.sub(r"<details.*?</details>", "", body, flags=re.S)
assert len(re.findall(r"<input|<select|<button", visible)) <= 5 and "More settings" in body and "A paper plan" in visible # at most five controls on show, and the caveat is always on show
assert len(HTML.splitlines()) < 300 and "light-dark(#ffffff, #0a0f17)" in HTML and "spicyshark:theme" in HTML and "prefers-reduced-motion" in HTML
assert not re.search(r"0072B2|56B4E9|D55E00|ht" r"tps?://|<img|@import", HTML, re.I)