On this page
A model server has settings, and three of them decide how many chats it holds at once. Take an AI model with 8 billion numbers inside, called weights, on one A100 chip with 80 GB (gigabytes, billions of bytes) of memory. A token is a small piece of text, often a word. Set to use 90% of the memory and accept chats of up to 8,000 tokens, the server has room for 205 chats of 2,000 tokens at once. A server that set aside the full 8,000 for each chat would hold 51. This lesson shows what each setting moves, and what it takes from something else.
In one minute
A server’s settings do not add memory. They move a fixed amount of it between the model’s weights, a working reserve, the notes for every chat in progress and an unused part. Picture a restaurant with a fixed floor: the kitchen takes one part, a working corner stays clear for the cooks, a strip is left for the neighbours and tables fill the rest. Using more of the floor adds tables but shrinks the strip. Seating more parties at once keeps the cooks busier but makes each party wait longer for its next dish. Allowing long dinners costs nothing until someone stays long, unless a table is held for the longest dinner from the start. Measure your own crowd to pick the numbers.
Explore the numbers
Open the widget. It runs a toy server on a chip like lesson 24’s, with 600 invented chats, and draws the memory as a bar. Set the longest chat to 128,000 tokens and lower the memory share: below 44% the server will not start. Then set the share back to 90%, raise the most chats at once from 64 and watch the total stop rising as the wait, in ms (thousandths of a second), grows. It shows a model at the chip’s peak speed, not a measurement.
How it works
One floor, four parts
The chip has 80 GB of memory. The weights take 16.06 GB, whatever the settings. The server also keeps a working reserve for the calculations inside each step; the toy sets 2 GB, an invented figure, as a real server measures its own. The memory share is the part of the chip the server may use. At 90% that is 72 GB, which leaves 53.9 GB after the weights and the reserve. That is the cache pool, the room for the notes about every earlier token in every chat (lesson 22). The other 8.0 GB stays unused, for other programs and for safety. Each token takes 131,072 bytes of cache, so the pool holds 411,520 tokens.
Figure 1. The chip’s memory at a 90% memory share. A higher share moves the line right and turns unused room into pool.
The longest chat is a ceiling, not a slice
The longest chat is the most tokens one chat may reach, prompt and reply together. A good server hands out cache in small blocks of 16 tokens, only as a chat grows (lesson 22), so an 8,000-token ceiling takes no room until a chat gets there. The pool holds 205 chats of 2,000 tokens. A server that set aside the full 8,000 for every chat on arrival would hold 51, and at 32,000 tokens only 12.
The ceiling still matters twice. The toy turns away a chat that would pass it. And the server will not start if the pool cannot hold one chat of that length: for 128,000 tokens the toy needs a memory share of at least 44%.
Figure 2. Chats of 2,000 tokens that fit in the 53.9 GB pool. Green hands out blocks as chats grow; orange holds the full ceiling for every chat.
More chats at once helps, then stops
The most chats at once caps how many chats the server works on in one step. Each added chat shares the same trip to read the model, so the total rises. The server also reads every chat’s cache, so each wait between words grows. The toy serves 600 chats at each cap. With one chat at a time it writes 117 tokens a second and a user waits 8.0 ms between words. At 64 it writes 1,202 and the wait is 47.3 ms. Doubling to 128 adds only 8% to the total and makes each wait 1.7 times as long, 79.4 ms. The total flattens because every chat’s prompt must still be read once, and that work does not shrink.
At 256 the chats in progress need more cache than the pool holds. The server pauses the newest chat, frees its room and rebuilds its notes later (vLLM calls this preemption): 11 pauses, 21,333 tokens of cache rebuilt. A cap of 512 gives the same result, because now the pool, not the cap, decides how many run.
Figure 3. The toy server at a 90% share and an 8,000-token ceiling. The grey line marks a cap of 64, the last before a doubling adds under 10%.
The share has a price too
A higher memory share gives the pool more room: 21.9 GB at 50% and 53.9 GB at 90%, or 167,376 and 411,520 tokens. That room comes out of the unused part, which other programs need too, and two servers on one chip must split the share. A small pool can also cancel a high cap: at 50% the total is 1,202 tokens a second at a cap of 64 but 1,168 at 128, because chats pause.
What this means when you build
- Know which part each setting moves. The memory share moves room between the pool and the unused part. The longest chat is a ceiling on one chat. The most chats at once caps how many share the pool. None adds memory.
- Set the longest chat to what users need. In the toy, a ceiling of 8,000 tokens starts at a 24% share, 32,000 at 28% and 128,000 at 44%.
- Raise the most chats at once until the pool is full or the wait passes your target, then stop. Here 64 to 128 added 8% and made each wait 1.7 times as long.
- Leave room if anything else runs on the chip. The shares of two servers must add up to less than the whole.
- Watch the server’s pause messages, and measure at your load. A paused chat shows up as a delay, not an error. Measure the slowest users, not the average (lesson 27 says why), with chats like your users send.
Optionalabout 15 min
Go deeper
Optional. This part is for readers who write Python. The code is in the code/ folder next to this lesson. From inside that folder, python3 show_numbers.py prints every number below, and python3 -B -m pytest -q -p no:cacheprovider runs the tests.
vLLM, an open-source model server, calls the three settings gpu_memory_utilization (the memory share), max_model_len (the longest chat) and max_num_seqs (the most chats at once), and calls a chat a request or a sequence.
The pool sum
Let G be the card’s memory, u the memory share (gpu_memory_utilization), W the weights, R the reserve and c the bytes of cache for one token. A block holds 16 tokens, the default of the vLLM paper:
allowed = u G
pool = allowed - W - R
blocks = pool // (16 c)
tokens = 16 blocksG is 80e9 bytes, the A100 data sheet’s “80GB” read as in lesson 24. W is 2 bytes for each of 8,030,261,248 weights, 16,060,522,496 bytes, and c is 131,072, the figures of lessons 22 and 24 (lessons 18 and 26 read the card as 80 GiB, lesson 24 and this lesson as 80 billion bytes; lesson 22 counts cache sizes in GiB). A block is 2,097,152 bytes. R is 2 GB and invented. memory_parts does this sum and rounds the pool down to whole blocks; each row of the table is one call. Under lesson 18’s reading, 80 GiB, memory_parts(90, GPU_BYTES_GIB) gives a pool of 59.2 GB, or 28,252 blocks, with room for 226 chats of 2,000 tokens, against 53.9 GB, 25,720 and 205. The card is 7.4% larger and the pool 9.8%, because the weights and the reserve do not change.
| Share | Pool | Chats of 2,000 |
|---|---|---|
| 30% | 5.9 GB | 22 |
| 50% | 21.9 GB | 83 |
| 90% | 53.9 GB | 205 |
| 98% | 60.3 GB | 230 |
At 90% the pool is 25,720 blocks, or 411,520 tokens, and a chat of 2,000 tokens takes 125 blocks, so 205 fit. Below a 23% share the weights and the reserve leave no block at all. The vLLM engine-arguments page (the “latest” version, read 2026-10-19) describes gpu_memory_utilization as “the fraction of GPU memory to be used for the model executor, which can range from 0 to 1” and gives “0.92” as the default when it is unspecified (the stable docs give the same 0.92). It is a per-instance limit: for two servers on one GPU the page suggests 0.5 each. In the source (main branch, read the same day), request_memory in vllm/v1/worker/utils.py multiplies the total memory by the share, and determine_available_memory in vllm/v1/worker/gpu_worker.py profiles a test pass and subtracts what it used from the request.
The server that will not start
One chat of the longest length must fit. The toy starts only if its blocks reach the blocks of that length, and a 128,000-token ceiling needs 8,000. At a 40% share the pool is 13.9 GB, or 6,646 blocks, so the server will not start. At 44% it has 8,172 blocks and starts. The lowest working share is 24% for ceilings of 4,000 and 8,000 tokens, 26% for 16,000, 28% for 32,000, 34% for 64,000 and 44% for 128,000. The max_model_len entry says “Model context length (prompt and output).” When the pool is too small, the source raises an error whose text includes: “To serve at least one request with the model’s max seq len”, the cache needed, the cache available, “Try increasing gpu_memory_utilization … or decreasing max_model_len”.
Blocks against reserving the full length
This is room, comments removed:
def room(share: int, longest: int, reserved: bool = False) -> dict:
parts = memory_parts(share)
per_longest = blocks_for(longest)
per_typical = blocks_for(min(TYPICAL_CHAT, longest))
at_longest = parts["blocks"] // per_longest
return {**parts, "starts": at_longest >= 1, "at_longest": at_longest, "typical": at_longest if reserved else parts["blocks"] // per_typical,
"per_typical": per_typical, "per_longest": per_longest}With blocks, chats of 2,000 tokens take per_typical blocks each and the ceiling does not change the count. With reserving, every chat takes per_longest. At a 90% share, chats at once by longest chat in tokens, reserved against blocks:
| Longest | Reserved | Blocks |
|---|---|---|
| 4,000 | 102 | 205 |
| 8,000 | 51 | 205 |
| 32,000 | 12 | 205 |
| 128,000 | 3 | 205 |
The speed follows. With the cap out of the way (512), the block server writes 1,284 tokens a second at either ceiling, with 194.9 chats running on average. The reserving server writes 1,161 at 8,000 with 49.7 running, and 732 at 32,000 with 11.9 running. Its users wait less per word, 39.9 ms and 15.6 ms against 106.6 ms, but only because fewer run at once.
The scheduler
serve is lesson 24’s continuous batching plus a pool of blocks. This is its memory part, comments removed:
paused_now = False
k = 0
while k < len(running):
i = running[k]
if not reserved and (requests[i].prompt + made[i]) % BLOCK_TOKENS == 0:
if free == 0:
j = running.pop()
free += held[j]
held[j] = 0
waiting.appendleft(j)
paused += 1
paused_now = True
continue
free -= 1
held[i] += 1
k += 1
joined = []
while not paused_now and waiting and len(running) < cap:
i = waiting[0]
need = blocks_for(longest) if reserved else blocks_for(requests[i].prompt + made[i] + 1)
if need > free:
break
waiting.popleft()
free -= need
held[i] = need
running.append(i)
joined.append(i)A running chat whose next token starts a new block takes one. If none is free, the newest running chat gives up all its blocks and goes to the front of the queue, keeping the tokens it wrote. Nobody joins in that step. A waiting chat joins, oldest first, only while fewer than cap run and the pool has the blocks for its next token; the first that does not fit holds up the rest. A chat that joins has its prompt and the tokens it wrote before read in that step, which lesson 24’s step time already counts. With the pool set to a billion blocks, serve gives lesson 24’s simulate the same times.
The cap sweep
The 600 chats are one seeded draw (seed 2026), all waiting at time zero, so the server never runs short of work: prompts of 305 to 5,965 tokens, mean 1,882, and replies of 40 to 400, mean 157. The pool is the 90% share with an 8,000-token ceiling. Tokens/s is tokens a second for everyone together; the wait is between words for one user.
| Cap | Tokens/s | Wait |
|---|---|---|
| 1 | 117 | 8.0 ms |
| 8 | 590 | 12.8 ms |
| 32 | 1,049 | 28.5 ms |
| 64 | 1,202 | 47.3 ms |
| 128 | 1,300 | 79.4 ms |
| 256 | 1,284 | 106.6 ms |
A cap of 512 equals 256. Every doubling up to 64 adds at least 14.6% to the total, and 64 to 128 adds 8.2% while the wait grows 1.68 times. The code defines “stops paying” as the first cap whose doubling adds under 10% (stops_paying), and it returns 64. Ten percent is a choice; a target for the wait picks a cap too, 32 for 30 ms and 64 for 50 ms.
The flattening has a floor under it. Reading the 1,129,059 prompt tokens takes at least 58.1 s of arithmetic at 51.5 microseconds a token, so no cap can pass 94,117 reply tokens over 58.1 s, 1,619 tokens a second. The wait per word also grows because a step that reads a new prompt is long: a prompt of the mean 1,882 tokens takes 96.9 ms of arithmetic.
A smaller pool moves the knee. At a 50% share, with 167,376 tokens, a cap of 64 writes 1,202 tokens a second with no pauses and a cap of 128 writes 1,168 with 26. At 35% (75,824 tokens) a cap of 32 writes 1,038 with 4 pauses, and caps of 64 and up write 1,018 with 39.
Figure 4. The best cap depends on the pool.
Preemption
When blocks run out the toy pauses the newest running chat. The vLLM paper (Kwon et al., SOSP 2023) says that vLLM serves requests first come, first served and that “the latest requests are preempted first”, and that it evicts all of a sequence’s blocks or none. It names two ways back, swapping the blocks to CPU memory and recomputing them; the toy recomputes. The optimization page (read 2026-10-19) says: “Preempted requests are recomputed when sufficient KV cache space becomes available again.” and “In vLLM V1, the default preemption mode is RECOMPUTE rather than SWAP”. It warns that preemption “can adversely affect end-to-end latency”. For frequent preemptions it advises more gpu_memory_utilization; a lower max_num_seqs or max_num_batched_tokens; or more GPUs (lesson 26). The count shows in Prometheus metrics, or in the log with disable_log_stats=False.
A hand-worked case is pinned by the test. Two chats have 30 tokens and will write 10, in a pool of 4 blocks, 2 each. After the second step each is at 32 tokens and needs a third block. None is free, so the newer chat is paused and its blocks are freed. The older one finishes. The newer one rejoins and reads its 32 tokens in one step. The run takes 18 steps, against 10 with room for both.
The second server’s names
Text Generation Inference (TGI), Hugging Face’s server, documents its launcher arguments on one page (the “main” docs, read 2026-10-19). The names differ, and not one to one.
- Memory share.
--cuda-memory-fraction: “Limit the CUDA available memory. The allowed value equals the total visible memory multiplied by cuda-memory-fraction”. Default 1.0, against vLLM’s 0.92. - Longest chat.
--max-total-tokens: “the most important value to set as it defines the “memory budget” of running clients requests”, with a default of “min(max_position_embeddings, 4096)”. The page says that the larger the value, “the less effective batching can be”. It does not say whether the server holds that many tokens for each chat, so Figure 2 is a claim about the toy, not about TGI. - Most chats at once. No twin.
--max-batch-total-tokenscounts tokens in a batch, “the largest possible amount that fits the remaining memory (after the model is loaded)”, and the page says TGI “cannot infer this number automatically”.--max-concurrent-requests(default 128) refuses new clients when full instead of making them wait.
What the toy leaves out
The reserve is fixed at 2 GB. A real server measures it at start-up: in the source, the start-up message splits the used memory into “consumed memory (weights + non-torch)”, “peak activation” and “CUDAGraph memory”. The max-num-active-seqs entry says “max_num_seqs sizes the model runner (per-request buffers and CUDA graph capture)”, so in a real server the cap can change the reserve and so the pool. The chats are invented and all waiting at the start, with the same mean length; real traffic arrives over time and changes. The toy turns away a whole chat whose prompt and reply together pass the longest chat. vLLM differs: _validate_prompt_len in vllm/v1/engine/input_processor.py rejects only a prompt that leaves no room for a reply token, and check_stop in vllm/v1/core/sched/utils.py ends a reply that reaches max_model_len as length-capped. A server that stored each cached number in fewer bits would have a smaller c: at 8 bits it would be 65,536 and the pool would hold twice the tokens. Other settings matter. max_num_batched_tokens is the “maximum number of tokens that can be processed in a single iteration”. The optimization page says smaller values such as 2048 give better inter-token latency and larger ones a better time to first token. The toy has no chunked prefill (lesson 24) and no second GPU.
Proof it works
test_settings_calc.py pins the numbers above: the bar adds up to the card at every share; the chats that fit and when a server will not start; reserving against blocks; lesson 24’s simulate matched on an unlimited pool; the pause case above, replayed step by step, and a chat of exactly the longest length, which is still served; the sweep’s gains, the stop at 64 and the smaller pool. A last test finds every number this lesson quotes in the code’s output and in the figures. test_widget.py runs the widget’s JavaScript in Node against the Python on a grid of settings, and drives the page, pools that fill included, with a stand-in for the browser.
Where this is used in practice
The vLLM paper (Kwon et al., SOSP 2023; read in full on 2026-10-19) states the problem as this lesson does. Its Figure 1 shows a 13B model on a 40 GB A100 with about 65% of memory for the weights and close to 30% for the requests’ states. It says earlier systems “statically allocate a chunk of memory for a request based on the request’s maximum possible sequence length”, and its profiling found only “20.4% - 38.2%” of the cache memory holding actual token states in them. It reports 2 to 4 times the throughput of those systems at the same latency; that is their setup, not this toy. Documentation changes between versions, so check the one you run.
Skills this lesson builds
- Split a GPU’s memory into weights, reserve, cache pool and unused part, and work out the pool from a memory share.
- Say how many chats of a given length fit, and whether a server with a given longest chat will start.
- Explain why a longest-chat ceiling costs little with blocks and a lot when the full length is reserved for each chat.
- Read a sweep of the most chats at once, find where it stops paying, and say what a full pool does.
- Match one server’s memory settings to another’s from their documentation.
Portfolio project: write a calculator that takes a model’s config file and a GPU’s data sheet, then reports the weights, the cache for one token and the pool at three memory shares, with the chats of 1,000, 4,000 and 16,000 tokens that fit. Run lesson 24’s simulator with a pool and a cap on your own invented chats. Publish the code, one chart of the total and the wait per word against the cap, and a note on where your knee differs from this lesson’s.
Check yourself
-
A server has 80 GB of memory, weights of 16.06 GB, a 2 GB reserve and a memory share of 90%. How big is the cache pool, and how much is left unused?
Answer
The server may use 0.9 x 80 = 72 GB. After the weights and the reserve, 72 - 16.06 - 2 = 53.94 GB is the pool. The other 80 - 72 = 8 GB is left unused.
-
Why does raising the longest chat from 8,000 to 32,000 tokens leave the 205 chats of 2,000 tokens unchanged, and what would cut the number?
Answer
With blocks, a chat takes room only as it grows, so the ceiling takes none until a chat reaches it. A server that held the full length for every chat would fit 51 chats at 8,000 and 12 at 32,000. A ceiling can also stop the server from starting if the pool cannot hold one chat of that length.
-
At a 40% share, can the toy start with a 128,000-token ceiling? Show the sum.
Answer
The server may use 32 GB. After 16.06 GB of weights and 2 GB of reserve, 13.94 GB is left, which is 6,646 blocks of 2,097,152 bytes, or 106,336 tokens. One chat of 128,000 tokens needs 8,000 blocks, so the server will not start. At 44% the pool is 8,172 blocks and it starts.
-
Why does a cap of 512 give exactly the result of a cap of 256?
Answer
At 256 the chats in progress already need more cache than the pool holds, so the server pauses chats (11 times) and about 195 chats run on average. The pool, not the cap, limits how many run, and a higher cap changes nothing.
-
What does the toy do when the pool is full, and how does the vLLM documentation describe the same event?
Answer
The toy pauses the newest running chat, frees its blocks, keeps the tokens it wrote and reads them again when the chat rejoins. The documentation says vLLM can preempt requests, that preempted requests are recomputed when space returns, and that V1 defaults to recompute. It advises more
gpu_memory_utilizationor a lowermax_num_seqsormax_num_batched_tokens.
Further reading
Gentle
- Why AI servers waste memory, and how paging fixes it (lesson 22, the cache in blocks)
- How does an AI server answer many people at once? (lesson 24, the server this lesson gives a pool)
- Optimization and Tuning (vLLM documentation, the “latest” version; preemption section; read 2026-10-19)
- Engine Arguments (vLLM documentation, the “latest” version; the stable docs give the same 0.92; read 2026-10-19)
Deep
- Efficient memory management for large language model serving with PagedAttention (Kwon, Li, Zhuang and others, SOSP 2023; read in full 2026-10-19)
- Text-generation-launcher arguments (Hugging Face, Text Generation Inference documentation, “main”; read 2026-10-19)
- vllm/v1/core/kv_cache_planning.py, vllm/v1/worker/gpu_worker.py and vllm/v1/worker/utils.py (vLLM source, main branch; the start-up checks, the memory profile and the memory request; read 2026-10-19)
- vllm/v1/engine/input_processor.py and vllm/v1/core/sched/utils.py (vLLM source, main branch; the length checks on a prompt and a reply; read 2026-10-19)
- Llama 3.1 8B configuration (a third-party copy of the model’s config file; read 2026-10-19)
- NVIDIA A100 Tensor Core GPU data sheet (peak speed, bandwidth and memory; the 80 GB SXM column of the “Jun21” revision)
Optional
Code
Each file runs on its own. Download it, or read it here.
Show code
"""Keep the copy of the model, the card and the chats inside ../widget.html equal to the ones in settings_calc.py.
python3 embed_widget.py rewrite the data block in widget.html
python3 embed_widget.py --check exit 1 if widget.html differs (test_widget.py runs the same check)
"""
import json
import re
import sys
from pathlib import Path
import settings_calc as sc
WIDGET = Path(__file__).resolve().parent.parent / "widget.html"
CONFIG_BLOCK = re.compile(r'(<script type="application/json" id="sw-config">)(.*?)(</script>)', re.S)
def config():
"""What the page needs to run the same server: the model, the card, the drawn chats and the settings it offers."""
return {
"params": sc.L24.PARAMS, "bytesPerWeight": sc.L24.FP16_BYTES, "kvBytes": sc.KV_BYTES, "peak": sc.L24.A100_PEAK_FLOPS, "bandwidth": sc.L24.A100_BANDWIDTH,
"gpuBytes": sc.GPU_BYTES, "weightBytes": sc.WEIGHT_BYTES, "reserveBytes": sc.RESERVE_BYTES, "blockTokens": sc.BLOCK_TOKENS, "blockBytes": sc.BLOCK_BYTES,
"typical": sc.TYPICAL_CHAT, "longest": sc.LONGEST, "caps": sc.CAPS, "opening": sc.OPENING, "prompts": sc.CHATS["prompts"], "replies": sc.CHATS["replies"],
}
def embed(html):
"""html with the data block rewritten from settings_calc.py."""
settings = json.dumps(config(), ensure_ascii=False, separators=(", ", ": "))
return CONFIG_BLOCK.sub(lambda m: m.group(1) + "\n" + settings + "\n" + m.group(3), html, count=1)
if __name__ == "__main__":
current = WIDGET.read_text(encoding="utf-8")
updated = embed(current)
if "--check" in sys.argv:
sys.exit(0 if updated == current else 1)
WIDGET.write_text(updated, encoding="utf-8")
print("widget.html data block now matches settings_calc.py")Show code
"""What a model server's three main settings change, on the GPU, model and step-time model of lesson 24 (how-an-ai-server-answers-many-people-at-once).
One A100 80GB holds three things: the model's weights, a working reserve, and the cache of every chat in progress. The server's settings do not make
memory; they decide how the same bytes are shared:
memory share how much of the card the server may use at all (vLLM: gpu_memory_utilization)
longest chat the most tokens one chat may reach, prompt and reply together (vLLM: max_model_len)
most chats the most chats the server runs in one step (vLLM: max_num_seqs)
cache pool = memory share * card - weights - reserve, cut into blocks of 16 tokens (lesson 22).
`serve` runs lesson 24's continuous-batching server with that pool: a waiting chat joins only if the pool has room for it, and when the running chats
need a block that is not there, the newest chat is paused, its cache dropped, and it is rebuilt later (vLLM V1's default, "recompute"). The chip
and the model are lesson 24's, so the times are best cases, not measurements. The reserve and the chats are invented.
Python 3 standard library only. The widget (../widget.html) repeats `serve` in JavaScript; test_widget.py runs both and compares them.
"""
from __future__ import annotations
import importlib.util
import math
import random
import sys
from collections import deque
from pathlib import Path
sys.dont_write_bytecode = True # loading lesson 24's file below must not leave a __pycache__ in its folder
LESSON_24 = Path(__file__).resolve().parents[2] / "how-an-ai-server-answers-many-people-at-once" / "code" / "batching_sim.py"
def _load_lesson_24():
spec = importlib.util.spec_from_file_location("batching_sim_lesson_24", LESSON_24)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module # dataclasses looks the module up by name
spec.loader.exec_module(module)
return module
L24 = _load_lesson_24()
Request = L24.Request
# --- the card and the model: lesson 24's ----------------------------------------------------------------------------------------------------
GPU_BYTES = int(L24.GPU_MEMORY_BYTES) # the data sheet says "80GB"; read there, and here, as 80 billion bytes
GPU_BYTES_GIB = 80 * 2**30 # the other reading, as lessons 18 and 26 take it: 80 GiB. Only `memory_parts(share, GPU_BYTES_GIB)` uses it
WEIGHT_BYTES = L24.FP16_BYTES * L24.PARAMS # 16-bit weights
KV_BYTES = L24.KV_BYTES_PER_TOKEN # 131,072 bytes of cache for each token of a chat
BLOCK_TOKENS = 16 # lesson 22 / the vLLM paper's default block size
BLOCK_BYTES = BLOCK_TOKENS * KV_BYTES # 2 MiB
RESERVE_BYTES = 2_000_000_000 # invented: working memory a pass needs plus the server's own overhead. A real server measures it at start-up.
TYPICAL_CHAT = 2000 # tokens in a typical chat (the invented chats below average about this)
# --- the settings the widget offers ---------------------------------------------------------------------------------------------------------
SHARES = list(range(30, 99)) # memory share, whole percent (vLLM's documented default is 92)
LONGEST = [4000, 8000, 16000, 32000, 64000, 128000] # longest chat, tokens
CAPS = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] # most chats at once
OPENING = {"share": 90, "longest": 8000, "cap": 64, "reserved": False}
KNEE_GAIN = 1.10 # raising the cap "stops paying" when doubling it brings less than 10% more tokens a second
MAX_STEPS = 1_000_000 # a guard, not a limit: the longest honest run (cap 1) takes about 94,000 steps; a bug that stops chats finishing must fail, not hang
def blocks_for(tokens: int) -> int:
return -(-tokens // BLOCK_TOKENS)
# --- the memory budget ----------------------------------------------------------------------------------------------------------------------
def memory_parts(share: int, card: int = GPU_BYTES) -> dict:
"""Bytes of the card in each part for a memory share given in whole percent. The pool is whole blocks; what is left over counts as unused."""
allowed = card * share // 100
blocks = max(0, allowed - WEIGHT_BYTES - RESERVE_BYTES) // BLOCK_BYTES
pool = blocks * BLOCK_BYTES
return {"weights": WEIGHT_BYTES, "reserve": RESERVE_BYTES, "pool": pool, "unused": card - WEIGHT_BYTES - RESERVE_BYTES - pool,
"allowed": allowed, "blocks": blocks, "pool_tokens": blocks * BLOCK_TOKENS}
def room(share: int, longest: int, reserved: bool = False) -> dict:
"""How many chats the pool holds. `starts` is False when it cannot hold even one chat of the longest length (vLLM then refuses to start).
`typical` chats of TYPICAL_CHAT tokens fit when blocks are handed out as a chat grows; a server that reserves the full length for each chat fits `at_longest`."""
parts = memory_parts(share)
per_longest = blocks_for(longest)
per_typical = blocks_for(min(TYPICAL_CHAT, longest))
at_longest = parts["blocks"] // per_longest
return {**parts, "starts": at_longest >= 1, "at_longest": at_longest, "typical": at_longest if reserved else parts["blocks"] // per_typical,
"per_typical": per_typical, "per_longest": per_longest}
# --- the invented chats ---------------------------------------------------------------------------------------------------------------------
SEED = 2026
N_CHATS = 600
PROMPT_RANGE = (300, 6000) # tokens already in the chat when the next reply starts: equal steps on a log scale
REPLY_RANGE = (40, 400)
def draw_chats(seed: int = SEED, n: int = N_CHATS) -> dict:
rng = random.Random(seed)
prompts, replies = [], []
for _ in range(n):
prompts.append(round(PROMPT_RANGE[0] * (PROMPT_RANGE[1] / PROMPT_RANGE[0]) ** rng.random()))
replies.append(round(REPLY_RANGE[0] * (REPLY_RANGE[1] / REPLY_RANGE[0]) ** rng.random()))
return {"prompts": prompts, "replies": replies}
CHATS = draw_chats()
def backlog(chats: dict = CHATS) -> list[Request]:
"""Every chat is waiting at time 0: a full queue, so the server is never short of work."""
return [Request(0.0, p, r) for p, r in zip(chats["prompts"], chats["replies"])]
# --- the server -----------------------------------------------------------------------------------------------------------------------------
def serve(requests: list[Request], cap: int, pool_blocks: int, longest: int, reserved: bool = False, cost=L24.step_seconds) -> dict:
"""Lesson 24's continuous batching with a cache pool of `pool_blocks` blocks and at most `cap` chats running. A chat's id is its index.
A step starts like this. Each running chat that needs a new block takes one; if none is free, the newest running chat is paused: its blocks are
freed and it goes to the front of the queue (the oldest chats are served first). In a step where a chat was paused, nobody new joins. Otherwise,
waiting chats join, oldest first, while fewer than `cap` run and the pool has the blocks the chat needs for the next token; the first one that
does not fit holds up the rest. A chat that joins has its whole cache (prompt, plus the tokens it had written before it was paused) read in that
step, as in lesson 24, and writes one token like every running chat. With `reserved`, a chat takes blocks for `longest` tokens when it joins, so it
is never paused. Chats longer than `longest` are turned away. A server that cannot hold one chat of `longest` tokens does not start.
The context used for the cost of a step is prompt + tokens written, as in lesson 24."""
assert pool_blocks >= blocks_for(longest), "the pool cannot hold one chat of the longest length"
n = len(requests)
refused = [i for i, r in enumerate(requests) if r.prompt + r.reply > longest]
turned_away = set(refused)
made, held = [0] * n, [0] * n
first, last = [-1.0] * n, [-1.0] * n
waiting, running = deque(), []
pending = [i for i in range(n) if i not in turned_away]
nxt, free, now, done = 0, pool_blocks, 0.0, 0
steps = paused = rebuilt = 0
area, used_area, lowest_free = 0.0, 0.0, pool_blocks
while done < len(pending):
while nxt < len(pending) and requests[pending[nxt]].arrival <= now:
waiting.append(pending[nxt])
nxt += 1
if not running and not waiting:
now = requests[pending[nxt]].arrival # nothing to do: wait for the next chat
continue
paused_now = False
k = 0
while k < len(running):
i = running[k]
if not reserved and (requests[i].prompt + made[i]) % BLOCK_TOKENS == 0: # the next token starts a new block
if free == 0:
j = running.pop() # the newest running chat gives up its blocks (when it is `i` itself, the scan ends)
free += held[j]
held[j] = 0
waiting.appendleft(j)
paused += 1
paused_now = True
continue
free -= 1
held[i] += 1
k += 1
joined = []
while not paused_now and waiting and len(running) < cap:
i = waiting[0]
need = blocks_for(longest) if reserved else blocks_for(requests[i].prompt + made[i] + 1)
if need > free:
break
waiting.popleft()
free -= need
held[i] = need
running.append(i)
joined.append(i)
joining = set(joined)
contexts = [requests[i].prompt + made[i] for i in running if i not in joining]
reread = sum(requests[i].prompt + made[i] for i in joined)
rebuilt += sum(requests[i].prompt + made[i] for i in joined if made[i] > 0) # a chat that was paused is read in again, in full
dt = cost(contexts, reread)
now += dt
steps += 1
if steps > MAX_STEPS:
raise RuntimeError(f"more than {MAX_STEPS:,} steps: some chat can never finish")
area += len(running) * dt
used_area += (pool_blocks - free) * dt
lowest_free = min(lowest_free, free)
still = []
for i in running:
made[i] += 1
if first[i] < 0:
first[i] = now
if made[i] == requests[i].reply:
last[i] = now
free += held[i]
held[i] = 0
done += 1
else:
still.append(i)
running = still
served = [i for i in range(n) if i not in turned_away]
tokens = per_word = 0.0
for i in served:
tokens += requests[i].reply
per_word += (last[i] - first[i]) / (requests[i].reply - 1)
return {"seconds": now, "steps": steps, "served": len(served), "refused": len(refused), "paused": paused, "rebuilt_tokens": rebuilt,
"tokens_per_second": tokens / now, "seconds_per_word": per_word / len(served), "running": area / now, "pool_used": used_area / (now * pool_blocks), "lowest_free": lowest_free, "first_token": first, "finished": last}
# --- what the widget shows -----------------------------------------------------------------------------------------------------------------
def gb(nbytes: float) -> str:
return f"{nbytes / 1e9:.1f}"
def commas(n: int) -> str:
return f"{n:,}"
def percent(x: float) -> int:
return math.floor(x * 100 + 0.5)
def room_line(r: dict, longest: int, reserved: bool) -> str:
if not r["starts"]:
return f"The server will not start. One chat of {commas(longest)} tokens needs room for {commas(r['per_longest'] * BLOCK_TOKENS)} tokens; the pool holds {commas(r['pool_tokens'])}."
if reserved:
return f"Room for {commas(r['at_longest'])} chats at once, because each one holds {commas(longest)} tokens whether it uses them or not."
return f"Room for {commas(r['typical'])} chats of {commas(min(TYPICAL_CHAT, longest))} tokens at once. {commas(r['at_longest'])} if every chat ran to {commas(longest)}."
def run_line(run: dict, cap: int) -> str:
return (f"Running on average: {commas(math.floor(run['running'] + 0.5))} chats (up to {commas(cap)}). Everyone together: {commas(math.floor(run['tokens_per_second'] + 0.5))} tokens a second. "
f"One user waits {run['seconds_per_word'] * 1000:.1f} ms between words.")
def extra_line(run: dict) -> str:
bits = [f"The chats in progress hold {percent(run['pool_used'])}% of the cache pool on average."]
if run["refused"]:
bits.append(f"{commas(run['refused'])} of {commas(run['refused'] + run['served'])} chats turned away: too long.")
if run["paused"]:
bits.append(f"{commas(run['paused'])} chats paused to free room; {commas(run['rebuilt_tokens'])} tokens of cache rebuilt.")
return " ".join(bits)
def describe(share: int, longest: int, cap: int, reserved: bool = False, requests: list[Request] | None = None) -> dict:
"""Everything the widget shows for one setting of its controls: the four parts of the bar, the legend, how full the pool is, and three lines of words.
`requests` defaults to the 600 invented chats; the widget test passes fewer to keep Python's side fast."""
r = room(share, longest, reserved)
out = {"grow": [r["weights"], r["reserve"], r["pool"], r["unused"]], "fill": 0, "room": room_line(r, longest, reserved), "run": "", "extra": "",
"legend": [f"Model weights {gb(r['weights'])} GB", f"Working reserve {gb(r['reserve'])} GB", f"Cache pool {gb(r['pool'])} GB", f"Left unused {gb(r['unused'])} GB"]}
if not r["starts"]:
out["extra"] = "Raise the memory share or lower the longest chat."
return out
run = serve(backlog() if requests is None else requests, cap, r["blocks"], longest, reserved)
out.update(run=run_line(run, cap), extra=extra_line(run), fill=percent(run["pool_used"]))
return out
def sweep(share: int = OPENING["share"], longest: int = OPENING["longest"], caps: list[int] = CAPS) -> list[dict]:
"""One run per cap on the same chats and pool: the rows of the lesson's table."""
r = room(share, longest)
rows = []
for cap in caps:
run = serve(backlog(), cap, r["blocks"], longest)
rows.append({"cap": cap, "tokens_per_second": run["tokens_per_second"], "seconds_per_word": run["seconds_per_word"], "running": run["running"],
"paused": run["paused"], "rebuilt_tokens": run["rebuilt_tokens"]})
return rows
def stops_paying(rows: list[dict]) -> int:
"""The first cap whose doubling brings under KNEE_GAIN times the tokens a second."""
by_cap = {row["cap"]: row["tokens_per_second"] for row in rows}
for cap in sorted(by_cap):
if 2 * cap in by_cap and by_cap[2 * cap] < KNEE_GAIN * by_cap[cap]:
return cap
return max(by_cap)Show code
"""Print every number the lesson quotes. Run: python3 show_numbers.py"""
import statistics
import settings_calc as sc
def main() -> None:
chats = sc.CHATS
print(f"card {sc.GPU_BYTES / 1e9:.0f} GB; weights {sc.WEIGHT_BYTES / 1e9:.2f} GB; cache {sc.KV_BYTES:,} bytes a token; block {sc.BLOCK_TOKENS} tokens = {sc.BLOCK_BYTES:,} bytes; reserve {sc.RESERVE_BYTES / 1e9:.1f} GB (invented)")
print(f"chats: seed {sc.SEED}, {sc.N_CHATS}; prompts {min(chats['prompts'])} to {max(chats['prompts'])} (mean {statistics.fmean(chats['prompts']):.0f}); "
f"replies {min(chats['replies'])} to {max(chats['replies'])} (mean {statistics.fmean(chats['replies']):.0f}, total {sum(chats['replies']):,}); "
f"longest whole chat {max(p + r for p, r in zip(chats['prompts'], chats['replies'])):,}; mean whole chat {statistics.fmean(p + r for p, r in zip(chats['prompts'], chats['replies'])):.0f}")
print()
print("memory parts by share (GB): weights, reserve, pool, unused; pool blocks and tokens; chats of 2,000 / of the longest (8,000 and 32,000)")
for share in (20, 23, 30, 40, 50, 60, 70, 80, 90, 92, 95, 98):
r8, r32 = sc.room(share, 8000), sc.room(share, 32000)
print(f" {share:3}%: allowed {sc.gb(r8['allowed']):>5} pool {sc.gb(r8['pool']):>5} unused {sc.gb(r8['unused']):>5} blocks {r8['blocks']:6,} tokens {r8['pool_tokens']:8,} "
f"typical {r8['typical']:4} at 8,000 {r8['at_longest']:3} at 32,000 {r32['at_longest']:3}")
print(" lowest share that leaves any pool:", next(s for s in range(1, 100) if sc.room(s, 16)["blocks"] > 0))
gb80, gib80 = sc.memory_parts(90), sc.memory_parts(90, sc.GPU_BYTES_GIB)
per_typical = sc.blocks_for(sc.TYPICAL_CHAT)
print(f" the card read as 80 GiB, at 90%: card {sc.GPU_BYTES_GIB / sc.GPU_BYTES - 1:+.1%}; pool {sc.gb(gib80['pool'])} GB against {sc.gb(gb80['pool'])} GB ({gib80['pool'] / gb80['pool'] - 1:+.1%}); "
f"blocks {gib80['blocks']:,} against {gb80['blocks']:,}; chats of {sc.TYPICAL_CHAT:,} {gib80['blocks'] // per_typical} against {gb80['blocks'] // per_typical}")
print()
print("opening state, and the room for each longest chat at 90%")
for longest in sc.LONGEST:
r, rr = sc.room(90, longest), sc.room(90, longest, True)
print(f" longest {longest:7,}: starts {r['starts']}; {sc.room_line(r, longest, False)} | reserved: {sc.room_line(rr, longest, True)}")
print(" lowest share that starts, by longest chat:", ", ".join(f"{longest:,}: {next(s for s in range(1, 100) if sc.room(s, longest)['starts'])}%" for longest in sc.LONGEST))
for share in (40, 44):
r = sc.room(share, 128000)
print(f" share {share}% with a 128,000-token longest chat: allowed {sc.gb(r['allowed'])} GB, pool {sc.gb(r['pool'])} GB = {r['blocks']:,} blocks, needs {r['per_longest']:,}; starts {r['starts']}")
print()
prompt_tokens = sum(chats["prompts"])
prefill_seconds = prompt_tokens * 2 * sc.L24.PARAMS / sc.L24.A100_PEAK_FLOPS
print(f"reading every prompt once: {prompt_tokens:,} tokens, at least {prefill_seconds:.1f} s of arithmetic, so no cap can pass {sum(chats['replies']) / prefill_seconds:,.0f} tokens a second")
print()
print("cap sweep at the opening share and longest chat (90%, 8,000)")
rows = sc.sweep()
print("cap | tokens/s | ms between words | chats running | pauses | cache rebuilt")
for row in rows:
print(f"{row['cap']:4} | {row['tokens_per_second']:8,.0f} | {row['seconds_per_word'] * 1000:6.1f} | {row['running']:6.1f} | {row['paused']:3} | {row['rebuilt_tokens']:,}")
print(" stops paying at cap", sc.stops_paying(rows))
for a, b in zip(rows, rows[1:]):
print(f" cap {a['cap']:3} to {b['cap']:3}: tokens/s x{b['tokens_per_second'] / a['tokens_per_second']:.3f}, wait x{b['seconds_per_word'] / a['seconds_per_word']:.2f}")
print()
print("the same sweep at other shares (tokens/s, ms per word, pauses)")
for share in (35, 50, 90):
out = sc.sweep(share)
print(f" share {share}%: pool {sc.room(share, 8000)['pool_tokens']:,} tokens;", "; ".join(f"cap {x['cap']}: {x['tokens_per_second']:,.0f}/{x['seconds_per_word'] * 1000:.0f}/{x['paused']}" for x in out if x['cap'] >= 32))
print()
print("blocks as chats grow, against reserving the full length (90%, cap 512)")
for longest in (8000, 32000):
r = sc.room(90, longest)
for reserved in (False, True):
run = sc.serve(sc.backlog(), 512, r["blocks"], longest, reserved)
print(f" longest {longest:6,} reserved={reserved!s:5}: {run['tokens_per_second']:,.0f} tokens/s, {run['seconds_per_word'] * 1000:.1f} ms, running {run['running']:.1f}, pauses {run['paused']}, turned away {run['refused']}")
print()
print("longest chat cuts off chats (90%, cap 64)")
for longest in sc.LONGEST:
run = sc.serve(sc.backlog(), 64, sc.room(90, longest)["blocks"], longest)
print(f" longest {longest:7,}: turned away {run['refused']}, tokens/s {run['tokens_per_second']:,.0f}")
if __name__ == "__main__":
main()Show code
"""Run: python3 -B -m pytest -q -p no:cacheprovider code/ -- each test guards a number the lesson or the widget shows a reader."""
import math
import re
import statistics
from functools import lru_cache
from pathlib import Path
import settings_calc as sc
from settings_calc import BLOCK_BYTES, BLOCK_TOKENS, GPU_BYTES, RESERVE_BYTES, WEIGHT_BYTES, Request, blocks_for, room, serve
LESSON = Path(__file__).resolve().parent.parent
@lru_cache(maxsize=None)
def swept(share=sc.OPENING["share"], longest=sc.OPENING["longest"]):
return sc.sweep(share, longest)
@lru_cache(maxsize=None)
def ran(share, longest, cap, reserved=False):
return serve(sc.backlog(), cap, room(share, longest)["blocks"], longest, reserved)
def test_the_parts_add_up_to_the_card_and_the_pool_is_whole_blocks():
"""A bar that does not add up to 80 GB, or a pool counted in half blocks, would put wrong numbers on every line of the widget."""
for share in sc.SHARES + [1, 20, 23]:
r = room(share, 4000)
assert r["weights"] + r["reserve"] + r["pool"] + r["unused"] == GPU_BYTES
assert r["pool"] == r["blocks"] * BLOCK_BYTES and r["pool"] <= max(0, r["allowed"] - WEIGHT_BYTES - RESERVE_BYTES)
assert r["allowed"] - WEIGHT_BYTES - RESERVE_BYTES - r["pool"] < BLOCK_BYTES or r["blocks"] == 0
assert (room(90, 8000)["blocks"], room(90, 8000)["pool_tokens"]) == (25_720, 411_520)
assert room(20, 8000)["blocks"] == 0 and room(23, 8000)["blocks"] == 161 # the card holds 18.06 GB of weights and reserve: 22.6% is the lowest share with any cache
def test_chats_that_fit_and_the_server_that_will_not_start():
r = room(90, 8000)
assert (r["typical"], r["at_longest"], r["starts"]) == (205, 51, True) # 25,720 // 125 and 25,720 // 500
assert room(90, 8000, reserved=True)["typical"] == 51
for share in sc.SHARES: # starts exactly when the pool can hold one chat of the longest length
for longest in sc.LONGEST:
r = room(share, longest)
assert r["starts"] == (r["blocks"] >= blocks_for(longest))
assert not room(40, 128_000)["starts"] and room(44, 128_000)["starts"]
assert not room(23, 128_000)["starts"] and "will not start" in sc.room_line(room(40, 128_000), 128_000, False)
def test_blocks_hold_more_chats_than_reserving_the_longest_length():
"""With blocks the longest chat is a ceiling; if each chat held a full-length slice, long ceilings would shrink the number of chats."""
for longest in sc.LONGEST[1:]:
r = room(90, longest)
assert r["typical"] == room(90, 8000)["typical"] > room(90, longest, True)["typical"] == r["at_longest"]
blocks = room(90, 32_000)["blocks"]
grown = serve(sc.backlog(), 512, blocks, 32_000)
held = serve(sc.backlog(), 512, blocks, 32_000, reserved=True)
assert held["paused"] == 0 and held["lowest_free"] >= 0 and held["running"] < 13 < grown["running"]
assert grown["tokens_per_second"] > held["tokens_per_second"]
def test_the_server_matches_lesson_24_when_the_pool_is_unlimited():
"""The new scheduler must be lesson 24's continuous batching plus a memory limit, or its numbers would not carry on from that lesson."""
for mix in ("same", "very mixed"):
requests = sc.L24.make_requests(8, mix)
theirs = sc.L24.simulate(requests, 8, "continuous")
mine = serve(requests, 8, 10**9, 10**5)
assert mine["first_token"] == theirs.first_token and mine["finished"] == theirs.finished
assert mine["steps"] == len(theirs.steps) and mine["paused"] == 0
def test_a_full_pool_pauses_the_newest_chat_and_rebuilds_it_later():
"""Two chats of 30 tokens each need a third block at token 32 and the pool has 4: the newer pauses, the older finishes, the newer reads its 32 tokens again. A chat of exactly the longest length is served; one token more is turned away."""
requests = [Request(0.0, 30, 10), Request(0.0, 30, 10)]
seen = []
run = serve(requests, 2, 4, 64, cost=lambda contexts, joining: seen.append((sorted(contexts), joining)) or 1.0)
assert run["paused"] == 1 and run["rebuilt_tokens"] == 32 and run["lowest_free"] == 0
assert run["finished"][0] < run["finished"][1] and run["first_token"][1] == 1.0 # the paused chat keeps its first-token time
assert seen[0] == ([], 60) and seen[1] == ([31, 31], 0) and seen[2] == ([32], 0) # step 3: only the older chat writes
assert seen[10] == ([], 32) and run["steps"] == 18 # step 11: the newer chat is read in again, then writes 8 more tokens
edge = serve([Request(0.0, 54, 10), Request(0.0, 55, 10)], 2, 4, 64) # "up to 64" means a chat of 64 tokens is served and one of 65 is turned away
assert (edge["served"], edge["refused"]) == (1, 1)
def test_pool_is_never_overdrawn_and_every_chat_gets_its_whole_reply():
for share in (30, 35, 50, 90):
r = room(share, 8000)
for cap in (16, 128, 512):
run = serve(sc.backlog(), cap, r["blocks"], 8000)
assert run["lowest_free"] >= 0 and run["served"] == sc.N_CHATS and all(t > 0 for t in run["finished"])
cut = serve(sc.backlog(), 64, room(90, 4000)["blocks"], 4000)
assert cut["refused"] == sum(p + rep > 4000 for p, rep in zip(sc.CHATS["prompts"], sc.CHATS["replies"])) == 89
assert cut["refused"] + cut["served"] == sc.N_CHATS
def test_raising_the_cap_stops_paying_where_the_lesson_says():
rows = {row["cap"]: row for row in swept()}
gains = [rows[2 * c]["tokens_per_second"] / rows[c]["tokens_per_second"] for c in sc.CAPS[:6]] # 1->2 ... 32->64
assert all(g > 1.10 for g in gains) and sc.stops_paying(list(rows.values())) == 64
assert 1.05 < rows[128]["tokens_per_second"] / rows[64]["tokens_per_second"] < 1.10
assert all(rows[2 * c]["seconds_per_word"] > rows[c]["seconds_per_word"] for c in sc.CAPS[:7]) # waiting keeps climbing up to 128
assert {**rows[256], "cap": 0} == {**rows[512], "cap": 0} and rows[256]["paused"] > 0 and rows[128]["paused"] == 0 # past what the pool holds, the cap changes nothing
assert rows[256]["running"] < 205 and rows[1]["running"] == 1
small = room(50, 8000)["blocks"] # a smaller pool: a higher cap is worse
low, high = (serve(sc.backlog(), cap, small, 8000) for cap in (64, 128))
assert high["tokens_per_second"] < low["tokens_per_second"] and high["paused"] > 0 and low["paused"] == 0
def test_every_number_in_the_lesson_and_figures_comes_from_the_code():
"""The prose and the four figures quote numbers by hand; a change to the constants or the scheduler must show up here, not in a reader's check."""
prose = (LESSON / "lesson.md").read_text(encoding="utf-8")
figures = {path.name: path.read_text(encoding="utf-8") for path in sorted((LESSON / "figs").glob("*.svg"))}
assert set(figures) == set(re.findall(r"\]\(figs/([\w-]+\.svg)\)", prose)) and len(figures) == 4
gb = lambda value: f"{value / 1e9:.1f}"
r, rows = room(90, 8000), {row["cap"]: row for row in swept()}
tps = lambda row: f"{row['tokens_per_second']:,.0f}"
ms = lambda row: f"{row['seconds_per_word'] * 1000:.1f}"
gains = [rows[2 * c]["tokens_per_second"] / rows[c]["tokens_per_second"] for c in sc.CAPS[:6]]
grow = rows[128]["tokens_per_second"] / rows[64]["tokens_per_second"]
wait = rows[128]["seconds_per_word"] / rows[64]["seconds_per_word"]
held = {longest: ran(90, longest, 512, True) for longest in (8000, 32000)}
grown = ran(90, 8000, 512)
prompts, replies = sc.CHATS["prompts"], sc.CHATS["replies"]
seconds = sum(prompts) * 2 * sc.L24.PARAMS / sc.L24.A100_PEAK_FLOPS
gib = sc.memory_parts(90, sc.GPU_BYTES_GIB) # the other reading of the card, quoted at the end of the pool sum
wanted = {"lesson.md": [
gb(r["pool"]), gb(r["unused"]), f"{r['weights'] / 1e9:.2f} GB", gb(RESERVE_BYTES), f"{r['pool_tokens']:,}", f"{r['blocks']:,}", f"{r['typical']} chats",
f"{room(90, 8000, True)['typical']}", f"{room(90, 32000, True)['typical']}", f"{room(90, 128000, True)['typical']}", f"{room(90, 4000, True)['typical']}",
f"{room(50, 8000)['pool_tokens']:,}", gb(room(50, 8000)["pool"]), f"{room(98, 8000)['typical']}", gb(room(98, 8000)["pool"]), gb(room(30, 8000)["pool"]),
f"{room(30, 8000)['typical']}", f"{room(50, 8000)['typical']}", f"{room(35, 8000)['pool_tokens']:,}",
f"{room(40, 128000)['blocks']:,} blocks", gb(room(40, 128000)["pool"]), f"{room(44, 128000)['blocks']:,} blocks", f"{blocks_for(128000):,}",
*(f"{next(s for s in range(1, 100) if room(s, longest)['starts'])}%" for longest in (8000, 16000, 32000, 64000, 128000)),
*(f"| {c} | {tps(rows[c])} | {ms(rows[c])} ms |" for c in (1, 8, 32, 64, 128, 256)),
f"{rows[256]['paused']} pauses", f"{rows[256]['rebuilt_tokens']:,}", f"{rows[256]['running']:.1f}", f"{(min(gains) - 1) * 100:.1f}%", f"{(grow - 1) * 100:.1f}%", f"{(grow - 1) * 100:.0f}%",
f"{wait:.2f} times", f"{wait:.1f} times", f"cap of {sc.stops_paying(list(rows.values()))}",
f"{sum(prompts):,}", f"{seconds:.1f} s", f"{sum(replies) / seconds:,.0f}", f"{sum(replies):,}", f"{min(prompts)} to {max(prompts):,}", f"mean {statistics.fmean(prompts):,.0f}",
f"{min(replies)} to {max(replies)}", f"mean {statistics.fmean(replies):.0f}", f"{statistics.fmean(prompts) * 2 * sc.L24.PARAMS / sc.L24.A100_PEAK_FLOPS * 1000:.1f} ms",
f"writes {tps(grown)} tokens", f"{grown['running']:.1f} chats", f"{tps(held[8000])} at 8,000", f"{held[8000]['running']:.1f} running", f"{held[32000]['running']:.1f} running",
f"{held[32000]['tokens_per_second']:,.0f} at 32,000", f"{grown['seconds_per_word'] * 1000:.1f} ms", f"{held[8000]['seconds_per_word'] * 1000:.1f} ms",
f"{held[32000]['seconds_per_word'] * 1000:.1f} ms",
f"{tps(ran(50, 8000, 128))} with {ran(50, 8000, 128)['paused']}", f"{tps(ran(50, 8000, 64))} tokens a second with no pauses",
f"{tps(ran(35, 8000, 32))} with {ran(35, 8000, 32)['paused']} pauses", f"{tps(ran(35, 8000, 64))} with {ran(35, 8000, 64)['paused']}",
f"{sc.BLOCK_BYTES:,}", f"{sc.WEIGHT_BYTES:,}", f"{sc.KV_BYTES:,}", f"{room(90, 8000)['per_typical']} blocks", f"{sc.OPENING['share']}%",
f"{gb(gib['pool'])} GB, or {gib['blocks']:,} blocks, with room for {gib['blocks'] // r['per_typical']} chats of 2,000 tokens, against {gb(r['pool'])} GB, {r['blocks']:,} and {r['typical']}",
f"The card is {(sc.GPU_BYTES_GIB / GPU_BYTES - 1) * 100:.1f}% larger and the pool {(gib['pool'] / r['pool'] - 1) * 100:.1f}%",
], "memory-parts.svg": [gb(r["pool"]), gb(r["unused"]), f"{r['weights'] / 1e9:.2f}", gb(RESERVE_BYTES), f"{r['pool_tokens']:,}", "72 GB", "90%", "80 GB"],
"blocks-vs-reserved.svg": [f"{room(90, 8000)['typical']} chats", f"{room(90, 8000, True)['typical']} chats", f"{room(90, 32000, True)['typical']} chats", gb(r["pool"]), f"{r['blocks']:,}"],
"cap-sweep.svg": [tps(rows[1]), tps(rows[64]), tps(rows[128]), ms(rows[1]), ms(rows[64]), ms(rows[128]), ms(rows[256]), tps(rows[256])],
"pool-moves-the-knee.svg": [f"{room(share, 8000)['pool_tokens']:,}" for share in (35, 50, 90)] + [tps(ran(share, 8000, cap)) for share in (35, 50, 90) for cap in (32, 64, 128)]}
missing = [f"{name}: {fragment!r}" for name, fragments in wanted.items() for fragment in fragments if fragment not in (prose if name == "lesson.md" else figures[name])]
assert not missing, "not found: " + "; ".join(missing)
for name, text in figures.items(): # the palette is fixed and colour-blind safe: none of the blue or vermillion of the other charts
assert not re.search(r"0072B2|56B4E9|D55E00", text, re.I) and "<title>" in text and "<desc>" in text and 'fill="#ffffff"' in text, nameShow code
"""Guards for ../widget.html: its data block, that its server gives the same numbers and words as settings_calc.py, and what a reader sees and can press."""
import json
import re
import subprocess
import settings_calc as sc
from embed_widget import CONFIG_BLOCK, WIDGET, config, embed
HTML = WIDGET.read_text(encoding="utf-8")
FEW = 40 # Python's side of the parity run uses this many chats, to stay fast; the page itself uses all 600
GRID = [(share, longest, cap, reserved) for share in (30, 35, 43, 44, 50, 70, 90, 98) for longest in sc.LONGEST for cap in (8, 64, 512) for reserved in (False, True)]
# Reads the model out of the widget between its marker comments, runs every setting with Node and compares the memory parts, the three lines of words and
# the legend with the ones Python wrote into the fixture, character for character.
PARITY = r"""
import fs from 'node:fs';
const [fixture, widget] = process.argv.slice(2);
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const html = fs.readFileSync(widget, 'utf8');
const cfg = JSON.parse(html.match(/<script type="application\/json" id="sw-config">([\s\S]*?)<\/script>/)[1]);
const js = html.slice(html.indexOf('/* ---------- the model ('), html.indexOf('/* ---------- display'));
const api = new Function('cfg', "'use strict';" + js + '; return {describe};')(cfg);
const reqs = cfg.prompts.slice(0, fx.n).map((p, i) => ({ prompt: p, reply: cfg.replies[i] }));
const bad = [];
fx.cases.forEach((c) => {
const got = api.describe(c.share, c.longest, c.cap, c.reserved, reqs);
if (JSON.stringify(got) !== JSON.stringify(c.want)) bad.push([c.share, c.longest, c.cap, c.reserved, JSON.stringify(got).slice(0, 300), JSON.stringify(c.want).slice(0, 300)]);
});
console.log(JSON.stringify({ checks: fx.cases.length, bad: bad.slice(0, 3) }));
process.exit(bad.length ? 1 : 0);
"""
# Runs the page script against a small stand-in for the DOM, moves the sliders, ticks the box and reads back what a reader sees.
PAGE = r"""
import fs from 'node:fs';
const [widget, fixture] = process.argv.slice(2);
const html = fs.readFileSync(widget, 'utf8');
const fx = JSON.parse(fs.readFileSync(fixture, 'utf8'));
const config = html.match(/<script type="application\/json" id="sw-config">([\s\S]*?)<\/script>/)[1];
const code = html.match(/<script>([\s\S]*?)<\/script>/)[1];
function el(tag) {
const e = { tag, children: [], listeners: {}, value: '', checked: false, style: {}, attrs: {}, textContent: '' };
e.setAttribute = (k, v) => { e.attrs[k] = v; };
e.getAttribute = (k) => e.attrs[k];
e.addEventListener = (type, f) => (e.listeners[type] = e.listeners[type] || []).push(f);
return e;
}
const page = { root: el('html'), messages: [], parent: { other: true } };
[...html.matchAll(/id="(sw-[^"]+)"/g)].forEach((m) => { page[m[1]] = el('x'); });
page['sw-config'].textContent = config;
const doc = { getElementById: (id) => page[id], documentElement: page.root };
const win = { addEventListener: (type, f) => page.messages.push(f) };
new Function('document', 'window', 'parent', code)(doc, win, page.parent);
const fire = (e, type) => (e.listeners[type] || []).forEach((f) => f({ target: e }));
const bad = [];
const want = (what, got, expected) => { if (JSON.stringify(got) !== JSON.stringify(expected)) bad.push([what, JSON.stringify(got).slice(0, 200), JSON.stringify(expected).slice(0, 200)]); };
const shown = () => {
const width = [0, 1, 2, 3].reduce((sum, k) => sum + parseFloat(page['sw-seg-' + k].style.width), 0); // the four parts fill the bar, no more and no less
if (Math.abs(width - 100) > 1e-6) bad.push(['bar widths add to 100', String(width), '100']);
return { grow: [0, 1, 2, 3].map((k) => Number(page['sw-seg-' + k].attrs['data-bytes'])), fill: Number(page['sw-fill'].style.width.replace('%', '')), room: page['sw-room'].textContent,
run: page['sw-run'].textContent, extra: page['sw-extra'].textContent, legend: [0, 1, 2, 3].map((k) => page['sw-leg-' + k].textContent) };
};
const cfg = JSON.parse(config);
want('opening controls', [page['sw-share'].value, page['sw-longest'].value, page['sw-cap'].value, page['sw-reserved'].checked, page['sw-longest'].max, page['sw-cap'].max],
[String(cfg.opening.share), String(cfg.longest.indexOf(cfg.opening.longest)), String(cfg.caps.indexOf(cfg.opening.cap)), false, String(cfg.longest.length - 1), String(cfg.caps.length - 1)]);
want('opening state', shown(), fx.opening);
want('nothing spoken yet', page['sw-live'].textContent, '');
page['sw-cap'].value = String(cfg.caps.indexOf(256));
fire(page['sw-cap'], 'input');
want('cap 256, dragging', shown(), fx.cap256);
want('dragging stays silent', page['sw-live'].textContent, '');
fire(page['sw-cap'], 'change');
want('release speaks', page['sw-live'].textContent, [fx.cap256.room, fx.cap256.run, fx.cap256.extra].join(' '));
want('cap words', [page['sw-cap-out'].textContent, page['sw-cap'].attrs['aria-valuetext']], ['256', '256 chats']);
page['sw-share'].value = '40'; page['sw-longest'].value = String(cfg.longest.length - 1);
fire(page['sw-share'], 'input');
want('will not start', shown(), fx.nostart);
want('share and longest words', [page['sw-share-out'].textContent, page['sw-longest-out'].textContent, page['sw-share'].attrs['aria-valuetext'], page['sw-longest'].attrs['aria-valuetext']],
['40%', '128,000 tokens', '40% of the card', '128,000 tokens']);
fx.full.forEach((c) => { // pools that fill up, with all 600 chats
page['sw-share'].value = String(c.share); page['sw-longest'].value = String(cfg.longest.indexOf(c.longest)); page['sw-cap'].value = String(cfg.caps.indexOf(c.cap));
fire(page['sw-share'], 'input');
want('pool full ' + [c.share, c.longest, c.cap].join('/'), shown(), c.want);
});
page['sw-share'].value = '90'; page['sw-longest'].value = String(cfg.longest.indexOf(8000)); page['sw-cap'].value = String(cfg.caps.indexOf(64)); page['sw-reserved'].checked = true;
fire(page['sw-reserved'], 'input');
want('reserved', shown(), fx.reserved);
const theme = (source, name) => page.messages.forEach((f) => f({ data: { type: 'spicyshark:theme', theme: name }, source }));
theme({ other: 'frame' }, 'dark');
want('theme from another sender is ignored', page.root.attrs['data-theme'], undefined);
theme(page.parent, 'dark');
want('theme from the host page', page.root.attrs['data-theme'], 'dark');
console.log(JSON.stringify({ bad: bad.slice(0, 3) }));
process.exit(bad.length ? 1 : 0);
"""
def run_node(tmp_path, script, *args):
path = tmp_path / "check.mjs"
path.write_text(script, encoding="utf-8")
done = subprocess.run(["node", str(path), *args], capture_output=True, text=True, timeout=120)
assert done.returncode == 0, done.stdout + done.stderr
return json.loads(done.stdout)
def test_widget_data_block_matches_the_python_module():
assert embed(HTML) == HTML
block = json.loads(CONFIG_BLOCK.search(HTML).group(2))
assert block == json.loads(json.dumps(config()))
assert block["opening"] == {"share": 90, "longest": 8000, "cap": 64, "reserved": False}
assert re.search(r'id="sw-share" min="%d" max="%d"' % (min(sc.SHARES), max(sc.SHARES)), HTML) # the slider covers the shares the tests cover
def test_widget_gives_the_same_numbers_and_words_as_python(tmp_path):
few = sc.backlog()[:FEW]
fixture = {"n": FEW, "cases": [{"share": a, "longest": b, "cap": c, "reserved": d, "want": sc.describe(a, b, c, d, few)} for a, b, c, d in GRID]}
assert any("paused" in case["want"]["extra"] for case in fixture["cases"]) and any("turned away" in case["want"]["extra"] for case in fixture["cases"])
assert any("will not start" in case["want"]["room"] for case in fixture["cases"]) # the grid reaches pauses, cut-off chats and a server that will not start
(tmp_path / "fx.json").write_text(json.dumps(fixture), encoding="utf-8")
assert run_node(tmp_path, PARITY, str(tmp_path / "fx.json"), str(WIDGET))["checks"] == len(GRID)
def test_widget_page_shows_the_opening_state_and_follows_the_controls(tmp_path):
state = lambda *a: sc.describe(*a)
fixture = {"opening": state(90, 8000, 64), "cap256": state(90, 8000, 256), "nostart": state(40, 128_000, 256), "reserved": state(90, 8000, 64, True),
"full": [{"share": a, "longest": b, "cap": c, "want": state(a, b, c)} for a, b, c in ((35, 8000, 64), (50, 8000, 128), (30, 8000, 64))]}
assert all("paused" in case["want"]["extra"] for case in fixture["full"]) # states where the pool fills and chats pause
assert fixture["opening"]["room"] == "Room for 205 chats of 2,000 tokens at once. 51 if every chat ran to 8,000."
(tmp_path / "fx.json").write_text(json.dumps(fixture), encoding="utf-8")
run_node(tmp_path, PAGE, str(WIDGET), str(tmp_path / "fx.json"))
def test_widget_is_small_themed_and_uses_only_the_site_palette():
body = HTML[HTML.index('<div class="sw-root">'):HTML.index("<script")]
visible = re.sub(r"<details.*?</details>", "", body, flags=re.S)
assert len(re.findall(r"<input|<select|<button", visible)) <= 5 and "More settings" in body # at most five controls on show; the rest sit under More settings
assert len(HTML.splitlines()) < 300 and "light-dark(#ffffff, #0a0f17)" in HTML and "spicyshark:theme" in HTML and "prefers-reduced-motion" in HTML
assert not re.search(r"0072B2|56B4E9|D55E00|https?://|<img|@import", HTML, re.I)