Serving many chats
See how memory and time are shared between users.
Why AI servers waste memory, and how paging fixes it
An AI server keeps working notes for every chat. Storing them in small equal blocks, not one big reserved stretch, lets one GPU serve far more chats.
Why does an AI wait a moment before the first word?
Before its first word, an AI reads your whole prompt in one pass. Long prompts wait longer, short ones do not, and later words arrive at a steady pace.
How does an AI server answer many people at once?
A server writes one word for many people in a single step, and a new request takes any free place at once. Here is how much that gains, and where it stops.
How can a small model speed up a big one?
A small AI model guesses the next few tokens and the big one checks them all at once, so the same reply arrives sooner. Here is how much, and when it fails.
Why does a big model need several GPUs?
A big AI model's weights do not fit in one GPU's memory, so several GPUs share them. See what the sharing costs in waiting as you add GPUs.
Why split reading the prompt from writing the reply?
One long prompt stops every reply on a shared GPU. A toy shows what splitting reading from writing fixes, what the hand-over costs and when it stops paying.