---
title: "Why does a big model need several GPUs?"
description: "A big AI model's weights do not fit in one GPU's memory, so several GPUs share them. See what the sharing costs in waiting as you add GPUs."
topic: "AI systems"
chapter: "serving-many-chats"
order: 26
minutes: { plain: 6, deep: 22 }
prerequisites: ["How much memory an AI model needs (lesson 18)", "Why a chip waits on memory (lesson 19)", "Reading short Python functions"]
seo_title: "Why does a big LLM need several GPUs?"
date: 2026-10-19
---
# Why does a big model need several GPUs?
A large language model (LLM) with 70 billion weights, the numbers it has learned, takes 131.42 GiB at 16 bits each (a bit is one 0 or 1; a GiB is a little over a billion bytes). One made-up GPU, a graphics chip of the kind that runs AI models, can use 73.6 GiB (92% of its 80 GiB). The model cannot sit on one GPU. Splitting it over several GPUs fixes that and lets each GPU read less, so words come sooner; but twice the GPUs does not give twice the speed.
## In one minute
A big LLM needs several GPUs because its weights do not fit in one GPU's memory, so each GPU holds a share. Think of a recipe book too big for one cook, torn into parts, one part for each cook. All the cooks read their own pages at the same moment, so the dish comes sooner. After every step, though, the cooks must call out their partial results to one another, and calling takes time. More cooks mean fewer pages each but more calling. A faster link between the GPUs shortens the wait.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/why-a-big-model-needs-several-gpus/widget.html). Choose a model size, a number of GPUs and a link, the connection between the GPUs. It shows the memory each GPU needs, the time for one word (a word here is one token, a chunk of text) and how much of that time is waiting. It opens on the 70B model (70 billion weights) on 4 GPUs with the fast link: 34.93 GiB each, 18.94 ms a word (a ms is a thousandth of a second) and 8.7% of that spent waiting. Choose the 405B model on 1 GPU and it says "Does not fit".
## How it works
### The weights must fit
To write a word, a GPU reads every weight it holds from its own memory, so the weights must sit there ([How much memory does an AI model need?](https://spicyshark.co/learn/model-memory/)). Fewer bits for each weight is one way to make a model fit ([What does quantization trade away?](https://spicyshark.co/learn/quantization/)). The other is to share the weights among several GPUs.
Figure 1 sets the 70B model's 131.42 GiB against the 73.6 GiB one GPU can use. Split four ways, each GPU holds 32.85 GiB. The widget adds a chat's notes (the attention cache) and working space, which brings it to 34.93 GiB. The 405B model takes 755.96 GiB and needs at least 11 GPUs.
*Figure 1. The 70B model's weights against one GPU's usable memory, then split four ways.*
### Slicing every layer
A model is a stack of layers, steps the numbers pass through; the 70B model has 80. The usual split cuts every layer into slices, one for each GPU. This is tensor parallelism. The GPUs work on the same layer at the same moment, and each reads only its share of the weights.
Reading is the slow part, because a chip waits on memory ([Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/)). One GPU would need 69.20 ms to read the 70B model's weights, if they fit. Four GPUs each read a quarter, in 17.30 ms, at the same time.
### Waiting for each other
A slice gives only part of the answer for a layer. The GPUs must add their parts, and every GPU needs the total. That swap is called an all-reduce. It happens twice in each layer, so 160 times for every word of the 70B model. Each message is small, 16,384 bytes, but a swap has a start-up time whatever the size: about 10 microseconds (a millionth of a second) on 4 GPUs. The message goes round a ring of GPUs, so more GPUs mean more steps and a longer start-up. The 160 swaps cost 1.64 ms.
Figure 2 shows reading shrinking and waiting growing. At 2, 4, 8 and 16 GPUs the waiting takes 3.5%, 8.7%, 21.8% and 47.7% of each word.
*Figure 2. One word from the 70B model on the fast link: reading (green), waiting (hatched).*
### When more GPUs stop helping
Doubling the GPUs halves the reading but not the waiting. The 8B model (8 billion weights, small enough for one GPU) shows where it ends. On the fast link, one user's words come 1.77, 1.69 and 1.35 times faster with each doubling from 1 GPU to 8, and 0.94 times for the last doubling, to 16: slower. Figure 3 shows words a second.
*Figure 3. The 8B model, one user. Among the counts shown (1 to 16), speed peaks at 8 GPUs on both links.*
The link decides how long the waiting lasts. For one user the messages are tiny, so the link matters little: the 70B model on 8 GPUs takes 11.06 ms a word on the fast link and 12.02 ms on the slow one. A server that handles many chats makes the messages bigger. With 64 words in each swap, the same split takes 11.54 ms on the fast link and 16.53 ms on the slow one.
### Another way to split: groups of layers
The other split gives each GPU a group of whole layers: with four GPUs, the first 20 layers go to the first one, and so on. This is pipeline parallelism. Only one short message passes from group to group, so waiting nearly vanishes: 69.23 ms a word on four GPUs, against 69.20 ms of reading alone. But the groups work one after another. One user's word takes as long as it would on one big GPU, and three of the four GPUs stand idle at any moment. Groups make a model fit; they do not speed up one chat. They pay off when many chats keep every group busy (a [later lesson](https://spicyshark.co/learn/how-an-ai-server-answers-many-people-at-once/) covers batching).
## What this means when you build
- **Count the fewest GPUs first.** Weights, the chat's notes and working space must fit in each GPU's usable memory. The 70B model needs 2 GPUs, the 405B model 11.
- **Try fewer bits before more GPUs.** A smaller file may need fewer GPUs, at a cost in accuracy ([lesson 20](https://spicyshark.co/learn/quantization/)).
- **Expect less than double.** Each doubling of GPUs gives less than twice the speed. Watch the share of time spent waiting.
- **Ask what links the GPUs.** For one user the link barely shows; for many chats at once it matters.
- **Keep layer groups for fitting, not speed.** They do not shorten one user's wait for a word.
## Go deeper
*Optional. For readers who write Python. The code is in the `code/` folder; every number below was printed by `python3 show_numbers.py`, worked out by hand, or comes from a cited source read on 2026-10-19.*
### Where the toy's numbers come from
The toy has one made-up GPU, two links and three models. The GPU has the 80 GiB of [lesson 18](https://spicyshark.co/learn/model-memory/), of which a server claims 92%: 73.6 GiB. It reads its memory at 2,039 GB/s and multiplies at 312 TFLOP/s, the figures of the A100 80 GB SXM column in the NVIDIA A100 data sheet (June 2021) that lessons [19](https://spicyshark.co/learn/tiled-matmul-roofline/) and [20](https://spicyshark.co/learn/quantization/) use. The same sheet lists NVLink at 600 GB/s, which the toy calls the fast link, and PCIe Gen4 at 64 GB/s, the slow link. It gives one figure for each link and does not say whether the figure counts both directions. NVIDIA's blog post "NVIDIA Ampere Architecture In-Depth" (Krashinsky et al., 14 May 2020) gives 25 GB/s in each direction for one NVLink and 600 GB/s in all, so the 600 probably adds the two directions. The toy uses the figures as written. If both figures count both directions, halving them changes little for one user: the 70B model on 8 GPUs goes from 11.06 to 11.07 ms on the fast link and from 12.02 to 12.09 ms on the slow one. With 64 words at once the slow link goes from 16.53 to 21.12 ms, 28% more.
The start-up times come from NCCL, NVIDIA's library for GPU-to-GPU communication. Its file `src/tuning/cost_model.cc` (master branch, last changed 2026-08-31) lists the library's default tuning constants: a base latency of 6.6 microseconds for a ring all-reduce with the small-message protocol, plus 0.6 microseconds for each step on NVLink and 1.0 on PCIe. These are planning estimates, not measurements. The model shapes are the `config.json` files of Llama 3.1 8B and 70B and Table 3 of the Llama 3 paper for the 405B, saved in `code/`; the weight counts match [lesson 18](https://spicyshark.co/learn/model-memory/).
### The memory sum, by hand
The 70B model on 4 GPUs, in bytes. Weights: 70,553,706,496 x 2 / 4 = 35,276,853,248. The chat's cache, for 8,192 tokens as in [lesson 18](https://spicyshark.co/learn/model-memory/): 2 x 80 layers x 8 key/value heads x 128 values x 2 bytes = 327,680 bytes a token, 2,684,354,560 for the chat, shared by key/value head: 671,088,640. Working space is not shared: 486,539,264 for 2,048 tokens of intermediate results plus a 1 GiB allowance, 1,560,281,088 in all. The sum is 37,508,222,976 bytes, or 34.93 GiB, under 73.6. The cache can be shared only by head, so 16 GPUs hold as much of it as 8 (the model has 8 key/value heads). This is `memory_per_gpu`, with its docstring left out:
```python
def memory_per_gpu(key: str, n: int) -> dict:
cfg = load_config(key)
parts = {
"weights": ceil_div(weight_bytes(cfg), n),
"cache": ceil_div(kv_bytes_per_token(cfg) * CHAT_TOKENS, min(n, cfg["num_key_value_heads"])),
"working": activation_bytes(cfg) + OVERHEAD_GIB * GIB,
}
parts["total"] = sum(parts.values())
return parts
```
The fewest GPUs that fit are 1 for the 8B model, 2 for the 70B and 11 for the 405B. The widget offers 16 so the 405B can fit at all: 49.60 GiB each.
### The swap, by hand
For the 70B model on 4 GPUs a message is one word's hidden vector, 8,192 numbers of 2 bytes: 16,384 bytes. A ring all-reduce over n GPUs takes 2(n - 1) steps, 6 here, and each GPU sends 2(n - 1) / n times the message, 1.5 x 16,384 = 24,576 bytes (the NCCL tests documentation gives the time as (S / B) x 2(n - 1) / n, for size S and bandwidth B). On the fast link the start-up is 6.6 + 6 x 0.6 = 10.2 microseconds and sending takes 24,576 / 600 GB/s = 0.041 microseconds, so a swap takes 10.24. Times 160 swaps is 1.64 ms. Start-up is 99.6% of it, so for one user the link's step latency matters and its speed does not. With 64 words in each swap on 8 GPUs, a swap takes 18.06 microseconds on the fast link and 49.27 on the slow one.
### One user, 8B model
Each cell is milliseconds a word:
| GPUs | Fast link | Slow link |
|---|---|---|
| 1 | 7.88 | 7.88 |
| 2 | 4.44 | 4.50 |
| 4 | 2.62 | 2.79 |
| 8 | 1.95 | 2.32 |
| 16 | 2.07 | 2.85 |
Waiting takes 11.3%, 24.9%, 49.4% and 76.2% of a word on the fast link at 2, 4, 8 and 16 GPUs, and 12.4%, 29.4%, 57.5% and 82.7% on the slow link.
Check one cell, 2 GPUs on the fast link. Each GPU reads 8,030,261,248 bytes at 2,039 GB/s: 3.94 ms. A message is 4,096 x 2 = 8,192 bytes. A swap over 2 GPUs takes 2 steps, so its start-up is 6.6 + 2 x 0.6 = 7.8 microseconds and it sends 8,192 bytes, 0.014 microseconds: 7.81 in all. The 8B model has 32 layers, so a word makes 64 swaps, 0.50 ms. The total is 3.94 + 0.50 = 4.44 ms, and waiting is 11.3% of it. Past 8 GPUs the waiting (1.58 ms on the fast link) outgrows the reading (0.49 ms). This sum accepts any whole number of GPUs and ignores the head rule under "What the toy leaves out". The fastest count for one user is 10 on the fast link and 8 on the slow one for the 8B model, 19 and 15 for the 70B model, 36 and 28 for the 405B. Among the counts the page offers, 8 is best for the 8B model and 16 for the others on both links.
### Many words at once, 70B on 8 GPUs
Each cell is milliseconds for one step that gives every word its next word:
| Words | Fast link | Slow link |
|---|---|---|
| 1 | 11.06 | 12.02 |
| 16 | 11.17 | 13.09 |
| 64 | 11.54 | 16.53 |
Waiting takes 21.8%, 22.6% and 25.0% of the step on the fast link at 1, 16 and 64 words, and 28.0%, 33.9% and 47.7% on the slow link. The slow link costs 0.96 ms more at 1 word and 4.99 ms more at 64. Calculation takes two operations for each weight and word. For 64 words on 8 GPUs that is 3.62 ms, still under the 8.65 ms of reading, so the toy counts the reading as the busy time, and for one word on one GPU calculation takes 0.45 ms against 69.20 ms of reading. Real servers also overlap the waiting with other work, which the toy does not.
### Groups of layers
*Figure 4. One word from a 70B model on four GPUs. Slices read together; groups read in turn.*
With groups the toy has each group read in turn: 4 x 17.30 = 69.20 ms, plus 3 hand-offs of 7.2 microseconds and a few bytes each, 69.23 ms in all, 0.03% of it waiting. Only one GPU in four works at any moment. In the toy the four groups run one after another, so a word takes about as long as one GPU reading the whole model, which is why groups do nothing for one chat.
The vLLM documentation (read 2026-10-19) gives the same order of advice: if the model fits on one GPU, distributed inference is probably not needed; if it needs several GPUs in one machine, use tensor parallelism; across machines, combine tensor parallelism inside each machine with pipeline parallelism between them. Its tuning page also names pipeline parallelism for very deep and narrow models, where distributing layers is more efficient than tensor sharding.
### What the toy leaves out
- Real systems overlap communication with calculation, so some waiting hides. The toy adds the two.
- A GPU also reads its share of the chat's cache for every word. The toy reads weights only.
- NCCL picks its method by message size. The toy always uses the small-message ring.
- The A100 data sheet lists servers of up to 16 GPUs on the 600 GB/s link. A model spread over several servers meets a slower link between them. The toy keeps one link for every count, so large counts may look better here than across servers.
- Serving software usually needs the number of GPUs to divide the number of attention heads, the 32, 64 and 128 the three models have, so 19 GPUs is not a usable count. Among the counts that do divide them, the fastest are 8 for the 8B model, 16 for the 70B and 32 for the 405B, on both links. Each model has 8 key/value heads, so past 8 GPUs each GPU keeps a copy of one, which the toy's cache sum allows for.
- Start-up times vary with load. The constants are planning estimates.
- Training also splits the data across GPUs. This lesson covers only a user waiting for the next word.
### Proof it works
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m pytest -q -p no:cacheprovider code/
```
The tests check these claims. The three weight totals match the published ones, and 70B on 4 GPUs adds up by hand to 37,508,222,976 bytes. The fewest GPUs that fit are 1, 2 and 11. Reading time times the GPU count stays constant. A ring all-reduce simulated for 2, 4, 8 and 16 GPUs ends with every GPU holding the sum in 2(n - 1) steps and sends the bytes the formula charges. A word makes 64, 160 and 252 swaps. Waiting grows with the GPU count and with a slower link, and the 8B model is slower on 16 GPUs than on 8 on both links. The best counts are 10 and 8, 19 and 15, and 36 and 28; among the counts that divide the heads they are 8, 16 and 32. Layer groups stay within 2% of one GPU's read. The slow link's extra cost is more than 5 times larger at 64 words than at 1. Every number this page works out appears in the code's output; the rest come from the sources named.
The widget repeats the arithmetic in JavaScript, so its tests keep it honest. Its data block must equal what `embed_widget.py` writes; every element its script looks up must exist; it must stay under 40 KiB and make no network call. A parity test runs the widget's number code under Node and compares it with Python for all 180 settings.
What the tests do not show: that a real server's waiting follows the planning constants, or that the data sheet's link figures hold in a given machine. To check, time one all-reduce of 16,384 bytes with the NCCL tests on your own hardware.
### Where it is used
- **Megatron-LM** (Shoeybi et al., arXiv 1909.08053, 2019) splits each layer's attention by heads and its feed-forward block by columns and rows. Its forward path has two all-reduces in each layer, after the attention block and after the feed-forward block, the two per layer this lesson counts. The paper reports 77% of linear scaling when the model grew with the number of GPUs, an 8.3-billion-parameter model on 8-way model parallelism against a 1.2-billion-parameter model on one GPU, in training, not in serving.
- **The NCCL tests** (NVIDIA, performance notes) give the all-reduce time used above.
- **vLLM** (documentation above) offers the tensor and pipeline settings the lesson describes.
## Skills this lesson builds
- Add up the memory one GPU needs for its share of the weights, the chat's cache and working space.
- Count the swaps a word needs from a model's layers, and the bytes in each.
- Estimate a swap's time from its start-up and its bytes, and read the waiting share from it.
- Explain why a split stops paying off, and what a faster link changes.
- Tell tensor parallelism from pipeline parallelism, and say what each does for one user.
**Portfolio project:** write a script that takes a model shape, a GPU's memory and a link's figures, and prints the fewest GPUs, the memory on each, and the time and waiting share for one word at 1, 2, 4, 8 and 16 GPUs. Add a chart and a test that checks one setting by hand, then compare the waiting with a timed all-reduce on hardware you own.
## Check yourself
1. Why can the 70B model not run on one GPU with 80 GiB of memory?
Answer
Its weights take 131.42 GiB at 16 bits, and one GPU can use 73.6 GiB. With two GPUs each holds about 65.71 GiB of weights, and with the chat's cache and working space 68.41 GiB, which fits.
Answer
To waiting. The GPUs swap partial results twice in each of the 80 layers, 160 swaps for each word, and each swap costs about 10 microseconds of start-up. That adds 1.64 ms.
Answer
Two swaps in each layer: 252 for each word. Each message is 16,384 x 2 = 32,768 bytes.
Answer
Doubling the GPUs halves the reading, but the waiting grows. In the toy the 70B model gets 1.34 times faster, from 11.06 to 8.27 ms a word on the fast link, and the 8B model gets slower, from 514 to 483 words a second.
Answer
They help a model fit across GPUs with little waiting, and they pay off when many chats keep every group busy. They do not shorten one user's wait: the 70B model takes 69.23 ms a word on four groups, against 69.20 ms of reading alone, with three GPUs in four idle at any moment.