---
title: "How much memory does an AI model need?"
description: "An AI model needs memory for its numbers, for notes on every chat, and for scratch space. Add the parts to see whether a model fits on a graphics card."
topic: "AI systems"
chapter: "inside-the-chip"
order: 18
minutes: { plain: 6, deep: 18 }
prerequisites: ["What the attention cache is (the notes a model keeps on a chat)", "Reading short Python functions", "Bytes, KiB, MiB and GiB (powers of 1,024)"]
seo_title: "How much GPU memory does an LLM need?"
date: 2026-10-19
---
# How much memory does an AI model need?
Llama 3.1 8B is an open AI model with 8,030,261,248 stored numbers, called weights. At 2 bytes each (a byte is the unit memory is counted in), they fill 14.96 GiB of the memory of a graphics chip (GPU) before the model reads a word. A GiB is 1,024 x 1,024 x 1,024 bytes. Let four people chat with it at once, each in a conversation of 8,192 tokens (a token is a chunk of text, often a short word), and the total reaches 20.18 GiB.
## In one minute
A model's memory is its weights times the bytes each one takes, plus working memory that grows with the length of each chat and the number of chats at once. Think of a workshop: the weights are machines bolted to the floor, taking the same room whether one person or fifty walk in, and each chat is a table of notes. Those notes are the attention cache (the KV cache; see [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/)): the record of every earlier word that saves the model from redoing its work. Two small parts finish the bill: scratch space and a fixed allowance for the program that runs the model. If the total is more than the graphics card holds, shrink the weights, cut the chats, or add cards.
## Explore the numbers
[Open the widget](https://spicyshark.co/learn/model-memory/widget.html). It shows the four parts as one bar and checks the total against five graphics cards. It opens on the 8B model at 16 bits (2 bytes each) with 8 chats, which does not fit the 24 GiB card. Drag the chats slider one step left, or switch to 8 bits (1 byte each), and watch it fit. Then pick the 70B model: at 16 bits, no card in the list holds it alone.
## How it works
The bill has four parts. Figure 1 shows one example to scale.
*Figure 1. Llama 3.1 8B in 16 bits with 4 chats of 8,192 tokens each. The last two parts are estimates.*
### Weights: the part you pay before anyone arrives
Weights (also called parameters) are set by training; afterwards they are only read. The "8B" in Llama 3.1 8B means about 8 billion. Each weight takes a fixed number of bytes, set by its precision, the number of bits used to store it (a bit is one 0 or 1; 8 bits make 1 byte). The usual choices are 16 bits (2 bytes), 8 bits (1 byte) and 4 bits (half a byte). Storing weights in fewer bits is called quantization; it saves memory at some cost in answer quality.
For Llama 3.1 8B the weights take 14.96 GiB at 16 bits, 8.46 GiB at 8 bits and 5.21 GiB at 4 bits. That is not half and a quarter, because about 1 billion weights, the token table (one row of numbers per token) and the output matrix (which scores the next token), stay at 16 bits in published files. Real 4-bit files also store scale values, so they run a few percent larger: the published 4-bit file of this model is 5.33 GiB.
*Figure 2. The weights of Llama 3.1 8B at three precisions.*
### The attention cache: a bill for every chat
To write a reply, a model must look back at every earlier word of the chat. Instead of redoing that work for each new word, it keeps notes on the earlier words: the attention cache. For Llama 3.1 8B the notes cost 131,072 bytes (128 KiB, where a KiB is 1,024 bytes) per token. A chat of 8,192 tokens therefore holds 1.00 GiB of notes, and one of 131,072 tokens holds 16.00 GiB.
Everyone shares the weights, but each chat has its own notes. Four chats of 8,192 tokens need 4.00 GiB, and thirty-two need 32.00 GiB, more than twice the weights.
*Figure 3. The weights stay put; each chat of 8,192 tokens adds 1.00 GiB of cache.*
### Working memory and a fixed allowance
While it calculates, the card also holds temporary results, called activations. A rough estimate for the 8B model, handling 2,048 tokens at once, is 0.23 GiB. The program that runs the model needs room too; this lesson sets aside 1.00 GiB as an example. Both are small next to the weights.
### Does it fit?
A card does not give all its memory away. vLLM, a free program that runs open models, claims 92% of a card by default, so a 24 GiB card offers 22.08 GiB. The five cards are the NVIDIA RTX 4090, RTX 5090, L40S, H100 and H200, with 24, 32, 45, 80 and 141 GiB. NVIDIA's pages say "GB". Card memory is built in binary sizes, so the nominal GB is close to GiB. The L40S is the exception: with error-correcting memory (ECC) on, which takes 6.25% of its 48 GB, it offers 45 GiB. If the total is within the usable share, the model runs on one card. If not, it can be split across several cards, at the cost of passing results between them. Figure 4 shows how many cards one chat needs.
*Figure 4. Cards of 24, 32, 45, 80 and 141 GiB needed for one chat of 8,192 tokens, split evenly.*
## What this means when you build
- **Add up the parts before you rent a GPU.** Take the weight count from the model's Hugging Face page (a site that hosts open models), multiply by the bytes per weight (a little more for 4-bit files), then add the cache: your longest chat times the chats you expect, with sizes from the model's `config.json` file.
- **Do not trust "weights plus 20%".** EleutherAI's Transformer Math 101 puts running a model at about 1.2 times the weights. That is close for one chat. For 32 chats of 8,192 tokens on the 8B model the real total is 3.22 times.
- **Fewer bits per weight is a cheap lever.** The 70B model needs 131.42 GiB at 16 bits, which no card in the list holds. At 8 bits it needs 67.67 GiB, which fits one 80 GiB card with room for 1 chat of 8,192 tokens. Test the smaller version's quality on your own task first.
- **Cap chat length and chats at once.** The cache grows in a straight line with both, so each limit is a memory limit. On a 24 GiB card, the 8B model at 16 bits holds 5 chats of 8,192 tokens.
- **Fitting is not speed.** A model that fits can still answer slowly, because every new word makes the chip read all the weights again. [Why a fast AI chip spends most of its time waiting](https://spicyshark.co/learn/tiled-matmul-roofline/) explains that limit. Renting a model through an API (a way for your program to call someone else's service) hides this bill: the provider carries it.
## Go deeper
*Optional. For readers who write Python. The code is in the `code/` folder; every number below was printed by `python3 show_numbers.py`, worked out by hand, or comes from a cited source read on 2026-10-19.*
### The naive approach: weights plus 20%
Two popular guides give a shortcut for "will this model fit for inference". EleutherAI's Transformer Math 101 puts inference at about 1.2 times the model's memory, and Hugging Face's Accelerate documentation says to expect up to 20% on top of its load estimate, crediting EleutherAI. As code, `params * bits // 8 * 1.2`.
For the 8B model at 16 bits that gives 17.95 GiB, whatever the number of chats. The full sum, below, disagrees as soon as the cache grows:
| chats x tokens | sum | x weights |
|---|---|---|
| 1 x 8,192 | 17.18 GiB | 1.15 |
| 4 x 8,192 | 20.18 GiB | 1.35 |
| 32 x 8,192 | 48.18 GiB | 3.22 |
| 1 x 131,072 | 32.18 GiB | 2.15 |
Against the rule's 17.95 GiB, the sum is 0.76 GiB lower in the first row and 2.24, 30.24 and 14.24 GiB higher in the others (from unrounded values; 48.18 minus 17.95 reads 30.23). The rule treats everything beyond the weights as a flat 20%, but the cache grows with tokens times chats: for 32 chats the rule says 17.95 GiB, so a 24 GiB card looks fine, while the sum says 48.18 GiB, which fits only the 80 GiB and 141 GiB cards.
### The sum, part by part
```text
embeddings = vocab x hidden x (1 if tied else 2) (kept at 16 bits)
weights = embeddings x 2 bytes
+ (parameters - embeddings) x bits / 8
cache = 2 x layers x kv_heads x head_dim
x 2 bytes x tokens x chats
working memory = 2,048 tokens x (3 x ffn + 4 x hidden)
x 2 bytes (rough estimate)
fixed = 1 GiB (example allowance)
usable = card memory x 0.92
fits = weights + cache + working memory + fixed
<= usable
```
The cache line is the one from [Why does a chatbot forget?](https://spicyshark.co/learn/why-chatbots-forget/): a 2 for one key and one value, then the layers, the key/value heads, the head size and 2 bytes per value. All four Llama models here share 8 key/value heads among 32 or more query heads (grouped-query attention), which shrinks the cache. It counts only the bytes the notes need; how a real server lays them out in memory, and how much that wastes, is the subject of [Why AI servers waste memory, and how paging fixes it](https://spicyshark.co/learn/paged-kv-cache/).
The weight count comes from the model's shape. Write `h` for the hidden size (the width of the vectors the model passes between layers) and `ffn` for the width of its feed-forward block. Per layer there are four attention matrices (query and output, each h x h; key and value, each h x kv_heads x head_dim), three feed-forward matrices of h x ffn, and two norm vectors of length h. Around the layers sit the token table (vocab x h), an output matrix of the same size unless the model reuses the table (called tied), and a final norm of length h. This matches the total that Hugging Face lists for each of the four models exactly: 1,235,814,400 for the 1B, 8,030,261,248 for the 8B, 70,553,706,496 for the 70B and 405,853,388,800 for the 405B.
The working memory is only a rough estimate. The large tensors one layer holds while it works are the feed-forward block's outputs (3 x ffn per token) and a few copies of size h, multiplied by the tokens processed at once. The 2,048 tokens and the 1 GiB are examples, not measurements; servers that must be exact measure instead, as the last section shows.
### Quantized weights, checked against published files
Quantization changes most of the weights, not all. The token table and the output matrix, together called the embeddings, are the first and last steps of the model, and the published copies below leave them at 16 bits. The formula does the same, counting them once when the model ties them. For the 8B model they are 1,050,673,152 weights, 13.1% of the total; for the 1B model, which ties them, 21.3%; for the 70B model, 3.0%.
To test this, here are published copies of Llama 3.1 models. The sizes are the totals of each repository's `.safetensors` weight files on Hugging Face, in GiB (read 2026-10-19). FP8 is an 8-bit number format, and AWQ and GPTQ are two common methods for 4-bit weights. In the table, "file" is the published size, "here" is the calculator's, and "gap" is the share of the file that the calculator misses.
| copy | file | here | gap |
|---|---|---|---|
| 8B FP8 | 8.46 | 8.46 | 0.0% |
| 8B AWQ | 5.33 | 5.21 | 2.4% |
| 8B GPTQ | 5.34 | 5.21 | 2.5% |
| 70B AWQ | 37.04 | 35.79 | 3.4% |
The 16-bit file of the 8B (NousResearch's copy of the Meta weights) is 14.96 GiB, the same as the calculator. The simpler formula, parameters x bits / 8, would be short by 11.6% for the FP8 file and about 30% for the 8B 4-bit files. The model page of the 8B AWQ copy lists 1,050,939,392 parameters as 16-bit and 6,979,321,856 as packed 4-bit integers; the 16-bit count is the embeddings plus 266,240, the norm vectors ((2 x 32 + 1) x 4,096). The 70B page agrees: 1,318,912 above its embeddings, which is (2 x 80 + 1) x 8,192.
The rest of the 4-bit gap is the scale values. The AWQ config files say `group_size` 128 with `zero_point` on: each group of 128 weights also stores one 16-bit scale and one 4-bit zero point (the numbers that turn stored integers back into weights), 2.5 extra bytes. Adding them to the tensors on the page gives 5,727,854,592 bytes for the 8B against a file of 5,727,938,576, and 39,767,785,472 against 39,767,996,256 for the 70B, within 0.002% both times. The calculator leaves them out on purpose, so it runs 2% to 4% under a 4-bit file.
### A hand-worked example
An invented tiny model (not a real one): 2 layers, hidden size 8, ffn 16, 4 heads, 2 key/value heads, head size 2 (hidden size divided by heads), vocabulary of 10 tokens, output matrix not tied.
1. Attention, per layer: 2 x 8 x 8 (query, output) + 2 x 8 x (2 x 2) (key, value) = 192.
2. Feed-forward, per layer: 3 x 8 x 16 = 384.
3. Norms, per layer: 2 x 8 = 16.
4. One layer: 192 + 384 + 16 = 592. Two layers: 1,184.
5. Other weights: token table 10 x 8, output matrix 10 x 8, final norm 8: 168.
6. Weight count: 1,184 + 168 = 1,352, which is 1,352 x 2 = 2,704 bytes at 16 bits.
7. Cache per token: 2 x 2 layers x 2 key/value heads x 2 head size x 2 bytes = 32 bytes. For 3 chats of 100 tokens: 32 x 100 x 3 = 9,600 bytes.
8. Working memory, 10 tokens: 10 x (3 x 16 + 4 x 8) x 2 bytes = 1,600 bytes.
9. Total, no allowance: 2,704 + 9,600 + 1,600 = 13,904 bytes.
Even in this toy, the cache of 3 chats (9,600 bytes) outgrows the weights (2,704 bytes), as in Figure 3.
### The real implementation
`weight_bytes` and `memory_need` from `code/memory_calc.py` (docstrings left out; `embedding_params` is the embeddings line of the formula above):
```python
def weight_bytes(cfg: dict, bits: int) -> int:
if bits not in BITS:
raise ValueError(f"bits must be one of {BITS}")
embeddings = embedding_params(cfg)
return embeddings * FULL_BITS // 8 + (param_count(cfg) - embeddings) * bits // 8
def memory_need(key: str, bits: int, tokens: int, chats: int) -> dict:
cfg = load_config(key)
parts = {
"weights": weight_bytes(cfg, bits),
"kv_cache": kv_cache_bytes(cfg, tokens, chats),
"activations": activation_bytes(cfg),
"overhead": overhead_bytes(),
}
parts["total"] = sum(parts.values())
return parts
```
The fit test is two lines more: `usable_bytes` is `int(gpu_gib * GIB * 0.92)`, `fits` is `total <= usable_bytes(gpu_gib)`, and `cards_needed` is `ceil(total / usable_bytes(gpu_gib))`. The shapes come from saved public files: the `config.json` of Llama 3.2 1B, Llama 3.1 8B and 70B on Hugging Face, and Table 3 of the Llama 3 paper for the 405B, whose config file needs a login (all read 2026-10-19). The results, in GiB, with the embeddings kept at 16 bits:
| model | 16 bits | 8 bits | 4 bits |
|---|---|---|---|
| 1B | 2.30 | 1.40 | 0.94 |
| 8B | 14.96 | 8.46 | 5.21 |
| 70B | 131.42 | 67.67 | 35.79 |
| 405B | 755.96 | 381.89 | 194.86 |
The cache per token is 32 KiB for the 1B, 128 KiB for the 8B, 320 KiB for the 70B and 504 KiB for the 405B.
The sum leaves out things a real server also pays for, and it takes four assumptions on trust: card sizes read as GiB, an even split across cards, a cache stored in 16 bits, and embeddings kept at 16 bits (the files above support the last).
### Proof it works
```bash
python3 -B -m pytest -q -p no:cacheprovider code/
```
The calculator tests check these claims. The counted weights equal the published total for each of the four models, and a tied model counts its token table once. The weights at each precision, the cache per token and the tiny model of the list above match values worked out on paper. The calculator stays at or just under the published 8-bit and 4-bit file sizes, by less than 4% for the two models in the table. The total is the sum of its parts. A total exactly at 92% of a card fits, one byte more does not, and the card count rounds up.
The widget is a separate copy of the arithmetic in JavaScript, so the widget tests keep it honest. Its data block must equal what `embed_widget.py` writes from the Python constants; every element its script looks up must exist; it must stay under 40 KiB and make no network call. Last, a parity test runs the widget's number code under Node and compares it with Python for all 756 combinations (4 models, 3 precisions, 9 chat lengths, 7 chat counts) on all 5 cards, plus 25 totals placed at each card's limit, one byte either side, and at twice the limit. Against deliberately broken copies (a wrong GiB, a strict comparison at the limit, a usable share rounded up, a tied model counted as untied, embeddings quantized), it failed each time.
What the tests do not show: that the working-memory estimate and the 1 GiB allowance match a real server. They are examples. To check them, start a model in vLLM on a real card and compare its start-up log with this sum.
### Where it is used
- **vLLM.** It works the same sum backwards. By default it requests 0.92 of each card (the `gpu_memory_utilization` setting), loads the weights, runs a test pass to measure the peak working memory, allows for captured CUDA graphs (a way to speed up repeated GPU work), and gives the attention cache what is left. Its log reports that figure as "Available KV cache memory". Sources: vLLM engine arguments, conserving-memory documentation and the `gpu_worker.py` source (read 2026-10-19).
- **Hugging Face Accelerate.** Its model memory estimator reports the memory to load a model at several precisions, and says inference can add up to 20% on top.
- **EleutherAI, Transformer Math 101.** The source of the 1.2 times rule, with the overhead put at 20% or less, from experience.
- **The Llama 3 paper.** Table 3 lists the layers, hidden size, feed-forward size and key/value heads that make the 405B shape above.
## Skills this lesson builds
- Count a model's weights from its published shape and check the count against the model page and against published file sizes.
- Convert between weight count, bits per weight, bytes and GiB, and keep GB and GiB apart.
- Compute the attention cache from a model's config and scale it with tokens and chats.
- Turn the parts into a fit test with a usable share of the card, count the cards needed, and say what the estimate leaves out.
- Check a second implementation (here, JavaScript) against a reference over every input with a parity test.
**Portfolio project:** build a command-line "will it fit" tool. It reads a `config.json` from a model page and a card size, prints the four parts and the total, and says how many chats of a given length fit. Then load a small open model on hardware you own, read the memory your framework reports, and publish a table of estimate against measurement. Mark which parts of your estimate are guesses.
## Check yourself
1. Llama 3.1 8B has about 8 billion weights. Roughly how much memory do its weights need at 16 bits and at 4 bits, and why is that not the whole bill?
Answer
At 16 bits each weight takes 2 bytes, so about 16 billion bytes, which is 14.96 GiB. At 4 bits each takes half a byte, which would give 3.74 GiB. But the token table and output matrix stay at 16 bits, so the lesson's figure is 5.21 GiB. Neither is the whole bill: it also holds the attention cache for every chat, working memory for the calculation under way, and a fixed allowance for the program that runs the model.
Answer
The attention cache. The weights are shared, but each chat keeps its own notes. For 32 chats of 8,192 tokens the cache is 32.00 GiB, against 14.96 GiB of weights, and the total is 48.18 GiB, which fits only the 80 GiB and 141 GiB cards.
Answer
2 x 16 x 8 x 64 x 2 = 32,768 bytes (32 KiB) per token. Two chats of 4,096 tokens are 8,192 tokens in all, so 32,768 x 8,192 = 268,435,456 bytes, which is 256 MiB or 0.25 GiB.
Answer
Usable memory is 24 x 0.92 = 22.08 GiB. Take away 8.46 + 0.23 + 1.00 = 9.69 GiB, leaving 12.39 GiB. Each chat takes 1.00 GiB, so 12 chats fit (the code prints 12).
Answer
The rule adds a flat 20%, but the cache grows with the number of chats times the tokens in each. For 32 chats the cache alone is 32.00 GiB, more than twice the weights, and the full sum is 48.18 GiB: about 30.2 GiB more than the rule says.