Glossary

The name for taking a model's refusal direction out, either from its vectors as it runs or by cutting it from its weights, shown in the lesson on a toy network only.

Taught in Can a model's refusals be removed?, in the section "Where this is used in practice".

The temporary results a model holds while it calculates one step; they take working memory on the chip, a small part next to the weights and the attention cache.

Taught in How much memory does an AI model need?, in the section "Working memory and a fixed allowance".

A swap in which every GPU sends its partial result to the others and gets back the total, so all of them end with the same sum; a model split into slices does it twice in each layer for every word.

Taught in Why does a big model need several GPUs?, in the section "Waiting for each other".

A short list of the only values the code will accept, such as the addresses an app may send to; a plain comparison in code that does not read the text the model saw.

Taught in What is prompt injection, and can a prompt stop it?, in the section "Defences made of limits".

A secret string that tells a service whose account to charge; whoever holds it can spend that account's money, so it belongs on a server, never in a web page or app.

Taught in Why an AI API key must never go in a web page or app, in the section "A key is a password that spends money".

The number of operations a program does for each byte it fetches from memory; a chip stays busy only if this number is high enough.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Two speed limits".

Each token takes a weighted average of the earlier tokens' value vectors; the weights come from how well its query matches their keys.

Taught in What does attention do inside an AI model?, in the section "Looking back at every word".

One way of looking in an attention layer. Each head has its own query table, and its key and value tables can be shared by several heads (grouped-query attention). A layer runs many heads side by side and joins their results.

Taught in What does attention do inside an AI model?, in the section "Several heads".

How common the harmful kind of message is in your traffic, such as 1 in 100; when it is rare, even a low false-alarm rate can mean that most blocked messages are harmless.

Taught in How does a safety filter decide, and where does it fail?, in the section "When harm is rare".

In serving, a group of requests from many users that a server runs together, so each weight fetched from memory serves all of them.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Why a chatbot waits".

The part of a model server that picks which waiting chats join the running group and then steps the whole group one word together, so one pass over the model serves many chats.

Taught in Build a tiny model server you can read end to end, in the section "The picture, step by step".

A short list, kept for each chat, of the memory blocks that hold its notes, in order.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "The idea: a block table".

The point where two ways of doing the same job cost the same, in money or in time: on one side the first is cheaper, on the other the second. For renting a pay-per-token API against running your own GPUs, it is the monthly token count where owning first costs no more than renting.

Taught in Should I rent a model API or run my own?, in the section "The picture, step by step".

The replies your GPUs served in a month divided by the replies they could have finished; every paid hour counts, busy or idle, so a low busy share raises the cost per token of running your own.

Taught in Should I rent a model API or run my own?, in the section "The picture, step by step".

A way to build a tokenizer's table: start from single bytes, then again and again give the most frequent pair of neighbouring ids in a training text a new id and replace that pair everywhere.

Taught in What is a token, and why is AI priced per token?, in the section "The picture, step by step".

The label a cache files an answer under and looks it up by; if the key leaves out who asked, a request from one user can find an answer stored for another.

Taught in Can one user's chat leak into another's?, in the section "A shelf that forgets who asked".

A value, often the user or account, mixed into a cache key so that only requests with the same value can reuse an entry.

Taught in Can one user's chat leak into another's?, in the section "A wait can give a cache away".

The rule that a token looks only at itself and the tokens before it: the scores of later tokens are set to minus infinity before softmax, so their weight is 0.

Taught in What does attention do inside an AI model?, in the section "Scores, then weights".

Limited by calculation speed rather than memory speed: the chip is busy, so faster calculators would help.

Taught in Why a fast AI chip spends most of its time waiting, in the section "The roofline".

The most text, counted in tokens, a model can handle at once: the prompt and its reply together.

Taught in Why does a chatbot forget?, in the section "The window has a fixed size".

A way of running a server in which a waiting request joins the running group as soon as there is room and a reply leaves the moment it ends, so room is not wasted waiting for a whole group to finish.

Taught in How does an AI server answer many people at once?, in the section "Fixed groups waste places".

A flaw in a speed test: a tester that waits for each reply before it sends the next request sends fewer requests while the server is stuck, so it records too few of the slow ones.

Taught in Why is the average speed of an AI service a misleading number?, in the section "Two ways to send test requests".

A rule for sharing a block of memory: many chats may read it, but a chat that must add to it first makes a private copy and writes there.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "Sharing is cheap".

Writing a reply one token at a time; each new token needs a fresh trip through the model's weights, so decode is memory-bound at small batch sizes.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Why a chatbot waits".

Taking one direction out of every internal vector as a model runs, by subtracting from each vector its part along that direction.

Taught in Can a model's refusals be removed?, in the section "The picture, step by step".

The small, cheap model in speculative decoding that writes the guesses; it must use the same tokens as the big model, and it pays off when its pass costs little and its guesses are often kept.

Taught in How can a small model speed up a big one?, in the section "Guess ahead, then check".

A retry rule in which the wait grows after each refusal, usually doubling up to a longest wait, so a client calls less often the longer a service stays busy.

Taught in What 'too many requests' means, and what your app should do, in the section "A toy to test retry rules".

Free memory split into gaps between reserved stretches, each too short for a new request even when the free total is large.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "Why one big buffer wastes it".

A harmless message that a safety filter blocks; the false-alarm rate is the share of harmless messages blocked.

Taught in How does a safety filter decide, and where does it fail?, in the section "Four things can happen".

An exact way to compute attention on a GPU in tiles, so the full table of scores is never written to main GPU memory.

Taught in How FlashAttention avoids storing the giant score table, in the section "In one minute".

One floating-point addition or multiplication; a multiply-add counts as 2 FLOPs.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Counting work and traffic".

One run of the model that adds one token to the text: it reads the text so far, gives each possible next token a chance, and one is picked; also called a pass.

Taught in Why do answers appear word by word?, in the section "The model makes one token at a time".

The share of requests that meet every speed target set for them, such as a first word within a set time and no gap between words over a set limit; a request that misses either target does not count.

Taught in Why split reading the prompt from writing the reply?, in the section "The picture, step by step".

A graphics processor, the kind of chip that runs most AI models; how fast it calculates and how fast its memory delivers data are two separate limits.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Two speed limits".

A design in which several query heads share one key/value head, which makes the cache smaller.

Taught in Why does a chatbot forget?, in the section "How much memory the cache takes".

An answer from an AI model that sounds right but is false, such as a study, a quote or a number that does not exist.

Taught in Why does AI make things up?, in the section "Where the text says little, a fluent answer can be wrong".

High-bandwidth memory: a GPU's large main memory, slower than its small on-chip memory (SRAM). Consumer GPUs usually use GDDR memory in this role instead.

Taught in How FlashAttention avoids storing the giant score table, in the section "Why reading once matters".

Capacity kept unused on purpose, such as filling each GPU to only 80% of what it can carry, so random bursts of requests do not push users past the speed target.

Taught in How many GPUs do I need for 1,000 users?, in the section "Count the GPUs".

The status code a service sends when you have passed a limit on your calls ("Too Many Requests"). For a temporary rate limit the reply may say how long to wait before you try again; a 429 that says a quota or spend cap is used up will not clear within a retry loop.

Taught in What 'too many requests' means, and what your app should do, in the section "Why a service says no".

Prompt injection in which the hidden text sits in something the app fetches, such as a page, an email or a file, so the person using the app may never see it.

Taught in What is prompt injection, and can a prompt stop it?, in the section "Two ways in".

Everything an app sends to an AI model in one call, including the system prompt and the earlier chat, which the service bills at its input price.

Taught in Why is my AI app bill higher than I expected?, in the section "What the bill counts".

Memory reserved for a request but left empty inside its own buffer, where no other request can use it.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "Why one big buffer wastes it".

A random part added to a wait, so clients that were refused together do not all come back at the same moment.

Taught in What 'too many requests' means, and what your app should do, in the section "Why a random wait helps".

A function that runs on a GPU; one piece of GPU work.

Taught in How FlashAttention avoids storing the giant score table, in the section "From softmax to attention".

The working notes, a key and a value for each earlier token, that a server keeps in GPU memory so it does not recompute them for every new token.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "The notes pile up".

The chats in progress on average equal the new chats a second times the seconds each one stays, from the moment it joins a batch to its last word; it lets you size a service for its busiest hour from the arrival rate and that time, not from the number of users.

Taught in How many GPUs do I need for 1,000 users?, in the section "Size for the busiest hour".

The fraction of a GPU's memory that a model server may use; it fills that part with the weights, a working reserve and the cache pool, and leaves the rest alone (vLLM calls it gpu_memory_utilization).

Taught in What do a model server's settings change?, in the section "One floor, four parts".

Limited by memory speed rather than calculation speed: the chip waits for data, so more work per byte would help but faster calculators would not.

Taught in Why a fast AI chip spends most of its time waiting, in the section "The roofline".

The share of harmful messages that a safety filter lets through, such as 5 missed out of 40 harmful ones, which is 12.5%.

Taught in How does a safety filter decide, and where does it fail?, in the section "Four things can happen".

The stored numbers a model learned in training; their memory is the count times the bytes each one takes, the same for one chat or fifty.

Taught in How much memory does an AI model need?, in the section "Weights: the part you pay before anyone arrives".

A way to find softmax's highest score and running total in one read of the list, by rescaling the total whenever a new highest score arrives.

Taught in How FlashAttention avoids storing the giant score table, in the section "One read and a running tally".

Everything an AI model writes back in one call, billed at its own price, usually higher than the input price; hidden reasoning tokens count here too.

Taught in Why is my AI app bill higher than I expected?, in the section "What the bill counts".

In floating-point arithmetic, a number too big for its format to store: the computer records infinity, and results built from it can turn into NaN (not a number). Fixed-size integers wrap around instead.

Taught in How FlashAttention avoids storing the giant score table, in the section "Big scores break the plain way".

A way to keep each chat's working notes in small, equal blocks handed out only when needed, instead of one big stretch reserved up front.

Taught in Why AI servers waste memory, and how paging fixes it, in the section "Small blocks fix both".

The share of test questions whose answers pass their check, such as 12 of 20 or 60%; with few questions one question moves it a lot, so read the fixed and broken lists and a range, not the rate alone.

Taught in How do I check my AI's answers with a small test set?, in the section "A fixed list of questions".

A cap on the money (or model tokens) that one signed-in account may use in a period, checked by the server before each call; unlike a request limit, it counts what calls cost and not how many there are.

Taught in How to stop one user running up your AI bill, in the section "Count money, per signed-in user".

A cut in the list of all measured times, sorted from fastest to slowest: the p95 is the time that 95 of every 100 requests beat, so only 1 in 20 is slower; the p50 is the middle one.

Taught in Why is the average speed of an AI service a misleading number?, in the section "In one minute".

A score of how surprised a language model is by real text it was not trained on: lower is better, and a rise after rounding the weights is one way to measure what the rounding cost.

Taught in What does quantization trade away?, in the section "What it costs in quality".

Python's way of saving an object as a list of steps that rebuild it; the steps can import and call any function Python can find, so opening a pickle from a stranger can run code.

Taught in Is it safe to download a model file?, in the section "What a pickle can do while it opens".

Splitting a model into groups of whole layers, one group for each GPU; it makes a big model fit, but one user's word still passes through the groups one after another.

Taught in Why does a big model need several GPUs?, in the section "Another way to split: groups of layers".

Pausing a running chat when the cache pool is full, so its blocks can go to others; the chat is rebuilt or resumed later, which costs extra work and delay.

Taught in What do a model server's settings change?, in the section "More chats at once helps, then stops".

Reading the prompt before the reply starts; the model handles all the prompt's words together, so each weight it fetches serves many words.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Why a chatbot waits".

Running the two jobs of a reply on separate GPUs, one that reads prompts and one that writes replies, and sending each prompt's cache from the first to the second; also called disaggregated serving.

Taught in Why split reading the prompt from writing the reply?, in the section "The picture, step by step".

The opening tokens of a prompt; two prompts share a prefix when they start with the same tokens. Prompt caching reuses a stored start only when the new request begins with all of it, so a shared prefix alone is not enough.

Taught in What is prompt caching, and when does it save money?, in the section "A stored start is reused whole or not at all".

Keeping the KV cache of a prompt's opening text, up to a mark or the end of a request, so a later request that begins with all of that text reads it back instead of computing it again; providers bill that reused input at a lower rate.

Taught in What is prompt caching, and when does it save money?, in the section "What the server keeps".

Text hidden in something an AI app reads, such as a web page or an email, that the model takes for instructions; a line in the prompt cannot reliably stop it, so limit what the app can do to bound the harm.

Taught in What is prompt injection, and can a prompt stop it?, in the section "One stream of text".

Storing numbers in fewer bytes, usually the weights: 1 byte instead of 2, for example, halves what must be read for the numbers quantized, at some cost in quality.

Taught in What does quantization trade away?, in the section "Smaller and faster".

A run of weights, often 128, that share one quantization scale, so a very large weight stretches the rounding steps of its own group only.

Taught in What does quantization trade away?, in the section "Why scales come in groups".

The stored number that sets the step between neighbouring rounded values in a quantization group: each weight is kept as a small whole number, and multiplying it by the scale gives back the approximate weight.

Taught in What does quantization trade away?, in the section "Rounding to a grid".

The three short lists attention makes from each token: the query is what the token looks for, the key is the label it holds up, and the value is the note it hands over.

Taught in What does attention do inside an AI model?, in the section "Query, key and value".

A cap on how often one client may call a service; a request over the cap is refused with the status 429 until enough time has passed.

Taught in Why an AI API key must never go in a web page or app, in the section "Limits only a server can add".

The tokens a reasoning model writes to work out an answer before it replies; they are often not shown, yet they are billed as output tokens.

Taught in Why is my AI app bill higher than I expected?, in the section "Hidden reasoning is billed as output".

The direction in a model's internal numbers that carries its refusals, found by subtracting the average vector of plain requests from the average vector of requests the model refuses.

Taught in Can a model's refusals be removed?, in the section "The picture, step by step".

A way to enforce a budget: before a call, set aside the most it can cost and refuse it if that does not fit; after the reply, charge the real cost and give back the rest, so calls still running are counted too.

Taught in How to stop one user running up your AI bill, in the section "Set aside first, then charge".

A cache that keeps whole answers, so that a repeated request skips the model; it is safe to share only for answers that are the same for everyone.

Taught in Can one user's chat leak into another's?, in the section "A shelf that forgets who asked".

A header a service may add to a reply to say when to try again; a program that sees it should wait at least that long.

Taught in What 'too many requests' means, and what your app should do, in the section "Why a service says no".

A plot of a chip's speed against the work done per byte fetched: a slope that climbs, then a flat roof.

Taught in Why a fast AI chip spends most of its time waiting, in the section "The roofline".

Quantizing each number to the closest step of the grid, with no search for a better choice; it is the plain baseline that smarter methods are compared with.

Taught in What does quantization trade away?, in the section "Rounding to a grid".

A model file format made of an 8-byte header length, a text header that names each table of numbers and says where its bytes sit, and then the numbers; it holds no steps to follow, so by design opening one runs no code.

Taught in Is it safe to download a model file?, in the section "Files that hold only numbers".

Drawing the next token at random by its chance, after the model has given a chance to every possible token; it is why the same question can give different answers.

Taught in Why does the same question give different answers?, in the section "The model gives chances and the app draws one".

The starting number of the generator that makes the random numbers for a draw; the same seed gives the same numbers, so the same draws.

Taught in Why does the same question give different answers?, in the section "A seed makes a run repeatable".

The few numbers a model server keeps about itself while it runs, such as the wait for the first word, the gap between words, the queue length and how full the cache is, read as percentiles.

Taught in Build a tiny model server you can read end to end, in the section "The picture, step by step".

A way to fit a long chat into the context window by keeping the newest messages that fit and dropping the oldest first.

Taught in Why does a chatbot forget?, in the section "A sliding window by hand".

A function that turns a list of scores into positive shares that add up to 1: each score is exponentiated and divided by the sum of all of them, so the biggest score gets the biggest share. In attention it turns scores into weights.

Taught in How FlashAttention avoids storing the giant score table, in the section "Scores become shares".

A way to write a reply faster: a small draft model guesses the next few tokens and the big model checks all of them in one pass, keeping the guesses up to the first one it would not have written and using its own token there, so the big model's text, or its odds when it picks at random, is unchanged.

Taught in How can a small model speed up a big one?, in the section "Guess ahead, then check".

Static random-access memory: a GPU's small, fast on-chip memory.

Taught in How FlashAttention avoids storing the giant score table, in the section "Why reading once matters".

Showing a reply token by token as it is made, instead of waiting for the whole reply; the server sends many small messages and the page adds each one to the text.

Taught in Why do answers appear word by word?, in the section "Why apps show each token as it arrives".

How long the slowest few requests wait, such as the slowest 1 in 20 or 1 in 100; an average hides it, and a high percentile such as p99 shows it.

Taught in Why is the average speed of an AI service a misleading number?, in the section "Many fast, a few slow".

A setting that controls how adventurous a model's word choice is: low makes it take its top choice almost every time, high lets in more varied words, and neither makes an answer more true.

Taught in Why does AI make things up?, in the section "Why a lower temperature does not make facts true".

Splitting every layer of a model into slices, one for each GPU, so the GPUs read their shares of the weights at the same time and swap partial results.

Taught in Why does a big model need several GPUs?, in the section "Slicing every layer".

A fixed list of questions, each with the right answer written down before you change anything; you run all of them after every change to the prompt or the model and compare the results question by question. In machine learning the term usually means data held back from tuning; here it is the list you rerun after each change.

Taught in How do I check my AI's answers with a small test set?, in the section "A fixed list of questions".

The score at which a safety filter starts to block: a message scoring at or above it is blocked and one below it passes; raising it blocks fewer messages, so it misses more harmful ones and blocks fewer harmless ones.

Taught in How does a safety filter decide, and where does it fail?, in the section "A score and a threshold".

A small, often square, block cut from a matrix; the chip keeps a few tiles in fast memory and finishes all the work they allow before fetching more.

Taught in Why a fast AI chip spends most of its time waiting, in the section "Reuse makes the difference".

The time from sending a prompt to the first token of the reply (end to end, as the user feels it); the model reads the whole prompt first, so a longer prompt usually makes this longer.

Taught in Why does an AI wait a moment before the first word?, in the section "Two numbers to watch".

A way to learn a hidden fact from how long a system takes instead of from what it returns; here, a shorter wait before the first word shows that a start was already stored.

Taught in Can one user's chat leak into another's?, in the section "What the toy leaves out".

A chunk of text, often a short word or part of a longer one; a model reads and writes text in tokens.

Taught in What is a token, and why is AI priced per token?, in the section "The picture, step by step".

A rate-limit method: a bucket holds up to a fixed number of tickets, gains tickets at a steady speed, and each request spends one (these tickets are not the tokens an AI model reads).

Taught in Why an AI API key must never go in a web page or app, in the section "The token bucket by hand".

The plain number a tokenizer gives to one token; the model is given these numbers, not the letters, and a number alone does not show which letters are inside its token.

Taught in Why can't AI count the letters in a word?, in the section "What the model is given".

The program that cuts text into tokens; different tokenizers split the same text differently, so token counts differ between models.

Taught in What is a token, and why is AI priced per token?, in the section "The picture, step by step".

How many tokens the model writes each second once the reply has started; each token needs one decode step, so this is at most one divided by the time of a step.

Taught in Why does an AI wait a moment before the first word?, in the section "Two numbers to watch".

A setting that keeps the fewest likeliest tokens whose chances add up to at least p, cuts the rest, and draws from what is left; also called nucleus sampling.

Taught in Why does the same question give different answers?, in the section "Top-p cuts the unlikely tail".

PyTorch's setting for torch.load that restricts its unpickler to tensors, primitive types, dictionaries and types you add. Its notes say the unpickler "is not allowed to dynamically import anything during unpickling". It is the default since PyTorch 2.6 unless you pass your own pickle module, it narrows the risk without removing it, and it is a separate unpickler from the simpler restricted one in this lesson's demo.

Taught in Is it safe to download a model file?, in the section "Closing the door in the loader".