Inside the chip

See why moving data, not arithmetic, sets the speed.

  1. How much memory does an AI model need?

    An AI model needs memory for its numbers, for notes on every chat, and for scratch space. Add the parts to see whether a model fits on a graphics card.

    6 min+18 min deeper

  2. Why a fast AI chip spends most of its time waiting

    A GPU can calculate far faster than its memory can feed it. See why reusing data sets the speed, and why one chatbot user keeps a chip under 1% busy.

    5 min+22 min deeper

  3. What does quantization trade away?

    Quantization rounds an AI model's numbers to fewer bits. See how much memory and speed that buys, and what it costs in accuracy.

    6 min+22 min deeper

  4. How FlashAttention avoids storing the giant score table

    Online softmax finds a list's highest score and running total in one read, so FlashAttention never stores the giant score table.

    5 min+14 min deeper