Plain: the big model writes every token
With a draft model: guesses, checked together
A model at the chip’s peak speed, not a measurement. You set the chance a guess is kept; a seeded draw decides which ones are.
The percentage is a share of a big pass. The opening stop, 1.2 ms, is a 1.24-billion-weight model at the chip’s peak. The big model’s pass takes 7.88 ms.
Real guesses are not independent chances: how often they are kept depends on the text and on the two models. The draft model needs its own memory (about 2.5 GB here) and cache. A small model usually runs further below the chip’s peak than a big one, so its passes cost more than shown. Software overhead adds more, and with many users sharing the chip the check stops being free. The words are a toy; the sizes are Llama 3.1 8B (big) and Llama 3.2 1B (draft) at 16 bits on one A100.