A new group starts only when every slot is empty.
A waiting request takes any empty slot at once.
A model at the chip’s peak speed, not a measurement. The 48 requests are invented.
Network time, the software around the model, running out of memory for the caches (real servers then pause a request), and the gap between the chip’s data-sheet peak and real speed. Real servers are slower than these times. Prompts and replies are a seeded random draw. The model is Llama 3.1 8B with 16-bit weights on one A100 chip.