Plan and measure

Turn the numbers into a plan and check it by measuring.

  1. Why is the average speed of an AI service a misleading number?

    Slow replies hide in a good average. Report the middle and the slowest 1 in 20 and 1 in 100. If users arrive on their own, send test requests on a schedule.

    5 min+14 min deeper

  2. What do a model server's settings change?

    A GPU's memory is one fixed budget. A model server's main settings only move room between the weights, a reserve and the chats in progress.

    5 min+15 min deeper

  3. How many GPUs do I need for 1,000 users?

    Size an AI service for its busiest hour, not its user count: chats at once, each GPU's memory and speed limits, a spare, then a load test.

    5 min+17 min deeper

  4. Should I rent a model API or run my own?

    Renting an AI model bills per token, and running your own bills per hour, busy or not. Where the two cost the same depends on tokens a month and how busy the GPUs are.

    6 min+16 min deeper

  5. Build a tiny model server you can read end to end

    A model server is a cache, a batcher and metrics around the model. Build all three in under 150 lines and read the waits it reports.

    6 min+19 min deeper