If you have ever tried to answer “how many GPUs do we need to serve this model?”, you have probably found that the answer depends on almost everything: the model, the precision, the prompt length, how many users arrive at once, and what latency they will put up with. It is tempting to skip the arithmetic and benchmark. You should benchmark eventually. But a back-of-the-envelope model explains what the benchmark will show and why, and it takes about a page of math.
This lab is that back-of-the-envelope model, made interactive. It uses real architecture numbers for six open-weight models and datasheet numbers for six GPUs. Everything is computed in your browser as you drag the controls. The model has two ideas. Memory is mostly weights plus the KV cache. Each phase of generation takes as long as the slower of moving bytes and doing arithmetic.
What has to fit in memory #
The weights are the easy part. A parameter stored in BF16 takes 2 bytes, so Llama 3.1 8B (8.03 billion parameters) needs 16.1 GB. Llama 3.1 70B needs 141 GB, which is why the lab reports that its weights alone don’t fit on one 80 GB H100. Switch to 2 GPUs, or switch the weights to INT4, and it fits.
The KV cache is where the workload comes in. A transformer generates one token at a time, and each new token attends to every earlier token. Recomputing the keys and values of all earlier tokens at every step would be wasteful, so the server keeps them in memory. For every token in every active conversation, every layer stores one key vector and one value vector per key/value head:
For Llama 3.1 8B that is bytes, or 128 KiB per token. A single 32K-token conversation needs 4 GiB, about a quarter of the model’s own weights.
The lab gives the server 90% of GPU memory, which is vLLM’s default, and puts the weights there first. Whatever is left holds KV cache, and that sets how many conversations can run at once. On one H100 with 2,048-token prompts and 512-token answers, Llama 3.1 8B has room for 166 conversations.
Why grouped-query attention matters #
Pick Llama 2 7B, then Llama 3.1 8B, and compare the KV cache per token: 512 KiB versus 128 KiB. The newer model is slightly larger, yet it uses a quarter of the cache.
The difference is grouped-query attention (GQA). Llama 2 7B gives each of its 32 attention heads its own keys and values. Llama 3.1 8B keeps 32 query heads but shares 8 key/value heads among them, four queries per key/value head. Quality barely changes, and the cache shrinks by 4×.
Set the prompt to 32K tokens and look at the maximum number of conversations on one H100: 3 for Llama 2 7B and 12 for Llama 3.1 8B. When serving cost comes up, architecture choices like this often matter more than raw parameter count.
Decoding is a memory problem #
Now the timing. Generating a token requires reading every weight once, since each one takes part in a matrix multiply, and doing about two floating-point operations per weight: a multiply and an add. For Llama 3.1 8B on an H100, at batch size 1:
- Moving 16 GB at 3.35 TB/s (the lab assumes 70% of that is achievable) takes about 7 ms.
- Doing 16 GFLOP at 989 TFLOP/s (assuming 50%) takes about 0.03 ms.
The GPU spends almost all of its time waiting for memory. The lab reports decode is memory-bound, at about 7 ms per token or 143 tokens per second. That is in the same range as published single-stream H100 measurements for 8B models. Try the L4, which has a tenth of the bandwidth: about 78 ms per token.
This is also why quantization speeds up decoding, not just saves memory. Switch the weights to INT4 and the time per token falls from 7 ms to about 1.9 ms, because there are about 4× fewer bytes to move. A lower-precision number is not faster to multiply here. There is simply less data to move.
Batching is nearly free, until it isn’t #
If each step spends most of its time loading weights, why not use each load for many requests? That is what batching does. The weights are read once per step, and the same matrices are applied to every sequence in the batch.
Set the prompt to 256 tokens and the output to 64, then drag Concurrent requests from 1 to 64. The time per token goes from 6.9 ms to 7.9 ms. Each user sees roughly the same speed, while total throughput grows from about 145 to about 8,100 tokens per second. Cost per million output tokens falls from about $5.83 to $0.21 at the example rate of $3 per GPU-hour.
The chart shows where this stops. Compute grows with the batch size while weight traffic does not, and at some point the arithmetic takes longer than the memory reads. The dashed line marks that crossover, about 411 concurrent requests for this setup. Beyond it, extra requests make every user wait longer.
Long contexts move the bottleneck again #
Now set the prompt back to 2,048 tokens with 64 requests. The time per token doubles to about 15 ms, and the dashed line disappears from the chart. What changed?
Each sequence has its own KV cache, and attention has to read all of it at every step. Weight reads are shared across the batch, but KV reads are not. At 64 conversations × 2,304 average tokens × 128 KiB, the cache read per step is about 19 GB, more than the weights. Adding a request also adds memory traffic, so decode stays memory-bound at any batch size. This is why long-context serving is expensive, and why techniques that shrink or share the cache, such as GQA, FP8 KV cache, prefix caching and paged attention, show up in every serving engine.
Prefill is a compute problem #
The first token is different. Before generating anything, the model processes the whole prompt in one pass. All 2,048 prompt tokens go through each matrix together, so each weight load is used 2,048 times. Now the arithmetic dominates. Llama 3.1 8B needs about 69 ms of compute versus 7 ms of memory traffic, so prefill is compute-bound.
That gives time to first token (TTFT). It grows with prompt length and, in this simplified model, with the number of prompts processed together. At 64 simultaneous 2K prompts, the lab reports 4.4 seconds. Real servers avoid this with continuous batching and chunked prefill, interleaving prompt chunks with other users’ decode steps. The underlying arithmetic still has to happen somewhere.
Mixture-of-experts models #
Pick Qwen3 30B-A3B. It has 30.5B parameters, but each token uses only about 3.3B of them: 8 of 128 expert blocks per layer, plus shared attention weights. At batch size 1 it decodes in about 2.9 ms per token, faster than Llama 3.1 8B, even though all 61 GB of weights must stay in memory.
Now raise the batch to 32. Different tokens choose different experts. With 32 tokens each picking 8 of 128 experts, most experts are needed by at least one token, so each step reads close to the full model. The lab assumes uniform, independent routing, so the expected fraction of experts used is . Time per token rises to about 26 ms. Mixture-of-experts models are cheap per token at low batch sizes. At high batch sizes they behave more like the dense model of the same total size.
What this model leaves out #
These estimates are optimistic lower bounds. They ignore communication between GPUs, which matters with tensor parallelism across several GPUs. They also ignore scheduler overhead, kernel launch costs, sampling and the memory taken by activations. The two efficiency sliders are where reality comes in. Move them until the lab matches a benchmark you trust, then use it to reason about changes you haven’t measured yet.
Compute is always BF16 in this lab, even when weights are INT4 or FP8. Many INT4 kernels convert weights to 16-bit before multiplying, while an H100 running true FP8 math has about twice the peak shown here. That only affects compute-bound phases, which mostly means prefill.
The main lesson is simple. For a given model and GPU, count bytes and FLOPs per step and see which one runs out first. That tells you whether a proposed optimization will help. Faster matrix multiplication does little for single-user decode on an H100, while a smaller KV cache does little for a short-prompt, compute-bound batch job.