Essay · Visual studies

From tensor shapes to inference bottlenecks

Use a decoder roofline to reason about weight reuse, cache traffic, quantization, and serving latency.

A tensor diagram tells you which operations exist. It does not tell you which one determines latency. The same decoder can be limited by weight reads at batch one, by matrix multiplication during a large prefill, or by cache traffic at long context. Calling all three workloads “inference” obscures the useful distinctions.

Brendan Bycroft’s visualization is a good place to inspect the operations and their shapes. The accompanying Inference roofline experiment asks a different question: under a stated bandwidth and compute budget, what is the most optimistic time for one decode step?

Read shapes before model names #

An embedding lookup produces a vector per position. Projections produce queries, keys and values. Attention combines positions; the MLP expands and contracts the channel dimension; residual connections add updates back into the stream. The final projection produces vocabulary logits, which are converted into probabilities before decoding selects a token.

Use the walkthrough’s stage controls to follow those transformations. The tiny sorting model makes the complete calculation tractable. Selecting a larger architecture changes the scale of the diagram; it should not be read as a benchmark of that model on your machine.

Logits are unnormalized scores. Their differences determine probability ratios after softmax. Calling each individual score a literal log-odds value is misleading because adding the same constant to every logit changes none of the probabilities.

A decode step has two competing floors #

Let PP be the parameter count, BB the number of sequences decoded together, sws_w bytes per weight, β\beta memory bandwidth and FF peak compute throughput. A first approximation is

tmemory≥Psw+KV bytes readβ,tprojection≥2PBF.t_{\mathrm{memory}}\geq\frac{Ps_w+\text{KV bytes read}}{\beta},\qquad t_{\mathrm{projection}}\geq\frac{2PB}{F}.

One multiply-accumulate is counted as two floating-point operations. The larger time is a lower bound for the step under ideal overlap. Summing them would assume no overlap; treating either as achieved latency would assume away utilization and the rest of the system.

At the default seven billion parameters and 16-bit weights, the weight read alone is 14 GB. At 1000 GB/s, that is 14 ms before cache traffic and other costs. The 100 TFLOP/s projection estimate for batch one is only 0.14 ms. More arithmetic throughput cannot eliminate the larger memory floor.

Batching changes the economics #

Increase batch size. The model’s weights can be reused across more output tokens in the same step, while the arithmetic grows with the batch. That amortization raises aggregate throughput. It does not imply that an individual sequence receives its next token sooner.

The cache grows with batch and context. At sufficiently long context, reducing weight precision has diminishing benefit because weights are no longer the only large read. The experiment keeps cache precision at FP16 when you choose 8- or 4-bit weights. Weight quantization and cache quantization are separate decisions with different quality and implementation costs.

The chart’s blue curve represents the memory floor; amber represents projection compute. The dashed line marks the chosen batch. Capacity is checked against an illustrative 80 GiB device using weights plus cache only. Activations, allocator overhead and temporary workspaces still need room.

What this estimate deliberately misses #

The 2PB2PB approximation covers dense parameter-driven projections. Attention arithmetic grows with context and is omitted, so long-context compute can be materially higher. Communication between devices, dequantization, activation reads, kernel launches, cache writes and scheduler overhead also add time. The estimate is a tool for rejecting implausible expectations, not promising a service-level objective.

The parameter count and fixed cache geometry are independent inputs here. They describe a hypothetical system, not a named 7B or 70B architecture. Real estimates should use the model’s layer count, head dimensions, expert routing if applicable and actual precision for each tensor.

Measure the serving system #

For an actual deployment, separate time to first token from inter-token latency, and report distributions rather than one throughput number. Continuous batching can improve utilization while changing queueing delay. A long prefill can interfere with decode work unless scheduling accounts for it. Prefix-cache hits change both memory and compute costs, so a benchmark with repeated prompts can flatter an unrepresentative workload.

I would use the roofline to choose the next measurement: device bandwidth, kernel utilization, cache occupancy or queue delay. Once the measured bottleneck moves, the optimization plan should move with it. A beautiful matrix-multiplication benchmark cannot explain time spent waiting in the scheduler.

Further reading #

LLM Visualization is the source of the three-dimensional walkthrough. The roofline model formalizes the relation between arithmetic intensity, bandwidth and compute. PagedAttention addresses another part of the system: allocating and sharing KV storage efficiently under dynamic serving workloads.

Related articles