Essay · Visual studies

Attention, interventions, and the cost of context

Follow the residual stream, test causal routing, and budget the KV memory behind long-context inference.

The most useful way to read a transformer diagram is to follow one token’s residual stream and ask where information can enter it. That separates routing between positions from computation within a position. It also exposes the memory traffic hidden by a neat sequence of boxes.

The visualization runs Polo Club’s GPT-2 explainer with local model assets. Attention & KV cache opens a smaller analytical experiment without loading the model. Its six-token heatmap is synthetic, which lets you intervene on masking and score scale without confusing the result with a measurement of GPT-2.

Follow the residual stream #

GPT-2 turns token IDs into learned embeddings and adds learned position embeddings. Each pre-normalized block then adds an attention update and an MLP update to the residual stream. The attention update mixes information across allowed positions; the MLP applies the same learned nonlinear transformation independently at each position.

For one attention head,

Q=XWQ,K=XWK,V=XWV,A=softmax⁡ ⁣(QK⊤dk+M),Y=AV.Q=XW_Q,\quad K=XW_K,\quad V=XW_V,\qquad A=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right),\quad Y=AV.

The mask MM excludes future positions in a causal decoder. The division by dk\sqrt{d_k} controls the variance of dot products under common initialization assumptions. It is distinct from a user-facing sampling temperature applied to final vocabulary logits.

In the small experiment, switch causal attention to bidirectional attention and inspect an early query. New positions become available. Increasing the logit scale concentrates its weights but does not alter which keys the mask permits. If ablation removes every allowed key, the row is undefined; the lab explicitly displays zero rather than quietly passing NaN through the rest of the calculation.

An attention map is not an explanation by itself #

A large attention weight says that a value vector contributes strongly to this head’s weighted sum. It does not say that the corresponding input token caused the final answer. Values can have different magnitudes and signs, heads are projected and combined, and later residual updates can reinforce or cancel the result.

Use Ablate key to remove a position. The remaining weights renormalize, so the intervention changes more than one coefficient. Compare the output with the unablated baseline. Even the scalar values in this experiment can cancel despite a visually strong attention link.

For a real model, causal claims need interventions and an outcome measure: activation patching, controlled input changes or ablation followed by evaluation. An appealing attention heatmap is a place to start an investigation, not the conclusion.

Prefill and decode are different workloads #

Prefill processes the prompt’s positions in parallel. Decode usually adds one position per sequence at a time. The new query attends to old keys and values, so caching those tensors avoids recalculating them from the entire prefix on every step.

For batch size BB, context length NN, LL layers, HkvH_{kv} key/value heads, head dimension dhd_h and ss bytes per element,

KV bytes=2BLNHkvdhs.\text{KV bytes}=2BLNH_{kv}d_hs.

The factor of two is for keys and values. At the experiment’s defaults—32 layers, 32 KV heads, head dimension 128, context 4096, batch one, FP16—the cache alone is 2 GiB. Change to eight KV heads and it becomes 0.5 GiB. Grouped-query attention shares keys and values among groups of query heads; multi-query attention shares one set. This is an architectural choice, not a lossless switch you can apply to arbitrary trained weights.

The quadratic tensor does not have to exist #

Materializing all attention scores costs BN2HqBN^2H_q elements per layer. The lab shows that hypothetical allocation beside the linear KV-cache budget. Doubling context quadruples the first and doubles the second.

FlashAttention-style algorithms tile the computation and maintain an online softmax, avoiding the full score matrix in high-bandwidth memory. They still compute dense attention; the improvement comes from data movement and intermediate storage, not from pretending the pairwise interactions disappeared. Sparse and sliding-window attention change the dependency pattern and require a separate quality argument.

The cache panel uses a hypothetical decoder geometry, not GPT-2’s dimensions. Likewise, rotating position embeddings, grouped-query attention and modern normalization variants should not be inferred from GPT-2’s picture. Follow the tensor shapes for the actual architecture before carrying an intuition into a capacity estimate.

Further reading #

Transformer Explainer supplies the main visualization. The Illustrated Transformer is a useful architectural companion. FlashAttention develops the I/O argument, while GQA describes the quality and efficiency tradeoff behind sharing KV heads.

Related articles