A sampler can be wrong while its chart looks completely reasonable. If the chart keeps only the fifty leading logits and then normalizes them, it has silently changed the probability space. Top-p now answers a different question.
That is why this lab starts with the full distribution: five recorded GPT-2 examples from Transformer Explainer, each containing all 50,257 logits. The controls recompute probabilities in the browser. There is no inference call, and choosing a token does not manufacture the next model state.
Patrick von Platen’s Hugging Face generation guide supplies the decoding framework and the greedy-versus-beam example. The local experiment adds something worth testing before shipping a sampler: whether display limits have leaked into the computation.
Name the pipeline, including its order #
The default path is:
- Subtract an additive presence penalty from each distinct token already in the recorded prompt.
- Divide the resulting logits by temperature.
- Retain the top k candidates, if enabled.
- Renormalize those candidates, then retain the smallest prefix reaching the top-p threshold.
- Normalize the survivors and sample.
The token crossing the nucleus threshold is included. At least one candidate survives. This lab does not use a frequency penalty or a multiplicative repetition penalty.
For adjusted logits and positive temperature ,
Subtracting the maximum avoids unnecessary overflow without changing probabilities. Temperature preserves score ordering; it changes probability ratios. Truncation changes which tokens can be selected at all.
Switch to k → p → temperature. The nucleus is now chosen before temperature reshapes the distribution. Applying the same numeric settings in these two orders does not specify the same sampler.
The tail is part of the denominator #
Leave top-k off, raise temperature to 1.5 and use a high top-p threshold. Compare the real nucleus size with the Top-50 shortcut TV distance.
The shortcut deliberately discards every token below rank fifty before processing the distribution. It reproduces the kind of approximation that is convenient in a visual explainer but unsafe to treat as a full-vocabulary sampling implementation. Transformer Explainer’s source trims the displayed candidates in this way; this lab retains the raw recorded logits instead.
Total variation is
It compares the two final distributions over the same complete vocabulary. A large value means that identical-looking temperature and top-p settings can lead to materially different draws.
The visible chart still shows only twenty tokens. The difference is that the remaining mass is reported and remains in every calculation. Hiding a bar is a presentation choice; deleting its probability is an algorithmic one.
“Top-p 0.9” is not always ninety percent of the model #
If top-k first removes half the mass, applying top-p 0.9 to the renormalized survivors need not retain ninety percent of the original distribution.
The Base mass retained readout measures the surviving support against the distribution after the presence penalty and before temperature. With a nonzero penalty, “base” is therefore not the untouched model distribution.
There are other quantities worth inspecting. Entropy measures uncertainty under the final distribution. The effective-choice count gives the number of equally likely outcomes with the same entropy. A nucleus containing hundreds of tokens may still behave like a much smaller set if a few candidates dominate.
None of these quantities measures truthfulness, usefulness or calibration against a task. A confidently wrong answer can have low entropy.
Reproducibility needs a contract, not just a seed #
Sample 1000 performs independent draws from the selected snapshot. Changing the controls resets the empirical counts, so samples from incompatible distributions cannot be mixed. Changing the seed changes the reproducible draw stream.
This is deliberately different from autoregressive generation. In generation, every sampled token changes the context and the next logits. A one-step histogram cannot measure sequence repetition, answer quality or stopping behavior.
The recorded context, tokenizer IDs and full float32 logits make this particular experiment reproducible without downloading GPT-2. A production run additionally needs model and tokenizer revisions, processor order, stopping rules, numerical precision and an execution environment. Hardware, batching and kernel changes can alter arithmetic even when a seed is fixed.
Tokens that are not readable ASCII are labelled by ID in this English interface. They are not removed from the vocabulary or assigned invented probabilities.
A locally likely token can lose at sequence level #
The small, separate beam-search experiment follows the source guide’s example:
- Greedy follows “The nice woman,” with probability .
- Width two also retains “The dog,” exposing “The dog has,” with probability .
Change the beam width and inspect the surviving hypotheses after each expansion. The branch table is a constructed teaching example, not a continuation of the selected GPT-2 prompt. Residual branches complete the conditional distributions locally.
This establishes a narrow point: optimizing the next step need not optimize the completed sequence. It does not establish that beam search always improves output quality, or that wider beams universally improve a production decoding objective.
Variable-length sequences introduce EOS, length penalties and stopping heuristics. Pruning decisions made earlier cannot necessarily be undone later. A beam score should state its exact objective; “most likely answer” is too imprecise.
What I would test before changing a production default #
First, test the implementation contract: numerical stability, normalization, threshold crossings, support size and behavior when several processors interact. Include flat distributions and heavy tails, not just one sharp next-token example.
Then test the task. For structured output, measure syntax and schema validity separately from semantic correctness. For extraction, measure field accuracy and abstention. For open-ended text, inspect repetition and human judgments alongside task success. A grammar can rule out malformed JSON without preventing a well-formed false claim.
Finally, compare latency and cost at the same stopping conditions. A setting that tends to emit EOS earlier can look cheaper for reasons unrelated to kernel efficiency.
References: Hugging Face’s generation guide, Transformer Explainer’s recorded examples, and Holtzman et al., The Curious Case of Neural Text Degeneration. The source guide is a starting point for decoding choices, not evidence that any one setting is best for your workload.