The average is not the experience #

Consider a synthetic workload: 95 requests take 40 ms each, and 5 take 800 ms. The mean weights every request equally; a percentile identifies a position in the sorted distribution.

Mean: 78 ms. P99: 800 ms. The mean hides a slow path that one in twenty requests takes.

Budget for the slow path #

A service that calls several dependencies can inherit the tail of each one. Deadlines, bounded retries, and queue limits should form one coherent budget.

Choose the measurement window #

This calculation uses the nearest-rank percentile: sort the observations and select rank ceil(p × n). With 100 requests, p99 is the 99th observation. Track percentiles alongside request volume and the measurement window; changing either can change how much evidence supports the reported tail.