What happens to a neural network when you round its weights

Every modern model release seems to come with a quantized version, and at some point we all nodded along to “4-bit is basically free”. This lab tests that claim on a real network. It takes the 4.4 million convolution weights of the SSDLite object detector from the object detection lab, rounds them to fewer bits, and runs the rounded model next to the original on the same photo.

Spoiler: for this model, 4-bit is not free. It isn’t even close.

What rounding means here #

A 32-bit float can represent billions of different values. An 8-bit integer can represent 256. Quantizing a layer means choosing a scale $s$ and mapping each weight $w$ to the nearest integer on a grid:

With symmetric quantization at $b$ bits, the integers run from $-(2^{b-1}-1)$ to $2^{b-1}-1$, and $s$ is chosen so the largest absolute weight maps to the edge. At 8 bits that gives 255 levels, and at 4 bits only 15. Asymmetric quantization adds a zero-point offset so the grid covers the range from minimum to maximum. That helps when a layer’s weights aren’t centred on zero.

The lab does simulated quantization. It rounds the weights, converts them back to floats and runs the normal model. You see the accuracy cost, not the speed or memory gain, which would need real integer kernels.

How many weights share a scale #

This is the setting that matters most, and the one people skip over.

• Per tensor: one scale for a whole layer. A single large weight stretches the grid for everyone, and the small weights round to zero.
• Per output channel: one scale for each filter. A filter with large weights no longer hurts its neighbours.
• Per group: one scale for every 32 consecutive weights inside a channel. This is the approach used by most 4-bit LLM formats, usually with groups of 64 or 128.

More scales mean better accuracy and a bit more storage. At 8 bits per tensor, the 88 layers need 88 scales in total. Per-group quantization needs tens of thousands, still a small fraction of 4.4 million weights.

What happened on this detector #

The lab opens at 8 bits per tensor, the setting most people assume is safe. On the street photo, the original model finds a bus at 0.95 and a car at 0.91. The 8-bit per-tensor model keeps the bus but loses the car, and adds a person who isn’t there. The park photo keeps all nine people, plus one extra, and the desk photo loses one of its four objects.

Switch to per output channel, still at 8 bits, and both street objects come back (bus 0.91, car 0.79). Every original detection on all three photos is back, along with two new false positives: a “boat” at 0.67 on the street and a “book” on the desk. The weight SQNR, the signal-to-quantization-noise ratio, explains why. The median layer goes from 35 dB per tensor to 44 dB per channel. Each 6 dB is roughly one extra bit of precision.

Now lower the bits. At 6 bits per channel, the park and desk photos still match perfectly, but both street objects disappear. At 4 bits per tensor, the model starts seeing people everywhere: six phantom detections on the street, seven on the park photo and seven on the desk. At 2 bits, even with per-group scales, it reports around 15 objects per photo, mostly people and dining tables. There are no dining tables.

The street photo is the first to break, and that’s not a coincidence. It’s an old, reddish, motion-blurred photo, unlike most of COCO. Quantization error is a kind of noise, and inputs that were already hard for the model have the least margin to absorb it. The park photo, full of clearly visible people, kept all nine detections down to 5 bits with per-channel or per-group scales. If you only evaluate quantization on your easiest examples, you’ll conclude it’s free.

Which layers break first #

Pick a layer from the Layers hurt most list. At 8 bits per tensor, the list is led by a 1×1 projection convolution in the middle of MobileNetV2, at about 20 dB. Look at its histogram and at how many distinct values survive rounding. When most weights sit near zero and a few sit far out in the tails, a per-tensor grid spends its levels covering the tails, and the crowded middle collapses onto a handful of values.

This is a known property of MobileNet-style networks. Depthwise and projection layers have very different ranges from one channel to the next. TensorFlow Lite and PyTorch quantize convolution weights per channel by default for exactly this reason. The “Keep first layer and prediction heads at 8 bits” option tries the other common fix, mixed precision. It raises the median SQNR slightly at 5 bits, but barely changes the detections, because the damage in this model mostly comes from the middle of the backbone, not its ends.

Why LLMs get away with 4 bits #

If a small detector falls apart at 4 bits, why do 4-bit LLMs work? Three reasons.

1. Size. A 7B-parameter model has a lot of redundancy. Rounding errors in thousands of weights feeding one output tend to average out. A 4.4M-parameter detector has far less slack.
2. Better rounding than round-to-nearest. Methods like GPTQ and AWQ use a small calibration set to choose roundings and scales that minimize the error in each layer’s output, not in each weight. This lab uses plain round-to-nearest, the naive baseline those methods improve on.
3. Groups. LLM formats use per-group scales, typically every 64 or 128 weights, plus special handling for the few outlier channels that large transformers are known to develop.

So “4-bit is free” is really three claims: the model is big enough, the method is smarter than rounding, and someone checked on data that looks like yours. This lab lets you break each one.

What to take away #

Quantization is a trade-off to measure, not a checkbox. Pick the granularity before the bit width, since per-channel or per-group scales often matter more than one extra bit. Evaluate on your hardest inputs, not your average ones. And look at outputs, not just aggregate scores: a tiny average change in scores sounds harmless until you notice the missing car.