Most people picture an object detector as a model that looks at a photo and draws a box around each thing it sees. That is roughly what you get at the end, but the model itself does something cruder. It makes a fixed number of guesses, the same number for every image, and scores each one. Turning those guesses into a clean set of boxes is a separate step, and it has a few settings that make a surprising amount of difference.
This lab runs a real detector in your browser: SSDLite with a MobileNetV2 backbone, trained on the COCO dataset. It is the model behind TensorFlow.js’s coco-ssd package. The difference is that this lab calls the network directly and does the post-processing in plain TypeScript, so you can watch every stage. Nothing is uploaded. The 17 MB of weights come from this site, and inference runs on your GPU through WebGL.
One pass, 1,917 guesses #
Load the detector and pick the Street photo. The network sees a 300 × 300 version of the image and returns two arrays:
- 1,917 boxes, each as four numbers (top, left, bottom, right), scaled to the image size;
- for every box, 90 scores, one per COCO category.
Why 1,917? SSD tiles the image at six scales. The finest grid is 19 × 19 cells, and each cell proposes 3 anchor boxes of different shapes. Coarser grids of 10 × 10, 5 × 5, 3 × 3, 2 × 2 and 1 × 1 propose 6 boxes per cell:
Small objects are caught by the fine grid and large ones by the coarse grids. The network does not invent boxes from nothing. It nudges each anchor’s position and size, then says how much the result looks like a person, a bus, a cup and so on.
This is why it is called a single-shot detector. Older approaches such as Faster R-CNN first proposed regions, then ran a second classifier on each one. SSD does both jobs in one forward pass. It is less accurate on small objects, but fast enough to run on a phone or in a browser tab.
The 90 score columns cover only 80 real categories. COCO’s original label list had 91 ids, and some, like “hat” and “shoe”, were dropped before the dataset shipped. The gaps stayed in the numbering, and the model kept the unused columns.
First filter: the score threshold #
For each anchor, the lab keeps the best-scoring class and discards anchors whose best score is below the score threshold. At the default of 0.30, the street photo goes from 1,917 anchors to 16 candidates.
Switch Show to “All candidates above the score threshold.” You will see stacks of nearly identical boxes. There are 9 on the bus and 7 on the car. Neighbouring anchors, at slightly different positions and scales, all overlap the bus well enough to fire. From the network’s point of view that is correct: each of those anchors really does contain a bus.
Drag the score threshold down to 0.10 on the Park photo. Roughly twice as many boxes survive: more people far down the path, plus low-scoring guesses for other objects. Check each one against the photo. Some are real and some are not. Drag it up to 0.50 and the faintest walkers drop out. No setting is right for every use. A pedestrian-safety system would rather see a false alarm than miss a person. A photo-tagging app would rather miss a small object than label a lamp post as a person.
These scores are not probabilities in any careful sense. Each one is a sigmoid output, trained so that higher usually means more likely. A score of 0.8 does not mean the model is right 80% of the time. Checking that would need a separate step called calibration.
Second filter: non-maximum suppression #
The stacks of duplicates are what non-maximum suppression (NMS) is for. The idea is simple and greedy:
- Sort the surviving candidates by score, highest first.
- Keep the top box.
- Remove every remaining box that overlaps a kept box by more than the IoU threshold.
- Move to the next box that is still around and repeat.
Overlap is measured with intersection over union:
Two identical boxes have an IoU of 1. Two boxes that don’t touch have an IoU of 0. A box shifted sideways by half its width has an IoU of 1/3, lower than most people guess.
Switch back to “After suppression” and click the bus. The panel lists the 8 boxes it removed and their IoU with it. All of them are above 0.85. They were real detections of the same bus, and NMS removed them because a higher-scoring box already covered it. The street photo ends with 2 boxes: the bus at about 0.94 and the car at about 0.92.
The exact scores can shift by a point or two between browsers and GPUs. The weights are the same, but the arithmetic runs at slightly different floating-point precision.
What the IoU threshold trades off #
Drag the IoU threshold up to 0.90. Two extra bus boxes reappear, the ones whose overlap with the best box was just under 0.9. Drag it down to 0.20 and nothing changes on this photo, because the car and the bus barely overlap.
In crowded scenes the low end hurts. Think of people standing in a queue. Their boxes can overlap by 0.5 or more, even though they are different people. With a low threshold, NMS decides they are duplicates and keeps only the highest-scoring person. That is a known weakness of greedy NMS in crowds, and variants such as Soft-NMS lower the neighbours’ scores instead of deleting them.
The Suppression setting controls whether a box can only suppress boxes of the same class. On these three samples the two settings give the same result. Try uploading a photo of someone riding a bicycle. The person and bicycle boxes overlap heavily. If suppression runs across all classes, the higher-scoring person can delete the bicycle. Per-class NMS is the common default for exactly this reason.
When the model is simply wrong #
Open the Desk photo. The detector finds the laptop at about 0.8, and the keyboard and the mug, labelled “cup”, at about 0.5. It also calls the trackpad a laptop, at about 0.6.
No threshold fixes that without also losing the mug. The trackpad is a flat grey aluminium shape next to a keyboard, which looks a lot like the laptops in COCO, and COCO has no “trackpad” class. The model has to choose one of its 80 labels or stay silent, and this time it chose a confident wrong answer.
This is worth remembering when you read benchmark numbers. COCO mean average precision summarises performance across many photos and thresholds. It says less about the specific objects, lighting and camera angles in your product. A few minutes of trying your own photos, as you can here, often tells you more about a model’s blind spots than the leaderboard does.
Speed in the browser #
The lab times the full round trip: copying the image to the GPU, running the graph and reading the outputs back. The first run is slow because the browser compiles WebGL shaders. After that, a laptop GPU usually takes tens of milliseconds per image. The Webcam option runs the detector in a loop and reports frames per second, so you can see what “real time” costs on your own machine.
NMS itself costs almost nothing here, because only a few dozen candidates pass the score threshold. That is one practical reason to apply the score threshold first. NMS compares each box against every kept box, and running it on all 1,917 anchors would do much more work.
Taking this to a real system #
A production detector would usually be larger and more accurate than SSDLite, such as a recent YOLO model or a DETR-style transformer. The concepts carry over. DETR-style models are trained to produce one box per object, so they need little or no NMS. Most YOLO versions still use it.
The two thresholds belong to your product, not to the model. Pick them on a validation set that looks like your real images, and pick them for the mistake you care about more: missed objects or false alarms. Then check the failure cases one image at a time. A box around a trackpad labelled “laptop 60%” is easy to catch when you look, and easy to miss in an average.