Hold the capture button, move your hand, let go. A few seconds later the lab above is telling your hand's position apart from an empty desk with no training loop you wrote, no labeled dataset you assembled in advance, and no round trip to a server. Nothing about that is magic, and it's worth taking apart, because the trick behind it — reuse a big model's features, train something tiny on top — is the same trick behind most practical computer vision systems you'll ship, not a toy simplification of them.
You are not training a neural network from scratch #
A convolutional network trained on ImageNet from random weights needs on the order of a million labeled images and days of GPU time to learn to separate a hand from a desk, a mug from a bottle, a cat from a dog. You gave this lab a few dozen frames. The reason that's enough is that the lab never asks a network to learn what an edge, a texture, or an object part looks like — it reuses a network that already learned that, and only learns the much smaller, much easier problem of drawing a boundary through the representation that network produces.
That's transfer learning, and the specific shape used here — freeze a pretrained feature extractor, train a lightweight classifier on its output — is the cheapest, most robust version of it. MobileNetV2 is the feature extractor: a convolutional network trained on ImageNet, built from depthwise separable convolutions specifically so it's small enough to run in a browser tab rather than a datacenter. mobilenet.infer(image, true) runs a frame through it and returns not a class prediction but the 1280-dimensional activation one layer before the final classification head — a vector that encodes, roughly, "what kind of visual stuff is in this image" without committing to any of ImageNet's thousand specific labels.
Why k-nearest neighbors, and not a trained layer #
The obvious next step would be to train a small dense layer on top of that embedding with backpropagation. This lab does something simpler: it stores every embedding you capture, tagged with the label you assigned, and classifies a new frame by finding its nearest neighbors among those stored embeddings and taking a majority vote. No gradient descent happens after you press the capture button.
That's a real trade, not just an implementation shortcut, and it's worth being explicit about which side of it you're on. A trained classification layer can, in principle, learn a decision boundary a nearest-neighbor vote can't represent — one that isn't well-approximated by "close in embedding space." What k-NN buys back is exactly what makes this lab feel instant: no epochs, no loss curve to babysit, no risk of overfitting a two-layer network to twelve examples, and a new example changes the model's behavior the moment it's added, not after a retraining pass. For a handful of visually distinct classes captured in one sitting — which is the entire use case this lab is built for — that trade is close to free. It stops being free the moment your classes are visually similar (two shades of the same object) or the examples per class run into the thousands, where a trained head's ability to shape a sharper boundary and k-NN's linear-in-dataset-size lookup cost both start to matter.
The embedding is doing more work than the classifier #
It's tempting to think of MobileNet as a preprocessing step and the k-NN vote as "the model." Reverse that framing. The entire capacity to distinguish anything here — texture, shape, color arrangement — lives in the frozen embedding; the k-NN layer contributes nothing but a distance metric and a vote. This is worth internalizing because it explains the failure mode you'll hit fastest if you experiment with this lab: label two classes that differ mainly in where they are in frame, or in lighting, rather than in what they are, and the classifier will confidently learn your background or your lighting rig, not your hand gesture. MobileNet's embedding was trained to be invariant to exactly the things ImageNet varied across its training set, and idiosyncratic to whatever a hand gesture against a fixed background actually varies in — which, uncontrolled, is mostly the background.
This is not a toy-lab-only problem. It's the same shortcut-learning failure that makes a production vision model trained on hospital X-rays from one scanner learn the scanner's metadata burn-in instead of the pathology, or a defect classifier trained on daytime factory-floor images learn the shift's lighting instead of the defect. The fix in both cases is the same one that works here: capture examples that vary everything except the label — move your hand across the frame, rotate it, change the background behind it — so the only consistent signal left for the classifier to lock onto is the one you actually labeled.
What's missing between this and a shipped model #
Two things separate this lab from a production image classifier, and naming them is more useful than pretending they're not there. First, there's no held-out evaluation set — every example you capture goes straight into the classifier's reference set, so the confidence you see is a measure of how close a new frame is to training examples, not a generalization estimate. A real evaluation would hold back a fraction of captured frames, ideally from a session captured at a different time or angle, and report accuracy on those. Second, this lab's k-NN classifier lives only in this browser tab's memory; the original Teachable Machine (and any serious deployment of this pattern) exports the trained layer as a portable graph — TensorFlow.js, TensorFlow Lite, or a Core ML model — so the same embedding-plus-head can run on a phone, a microcontroller, or a server without re-teaching it. The interesting engineering happens after the demo: turning "a classifier that works in my tab, on my lighting, on my examples" into one that survives export, a different camera, and a different day.