Most image classifiers are trained with a fixed list of labels. If a model was trained on ImageNet’s 1,000 classes and you want to know whether a photo shows a “broken projector”, you’re out of luck: that label doesn’t exist, and adding it means collecting examples and training again.
CLIP, released by OpenAI in 2021, avoids the fixed list. You write the labels at the moment you ask the question, in plain English. This lab runs the original CLIP ViT-B/32 model in your browser, in its 8-bit version, and shows every number it uses to decide.
Two encoders, one space #
CLIP has two separate networks. An image encoder (a Vision Transformer) turns a photo into a vector of 512 numbers. A text encoder (a small transformer) turns a sentence into another 512 numbers. Both vectors are normalized to length 1, so comparing them only depends on the angle between them. The comparison is the cosine similarity, the dot product of the two unit vectors.
The two networks were trained together on about 400 million image–caption pairs from the web. In each training batch, CLIP saw N images and their N captions and learned to make each image most similar to its own caption and less similar to the other N − 1. That’s called a contrastive objective. The model never learned “this is class 17”. It learned which pictures and which sentences tend to appear together.
Zero-shot classification falls out of this almost for free. Turn every label into a sentence, embed all of them, and pick the one closest to the image.
Cosines are small, and that’s fine #
Load the model and select Street. With the default labels, “a bus” wins with about 96% probability. Now look at the cosine column. In my browser it read 0.268 for the bus, 0.228 for “people walking in a park” and 0.191 for “a cup of coffee”. Your numbers may differ in the second decimal place, because browsers decode and resize images slightly differently.
Those differences look tiny, and everything is between 0.19 and 0.27. That is normal for CLIP. Image and text vectors never line up closely. What matters is the ranking, and how far apart the scores are relative to each other.
The probabilities come from a softmax over the cosines multiplied by 100, CLIP’s learned temperature:
Multiplying by 100 turns a gap of 0.04 into a factor of about . Drag the softmax scale down to 1 and the same cosines give almost equal probabilities. With four vehicle labels on the street photo, the four came out at 25.5%, 25.0%, 24.9% and 24.6%. The model’s preference hasn’t changed. Only the way we present it has.
The probabilities depend on your list #
A softmax only compares the options it’s given, so the probability of a label depends on what else is on the list. With six mixed labels, “a bus” got about 96%. Replace them with “a bus”, “a car”, “a police officer” and “a traffic jam”, which all fit the photo, and the bus drops to 76.1%. Its cosine barely moved, from 0.268 to 0.267. The competition got better.
This is the most common way zero-shot classifiers are misread. A 96% probability does not mean 96% sure it’s a bus. It means that, of these six sentences, the bus sentence matched best by a wide margin. To ask a yes/no question like “is there a bus?”, compare against a sensible “not a bus” alternative, or calibrate a threshold on the cosine itself using labelled examples.
Prompts matter more than you’d expect #
The label isn’t embedded alone. By default it goes into the template “a photo of {}”. The CLIP authors found that this simple wrapper improved ImageNet accuracy by more than a point, because web captions rarely consist of a bare noun.
Tick Average 8 templates to try their other trick, prompt ensembling. Each label is embedded with eight templates (“a blurry photo of {}”, “a close-up photo of {}” and so on), and the eight vectors are averaged into one. On the four vehicle labels, the bus went from 76.1% to 78.8%. That’s a small gain, and it cost 32 text encodings instead of 4. On ImageNet, the paper reports that ensembling added a few more points of accuracy, which is worth that price. For a demo, you can take it or leave it.
Try clearing the template to a bare “{}” and watch the rankings shift. The model is sensitive to phrasing in ways that have nothing to do with the image. That is useful to know before you put label names in front of it in production.
Search works the same way, backwards #
The text-to-image search at the bottom reuses the same vectors. Instead of ranking labels for one image, it ranks images for one sentence. “A vehicle for many passengers” picked the street photo with a cosine of 0.248, ahead of the desk (0.210) and the park (0.205). The word “bus” appears nowhere in that query.
This is how many image search systems work: embed every image once, store the vectors, then embed each query and find the nearest vectors. The expensive part, the image encoder, runs once per image. In this browser it took about 230 ms per image. The text encoder handled six prompts in about 124 ms.
Where CLIP goes wrong #
CLIP inherits the web’s habits. It is good at objects, scenes and styles that people often caption, and weak at counting, reading small text, spatial relations (“the cup is left of the laptop”) and anything rarely photographed or captioned. It also absorbs the web’s biases, so labels involving people deserve extra caution.
For production use, the common pattern is to use CLIP’s embeddings rather than its zero-shot guesses. Add a small classifier trained on a few hundred of your own labelled examples, or use CLIP to retrieve candidates for a person or a larger model to check. The zero-shot demo is where you find out what the embeddings already know. It is not where you stop.