Embeddings are everywhere now: search, recommendations, retrieval for LLMs, clustering, deduplication. And almost every blog post about them contains the same picture, a 2-D scatter plot where similar things sit together in tidy coloured clusters. This lab builds that picture from scratch in your browser, then shows how much of it you should believe.
Embedding the site #
Press Embed the site and your browser downloads all-MiniLM-L6-v2, a 22 MB sentence-embedding model, and runs it on every passage from this site’s writing: 292 passages, each turned into a vector of 384 numbers. The model runs in a Web Worker with WebAssembly, so the page stays responsive. On my laptop it took a minute or two. A GPU server would do it in a blink, but that server isn’t running in your tab.
Each vector is normalized, so closeness between two passages is their cosine similarity. That similarity is real and useful. Type into Search by meaning: “how do I stop an optimizer from diverging” returns five passages from the gradient descent article, led by “A steep direction can make the same rate fail” at a cosine of 0.48. That’s semantic search, and it works with no projection at all.
The trouble starts when we try to look at 384 dimensions.
PCA: honest and a bit boring #
Principal component analysis finds the two directions in which the data varies most, and projects every point onto them. It is linear, deterministic and has no settings to tune. It is also brutally honest about how much it throws away. Here, the two PCA axes keep 12.9% of the variance. The other 87.1% is spread over the remaining 382 directions.
To measure what that costs, the lab checks each passage’s 10 nearest neighbours in the original 384-D space, then counts how many are still among its 10 nearest on screen. For PCA the answer is 28%. Click a point: the lines connect it to its true five nearest neighbours, and several of them sit on the other side of the plot.
That doesn’t mean PCA is wrong. It is the best linear 2-D view, and distances along its axes mean something. It just isn’t very informative when the structure you care about lives in many directions at once, which is the usual case for text embeddings.
t-SNE: good neighbourhoods, meaningless distances #
t-SNE (van der Maaten and Hinton, 2008) takes a different approach. For each point it builds a probability distribution over its neighbours in high-D space. It then moves the 2-D points around until their neighbour distributions match as closely as possible. It cares about who is near whom, and almost nothing about how far apart the clusters are.
Switch to t-SNE and watch it settle. The lab runs the exact algorithm, which is fine for a few hundred points, and animates it step by step. At the default perplexity of 15, roughly the number of neighbours each point considers, neighbour preservation rises to about 54%, nearly double PCA’s. Passages from the same article gather into tight little islands.
The islands are also where t-SNE misleads people. The distance between islands is mostly meaningless, the size of an island says nothing about how spread out that topic is, and changing the perplexity can split or merge clusters. Try perplexity 3 and then 50, and compare both the shapes and the neighbour score. A tidy t-SNE plot is a claim about neighbourhoods, not a map.
UMAP: t-SNE’s faster cousin #
UMAP (McInnes et al., 2018) has similar goals. It builds a graph of each point’s nearest neighbours and lays that graph out in 2-D. In practice it runs much faster than t-SNE on large datasets and tends to keep a bit more of the large-scale layout. Its neighbours setting plays a similar role to perplexity: small values emphasize local detail, and large values emphasize the overall shape.
Compare the neighbour-preservation readout for UMAP and t-SNE yourself, and try a few settings. The honest summary from the research literature is that neither method is uniformly better. Both need to be checked, and both can be tuned into showing almost any story you want.
How to use these plots without fooling yourself #
A few rules I follow:
- Do the real work in the original space. Search, clustering and deduplication should use the 384-D vectors directly. The 2-D plot is for looking, not for measuring.
- Check neighbours, not pictures. Click a few points and see whether their true nearest neighbours are nearby on screen. A plot where they aren’t is decorative.
- Try several settings and seeds. Structure that survives changes to perplexity or neighbour count is probably real. Structure that appears only at one setting is probably an artifact.
- Report what was lost. “The first two principal components explain 12.9% of the variance” is a sentence more embedding plots deserve.
The colours here come from each article’s first tag, not from the model. Where colours mix inside a cluster, the model thinks those passages are about the same thing even though I filed them differently. Sometimes the model is right and my tags are the problem. That’s a useful thing to learn from an embedding plot, as long as you don’t read it like a map.