I looked for jets in nuclear collisions for a few years at university. Last week I was debugging a product search and realised I was looking at the same picture again. The panels on the right follow along; the underlined words are clickable.
The bug was boring. A search kept returning the wrong items from a product catalogue. I plotted the documents in 2D, put the query vector in there too, sat back, and got a very specific kind of déjà vu. I have stared at this plot before. Thousands of times. It was about quarks then.
Quick recap for people who did not spend their student years on this. You collide two nuclei, a quark comes flying out, and you never get to see it. Colour confinement will not allow it, so on the way out it pulls new particles from the vacuum and arrives as a spray, all going roughly the same direction. The spray is a . You reconstruct it because the part you actually wanted was over long before anything reached your detector.
The algorithm doing that up there is : repeatedly merge the two closest fragments, where closest means small angle and weighted so the . Watch the counter. Every merge is one line of arithmetic.
Vector search does the same trick with text. Every document gets a direction, documents about the same thing point about the same way, and your is just another direction. Retrieval keeps whatever sits inside a of it. Fragments, angles, and a thing you never observe directly. Same picture, worse units.
The picture is a lie in two respects. Real embeddings have a few thousand dimensions, where everything is nearly perpendicular to everything and none of this looks like anything; we all draw it in 2D anyway, and we all know. And these particular documents were never embedded by anything — I generated them, one direction per topic plus some scatter. A real catalogue is messier and the clusters do not sit still.
If the two really are one algorithm, the jet finder should work on the documents. It does. The right panel is now running , on embeddings instead of particles, and the wedges it draws are topics. It merges on angle and length alone — nobody told it about catalogues. The names on the wedges are the only part that cheats: those come from the catalogue's own tags, counted up after the fact.
k-means, DBSCAN, mean shift and anti-kT are the same errand with different paperwork.
I was pleased with myself for roughly a day. Then I started seeing it in fields where I have no business having opinions:
All of it fits in one fairly dull sentence. Put your things in a space where distance means similarity, turn each one into a little hill, add the hills up, look for the peaks. A peak is a jet if you do physics, a topic if you do search, a cluster if you do astronomy. Statisticians called it a mode of a density in the 1950s and have been watching everyone rename it ever since.
Nobody invented this. It just kept showing up, in places that were definitely not reading each other's journals.
None of them were copying each other. Jet physics needed something statistics had no reason to worry about, and search needed to do it a billion times before lunch, so both built their own. Everyone landed on angles and neighbours anyway, and then named it after their own department.
Jet physics has a rule called . If one particle splits into two going the same way, or some pathetically soft particle wanders in, your jets have to come out exactly the same. It sounds pedantic. It killed off a whole generation of otherwise sensible algorithms.
Split a document into two chunks, the results should not change. Add a filler word to the query, the results should not change. Physicists prove this. Search teams call it chunking and hope.
Clicking that term does it to both panels at once: it splits the hardest particle in two on the left, and cuts one document into two chunks on the right. The jets come back identical. The top-k does not — the second chunk takes a slot, and whatever was last falls out of the results. Nothing about the corpus changed, only how it was sliced. I tried the same thing on the search I was debugging, mostly out of curiosity. It did not pass either. That is a separate post and a separate sulk.
Colliders dump dozens of boring soft collisions on top of the one you actually wanted, so you subtract them. Every physicist has seen somebody subtract too enthusiastically and delete their own signal. The grey dots on the left are ; the grey dots on the right are the catalogue's equivalent, and both panels are highlighting them now. Search filters go wrong in exactly that way. The word over-subtraction was already there, waiting, invented in a building with no e-commerce in it.
Anyone who has done clustering is reading this and thinking it is trivial. It is. Angles, neighbours, peaks. Chapter one, with exercises. Several people have already opened a reply box to tell me about k-means.
I still think it is beautiful. A quark that existed for 10-24 seconds and somebody typing a query into a box get handed the same geometry, and the geometry could not care less which is which. Snow drew it on a map of Soho, Firth wrote it about words, a redshift survey found it in galaxies, and none of them had any idea they were doing each other's job. One idea, wearing whatever coat the local field hands it.
The two panels have a full-screen version with more knobs, if you would rather poke at it than read about it.