Raising the CRF until search breaks

The question from the previous night was simple to ask and I took my time asking it. The app keeps every card's JPEG cached for display. The .nest doesn't need to be pretty, it needs to answer which card it is. So the CRF, the codec's quality setting, can go up until the vector breaks, and the vector breaks well after the eye does. If that were true, it would even be worth thinking about two tiers: a thumbnail inside Nest for search, and the full card on demand when the app needs to show it.

I had a way to measure this, or thought I did. Nest's quality gate has two sides: a visual metric, which compares the decoded frame with the source, and vector drift, which measures how much the CLIP vector moves between the original image and the decoded one. The idea behind drift is that an image can look fine and still have moved in search space. I trusted that metric. It was mine, I had insisted on it in the plan, and for good reason: on dermoscopy it had caught a case where the image passed by eye and search by clinical text dropped by half.

The gate rejected the whole ladder

I took the visual side out of the gate and left only drift, with the floor Nest has used since dermoscopy. I had the CRF climb a ladder from 40 to 60. The gate rejected the whole ladder, including the first rung, and fell back to the most conservative one because it had nowhere else to go. Drift was already below the floor at CRF 40 and only got worse. And the visual metric, which I had taken out of the gate but kept computing, told the same story from another angle: from CRF fifty on it gave negative values on some cards, something I didn't even know it could do.

The file the gate delivered, at CRF 40, the lowest on the ladder, came out close to one gig, four times smaller than the source. It was already smaller than the recommended build, and the gate had rejected it too, it just had nowhere lower to fall.

By the gate's logic, none of these files was usable. I built one at CRF fifty anyway, no gate, just to see the size. Half a gig. Seven and a half times smaller than the source, and the cards turned into a blur, rules text unreadable, the art recognizable only from a distance. A file I wouldn't show anyone.

Then I ran the metric that actually matters, which I hadn't run before because I was too lazy to set it up: search by name. A hundred queries like "the art of such-and-such card", on the five files I had of the full corpus, from the 3.6 GB lossless one down to the half-gig blur. The first round gave zero hits on two of the five files, the lossless and the recommended one, and for a minute I thought I had broken search. It was the bench: it matched results by the card's position in the stream, and the two files with similarity sorting have a different order. Nest has a content id per item for exactly this, and once the bench matched on that, all five files answered again.

And accuracy stayed within the noise on all of them. The half-gig file finds the right card as well as the 1.3 GB one, and both stay within the noise of the lossless file.

Search survives far more compression than the eye

I stared at that on screen for a while. Cosine drift measures whether the vector moved. Search measures whether the vector still finds. I was using the first as if it were the second, and it would have vetoed a file that works. The vector does move, and it still stays closer to the right card than to any other. How far it travels doesn't matter if its neighbors don't change. A gate that's any good for this use needs hit@k as its floor, the fraction of queries where the right card shows up in the top k results, not cosine.

That became a profile in Nest, retrieval-only, which writes into the manifest that the file serves search and never display. And AVIF, which in the benchmark had come out about 12% smaller than AV1 at the same quality, held up on the full corpus: 13% smaller than the recommended build. It's the candidate to become the default backend for stills, and I haven't decided yet, because AVIF takes four to ten times longer to build and embed.

There's a third metric I ran alongside that's worth nothing, and it's worth saying why. Identity: give the image itself as the query and see if it finds itself. It gave a hundred percent on every file, even the blur. Looks great and says nothing, because the query is the very image that produced the frame, and the uncompressed control also gives a hundred. It only proves the codec didn't destroy the data. I left it in the bench so nobody confuses pipeline stability with usefulness, which is exactly the confusion I made with drift.

The metric is weak and I know it. It's a hundred queries with a model that does poorly on this corpus, CLIP, which only looks at the art. "Didn't drop" means "I couldn't measure a drop", not "it's the same". And WeMM, which reads the name printed on the card and gets nine in ten right, might suffer far more from the blur than CLIP, because what it reads is exactly what the CRF destroys first. I didn't measure it. That's next, along with the full reprint corpus, which is where inter has something to predict.

What stuck with me that week was less about compression and more about what I choose to measure. I had a metric I designed myself, one that looked rigorous, and it answered a question I wasn't asking. And before publishing any of these numbers, I wanted to validate each one against the disk, because a report written in the middle of the night gets things wrong.