Validating on disk the benchmark I wrote in the middle of the night

I opened a new session, with no context at all, and asked as if I were someone who had just cloned the repo: how do you use Nest, does it have an interface? The first command the answer suggested returned command not found on my own machine. The second one showed a default model I was sure I had swapped out weeks earlier. Two holes no test catches, found in five minutes, and that's when I realized the topic of the week wasn't compression.

The benchmark was just the place where it became visible. So, before publishing any number from the series, I spent a whole day validating: every number against the file on disk.

What was wrong

The first thing that didn't match wasn't even a number. I ran strings on the release file and out came "Spellbook MTG corpus". The file is called mtgdataset. The rename from a week earlier, which swapped Felipe's app name for the generic name in every text file, never made it into the binary. The lock next to the release described a file with the new name while the file had the old name inside, and no tool flagged it, because validation checks the file against itself. I rebuilt both releases with the right name inside, and the media bytes came out identical, which at least proved the encode is reproducible.

The sizes of the three most aggressive builds were in mebibytes labeled as gigabytes. du talks in MiB and I pasted it as GB. The difference is five percent per row, but the ratio inherits the error: the search build I had announced as 7.8 times smaller than the source is 7.5. The AVIF I said was 20% smaller than the recommended build is 13%. No conclusion changes, and it still made me ashamed, because it's the kind of mistake I'd point out in someone else's report.

The strangest one was the manifest of the AVIF build, which records a 21 GB source. The source is 4. I couldn't make sense of it until I divided by the number of cards: it gives the size of an uncompressed image, which is what the encoder receives midway through. The backend counted the intermediate instead of the JPEG. One line to fix, one test to write, and until then the ratio in that manifest was made up.

And there's what can no longer be verified. I deleted the first full build when the disk filled up, so the sentence "the three builds share the same hash" is now two-thirds checkable. The CapCut experiment and the reprint matrix left no artifacts, only text. I told it to delete them myself, to free up space. Disk space is cheaper than trust and I should have known that.

The other times I read the summary and not the file

The 200-card summary. I asked for the whole dataset, got a summary that talked about the corpus as if it were the corpus, and it was a pilot. I noticed with a meeting already scheduled. The agent didn't lie, it summarized the scope without saying so, and I read the summary instead of the disk. Since then every build delivery comes with validation of the real file, and this post exists for the same reason.

My machine's paths. Days after the benchmark I found my home directory path in dozens of files headed for GitHub, inside manifests, locks, specs. The fix was an environment variable and a script that rewrites the sidecars before committing. And before that the agent had wanted to store my preferences in a local memory file. I asked it to delete it. If the workflow depends on what it remembers about my machine, it won't run in the next repo, and it won't run on the machine of whoever clones it.

The five-gig repo. At some point I asked how ML engineers deal with this, because GitHub won't take it. The answer I adopted: not a single image byte goes into git. Each sample is a list of ids with the rule that generated it, and whoever has the SQLite database rebuilds the exact sample. The .nest files go to Hugging Face with a hash alongside, and the repo keeps the lock, the clean manifest and the hash. The whole report is generated from each experiment's JSON, and a CI check fails if the markdown differs from the JSON. I can no longer paste a wrong number into a report without CI complaining, which is the kind of protection I needed against myself.

The commits. The agent prefers one fat commit at the end of the day. I asked for sixteen, separate, with descriptions, because the history has to be readable for whoever comes later. And the plans: I rejected three in a row because they treated the model comparison as a footnote to a build already decided. The model list that exists in Nest today only exists because I insisted that research becomes a feature, never a footnote.

What these episodes have in common is just one thing: in all of them I accepted a layer between me and the file. Summary, manifest, memory, fat commit. And the layer was wrong and didn't say so.

What I'll try next time

Something that put me more at ease, in the middle of the validation, was finding out that the two findings I liked most already had names. The redundancy between images in a collection, which inter exploits, has been called set redundancy since Kosmas Karadimitriou's 1996 doctoral thesis, and the trick of lining images up as a pseudo-video has a decade of literature behind it. And the distinction between visual quality, vector stability and search usefulness, which I thought I had invented, is everyday business in the field the standards call coding for machines: it evaluates with task metrics, like mAP, precisely because feature fidelity and task performance can drift apart. I didn't discover anything. I confirmed two things on a corpus nobody had looked at, and that makes me calmer about publishing than if it were something new.

The 1.3 GB file is already in the pasta.cards repo, waiting for the screen that will use it. What's left of the benchmark fits in a paragraph: measure search with WeMM, build the full reprint corpus, decide whether AVIF becomes the default. None of the three has a date.

What I'll try is the cheapest thing of all: open a clean session as a user before every release and type the first command in the readme. Five minutes, and it found two holes a week of agents didn't. I don't know if I'll remember to do it. But now it's written down, and I trust what's written more than what I remember.