Brenner Cruvinel - compressão

The encoder had ignored tune=still since the first build

2026-09-02T00:00:00+00:00

The symptom was a number that didn't add up. I had a battery of encoders running in parallel: nineteen configurations across AV1, AVIF, x264, x265, WebP and lossy JPEG XL. Even VVC got in, more out of curiosity than hope. Each one calibrated to the same visual quality, to find out whether Nest's AV1 was the most compact option or just the one closest to home. One of the agents reproduced the Nest build by hand, with the same parameters, and the file came out a different size. Same card, same CRF, same preset, different bytes. I read that in the morning and my first reaction was that the agent had gotten the command wrong.

The probe failed and nobody said a word

It hadn't. The forge asks the encoder for tune=still, the still-picture mode, which is the right one for a stream where every frame is an independent photo. Before asking, it runs a probe to check whether the encoder accepts that mode. The probe ran without a parameter the mode requires, failed, and the code fell back to a generic mode without telling anyone. The manifest recorded the spec, which said "still", not what the encoder had actually done. Since the first build I had been reading a manifest that didn't match the binary, and the only place that recorded the mode was the wrong place.

The difference measured eleven points of visual quality for ten percent more bytes, at the same CRF. The old build was the worst point on the curve: smaller, but ugly in a way that isn't worth it. And nobody would have caught it by watching the video, because what a frame looks like in the player doesn't tell you which tune was used. Only bytes against the bytes of an independent encode.

And it was by chance. The encoder battery existed to answer a different question, and the manual baseline encode was just its reference point. If I had calibrated the battery against the manifest instead of against an encode done from scratch, the bug would have stayed invisible, and every number in the series would have come out ten percent optimistic on bytes and eleven points pessimistic on quality. Now the manifest records the mode the encoder actually used, and the probe passes the right parameter.

Even after the fix there was a second layer. SVT-AV1 has two modes you could call "still": tune=2, optimized for a structural similarity metric, which is what the fixed probe started picking, and tune=3, called image quality, which the documentation itself marks as still images only. The second one gives one more point of quality for one percent more bytes. Not much, but it's the right mode for still pictures, and the recommended build of the full corpus went out with it. The manifest now says which of the two, by number, because "still" on its own already fooled me once.

Variant on the same 2,048 cards	Size	Visual quality
JPEG XL repack, lossless	1.12x smaller	same as source
AV1 from the first build	3.3x smaller	52
AV1 with tune=still	3.0x smaller	63
AV1 with inter	2.5x smaller	62
AVIF	5.5x smaller	43

The AVIF row looks like the winner and it isn't. It sits at a much lower point on the quality curve, so you can't compare it on the same number. Calibrated to the same quality as AV1, it comes out about 12% smaller, and that left an open question about which backend to use for stills. And inter, my favorite bet, came out bigger than tune=still. I explain that in the next post, because the configuration was wrong and the problem was something else.

I won't table the rest of the nineteen, but three findings stayed with me. x264, the encoder for a 2003 standard, was the fastest inside the quality window by an enormous margin, and VVC, the newest codec on the list, cost a hundred times x264's time to land in the middle of the table.

Two different SVT-AV1 presets spat out byte-for-byte identical streams, because the version I use maps both to the same internal mode when the stream is all keyframes. One more case of a parameter I was passing and the encoder was ignoring, this time documented. And lossy JPEG XL, which I expected to compete, only competes at high quality, well above the window that matters for a deck tile. It's a good codec for a target this corpus doesn't ask for.

CapCut was throwing cards away

I had written in the bets that speeding up the video cut bytes, because I had done it in CapCut and the file came out half the size. I reproduced it programmatically: speed it up sixty times, export at thirty frames per second in H.264, the way the editor does. The file came out with half the bytes. And half the cards.

The export samples the sped-up timeline and drops whatever falls between one frame and the next. Of the 2,048 cards, 993 disappeared. What was left got re-encoded to H.264 on top of AV1 that was already lossy. CapCut didn't compress anything, it threw half the dataset away and called it optimization, and I looked at the file size and believed it. Changing only the fps, without speeding up, doesn't change a single byte. It's a timestamp annotation in the container, as ffprobe had already told me on day one and I hadn't properly written down.

The wrong bet yielded a real bug. With an fps other than one in the manifest, the code that fetches a frame by index asked for an instant that didn't exist, because it assumed ordinal and timestamp were the same thing. Fixed, and tested with the fps I never should have changed. The real gain CapCut was pointing at, redundancy between neighboring frames, does exist. It just has to be measured with inter, without losing cards, and that's the subject of the next late night.

What else is the manifest getting wrong

I closed the day with two things validated by hash and a bigger doubt than when I started. The encoder mode was a constant I thought was on, and it had been off since the beginning. CapCut was a number I thought was compression, and it was data loss. Both lived in a manifest or an export I had read and accepted without checking against anything.

If the encoder wasn't doing what the manifest said, the question left over was what else I was reading as a summary instead of validating on disk. I didn't know yet that the answer would come in the middle of the night, with a meeting on the calendar.

My bets before compressing the cards

2026-09-01T00:00:00+00:00

The compression can't lose anything, even if we have to invent it. I wrote that sentence in the plan, that confidently, after the first build. Two lines later I was saying lossless video with the cards sorted by similarity was a path, and further up I had dismissed jpegtran because "lossless doesn't count". I only noticed when I sat down to write the bets and had to define what I was going to measure, and measuring requires saying against what. Lossless relative to what? I had no answer. I had used the same word for three things and hadn't noticed.

Why zstd can't bite into a JPEG

A JPEG has already gone through three stages that exhaust the redundancy a general-purpose compressor knows how to find. The image becomes frequency coefficients, the high-frequency coefficients get rounded (that's where the loss happens), and the result goes through an entropy coder. What comes out is a bitstream where the repeated patterns have already been eaten. zstd looks for exactly that, repeated patterns. It's like trying to fold a sheet of paper that's already folded to the limit: it doesn't get smaller, it just crumples.

So "lossless" on top of JPEG has three meanings, and only one deserves the name. The first is repacking the JPEG in a way where the inverse operation gives back the original file byte for byte. JPEG XL does this: it takes the already-rounded coefficients and rewrites them with a better entropy coder, keeping enough to undo it. The JPEG's loss is still there, but the packaging is tighter.

The second is re-saving the same JPEG with jpegtran or jpegoptim. The pixels stay identical and the bytes change. Lossless for whoever looks, not for whoever checks the hash. And if the source is already optimized, like the files from Scryfall, the gain tends to zero.

The third is what I was proposing without knowing it: decode the pixels and recompress them with another lossless codec, PNG, WebP, or FFV1 video. It looks like the purest meaning and it's the worst. The new codec receives pixels carrying the JPEG's rounding noise as if it were signal, and spends bits representing that noise faithfully. Everything points to the result being bigger than the source JPEG, not smaller. I wrote that and still didn't believe it, hence the bet.

There's a fourth thing, which isn't a meaning but a gap. In 2016, a group from Tianjin University and Microsoft Research Asia published "Lossless Compression of JPEG Coded Photo Collections": they arrange a collection of JPEG photos as a pseudo-video, compensate the difference between neighboring photos, remove the remaining redundancy in the frequency domain, and still give back every original JPEG without losing a bit. Average savings above 31%. That's exactly what my invented idea wanted to be, done the right way. Except I couldn't find an open implementation, and I wasn't going to write a codec that week. So what I could measure was the ceiling with the tools that exist. The real ceiling, if that paper reproduces on Magic cards, is higher, and I want to say that before giving my number.

What I bet and why

I wrote it down before running anything, because the last time I published a number without a written bet I had to correct it two days later, and correcting a published number is worse than losing a bet in public.

Does lossy AV1 video compress much better than loose JPEGs? Yes, that's what the first build showed and what I had seen with dermoscopy. The more alike neighboring cards are, the more the video gains? Yes, and strongly: the database has 100 thousand printings for 52 thousand illustrations, almost half the cards repeat art that already exists on another, and they all share the same frame. If there's a video gain, it lives there.

Does speeding up the video cut bytes? I had dropped one of the MP4s into CapCut, sped it up sixty times, and the file came out half the size. I bet yes, and if it's true it's the cheapest item on the list. Does tar plus zstd gain anything? I think not, for the reason in the section above. And does sorting by similarity help with unique cards too? I think not, I've seen sorting do nothing on another dataset, but the shared frame leaves me unsure.

And the one I wanted most: there's a lossless option better than JPEG, "even if we invent it". After writing out the three meanings, I changed my mind halfway through the bet. The candidate became JPEG XL, and the lossless video sorted by similarity, the invented idea, is going to lose. The two I most wanted to be true, CapCut and lossless, were the two I could least defend. That's usually a sign.

The lossless run went that same night, ten variants on the same 2,048 cards. Tar plus zstd came out one to one, exactly. Re-saving with jpegtran gave less than one percent, the Scryfall JPEGs are already optimized. The ceiling was the JPEG XL repack, about 11% smaller, with all 2,048 original files rebuilt and validated by hash. Pushing the encoder effort to the maximum gives less than one percent more for six times the time, so the default stayed at medium effort.

And the invented idea lost badly. FFV1 and x264 without quantization, with the cards sorted by similarity, came out three to four times bigger than the source JPEG. Sorting, in that regime, barely moved the size at all, a fraction of a megabyte on a file of hundreds. A decoded JPEG pixel carries noise and no codec predicts noise, just as I had written that morning without believing it.

That became Nest's archive profile: anyone who won't accept losing anything gets the JPEG XL repack, 3.6 GB for the whole corpus, and every JPEG comes back bit for bit on request. It's not the number I wanted. It's the number I can defend with a hash, and I prefer that one. The other bets, the lossy ones, were running while I wrote this. The CapCut one, I was already afraid to look at.

Nest has the potential to be a Magic card archiver

2026-08-31T00:00:00+00:00

Felipe Navarro, a friend of mine, is building pasta.cards, an open source app for managing your collection and Commander decks. It runs locally, keeps everything in SQLite, and to work offline it downloads the art of every Magic card ever printed, in two versions: the full card and just the illustration. On my Mac that came to 83 thousand JPGs and 7 GB. Every card that exists, twice, on my disk, and Finder dragging its feet to open the folder.

What caught my attention in that number wasn't the size, disk is cheap. It was the structure. Cards share the frame, the text box, the typography, the mana symbols. And the database next to it shows that almost half of all printings reuse art that already appeared on another card. A loose JPG knows nothing about that. A good catalog points instead of copying, and a good codec doesn't spend bits on a pixel it can predict from its neighbor. I like both for the same reason.

I had Nest at hand, the single-file vector database I've been building this year, and since August it stores images too. The question I asked that morning was far more naive than everything that came after: how does it work, does it turn this into a video? I have 38 thousand images and a database, what does it do with them?

Does it turn this into a video?

It really does turn it into a video. Each image becomes a frame of an AV1 stream inside the file, and search runs on vectors taken from the decoded frames, not from the source JPEG. That's on purpose: the index has to describe what the file serves, and what the file serves is the frame after the codec. Video codecs are the most optimized image compression technology there is, so what looks like a hack is just borrowing decades of other people's work.

I had already measured this on skin. Dermoscopy, biopsy slides, pages of old books, the datasets I had on hand from my work at Hoff. What those numbers taught me is that how much something compresses is a property of the domain, not the codec. Same encoder, same settings: dermoscopy compressed 151 times and the book page compressed 4. Smooth skin compresses well because there's little detail to keep. Small print is the opposite, every edge costs bits.

A Magic card has small print. It has the same frame on every card, which the codec may or may not exploit. It has reprints of the same art, something dermoscopy doesn't have. And it has a product constraint that no search metric measures: the deck tile has to look good on the app's screen. I thought the question was how much it compresses. It was, but not for long.

Before running anything, I took a calmer look at the raw folder, and it had surprises of its own. Half the weight was the cropped version of the art, and my first idea was to throw it away and crop from the full card when needed. An agent read the app's code and came back with bad news: the database doesn't store any crop coordinates, and the app uses the crop directly in the deck tiles. The cropped art is a class of its own, not a derivative. And next to the image folder there was a directory called _redundant, holding another seven gigs of copies of things that were already there. That one I deleted without mercy.

The first build

The pilot with a few hundred cards died with a segfault before doing anything. Nest's Python binding had been compiled without the flag that says "use the process's Python, don't bring your own", and the Python I use is static, so two interpreters were fighting over the same memory. One flag and done. A small bug, and the kind that makes me suspicious: how many other environment assumptions does the build carry without declaring them?

With the flag, the full build ran in the background and I opened one of the videos in the player. Cards going by, one per second, forty minutes of cards. My first hypothesis was that the encoder was repeating frames to hold each card on screen, and that this could be cut. One line of ffprobe killed the hypothesis: one frame per card, and the duration is just a timestamp annotation in the container, it costs no bytes.

7 GB became 1.9 GB, with image search working on top. It ran on the first try, which in a project of mine is rare enough to make me suspicious. Then I opened the output folder.

An archiver with 38 MP4s next to it isn't an archiver

38 video files, a .nest with the indexes, a JSON manifest and a cache directory. Four kinds of file to serve one thing. Nest exists to be the database in one file, and what came out was a database with forty things next to it. It worked, the way a piece of furniture assembled with leftover screws works. The point of Nest was getting lost there, and compression wasn't even the problem. Nest has the potential to be an archiver for Magic cards, for maps, for any image dataset with a database beside it. But an archiver is a file, not a folder.

The second question came from the model. The script used CLIP because that was what the forge had, and I didn't really know what CLIP did with a card. I looked into it: it looks at the art and ignores the printed text. For a Magic card that means looking at the less useful half, because what identifies the card is the name written at the top. I asked for a search on Hugging Face and GitHub of the open multimodal models available that month, and the answer was a Tencent family called WeMM, which reads text inside images, plus SigLIP and Jina. I already had the two-billion-parameter WeMM downloaded for another reason.

In a comparison with 1,500 cards and sixty queries like "the art of such-and-such card", CLIP found the right card one time in four. WeMM got nine in ten, SigLIP came close, Jina landed in the middle. WeMM's price is being about thirty times slower than CLIP, and running it over the whole corpus would take almost a day on my machine. So the comparison stayed on the sample, and the full corpus kept CLIP for now.

What came out of this wasn't "switch to WeMM". It was that the model can't be a constant at the top of a script. It has to be a list of presets inside the engine, and whoever builds picks which one, or which ones, at build time. And the video has to fit inside the file, not next to it. Before either of those two things, I wanted to know how far I could compress without losing anything. Even if we had to invent it.

Image compression for computer vision: February's proposal and September's answer

2026-03-12T00:00:00+00:00

In February 2026 I wrote a proposal for dermML's dermatology dataset, which was already stored as AV1 video: sort the images by visual similarity before encoding, so neighbouring frames would look alike and the codec could take advantage of it. I estimated compression would go from about 30 times to 50 to 80 times. I did not implement it at the time.

In September I tested the same idea in Nest, with Magic cards, and it did not survive the real corpus. This post brings both ends together: what I was reading, what I proposed and what the data answered.

What I was reading

Two readings convinced me that lossy compression of training data was safe and that video was a real dataset format.

The first was a two-part series by Kaleigh Mentzer at Granica. In part 1, from November 2023, Food101 classification with a Vision Transformer barely lost accuracy until compression went past about 70%, and segmentation models on Cityscapes could be trained on data compressed by 97%, using only 3% of the original storage, with no relevant drop in mIoU. Their explanation is that lossy compression throws away high frequencies first, and the model depends little on them too; in a test with 32 images, perturbing high-frequency DCT coefficients barely moved the prediction. In part 2, with a fixed budget of about 0.5 GB, the original JPEGs fit 9,985 images, and compressing to Butteraugli distance 10 fit about 5.6 times more. With fewer than 100 images per class, more worse images beat fewer better ones by more than 10 accuracy points.

The second was Hugging Face. In the post on encoding robotics datasets as video, from August 2024, datasets ended up at 14% of their original size on average, reaching 0.2% in the best case, and loading several consecutive frames took 25% to 50% of the time of loading the same frames from PNG. LeRobotDataset v3.0, from September 2025, went further: several episodes per file, metadata in Parquet and streaming straight from the Hub.

The proposal

The problem I saw in dermML was frame order. Images went into the video in file system order, alphabetical by name:

Frame 0: ISIC_0024306.jpg  (dark melanoma, light skin)
Frame 1: ISIC_0024307.jpg  (benign nevus, dark skin)
Frame 2: ISIC_0024308.jpg  (dermatofibroma, medium skin)

AV1 encodes a frame from its difference to a reference frame. If the neighbour is a different lesion on different skin, the difference is large and inter-frame prediction does not help. My reading was that the 30 times came almost entirely from compression inside each image.

The idea: DermLIP embeddings, with 512 dimensions, already existed in the indexing pipeline. Finding the order that minimizes the distance between consecutive frames is a travelling salesman problem in embedding space, intractable to solve exactly for 58 thousand points, so the proposal used the simplest heuristic, greedy nearest neighbour:

import numpy as np
from sklearn.metrics.pairwise import cosine_distances

def sort_by_visual_similarity(embeddings):
    order, visited, current = [0], {0}, 0
    for _ in range(len(embeddings) - 1):
        dists = cosine_distances(embeddings[current:current + 1], embeddings)[0]
        dists[list(visited)] = float("inf")
        current = int(np.argmin(dists))
        visited.add(current)
        order.append(current)
    return order

And the estimate I wrote down:

Scenario	Estimated compression
File system order	~30x
Sorted by similarity	~50 to 80x
Sorted, with CRF and GOP tuning	~60 to 100x

For the full dataset, that would take the 52 GB from something like 1.7 GB to somewhere between 650 MB and 1 GB. The proposal already called for keyint=250 and scd=0 in SVT-AV1, to turn off scene cut detection and avoid an extra keyframe at every new lesion, and for one more SQLite column to map each sorted frame back to its original image. The validation plan was to encode with and without sorting, compare size and quality, and check that search still found the same things.

What September answered

In August Nest started storing images, and in September the Magic card series became the test I had not done in February.

The first round with inter-frame prediction came out bigger than without it. The reason was exactly what the proposal had warned about: scene cut detection was on, and one different card after another is a scene cut on every frame. With it off, on a corpus of almost three thousand reprints of the same art, prediction made the file 29% smaller. Shuffling the reprints cost only two points, because part of the gain came from the frame every card shares. And that 29% compares bytes only, not equivalent quality.

On the real corpus, of unique cards, the sorted build and the plain build came out at the same 1.3 GB. Nest tests both ways per segment before deciding, and it vetoed prediction in all of them. There was nothing to predict: the next card is another card, with another artwork. In the lossless regime it was worse: FFV1 and x264 without quantization, with the cards sorted, came out three to four times bigger than the source JPEGs, and sorting barely moved the size.

What remains

The thesis from the readings still stands: training data tolerates a lot of compression, and video is a good dataset container. What fell was my extrapolation. Sorting by similarity only pays off when the corpus has near-duplicate images, like card reprints or consecutive robot frames. In dermoscopy, each image is a different lesion from a different person, and the estimate of 50 to 80 times was probably optimistic for the same reason as the card one.

The other lesson is about method. Compression is a property of the domain before it is a property of the codec, and the only way to know is to encode the real corpus and look at the file, not the summary.