Image compression for computer vision: February's proposal and September's answer

In February 2026 I wrote a proposal for dermML's dermatology dataset, which was already stored as AV1 video: sort the images by visual similarity before encoding, so neighbouring frames would look alike and the codec could take advantage of it. I estimated compression would go from about 30 times to 50 to 80 times. I did not implement it at the time.

In September I tested the same idea in Nest, with Magic cards, and it did not survive the real corpus. This post brings both ends together: what I was reading, what I proposed and what the data answered.

What I was reading

Two readings convinced me that lossy compression of training data was safe and that video was a real dataset format.

The first was a two-part series by Kaleigh Mentzer at Granica. In part 1, from November 2023, Food101 classification with a Vision Transformer barely lost accuracy until compression went past about 70%, and segmentation models on Cityscapes could be trained on data compressed by 97%, using only 3% of the original storage, with no relevant drop in mIoU. Their explanation is that lossy compression throws away high frequencies first, and the model depends little on them too; in a test with 32 images, perturbing high-frequency DCT coefficients barely moved the prediction. In part 2, with a fixed budget of about 0.5 GB, the original JPEGs fit 9,985 images, and compressing to Butteraugli distance 10 fit about 5.6 times more. With fewer than 100 images per class, more worse images beat fewer better ones by more than 10 accuracy points.

The second was Hugging Face. In the post on encoding robotics datasets as video, from August 2024, datasets ended up at 14% of their original size on average, reaching 0.2% in the best case, and loading several consecutive frames took 25% to 50% of the time of loading the same frames from PNG. LeRobotDataset v3.0, from September 2025, went further: several episodes per file, metadata in Parquet and streaming straight from the Hub.

The proposal

The problem I saw in dermML was frame order. Images went into the video in file system order, alphabetical by name:

Frame 0: ISIC_0024306.jpg  (dark melanoma, light skin)
Frame 1: ISIC_0024307.jpg  (benign nevus, dark skin)
Frame 2: ISIC_0024308.jpg  (dermatofibroma, medium skin)

AV1 encodes a frame from its difference to a reference frame. If the neighbour is a different lesion on different skin, the difference is large and inter-frame prediction does not help. My reading was that the 30 times came almost entirely from compression inside each image.

The idea: DermLIP embeddings, with 512 dimensions, already existed in the indexing pipeline. Finding the order that minimizes the distance between consecutive frames is a travelling salesman problem in embedding space, intractable to solve exactly for 58 thousand points, so the proposal used the simplest heuristic, greedy nearest neighbour:

import numpy as np
from sklearn.metrics.pairwise import cosine_distances

def sort_by_visual_similarity(embeddings):
    order, visited, current = [0], {0}, 0
    for _ in range(len(embeddings) - 1):
        dists = cosine_distances(embeddings[current:current + 1], embeddings)[0]
        dists[list(visited)] = float("inf")
        current = int(np.argmin(dists))
        visited.add(current)
        order.append(current)
    return order

And the estimate I wrote down:

For the full dataset, that would take the 52 GB from something like 1.7 GB to somewhere between 650 MB and 1 GB. The proposal already called for keyint=250 and scd=0 in SVT-AV1, to turn off scene cut detection and avoid an extra keyframe at every new lesion, and for one more SQLite column to map each sorted frame back to its original image. The validation plan was to encode with and without sorting, compare size and quality, and check that search still found the same things.

What September answered

In August Nest started storing images, and in September the Magic card series became the test I had not done in February.

The first round with inter-frame prediction came out bigger than without it. The reason was exactly what the proposal had warned about: scene cut detection was on, and one different card after another is a scene cut on every frame. With it off, on a corpus of almost three thousand reprints of the same art, prediction made the file 29% smaller. Shuffling the reprints cost only two points, because part of the gain came from the frame every card shares. And that 29% compares bytes only, not equivalent quality.

On the real corpus, of unique cards, the sorted build and the plain build came out at the same 1.3 GB. Nest tests both ways per segment before deciding, and it vetoed prediction in all of them. There was nothing to predict: the next card is another card, with another artwork. In the lossless regime it was worse: FFV1 and x264 without quantization, with the cards sorted, came out three to four times bigger than the source JPEGs, and sorting barely moved the size.

What remains

The thesis from the readings still stands: training data tolerates a lot of compression, and video is a good dataset container. What fell was my extrapolation. Sorting by similarity only pays off when the corpus has near-duplicate images, like card reprints or consecutive robot frames. In dermoscopy, each image is a different lesion from a different person, and the estimate of 50 to 80 times was probably optimistic for the same reason as the card one.

The other lesson is about method. Compression is a property of the domain before it is a property of the codec, and the only way to know is to encode the real corpus and look at the file, not the summary.