Brenner Cruvinel - vector db

At the end of the day it's the same thing, 1.3 GB

2026-09-03T00:00:00+00:00

I was sure. Sorting the cards by similarity before encoding would multiply the compression, because similar neighboring frames are exactly what video codecs love. I had written it in the bets, I had repeated it to the agent, and the benchmark had handed me back a strange row: the variant with inter came out bigger than the one without. The configuration was wrong, scene cut detection was on, and different card after different card is one scene cut per frame. The encoder was burning cycles looking for a scene that didn't exist. I turned it off and put together a second corpus, almost three thousand real reprints of the same art, grouped together.

That did it. With prediction in groups of sixteen frames, the file came out 29% smaller than without prediction. Groups of sixteen beat a single keyframe for the whole stream, which surprised me, and they still keep any card reachable by decoding at most sixteen frames. And a detail I didn't expect: shuffling the reprints instead of grouping them cost only two points. Part of the gain comes from the frame every card shares, not from the repeated art. The codec was exploiting the chrome, as I had suspected on day one, and that held for any order.

Building the whole corpus at night

With 29% in hand, I built the full corpus with similarity sorting and prediction on, and next to it the same corpus with none of that, to compare. And I built it with the media inside the file, because by then I no longer accepted MP4s sitting next to the database. Before getting there I opened the folder of an intermediate build and found .build, .cache, .forge-state, .tmp, a lock, a manifest, all next to a .nest of a few megabytes. What is all this for? I asked, and the honest answer was that every piece had a reason and none of them needed to be there once the build finished. Media inside the file was half the answer, the other half was cleaning up the output.

A .nest is a container of sections, each with its own hash. The new section is the simplest of all: a table of offsets and the raw bytes of the videos, glued one after another. An MP4 stays an MP4, nothing is re-encoded to fit. Opening the file doesn't touch the media, the system only maps the slice when someone asks for a card, and you can export the original MP4 back out with the hash checked before writing.

That section doesn't count toward Nest's content hash, and this matters more than it seems. The content hash is computed over what's inside the file, not over the packaging, and it's what makes a Nest citation point to a piece of knowledge instead of a place on disk. The same corpus built with the media inside and with the media outside gives the same hash. It cost three percent: the indexes, the card text and the vectors add up to about forty megabytes on top of 1.3 GB of video.

The builds ran overnight, and the night was rough. Two died with a full disk, because macOS was taking system snapshots in waves, eating and releasing gigabytes without warning, and at one point I was down to a hundred and thirty megabytes free. And in the middle of that I read a summary from the agent saying the corpus had 200 cards. I had a client meeting in a few hours.

"Seriously, you used 200 demo cards? I have a client meeting and I still don't have the dataset." I had it run validation on the 1.3 GB file. All 38 thousand blobs checked out, one by one. The corpus was complete, the summary was describing the pilot, and I had read the summary instead of the disk. After the encoder that wasn't using the tune the manifest claimed, it was the second time that week that what I read wasn't what was actually there.

At the end of the day

The sorted build sat next to the plain build. 1.3 GB both, no difference at all. I sent it to the agent at four in the morning: at the end of the day it's the same thing, right, 1.3 GB.

The reason was in the manifest, correct this time. Nest decides per segment whether to use prediction, encoding a sample both ways. On the real corpus, made of unique cards, the probe vetoed prediction in every segment. There's nothing to predict: the next card is another card, with other art, and the shared chrome doesn't pay for the cost. The 29% gain only exists on the reprint corpus, and I haven't built that one in full yet.

There was a worse part, which I only understood later. In an earlier version of the probe, it decided on bytes alone, and in a build of the sample it picked prediction for every segment because it tied on size. Median quality dropped sixteen points. The encoder compresses predicted frames with coarser quantization, and with unique cards a predicted frame is just a worse keyframe in disguise. Equal bytes aren't equal quality. The probe now measures both sides, accepts losing at most two points, and records the reason when it vetoes. The 29% matrix also compares bytes only, so the number is real but it isn't equivalent quality. I'm putting that down here because I forgot it myself for a whole day.

What if the file doesn't need to be pretty

What I take from that night isn't the 1.3 GB. It's that my intuition about codecs was wrong twice, and that I read summaries when I should read files. Now I only turn sorting on when the corpus has near-duplicates, and no build goes in without validation of the real file running alongside it.

And one question was left over that seemed more promising than any sorting. Nest's quality gate locks the codec at a conservative level because the visual metrics never pass on small cards with tiny print. But Felipe's app keeps every card's JPEG cached for display. The .nest only needs to answer which card it is. If the file doesn't need to be pretty, how far can you push the setting before search starts complaining?

Nest: a sovereign, content-addressable, offline-first vector DB

2026-01-15T00:00:00+00:00

Sharing the concept of something I am working on, tired of hacking around vector DBs with no stability. In an ordinary database, a chunk's ID can change when the index is rebuilt, and a saved citation starts pointing somewhere else without telling anyone.

Nest is my answer to that. It is a single-file vector DB, content-addressable, offline-first and hash-verified. Python builds the file, Rust serves the file. The address of any piece is:

nest://content_hash/chunk_id

You cite by content, not by location. If the content changes, the hash changes, and it is impossible to receive something other than what you asked for.

The hash as the address

The central idea is just one: the hash of the content becomes the address of the content. Instead of "give me the file in such-and-such folder", which could have been swapped without you knowing, you say "give me the file whose content has this hash". If something else came back, it would have a different hash. Verification is not an extra step; it is the very way you ask.

In Nest, the two parts of the address play different roles:

content_hash identifies the whole file by the hash of what it contains
chunk_id points to the piece inside that file

The practical consequence is the one I wanted: a citation does not break when the file moves to another folder, disk or machine. It is only valid for that content. If the content changed, the hash changed, and you know right away you are talking about something else.

The family

This is not my invention. It is content-addressable storage, and it has famous relatives.

The most everyday one is Git. Each commit is identified by a hash of its content, and a git commit is a sealed snapshot of the code at that moment. If someone rewrites history, the hashes do not match and Git flags it. IPFS does the same thing at scale, on a distributed network of many machines.

Nest sits between the two:

	What it seals	What it is for	Scale
Git	Code	Navigating versions	Repository
IPFS	Any file	Distribution	Network of machines
Nest	Vectorized knowledge	Search	One local file

Same mechanism, different purpose. Of the analogies I collected, this is the only one that describes Nest without a metaphor: content addressing for the same reason, so that asking for the data and verifying the data are the same operation.

Shipping versions without downloading everything again

A file that never changes is great for citing and bad for updating. If every new version forces you to download the whole file, nobody updates.

The most technical option is binary diffing. Tools like bsdiff, xdelta and the --patch-from mode of zstd compute the difference between yesterday's file and today's and produce a small patch. Whoever already has yesterday's version downloads only the patch and rebuilds today's locally, the same way an operating system updates without downloading everything again.

The problem is that this only works if the format is append-friendly, meaning that adding content at the end does not mess up what was already there. A deterministic build that reorders things internally, and compressed, indexed formats tend to reorganize a lot, can produce a huge diff for a small change: you change one chunk, the whole index moves, and the patch is no longer small. That is why I think this route is more fragile than splitting the file into shards and shipping only the new shard. And you only find out by testing.

Beyond the hash

The hash proves that something did not change. It does not prove who vouched for it or when the content existed. There are three known layers for that:

Digital signatures and certificates, the same idea behind https and signed software, which adds the "who"
Trusted timestamping, such as the RFC 3161 protocol or OpenTimestamps, which anchors the hash in Bitcoin and proves the content existed on a date without revealing the content
Blockchain, which solves immutability with no central party to trust

Of the three, timestamping fits Nest best. Blockchain, for storing papers and citing passages, would be a sledgehammer to crack a nut: Nest wants content addressing and local verification, not consensus among machines that do not trust each other.

In August, Nest started storing images too, and that became the Magic card series. More soon.