Nest: a sovereign, content-addressable, offline-first vector DB

Sharing the concept of something I am working on, tired of hacking around vector DBs with no stability. In an ordinary database, a chunk's ID can change when the index is rebuilt, and a saved citation starts pointing somewhere else without telling anyone.

Nest is my answer to that. It is a single-file vector DB, content-addressable, offline-first and hash-verified. Python builds the file, Rust serves the file. The address of any piece is:

nest://content_hash/chunk_id

You cite by content, not by location. If the content changes, the hash changes, and it is impossible to receive something other than what you asked for.

The hash as the address

The central idea is just one: the hash of the content becomes the address of the content. Instead of "give me the file in such-and-such folder", which could have been swapped without you knowing, you say "give me the file whose content has this hash". If something else came back, it would have a different hash. Verification is not an extra step; it is the very way you ask.

In Nest, the two parts of the address play different roles:

• content_hash identifies the whole file by the hash of what it contains
• chunk_id points to the piece inside that file

The practical consequence is the one I wanted: a citation does not break when the file moves to another folder, disk or machine. It is only valid for that content. If the content changed, the hash changed, and you know right away you are talking about something else.

The family

This is not my invention. It is content-addressable storage, and it has famous relatives.

The most everyday one is Git. Each commit is identified by a hash of its content, and a git commit is a sealed snapshot of the code at that moment. If someone rewrites history, the hashes do not match and Git flags it. IPFS does the same thing at scale, on a distributed network of many machines.

Nest sits between the two:

Same mechanism, different purpose. Of the analogies I collected, this is the only one that describes Nest without a metaphor: content addressing for the same reason, so that asking for the data and verifying the data are the same operation.

Shipping versions without downloading everything again

A file that never changes is great for citing and bad for updating. If every new version forces you to download the whole file, nobody updates.

The most technical option is binary diffing. Tools like bsdiff, xdelta and the --patch-from mode of zstd compute the difference between yesterday's file and today's and produce a small patch. Whoever already has yesterday's version downloads only the patch and rebuilds today's locally, the same way an operating system updates without downloading everything again.

The problem is that this only works if the format is append-friendly, meaning that adding content at the end does not mess up what was already there. A deterministic build that reorders things internally, and compressed, indexed formats tend to reorganize a lot, can produce a huge diff for a small change: you change one chunk, the whole index moves, and the patch is no longer small. That is why I think this route is more fragile than splitting the file into shards and shipping only the new shard. And you only find out by testing.

Beyond the hash

The hash proves that something did not change. It does not prove who vouched for it or when the content existed. There are three known layers for that:

• Digital signatures and certificates, the same idea behind https and signed software, which adds the "who"
• Trusted timestamping, such as the RFC 3161 protocol or OpenTimestamps, which anchors the hash in Bitcoin and proves the content existed on a date without revealing the content
• Blockchain, which solves immutability with no central party to trust

Of the three, timestamping fits Nest best. Blockchain, for storing papers and citing passages, would be a sledgehammer to crack a nut: Nest wants content addressing and local verification, not consensus among machines that do not trust each other.

In August, Nest started storing images too, and that became the Magic card series. More soon.