Pack per-file embedding store into a single mmap matrix #3

New Issue

2026-05-31T18:31:19+02:00

Brummel commented

2026-05-31 18:31:19 +02:00

Concern

The semantic index is stored as one file per corpus entry: index/embeddings/<model-sanitized>/<sha256>.f32, each holding a single 1024-dim bge-m3 vector. The current local index is 90 896 files of 4096 bytes each (~356 MB total) — verified by find index/embeddings -name '*.f32' | wc -l and stat of one file. The per-file layout is written and read by EmbeddingStore (src/embed.rs), keyed by SHA256(model + text) (src/embed.rs:20-26).

This layout is a deployment liability. ~91 000 separate inodes are hostile to container image layers (slow unpack, poor caching), object storage (per-object PUT/GET count), and Git-LFS. The payload is only 356 MB but its file count, not its byte size, is the cost driver.

Already a known post-eval step

The per-file store is the proof-of-concept resume mechanism — it lets an interrupted embedding run skip already-computed vectors. Packing it into a single contiguous, mmap-able matrix was always intended as a post-evaluation step, not part of the PoC. This issue lifts that deferred step into the forward queue now that deployment shape is under discussion.

Shape of the work (unverified — design to be ratified)

A packed artifact: one contiguous file of N × 1024 × f32 plus a row-order manifest (corpus-entry order, or an index keyed the same way as today) so VectorIndex::from_vectors can load it in one mmap rather than 90 896 reads.
Keep the per-file store as the build/resume path; the packed matrix is the serve path. They are not mutually exclusive: build with the resumable store, then pack once for deployment.
Pipeline::load currently reads every entry's vector via store.get(&e.text) and builds the in-RAM matrix all-or-nothing (src/pipeline.rs:43-50); the packed-load path would replace that loop with a single mapped read.

Acceptance

A single-file packed matrix can be produced from a complete per-file store.
Pipeline::load can build its VectorIndex from the packed file without reading the per-file store.
Row order / keying is documented so the packed file is verifiably aligned with the corpus snapshot it was built from (see the version-binding concern, issue to be cross-linked).

## Concern The semantic index is stored as one file per corpus entry: `index/embeddings/<model-sanitized>/<sha256>.f32`, each holding a single 1024-dim `bge-m3` vector. The current local index is **90 896 files of 4096 bytes each (~356 MB total)** — verified by `find index/embeddings -name '*.f32' | wc -l` and `stat` of one file. The per-file layout is written and read by `EmbeddingStore` (`src/embed.rs`), keyed by `SHA256(model + text)` (`src/embed.rs:20-26`). This layout is a deployment liability. ~91 000 separate inodes are hostile to container image layers (slow unpack, poor caching), object storage (per-object PUT/GET count), and Git-LFS. The payload is only 356 MB but its file count, not its byte size, is the cost driver. ## Already a known post-eval step The per-file store is the proof-of-concept resume mechanism — it lets an interrupted embedding run skip already-computed vectors. Packing it into a single contiguous, mmap-able matrix was always intended as a *post-evaluation* step, not part of the PoC. This issue lifts that deferred step into the forward queue now that deployment shape is under discussion. ## Shape of the work (unverified — design to be ratified) - A packed artifact: one contiguous file of `N × 1024 × f32` plus a row-order manifest (corpus-entry order, or an index keyed the same way as today) so `VectorIndex::from_vectors` can load it in one mmap rather than 90 896 reads. - Keep the per-file store as the *build/resume* path; the packed matrix is the *serve* path. They are not mutually exclusive: build with the resumable store, then pack once for deployment. - `Pipeline::load` currently reads every entry's vector via `store.get(&e.text)` and builds the in-RAM matrix all-or-nothing (`src/pipeline.rs:43-50`); the packed-load path would replace that loop with a single mapped read. ## Acceptance - [ ] A single-file packed matrix can be produced from a complete per-file store. - [ ] `Pipeline::load` can build its `VectorIndex` from the packed file without reading the per-file store. - [ ] Row order / keying is documented so the packed file is verifiably aligned with the corpus snapshot it was built from (see the version-binding concern, issue to be cross-linked).

Brummel commented

2026-05-31 18:31:30 +02:00

Cross-link: the version-binding concern referenced in the third acceptance item is now #4. The packed matrix produced here is the natural carrier for the manifest described there.

Brummel referenced this issue

2026-05-31 18:31:30 +02:00

Bind corpus version, embed model, and index as one versioned bundle #4

Brummel commented

2026-05-31 19:19:48 +02:00

Coupled with #4: this issue's third acceptance ('verifiably aligned with the corpus snapshot it was built from') is exactly #4's version-binding concern. Decision (2026-05-31): #3 and #4 will be brainstormed and specced together as a single 'versioned bundle' cycle in a fresh session, rather than packing in isolation and reworking the alignment under #4. Open design forks to ratify there: row keying/manifest (implicit corpus-entry order — as Pipeline::load reads today — vs. explicit SHA256 index), and mmap vs. one-shot read for VectorIndex (the from_vectors API would change). Prereq met: eval is GO, so post-eval packing is now in scope.