Bind corpus version, embed model, and index as one versioned bundle #4

New Issue

2026-05-31T18:31:19+02:00

Brummel commented

2026-05-31 18:31:19 +02:00

Concern

The semantic index is addressed by SHA256(model + exact corpus text) (src/embed.rs:20-26). Two independent inputs therefore silently invalidate the entire index:

A change to the corpus text (e.g. a new ICD-10-GM annual release replacing data/icd10gm2026_alphaidse_edvtxt_20250926.txt) changes every entry's text → every key misses.
A change to embed_model in config (currently BAAI/bge-m3, config/*.toml) changes the model component of every key → every key misses.

Either case forces a full re-embed of all ~90 896 corpus entries against IONOS — expensive in calls, time, and (per the user-level IONOS rule) explicit per-session consent.

Why this is a deployment hazard, not just a rebuild cost

Pipeline::load builds the semantic index all-or-nothing: it constructs the VectorIndex only when every corpus entry already has a cached embedding, otherwise it leaves the vector index None and Mode A degrades to Mode C (src/pipeline.rs:43-50). A partial or mismatched index does not error — it silently downgrades the hybrid retrieval mode to the lexical baseline.

Consequence for deployment: if the deployed index/ does not exactly match the deployed corpus snapshot and the configured embed_model, the service runs the cheaper baseline while reporting Mode A was requested, and the only signal is degraded: true in diagnostics (src/pipeline.rs:233). There is currently no artifact-level binding that prevents shipping a mismatched triple.

What is missing

There is no single versioned unit binding {corpus snapshot, embed model, packed index} together. The three are gitignored and sourced independently (.gitignore excludes /data/, /icd-claml/, /index/), so nothing structurally prevents a corpus update from landing without its matching index.

Shape of the work (unverified — design to be ratified)

A manifest recording the corpus file identity (hash or release id), the embed_model, and the index identity, checked at Pipeline::load time.
A load-time hard check: if a Mode A request is made but the index does not match the corpus+model manifest, fail loudly (or log a prominent warning) rather than silently degrading.
Cross-links to the packed-matrix work: the packed index file is the natural place to embed this manifest.

Acceptance

A mismatch between the deployed index and the corpus+model is detectable at load time, not only via degraded: true after a request.
The intended-mode-vs-actual-mode gap (Mode A requested, Mode C served due to a missing/mismatched index) is observable in operation.

## Concern The semantic index is addressed by `SHA256(model + exact corpus text)` (`src/embed.rs:20-26`). Two independent inputs therefore silently invalidate the entire index: - A change to the **corpus** text (e.g. a new ICD-10-GM annual release replacing `data/icd10gm2026_alphaidse_edvtxt_20250926.txt`) changes every entry's text → every key misses. - A change to **`embed_model`** in config (currently `BAAI/bge-m3`, `config/*.toml`) changes the model component of every key → every key misses. Either case forces a full re-embed of all ~90 896 corpus entries against IONOS — expensive in calls, time, and (per the user-level IONOS rule) explicit per-session consent. ## Why this is a deployment hazard, not just a rebuild cost `Pipeline::load` builds the semantic index all-or-nothing: it constructs the `VectorIndex` only when **every** corpus entry already has a cached embedding, otherwise it leaves the vector index `None` and Mode A degrades to Mode C (`src/pipeline.rs:43-50`). A partial or mismatched index does not error — it silently downgrades the hybrid retrieval mode to the lexical baseline. Consequence for deployment: if the deployed `index/` does not exactly match the deployed corpus snapshot and the configured `embed_model`, the service runs the cheaper baseline while reporting Mode A was requested, and the only signal is `degraded: true` in diagnostics (`src/pipeline.rs:233`). There is currently no artifact-level binding that prevents shipping a mismatched triple. ## What is missing There is no single versioned unit binding {corpus snapshot, embed model, packed index} together. The three are gitignored and sourced independently (`.gitignore` excludes `/data/`, `/icd-claml/`, `/index/`), so nothing structurally prevents a corpus update from landing without its matching index. ## Shape of the work (unverified — design to be ratified) - A manifest recording the corpus file identity (hash or release id), the `embed_model`, and the index identity, checked at `Pipeline::load` time. - A load-time hard check: if a Mode A request is made but the index does not match the corpus+model manifest, fail loudly (or log a prominent warning) rather than silently degrading. - Cross-links to the packed-matrix work: the packed index file is the natural place to embed this manifest. ## Acceptance - [ ] A mismatch between the deployed index and the corpus+model is detectable at load time, not only via `degraded: true` after a request. - [ ] The intended-mode-vs-actual-mode gap (Mode A requested, Mode C served due to a missing/mismatched index) is observable in operation.

Brummel referenced this issue

2026-05-31 18:31:30 +02:00

Pack per-file embedding store into a single mmap matrix #3

Brummel commented

2026-05-31 18:31:30 +02:00

Cross-link: the packed-matrix work referenced under 'Shape of the work' is now #3. The manifest described here should be embedded in the packed index file from #3.

Brummel referenced this issue

2026-05-31 19:19:48 +02:00

Pack per-file embedding store into a single mmap matrix #3

Brummel commented

2026-05-31 19:19:48 +02:00

Coupled with #3: the packed single-file matrix from #3 is the artifact this versioned bundle must bind (corpus version + embed model + index). Decision (2026-05-31): #3 and #4 will be brainstormed/specced together as one cycle in a fresh session, so the pack format and the version-binding scheme are designed coherently instead of #4 reworking #3's alignment.