diff --git a/docs/specs/2026-05-31-packed-versioned-index-bundle.md b/docs/specs/2026-05-31-packed-versioned-index-bundle.md new file mode 100644 index 0000000..d4610e6 --- /dev/null +++ b/docs/specs/2026-05-31-packed-versioned-index-bundle.md @@ -0,0 +1,289 @@ +# Packed, Versioned Index Bundle — Design Spec + +**Date:** 2026-05-31 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Turn the semantic index from a deployment liability into a single +self-describing, version-bound artifact, without disturbing the build path. + +Two concerns, cross-linked by their natural shared artifact (issues #3, #4): + +- **#3 — pack.** The index is stored as roughly one file per corpus entry + (`index/embeddings//.f32`, ~90 896 files of 4096 B + each, ~356 MB) — "roughly" because the store is keyed by + `SHA256(model + text)`, so text-identical corpus entries collapse to a + single file: the corpus has 90 898 entries + (`tests/corpus_tests.rs::loads_full_real_corpus`) but ~90 896 unique store + files. The file *count*, not the byte size, is hostile to container image + layers, object storage, and Git-LFS. `Pipeline::load` opens every one of + those files individually (`src/pipeline.rs:43-50`). +- **#4 — bind.** The index is keyed by `SHA256(model + text)` + (`src/embed.rs:19-25`). A new corpus release or a changed `embed_model` + silently invalidates every key; `Pipeline::load` then leaves the vector + index `None` and Mode Hybrid degrades to Mode Lexical with only + `degraded: true` to show for it — the same anonymous signal a transient + IONOS outage raises. Nothing binds `{corpus snapshot, embed model, index}` + as one unit. + +This cycle delivers a single packed matrix file that carries a manifest in +its header, and a load-time check that makes a corpus/model mismatch a +distinct, observable state — loud but not fatal, so Mode Lexical (which never +needs the index) keeps serving. + +The per-file store is **not** removed. It stays the build/resume path; the +packed file is an additive serve-path artifact. + +## Architecture + +Two iterations, one file format. The format (magic + manifest header + +payload) is born in iteration 1 and is stable from then on; iteration 2 adds +only the *checking* and the *observability*, never a format rewrite. + +- **Iteration 1 (#3):** a new `src/packed.rs` module and an + `alpha-id index --pack` subcommand. Packing reads a *complete* per-file + store and writes `index/packed..bin`. `Pipeline::load` prefers + the packed file and builds the `VectorIndex` by **positional order** + (row `i` = corpus entry `i`), falling back to the per-file store (today's + path) when no packed file is present. +- **Iteration 2 (#4):** the corpus-hash binding in the header is now + *enforced* at load: a load-time mismatch check, a distinct `index_status` + carried in diagnostics, a prominent load-time stderr warning, and a Mode + Hybrid degrade that names the reason instead of an anonymous `degraded`. + +The packed-file path is derived, not configured: `` + +sanitized `embed_model`, exactly the sanitization `EmbeddingStore::open` +already applies (`src/embed.rs:27-33`). No new config field; `load` and +`--pack` compute the same path independently. + +Row order is positional in corpus-load order (`corpus::load`, +`src/corpus.rs`): the packed matrix has exactly `n_rows = entries.len()` +rows (90 898 for the current corpus), one per corpus entry in load order. +This is the *entry* count, which exceeds the unique store-file count when +two entries share identical text — each such entry still gets its own row, +re-read from the one shared store file, so positional alignment holds. +Positional addressing is safe *because* the header binds the corpus +identity: if `corpus_sha256` matches the current corpus file, the +row-to-entry alignment is guaranteed; if it does not, the check fires and +the index is refused. The hash binding is what licenses dropping the +per-row SHA256 key. + +## Concrete code shapes + +### User-facing surface (the empirical evidence) + +Producing the bundle, once, after a complete embedding run: + +```text +$ alpha-id index --pack +packed 90898 rows × 1024 dims → index/packed.BAAI_bge-m3.bin (356 MB) +manifest: corpus=icd10gm2026_alphaidse_edvtxt_20250926.txt (sha256 4f3a…b1), model=BAAI/bge-m3 +``` + +Packing an incomplete store is refused loudly: + +```text +$ alpha-id index --pack +error: store incomplete: 41020 of 90898 entries embedded; run `index --full --confirm` first +$ echo $status +2 +``` + +`suggest` and `eval` are unchanged at the CLI; `load` picks up the packed +file automatically. With a matching bundle: + +```text +$ alpha-id suggest note.txt --mode hybrid +… suggestions … +[3 segments, mode Hybrid, 41 ms, degraded=false, index=ok] +``` + +With a corpus that no longer matches the deployed index — the formerly +silent downgrade is now named at load time and in the per-request line: + +```text +$ alpha-id suggest note.txt --mode hybrid +warning: packed index index/packed.BAAI_bge-m3.bin does not match the current corpus + (corpus sha256 mismatch); semantic index disabled, Mode Hybrid will degrade to Lexical +… suggestions … +[3 segments, mode Hybrid, 12 ms, degraded=true, index=mismatch: corpus sha256] +``` + +### Must-fail fixture (the criterion's evidence for the binding) + +A packed file whose header model disagrees with the configured model MUST +NOT yield a vector index. The test pins exactly that rejection: + +```rust +// A packed file built declaring embed_model "other/model", loaded under a +// config whose embed_model is "BAAI/bge-m3", must refuse the index. +let (vector, status) = load_packed_or_store(&cfg, &entries); +assert!(vector.is_none()); +assert!(matches!(status, IndexStatus::Mismatch(_))); +``` + +### Header (written at pack, checked at load) + +```rust +#[derive(serde::Serialize, serde::Deserialize)] +pub struct PackedHeader { + pub n_rows: u32, + pub dim: u32, + pub embed_model: String, // must equal cfg.embed_model + pub corpus_sha256: String, // must equal corpus_sha256(cfg.alpha_id_path) + pub corpus_name: String, // human-readable; not part of the check +} +``` + +On-disk layout: + +```text +0..4 magic b"AIPK" +4..8 format_version u32 LE (fast reject without a JSON parse) +8..12 header_len u32 LE +12.. header (JSON, UTF-8, the PackedHeader above) + pad to a 4-byte boundary +payload n_rows × dim × f32, little-endian, row-major, corpus order +``` + +### `Pipeline::load` — before → after + +The load-bearing change replaces the per-entry collection +(`src/pipeline.rs:43-50`) with a packed-first helper. + +```rust +// before +let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) { + Ok(store) if entries.iter().all(|e| store.has(&e.text)) => { + let vecs: Option>> = + entries.iter().map(|e| store.get(&e.text)).collect(); + vecs.map(VectorIndex::from_vectors) + } + _ => None, +}; +``` + +```rust +// after — packed file preferred; manifest enforced; per-file store is the fallback +let (vector, index_status) = load_packed_or_store(cfg, &entries); +if let IndexStatus::Mismatch(reason) = &index_status { + eprintln!("warning: packed index {} does not match the current corpus/model \ + ({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical", + packed_path(&cfg.index_dir, &cfg.embed_model).display()); +} +// `index_status` is stored on the Pipeline and copied into Diagnostics per request. +``` + +### `IndexStatus` and the diagnostics surface + +```rust +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub enum IndexStatus { + Ok, // packed file present and matches corpus + model + Absent, // no packed file and store incomplete — intentional lexical-only + Mismatch(String), // packed file present but corpus/model/shape disagree; reason carried +} +``` + +`Diagnostics` (`src/model.rs:114-121`) gains an `index_status: IndexStatus` +field. The transient `degraded` flag keeps its meaning (a runtime IONOS step +failed); `index_status` answers the orthogonal question "is the deployed +index structurally the right one". A request can be `degraded: true` for +*either* reason, but `index=mismatch` names the structural one. + +## Components + +| File | Change | +|---|---| +| `src/packed.rs` *(new)* | `PackedHeader`; `write_packed(path, &PackedHeader, rows: &[Vec])`; `read_packed(path) -> Result<(PackedHeader, Vec>), AppError>`; `packed_path(index_dir, model) -> PathBuf`; `corpus_sha256(path) -> Result` | +| `src/pipeline.rs` | `load_packed_or_store(cfg, entries) -> (Option, IndexStatus)` replacing the `:43-50` match; `Pipeline` stores `index_status`; `suggest` copies it into `Diagnostics` | +| `src/model.rs` | `enum IndexStatus`; `index_status` field on `Diagnostics` | +| `src/bin/alpha_id.rs` | `--pack` flag on the `Index` subcommand; the load-time mismatch warning; `index=…` in the human diagnostics line | +| `src/vector.rs` | **unchanged** — still consumes `Vec>` via `from_vectors` | +| `src/embed.rs` | **unchanged** — the per-file store stays the build/resume path | + +## Data flow + +- **pack** (`index --pack`): `corpus::load` → open the per-file store → + require every entry cached (else refuse, exit 2) → collect rows in corpus + order → `write_packed(header { embed_model, corpus_sha256, n_rows, dim }, + rows)`. +- **load** (`Pipeline::load`): if `packed_path` exists → `read_packed` → + validate `header.embed_model == cfg.embed_model`, + `header.corpus_sha256 == corpus_sha256(cfg.alpha_id_path)`, + `header.n_rows as usize == entries.len()`, payload length consistent → + on full match build `VectorIndex::from_vectors`, status `Ok`; on any + disagreement status `Mismatch(reason)`, vector `None`. If no packed file → + per-file store fallback (today's all-or-nothing build); complete ⇒ build, + status `Ok`; incomplete ⇒ `None`, status `Absent`. +- **suggest**: Mode Hybrid with `vector == None` degrades to + `collect_lexical` (unchanged); `Diagnostics.index_status` carries the + structural reason; `Diagnostics.degraded` stays the transient signal. + +## Error handling + +- **`--pack` on an incomplete store** → loud refusal, exit code 2, a message + naming the cached/total counts and pointing at `index --full --confirm`. + Packing a partial store is forbidden: it would mispair rows against the + corpus. +- **Corrupt or truncated packed file** (payload length ≠ + `n_rows × dim × 4`), **wrong magic**, or a **`format_version` newer than + supported** → `IndexStatus::Mismatch(reason)`, prominent load warning, no + index built, Mode Lexical serves. Not fatal — a structurally wrong index + must not take down the lexical service (chosen mismatch posture). +- **Corpus file unreadable** for hashing → surfaces as the existing + `corpus::load` error before any packed-load work (corpus load precedes it). + +## Testing strategy + +- **Round-trip (unit):** `write_packed` then `read_packed` reproduces the + header and every row bit-exactly (f32 little-endian preserved). +- **Corpus hash (unit):** `corpus_sha256` is stable for identical bytes and + differs when the corpus file changes by one byte. +- **Must-fail binding (integration):** a packed file built declaring one + `embed_model` (or one `corpus_sha256`), loaded under a config that + disagrees, yields `vector == None` and `IndexStatus::Mismatch(_)`. Wrong + bundle never builds an index. +- **Positional alignment (integration):** after `read_packed`, row `i` + equals `store.get(&entries[i].text)` for sampled `i` — packing preserves + corpus-order alignment. +- **Observable degrade (integration):** under a corpus-hash mismatch, a Mode + Hybrid `suggest` returns `degraded == true` *and* `index_status == + Mismatch(_)`, distinguishable from a transient-IONOS degrade (which leaves + `index_status == Ok`). +- **Packed beats store (integration):** with a valid packed file present and + the per-file store absent, `Pipeline::load` builds the `VectorIndex` from + the packed file alone (acceptance #3b). + +## Acceptance criteria + +**Issue #3 (pack):** + +- [ ] A single-file packed matrix is produced from a complete per-file store + via `alpha-id index --pack`. +- [ ] `Pipeline::load` builds its `VectorIndex` from the packed file without + reading the per-file store. +- [ ] Row order is positional in corpus order, documented here, and + verifiably aligned to the corpus snapshot via the header's `corpus_sha256`. + +**Issue #4 (bind):** + +- [ ] A mismatch between the deployed index and the corpus+model is + detectable at load time (the load-time warning and `IndexStatus`), not only + via `degraded: true` after a request. +- [ ] The intended-mode-vs-actual-mode gap (Mode Hybrid requested, Mode + Lexical served because the index is structurally wrong) is observable as a + distinct `index_status` in diagnostics, separate from the transient + `degraded` flag. + +**Out of scope:** + +- True `mmap` (a single `fs::read` is loaded; the brute-force full-scan + cosine search touches every row per query, so memory-mapping yields no + runtime benefit and is rejected to avoid an `unsafe` dependency). +- Incremental / delta re-pack (a corpus change triggers a full re-embed of + the changed entries via the existing hash-keyed store, then a full + re-pack; partial repack is not built). +- Removing or replacing the per-file store.