Design spec for the "versioned bundle" cycle binding issues #3 (pack the per-file embedding store into one contiguous matrix file) and #4 (bind {corpus snapshot, embed model, index} as one versioned artifact). Ratified decisions (user-approved across the brainstorm): - Approach 1: single self-describing file (magic + JSON-header manifest + row-major f32 payload), loaded with one fs::read. No mmap (a brute-force full-scan cosine index touches every row per query, so mmap yields no runtime benefit and is rejected to avoid an unsafe dependency). VectorIndex and the per-file store are untouched — the store stays the build/resume path, the packed file is an additive serve-path artifact. - Mismatch posture: loud but not fatal. A structurally wrong index (corpus sha256 or embed_model disagreement) builds no VectorIndex and surfaces as a distinct IndexStatus::Mismatch in diagnostics plus a load-time warning, instead of silently degrading Mode Hybrid to Lexical. Mode Lexical, which never needs the index, keeps serving. - Two iterations, one format: iter 1 delivers the format + `index --pack` + packed-first load; iter 2 enforces the manifest check and the observable index_status surface. grounding-check: PASS (7 load-bearing assumptions ratified by green tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,289 @@
|
||||
# Packed, Versioned Index Bundle — Design Spec
|
||||
|
||||
**Date:** 2026-05-31
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
## Goal
|
||||
|
||||
Turn the semantic index from a deployment liability into a single
|
||||
self-describing, version-bound artifact, without disturbing the build path.
|
||||
|
||||
Two concerns, cross-linked by their natural shared artifact (issues #3, #4):
|
||||
|
||||
- **#3 — pack.** The index is stored as roughly one file per corpus entry
|
||||
(`index/embeddings/<safe-model>/<sha256>.f32`, ~90 896 files of 4096 B
|
||||
each, ~356 MB) — "roughly" because the store is keyed by
|
||||
`SHA256(model + text)`, so text-identical corpus entries collapse to a
|
||||
single file: the corpus has 90 898 entries
|
||||
(`tests/corpus_tests.rs::loads_full_real_corpus`) but ~90 896 unique store
|
||||
files. The file *count*, not the byte size, is hostile to container image
|
||||
layers, object storage, and Git-LFS. `Pipeline::load` opens every one of
|
||||
those files individually (`src/pipeline.rs:43-50`).
|
||||
- **#4 — bind.** The index is keyed by `SHA256(model + text)`
|
||||
(`src/embed.rs:19-25`). A new corpus release or a changed `embed_model`
|
||||
silently invalidates every key; `Pipeline::load` then leaves the vector
|
||||
index `None` and Mode Hybrid degrades to Mode Lexical with only
|
||||
`degraded: true` to show for it — the same anonymous signal a transient
|
||||
IONOS outage raises. Nothing binds `{corpus snapshot, embed model, index}`
|
||||
as one unit.
|
||||
|
||||
This cycle delivers a single packed matrix file that carries a manifest in
|
||||
its header, and a load-time check that makes a corpus/model mismatch a
|
||||
distinct, observable state — loud but not fatal, so Mode Lexical (which never
|
||||
needs the index) keeps serving.
|
||||
|
||||
The per-file store is **not** removed. It stays the build/resume path; the
|
||||
packed file is an additive serve-path artifact.
|
||||
|
||||
## Architecture
|
||||
|
||||
Two iterations, one file format. The format (magic + manifest header +
|
||||
payload) is born in iteration 1 and is stable from then on; iteration 2 adds
|
||||
only the *checking* and the *observability*, never a format rewrite.
|
||||
|
||||
- **Iteration 1 (#3):** a new `src/packed.rs` module and an
|
||||
`alpha-id index --pack` subcommand. Packing reads a *complete* per-file
|
||||
store and writes `index/packed.<safe-model>.bin`. `Pipeline::load` prefers
|
||||
the packed file and builds the `VectorIndex` by **positional order**
|
||||
(row `i` = corpus entry `i`), falling back to the per-file store (today's
|
||||
path) when no packed file is present.
|
||||
- **Iteration 2 (#4):** the corpus-hash binding in the header is now
|
||||
*enforced* at load: a load-time mismatch check, a distinct `index_status`
|
||||
carried in diagnostics, a prominent load-time stderr warning, and a Mode
|
||||
Hybrid degrade that names the reason instead of an anonymous `degraded`.
|
||||
|
||||
The packed-file path is derived, not configured: `<index_dir>` +
|
||||
sanitized `embed_model`, exactly the sanitization `EmbeddingStore::open`
|
||||
already applies (`src/embed.rs:27-33`). No new config field; `load` and
|
||||
`--pack` compute the same path independently.
|
||||
|
||||
Row order is positional in corpus-load order (`corpus::load`,
|
||||
`src/corpus.rs`): the packed matrix has exactly `n_rows = entries.len()`
|
||||
rows (90 898 for the current corpus), one per corpus entry in load order.
|
||||
This is the *entry* count, which exceeds the unique store-file count when
|
||||
two entries share identical text — each such entry still gets its own row,
|
||||
re-read from the one shared store file, so positional alignment holds.
|
||||
Positional addressing is safe *because* the header binds the corpus
|
||||
identity: if `corpus_sha256` matches the current corpus file, the
|
||||
row-to-entry alignment is guaranteed; if it does not, the check fires and
|
||||
the index is refused. The hash binding is what licenses dropping the
|
||||
per-row SHA256 key.
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### User-facing surface (the empirical evidence)
|
||||
|
||||
Producing the bundle, once, after a complete embedding run:
|
||||
|
||||
```text
|
||||
$ alpha-id index --pack
|
||||
packed 90898 rows × 1024 dims → index/packed.BAAI_bge-m3.bin (356 MB)
|
||||
manifest: corpus=icd10gm2026_alphaidse_edvtxt_20250926.txt (sha256 4f3a…b1), model=BAAI/bge-m3
|
||||
```
|
||||
|
||||
Packing an incomplete store is refused loudly:
|
||||
|
||||
```text
|
||||
$ alpha-id index --pack
|
||||
error: store incomplete: 41020 of 90898 entries embedded; run `index --full --confirm` first
|
||||
$ echo $status
|
||||
2
|
||||
```
|
||||
|
||||
`suggest` and `eval` are unchanged at the CLI; `load` picks up the packed
|
||||
file automatically. With a matching bundle:
|
||||
|
||||
```text
|
||||
$ alpha-id suggest note.txt --mode hybrid
|
||||
… suggestions …
|
||||
[3 segments, mode Hybrid, 41 ms, degraded=false, index=ok]
|
||||
```
|
||||
|
||||
With a corpus that no longer matches the deployed index — the formerly
|
||||
silent downgrade is now named at load time and in the per-request line:
|
||||
|
||||
```text
|
||||
$ alpha-id suggest note.txt --mode hybrid
|
||||
warning: packed index index/packed.BAAI_bge-m3.bin does not match the current corpus
|
||||
(corpus sha256 mismatch); semantic index disabled, Mode Hybrid will degrade to Lexical
|
||||
… suggestions …
|
||||
[3 segments, mode Hybrid, 12 ms, degraded=true, index=mismatch: corpus sha256]
|
||||
```
|
||||
|
||||
### Must-fail fixture (the criterion's evidence for the binding)
|
||||
|
||||
A packed file whose header model disagrees with the configured model MUST
|
||||
NOT yield a vector index. The test pins exactly that rejection:
|
||||
|
||||
```rust
|
||||
// A packed file built declaring embed_model "other/model", loaded under a
|
||||
// config whose embed_model is "BAAI/bge-m3", must refuse the index.
|
||||
let (vector, status) = load_packed_or_store(&cfg, &entries);
|
||||
assert!(vector.is_none());
|
||||
assert!(matches!(status, IndexStatus::Mismatch(_)));
|
||||
```
|
||||
|
||||
### Header (written at pack, checked at load)
|
||||
|
||||
```rust
|
||||
#[derive(serde::Serialize, serde::Deserialize)]
|
||||
pub struct PackedHeader {
|
||||
pub n_rows: u32,
|
||||
pub dim: u32,
|
||||
pub embed_model: String, // must equal cfg.embed_model
|
||||
pub corpus_sha256: String, // must equal corpus_sha256(cfg.alpha_id_path)
|
||||
pub corpus_name: String, // human-readable; not part of the check
|
||||
}
|
||||
```
|
||||
|
||||
On-disk layout:
|
||||
|
||||
```text
|
||||
0..4 magic b"AIPK"
|
||||
4..8 format_version u32 LE (fast reject without a JSON parse)
|
||||
8..12 header_len u32 LE
|
||||
12.. header (JSON, UTF-8, the PackedHeader above)
|
||||
pad to a 4-byte boundary
|
||||
payload n_rows × dim × f32, little-endian, row-major, corpus order
|
||||
```
|
||||
|
||||
### `Pipeline::load` — before → after
|
||||
|
||||
The load-bearing change replaces the per-entry collection
|
||||
(`src/pipeline.rs:43-50`) with a packed-first helper.
|
||||
|
||||
```rust
|
||||
// before
|
||||
let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) {
|
||||
Ok(store) if entries.iter().all(|e| store.has(&e.text)) => {
|
||||
let vecs: Option<Vec<Vec<f32>>> =
|
||||
entries.iter().map(|e| store.get(&e.text)).collect();
|
||||
vecs.map(VectorIndex::from_vectors)
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
```
|
||||
|
||||
```rust
|
||||
// after — packed file preferred; manifest enforced; per-file store is the fallback
|
||||
let (vector, index_status) = load_packed_or_store(cfg, &entries);
|
||||
if let IndexStatus::Mismatch(reason) = &index_status {
|
||||
eprintln!("warning: packed index {} does not match the current corpus/model \
|
||||
({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical",
|
||||
packed_path(&cfg.index_dir, &cfg.embed_model).display());
|
||||
}
|
||||
// `index_status` is stored on the Pipeline and copied into Diagnostics per request.
|
||||
```
|
||||
|
||||
### `IndexStatus` and the diagnostics surface
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
|
||||
pub enum IndexStatus {
|
||||
Ok, // packed file present and matches corpus + model
|
||||
Absent, // no packed file and store incomplete — intentional lexical-only
|
||||
Mismatch(String), // packed file present but corpus/model/shape disagree; reason carried
|
||||
}
|
||||
```
|
||||
|
||||
`Diagnostics` (`src/model.rs:114-121`) gains an `index_status: IndexStatus`
|
||||
field. The transient `degraded` flag keeps its meaning (a runtime IONOS step
|
||||
failed); `index_status` answers the orthogonal question "is the deployed
|
||||
index structurally the right one". A request can be `degraded: true` for
|
||||
*either* reason, but `index=mismatch` names the structural one.
|
||||
|
||||
## Components
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `src/packed.rs` *(new)* | `PackedHeader`; `write_packed(path, &PackedHeader, rows: &[Vec<f32>])`; `read_packed(path) -> Result<(PackedHeader, Vec<Vec<f32>>), AppError>`; `packed_path(index_dir, model) -> PathBuf`; `corpus_sha256(path) -> Result<String, AppError>` |
|
||||
| `src/pipeline.rs` | `load_packed_or_store(cfg, entries) -> (Option<VectorIndex>, IndexStatus)` replacing the `:43-50` match; `Pipeline` stores `index_status`; `suggest` copies it into `Diagnostics` |
|
||||
| `src/model.rs` | `enum IndexStatus`; `index_status` field on `Diagnostics` |
|
||||
| `src/bin/alpha_id.rs` | `--pack` flag on the `Index` subcommand; the load-time mismatch warning; `index=…` in the human diagnostics line |
|
||||
| `src/vector.rs` | **unchanged** — still consumes `Vec<Vec<f32>>` via `from_vectors` |
|
||||
| `src/embed.rs` | **unchanged** — the per-file store stays the build/resume path |
|
||||
|
||||
## Data flow
|
||||
|
||||
- **pack** (`index --pack`): `corpus::load` → open the per-file store →
|
||||
require every entry cached (else refuse, exit 2) → collect rows in corpus
|
||||
order → `write_packed(header { embed_model, corpus_sha256, n_rows, dim },
|
||||
rows)`.
|
||||
- **load** (`Pipeline::load`): if `packed_path` exists → `read_packed` →
|
||||
validate `header.embed_model == cfg.embed_model`,
|
||||
`header.corpus_sha256 == corpus_sha256(cfg.alpha_id_path)`,
|
||||
`header.n_rows as usize == entries.len()`, payload length consistent →
|
||||
on full match build `VectorIndex::from_vectors`, status `Ok`; on any
|
||||
disagreement status `Mismatch(reason)`, vector `None`. If no packed file →
|
||||
per-file store fallback (today's all-or-nothing build); complete ⇒ build,
|
||||
status `Ok`; incomplete ⇒ `None`, status `Absent`.
|
||||
- **suggest**: Mode Hybrid with `vector == None` degrades to
|
||||
`collect_lexical` (unchanged); `Diagnostics.index_status` carries the
|
||||
structural reason; `Diagnostics.degraded` stays the transient signal.
|
||||
|
||||
## Error handling
|
||||
|
||||
- **`--pack` on an incomplete store** → loud refusal, exit code 2, a message
|
||||
naming the cached/total counts and pointing at `index --full --confirm`.
|
||||
Packing a partial store is forbidden: it would mispair rows against the
|
||||
corpus.
|
||||
- **Corrupt or truncated packed file** (payload length ≠
|
||||
`n_rows × dim × 4`), **wrong magic**, or a **`format_version` newer than
|
||||
supported** → `IndexStatus::Mismatch(reason)`, prominent load warning, no
|
||||
index built, Mode Lexical serves. Not fatal — a structurally wrong index
|
||||
must not take down the lexical service (chosen mismatch posture).
|
||||
- **Corpus file unreadable** for hashing → surfaces as the existing
|
||||
`corpus::load` error before any packed-load work (corpus load precedes it).
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **Round-trip (unit):** `write_packed` then `read_packed` reproduces the
|
||||
header and every row bit-exactly (f32 little-endian preserved).
|
||||
- **Corpus hash (unit):** `corpus_sha256` is stable for identical bytes and
|
||||
differs when the corpus file changes by one byte.
|
||||
- **Must-fail binding (integration):** a packed file built declaring one
|
||||
`embed_model` (or one `corpus_sha256`), loaded under a config that
|
||||
disagrees, yields `vector == None` and `IndexStatus::Mismatch(_)`. Wrong
|
||||
bundle never builds an index.
|
||||
- **Positional alignment (integration):** after `read_packed`, row `i`
|
||||
equals `store.get(&entries[i].text)` for sampled `i` — packing preserves
|
||||
corpus-order alignment.
|
||||
- **Observable degrade (integration):** under a corpus-hash mismatch, a Mode
|
||||
Hybrid `suggest` returns `degraded == true` *and* `index_status ==
|
||||
Mismatch(_)`, distinguishable from a transient-IONOS degrade (which leaves
|
||||
`index_status == Ok`).
|
||||
- **Packed beats store (integration):** with a valid packed file present and
|
||||
the per-file store absent, `Pipeline::load` builds the `VectorIndex` from
|
||||
the packed file alone (acceptance #3b).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
**Issue #3 (pack):**
|
||||
|
||||
- [ ] A single-file packed matrix is produced from a complete per-file store
|
||||
via `alpha-id index --pack`.
|
||||
- [ ] `Pipeline::load` builds its `VectorIndex` from the packed file without
|
||||
reading the per-file store.
|
||||
- [ ] Row order is positional in corpus order, documented here, and
|
||||
verifiably aligned to the corpus snapshot via the header's `corpus_sha256`.
|
||||
|
||||
**Issue #4 (bind):**
|
||||
|
||||
- [ ] A mismatch between the deployed index and the corpus+model is
|
||||
detectable at load time (the load-time warning and `IndexStatus`), not only
|
||||
via `degraded: true` after a request.
|
||||
- [ ] The intended-mode-vs-actual-mode gap (Mode Hybrid requested, Mode
|
||||
Lexical served because the index is structurally wrong) is observable as a
|
||||
distinct `index_status` in diagnostics, separate from the transient
|
||||
`degraded` flag.
|
||||
|
||||
**Out of scope:**
|
||||
|
||||
- True `mmap` (a single `fs::read` is loaded; the brute-force full-scan
|
||||
cosine search touches every row per query, so memory-mapping yields no
|
||||
runtime benefit and is rejected to avoid an `unsafe` dependency).
|
||||
- Incremental / delta re-pack (a corpus change triggers a full re-embed of
|
||||
the changed entries via the existing hash-keyed store, then a full
|
||||
re-pack; partial repack is not built).
|
||||
- Removing or replacing the per-file store.
|
||||
Reference in New Issue
Block a user