Brummel/alpha-id
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec: packed versioned index bundle (refs #3 #4)

Browse Source 
Design spec for the "versioned bundle" cycle binding issues #3 (pack the
per-file embedding store into one contiguous matrix file) and #4 (bind
{corpus snapshot, embed model, index} as one versioned artifact).

Ratified decisions (user-approved across the brainstorm):
- Approach 1: single self-describing file (magic + JSON-header manifest +
  row-major f32 payload), loaded with one fs::read. No mmap (a brute-force
  full-scan cosine index touches every row per query, so mmap yields no
  runtime benefit and is rejected to avoid an unsafe dependency). VectorIndex
  and the per-file store are untouched — the store stays the build/resume
  path, the packed file is an additive serve-path artifact.
- Mismatch posture: loud but not fatal. A structurally wrong index (corpus
  sha256 or embed_model disagreement) builds no VectorIndex and surfaces as a
  distinct IndexStatus::Mismatch in diagnostics plus a load-time warning,
  instead of silently degrading Mode Hybrid to Lexical. Mode Lexical, which
  never needs the index, keeps serving.
- Two iterations, one format: iter 1 delivers the format + `index --pack` +
  packed-first load; iter 2 enforces the manifest check and the observable
  index_status surface.

grounding-check: PASS (7 load-bearing assumptions ratified by green tests).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brummel
2026-06-01 00:15:18 +02:00
parent d83b10f939
commit 2bb1bfc534
1 changed files with 289 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/2026-05-31-packed-versioned-index-bundle.md
+289
View File
@@ -0,0 +1,289 @@
# Packed, Versioned Index Bundle — Design Spec
**Date:** 2026-05-31
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude
## Goal
Turn the semantic index from a deployment liability into a single
self-describing, version-bound artifact, without disturbing the build path.
Two concerns, cross-linked by their natural shared artifact (issues #3, #4):
- **#3 — pack.** The index is stored as roughly one file per corpus entry
(`index/embeddings/<safe-model>/<sha256>.f32`, ~90 896 files of 4096 B
each, ~356 MB) — "roughly" because the store is keyed by
`SHA256(model + text)`, so text-identical corpus entries collapse to a
single file: the corpus has 90 898 entries
(`tests/corpus_tests.rs::loads_full_real_corpus`) but ~90 896 unique store
files. The file *count*, not the byte size, is hostile to container image
layers, object storage, and Git-LFS. `Pipeline::load` opens every one of
those files individually (`src/pipeline.rs:43-50`).
- **#4 — bind.** The index is keyed by `SHA256(model + text)`
(`src/embed.rs:19-25`). A new corpus release or a changed `embed_model`
silently invalidates every key; `Pipeline::load` then leaves the vector
index `None` and Mode Hybrid degrades to Mode Lexical with only
`degraded: true` to show for it — the same anonymous signal a transient
IONOS outage raises. Nothing binds `{corpus snapshot, embed model, index}`
as one unit.
This cycle delivers a single packed matrix file that carries a manifest in
its header, and a load-time check that makes a corpus/model mismatch a
distinct, observable state — loud but not fatal, so Mode Lexical (which never
needs the index) keeps serving.
The per-file store is **not** removed. It stays the build/resume path; the
packed file is an additive serve-path artifact.
## Architecture
Two iterations, one file format. The format (magic + manifest header +
payload) is born in iteration 1 and is stable from then on; iteration 2 adds
only the *checking* and the *observability*, never a format rewrite.
- **Iteration 1 (#3):** a new `src/packed.rs` module and an
`alpha-id index --pack` subcommand. Packing reads a *complete* per-file
store and writes `index/packed.<safe-model>.bin`. `Pipeline::load` prefers
the packed file and builds the `VectorIndex` by **positional order**
(row `i` = corpus entry `i`), falling back to the per-file store (today's
path) when no packed file is present.
- **Iteration 2 (#4):** the corpus-hash binding in the header is now
*enforced* at load: a load-time mismatch check, a distinct `index_status`
carried in diagnostics, a prominent load-time stderr warning, and a Mode
Hybrid degrade that names the reason instead of an anonymous `degraded`.
The packed-file path is derived, not configured: `<index_dir>` +
sanitized `embed_model`, exactly the sanitization `EmbeddingStore::open`
already applies (`src/embed.rs:27-33`). No new config field; `load` and
`--pack` compute the same path independently.
Row order is positional in corpus-load order (`corpus::load`,
`src/corpus.rs`): the packed matrix has exactly `n_rows = entries.len()`
rows (90 898 for the current corpus), one per corpus entry in load order.
This is the *entry* count, which exceeds the unique store-file count when
two entries share identical text — each such entry still gets its own row,
re-read from the one shared store file, so positional alignment holds.
Positional addressing is safe *because* the header binds the corpus
identity: if `corpus_sha256` matches the current corpus file, the
row-to-entry alignment is guaranteed; if it does not, the check fires and
the index is refused. The hash binding is what licenses dropping the
per-row SHA256 key.
## Concrete code shapes
### User-facing surface (the empirical evidence)
Producing the bundle, once, after a complete embedding run:
```text
$ alpha-id index --pack
packed 90898 rows × 1024 dims → index/packed.BAAI_bge-m3.bin (356 MB)
manifest: corpus=icd10gm2026_alphaidse_edvtxt_20250926.txt (sha256 4f3a…b1), model=BAAI/bge-m3
```
Packing an incomplete store is refused loudly:
```text
$ alpha-id index --pack
error: store incomplete: 41020 of 90898 entries embedded; run `index --full --confirm` first
$ echo $status
2
```
`suggest` and `eval` are unchanged at the CLI; `load` picks up the packed
file automatically. With a matching bundle:
```text
$ alpha-id suggest note.txt --mode hybrid
… suggestions …
[3 segments, mode Hybrid, 41 ms, degraded=false, index=ok]
```
With a corpus that no longer matches the deployed index — the formerly
silent downgrade is now named at load time and in the per-request line:
```text
$ alpha-id suggest note.txt --mode hybrid
warning: packed index index/packed.BAAI_bge-m3.bin does not match the current corpus
(corpus sha256 mismatch); semantic index disabled, Mode Hybrid will degrade to Lexical
… suggestions …
[3 segments, mode Hybrid, 12 ms, degraded=true, index=mismatch: corpus sha256]
```
### Must-fail fixture (the criterion's evidence for the binding)
A packed file whose header model disagrees with the configured model MUST
NOT yield a vector index. The test pins exactly that rejection:
```rust
// A packed file built declaring embed_model "other/model", loaded under a
// config whose embed_model is "BAAI/bge-m3", must refuse the index.
let (vector, status) = load_packed_or_store(&cfg, &entries);
assert!(vector.is_none());
assert!(matches!(status, IndexStatus::Mismatch(_)));
```
### Header (written at pack, checked at load)
```rust
#[derive(serde::Serialize, serde::Deserialize)]
pub struct PackedHeader {
pub n_rows: u32,
pub dim: u32,
pub embed_model: String, // must equal cfg.embed_model
pub corpus_sha256: String, // must equal corpus_sha256(cfg.alpha_id_path)
pub corpus_name: String, // human-readable; not part of the check
}
```
On-disk layout:
```text
0..4 magic b"AIPK"
4..8 format_version u32 LE (fast reject without a JSON parse)
8..12 header_len u32 LE
12.. header (JSON, UTF-8, the PackedHeader above)
pad to a 4-byte boundary
payload n_rows × dim × f32, little-endian, row-major, corpus order
```
### `Pipeline::load` — before → after
The load-bearing change replaces the per-entry collection
(`src/pipeline.rs:43-50`) with a packed-first helper.
```rust
// before
let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) {
Ok(store) if entries.iter().all(|e| store.has(&e.text)) => {
let vecs: Option<Vec<Vec<f32>>> =
entries.iter().map(|e| store.get(&e.text)).collect();
vecs.map(VectorIndex::from_vectors)
}
_ => None,
};
```
```rust
// after — packed file preferred; manifest enforced; per-file store is the fallback
let (vector, index_status) = load_packed_or_store(cfg, &entries);
if let IndexStatus::Mismatch(reason) = &index_status {
eprintln!("warning: packed index {} does not match the current corpus/model \
({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical",
packed_path(&cfg.index_dir, &cfg.embed_model).display());
}
// `index_status` is stored on the Pipeline and copied into Diagnostics per request.
```
### `IndexStatus` and the diagnostics surface
```rust
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
pub enum IndexStatus {
Ok, // packed file present and matches corpus + model
Absent, // no packed file and store incomplete — intentional lexical-only
Mismatch(String), // packed file present but corpus/model/shape disagree; reason carried
}
```
`Diagnostics` (`src/model.rs:114-121`) gains an `index_status: IndexStatus`
field. The transient `degraded` flag keeps its meaning (a runtime IONOS step
failed); `index_status` answers the orthogonal question "is the deployed
index structurally the right one". A request can be `degraded: true` for
*either* reason, but `index=mismatch` names the structural one.
## Components
| File | Change |
|---|---|
| `src/packed.rs` *(new)* | `PackedHeader`; `write_packed(path, &PackedHeader, rows: &[Vec<f32>])`; `read_packed(path) -> Result<(PackedHeader, Vec<Vec<f32>>), AppError>`; `packed_path(index_dir, model) -> PathBuf`; `corpus_sha256(path) -> Result<String, AppError>` |
| `src/pipeline.rs` | `load_packed_or_store(cfg, entries) -> (Option<VectorIndex>, IndexStatus)` replacing the `:43-50` match; `Pipeline` stores `index_status`; `suggest` copies it into `Diagnostics` |
| `src/model.rs` | `enum IndexStatus`; `index_status` field on `Diagnostics` |
| `src/bin/alpha_id.rs` | `--pack` flag on the `Index` subcommand; the load-time mismatch warning; `index=…` in the human diagnostics line |
| `src/vector.rs` | **unchanged** — still consumes `Vec<Vec<f32>>` via `from_vectors` |
| `src/embed.rs` | **unchanged** — the per-file store stays the build/resume path |
## Data flow
- **pack** (`index --pack`): `corpus::load` → open the per-file store →
require every entry cached (else refuse, exit 2) → collect rows in corpus
order → `write_packed(header { embed_model, corpus_sha256, n_rows, dim },
rows)`.
- **load** (`Pipeline::load`): if `packed_path` exists → `read_packed` →
validate `header.embed_model == cfg.embed_model`,
`header.corpus_sha256 == corpus_sha256(cfg.alpha_id_path)`,
`header.n_rows as usize == entries.len()`, payload length consistent →
on full match build `VectorIndex::from_vectors`, status `Ok`; on any
disagreement status `Mismatch(reason)`, vector `None`. If no packed file →
per-file store fallback (today's all-or-nothing build); complete ⇒ build,
status `Ok`; incomplete ⇒ `None`, status `Absent`.
- **suggest**: Mode Hybrid with `vector == None` degrades to
`collect_lexical` (unchanged); `Diagnostics.index_status` carries the
structural reason; `Diagnostics.degraded` stays the transient signal.
## Error handling
- **`--pack` on an incomplete store** → loud refusal, exit code 2, a message
naming the cached/total counts and pointing at `index --full --confirm`.
Packing a partial store is forbidden: it would mispair rows against the
corpus.
- **Corrupt or truncated packed file** (payload length ≠
`n_rows × dim × 4`), **wrong magic**, or a **`format_version` newer than
supported** → `IndexStatus::Mismatch(reason)`, prominent load warning, no
index built, Mode Lexical serves. Not fatal — a structurally wrong index
must not take down the lexical service (chosen mismatch posture).
- **Corpus file unreadable** for hashing → surfaces as the existing
`corpus::load` error before any packed-load work (corpus load precedes it).
## Testing strategy
- **Round-trip (unit):** `write_packed` then `read_packed` reproduces the
header and every row bit-exactly (f32 little-endian preserved).
- **Corpus hash (unit):** `corpus_sha256` is stable for identical bytes and
differs when the corpus file changes by one byte.
- **Must-fail binding (integration):** a packed file built declaring one
`embed_model` (or one `corpus_sha256`), loaded under a config that
disagrees, yields `vector == None` and `IndexStatus::Mismatch(_)`. Wrong
bundle never builds an index.
- **Positional alignment (integration):** after `read_packed`, row `i`
equals `store.get(&entries[i].text)` for sampled `i` — packing preserves
corpus-order alignment.
- **Observable degrade (integration):** under a corpus-hash mismatch, a Mode
Hybrid `suggest` returns `degraded == true` *and* `index_status ==
Mismatch(_)`, distinguishable from a transient-IONOS degrade (which leaves
`index_status == Ok`).
- **Packed beats store (integration):** with a valid packed file present and
the per-file store absent, `Pipeline::load` builds the `VectorIndex` from
the packed file alone (acceptance #3b).
## Acceptance criteria
**Issue #3 (pack):**
- [ ] A single-file packed matrix is produced from a complete per-file store
via `alpha-id index --pack`.
- [ ] `Pipeline::load` builds its `VectorIndex` from the packed file without
reading the per-file store.
- [ ] Row order is positional in corpus order, documented here, and
verifiably aligned to the corpus snapshot via the header's `corpus_sha256`.
**Issue #4 (bind):**
- [ ] A mismatch between the deployed index and the corpus+model is
detectable at load time (the load-time warning and `IndexStatus`), not only
via `degraded: true` after a request.
- [ ] The intended-mode-vs-actual-mode gap (Mode Hybrid requested, Mode
Lexical served because the index is structurally wrong) is observable as a
distinct `index_status` in diagnostics, separate from the transient
`degraded` flag.
**Out of scope:**
- True `mmap` (a single `fs::read` is loaded; the brute-force full-scan
cosine search touches every row per query, so memory-mapping yields no
runtime benefit and is rejected to avoid an `unsafe` dependency).
- Incremental / delta re-pack (a corpus change triggers a full re-embed of
the changed entries via the existing hash-keyed store, then a full
re-pack; partial repack is not built).
- Removing or replacing the per-file store.