alpha-id/docs/specs/2026-05-31-packed-versioned-index-bundle.md

Packed, Versioned Index Bundle — Design Spec

Date: 2026-05-31 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Goal

Turn the semantic index from a deployment liability into a single self-describing, version-bound artifact, without disturbing the build path.

Two concerns, cross-linked by their natural shared artifact (issues #3, #4):

• #3 — pack. The index is stored as roughly one file per corpus entry (index/embeddings/<safe-model>/<sha256>.f32, ~90 896 files of 4096 B each, ~356 MB) — "roughly" because the store is keyed by SHA256(model + text), so text-identical corpus entries collapse to a single file: the corpus has 90 898 entries (tests/corpus_tests.rs::loads_full_real_corpus) but ~90 896 unique store files. The file count, not the byte size, is hostile to container image layers, object storage, and Git-LFS. Pipeline::load opens every one of those files individually (src/pipeline.rs:43-50).
• #4 — bind. The index is keyed by SHA256(model + text) (src/embed.rs:19-25). A new corpus release or a changed embed_model silently invalidates every key; Pipeline::load then leaves the vector index None and Mode Hybrid degrades to Mode Lexical with only degraded: true to show for it — the same anonymous signal a transient IONOS outage raises. Nothing binds {corpus snapshot, embed model, index} as one unit.

This cycle delivers a single packed matrix file that carries a manifest in its header, and a load-time check that makes a corpus/model mismatch a distinct, observable state — loud but not fatal, so Mode Lexical (which never needs the index) keeps serving.

The per-file store is not removed. It stays the build/resume path; the packed file is an additive serve-path artifact.

Architecture

Two iterations, one file format. The format (magic + manifest header + payload) is born in iteration 1 and is stable from then on; iteration 2 adds only the checking and the observability, never a format rewrite.

• Iteration 1 (#3): a new src/packed.rs module and an alpha-id index --pack subcommand. Packing reads a complete per-file store and writes index/packed.<safe-model>.bin. Pipeline::load prefers the packed file and builds the VectorIndex by positional order (row i = corpus entry i), falling back to the per-file store (today's path) when no packed file is present.
• Iteration 2 (#4): the corpus-hash binding in the header is now enforced at load: a load-time mismatch check, a distinct index_status carried in diagnostics, a prominent load-time stderr warning, and a Mode Hybrid degrade that names the reason instead of an anonymous degraded.

The packed-file path is derived, not configured: <index_dir> + sanitized embed_model, exactly the sanitization EmbeddingStore::open already applies (src/embed.rs:27-33). No new config field; load and --pack compute the same path independently.

Row order is positional in corpus-load order (corpus::load, src/corpus.rs): the packed matrix has exactly n_rows = entries.len() rows (90 898 for the current corpus), one per corpus entry in load order. This is the entry count, which exceeds the unique store-file count when two entries share identical text — each such entry still gets its own row, re-read from the one shared store file, so positional alignment holds. Positional addressing is safe because the header binds the corpus identity: if corpus_sha256 matches the current corpus file, the row-to-entry alignment is guaranteed; if it does not, the check fires and the index is refused. The hash binding is what licenses dropping the per-row SHA256 key.

Concrete code shapes

User-facing surface (the empirical evidence)

Producing the bundle, once, after a complete embedding run:

$ alpha-id index --pack
packed 90898 rows × 1024 dims → index/packed.BAAI_bge-m3.bin (356 MB)
manifest: corpus=icd10gm2026_alphaidse_edvtxt_20250926.txt (sha256 4f3a…b1), model=BAAI/bge-m3

Packing an incomplete store is refused loudly:

$ alpha-id index --pack
error: store incomplete: 41020 of 90898 entries embedded; run `index --full --confirm` first
$ echo $status
2

suggest and eval are unchanged at the CLI; load picks up the packed file automatically. With a matching bundle:

$ alpha-id suggest note.txt --mode hybrid
… suggestions …
[3 segments, mode Hybrid, 41 ms, degraded=false, index=ok]

With a corpus that no longer matches the deployed index — the formerly silent downgrade is now named at load time and in the per-request line:

$ alpha-id suggest note.txt --mode hybrid
warning: packed index index/packed.BAAI_bge-m3.bin does not match the current corpus
         (corpus sha256 mismatch); semantic index disabled, Mode Hybrid will degrade to Lexical
… suggestions …
[3 segments, mode Hybrid, 12 ms, degraded=true, index=mismatch: corpus sha256]

Must-fail fixture (the criterion's evidence for the binding)

A packed file whose header model disagrees with the configured model MUST NOT yield a vector index. The test pins exactly that rejection:

// A packed file built declaring embed_model "other/model", loaded under a
// config whose embed_model is "BAAI/bge-m3", must refuse the index.
let (vector, status) = load_packed_or_store(&cfg, &entries);
assert!(vector.is_none());
assert!(matches!(status, IndexStatus::Mismatch(_)));

Header (written at pack, checked at load)

#[derive(serde::Serialize, serde::Deserialize)]
pub struct PackedHeader {
    pub n_rows: u32,
    pub dim: u32,
    pub embed_model: String,   // must equal cfg.embed_model
    pub corpus_sha256: String, // must equal corpus_sha256(cfg.alpha_id_path)
    pub corpus_name: String,   // human-readable; not part of the check
}

On-disk layout:

0..4    magic  b"AIPK"
4..8    format_version  u32 LE      (fast reject without a JSON parse)
8..12   header_len      u32 LE
12..    header  (JSON, UTF-8, the PackedHeader above)
        pad to a 4-byte boundary
payload n_rows × dim × f32, little-endian, row-major, corpus order

Pipeline::load — before → after

The load-bearing change replaces the per-entry collection (src/pipeline.rs:43-50) with a packed-first helper.

// before
let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) {
    Ok(store) if entries.iter().all(|e| store.has(&e.text)) => {
        let vecs: Option<Vec<Vec<f32>>> =
            entries.iter().map(|e| store.get(&e.text)).collect();
        vecs.map(VectorIndex::from_vectors)
    }
    _ => None,
};

// after — packed file preferred; manifest enforced; per-file store is the fallback
let (vector, index_status) = load_packed_or_store(cfg, &entries);
if let IndexStatus::Mismatch(reason) = &index_status {
    eprintln!("warning: packed index {} does not match the current corpus/model \
               ({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical",
              packed_path(&cfg.index_dir, &cfg.embed_model).display());
}
// `index_status` is stored on the Pipeline and copied into Diagnostics per request.

IndexStatus and the diagnostics surface

#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
pub enum IndexStatus {
    Ok,               // packed file present and matches corpus + model
    Absent,           // no packed file and store incomplete — intentional lexical-only
    Mismatch(String), // packed file present but corpus/model/shape disagree; reason carried
}

Diagnostics (src/model.rs:114-121) gains an index_status: IndexStatus field. The transient degraded flag keeps its meaning (a runtime IONOS step failed); index_status answers the orthogonal question "is the deployed index structurally the right one". A request can be degraded: true for either reason, but index=mismatch names the structural one.

Components

File	Change
src/packed.rs (new)	PackedHeader; write_packed(path, &PackedHeader, rows: &[Vec<f32>]); read_packed(path) -> Result<(PackedHeader, Vec<Vec<f32>>), AppError>; packed_path(index_dir, model) -> PathBuf; corpus_sha256(path) -> Result<String, AppError>
src/pipeline.rs	load_packed_or_store(cfg, entries) -> (Option<VectorIndex>, IndexStatus) replacing the :43-50 match; Pipeline stores index_status; suggest copies it into Diagnostics
src/model.rs	enum IndexStatus; index_status field on Diagnostics
src/bin/alpha_id.rs	--pack flag on the Index subcommand; the load-time mismatch warning; index=… in the human diagnostics line
src/vector.rs	unchanged — still consumes Vec<Vec<f32>> via from_vectors
src/embed.rs	unchanged — the per-file store stays the build/resume path

Data flow

• pack (index --pack): corpus::load → open the per-file store → require every entry cached (else refuse, exit 2) → collect rows in corpus order → write_packed(header { embed_model, corpus_sha256, n_rows, dim }, rows).
• load (Pipeline::load): if packed_path exists → read_packed → validate header.embed_model == cfg.embed_model, header.corpus_sha256 == corpus_sha256(cfg.alpha_id_path), header.n_rows as usize == entries.len(), payload length consistent → on full match build VectorIndex::from_vectors, status Ok; on any disagreement status Mismatch(reason), vector None. If no packed file → per-file store fallback (today's all-or-nothing build); complete ⇒ build, status Ok; incomplete ⇒ None, status Absent.
• suggest: Mode Hybrid with vector == None degrades to collect_lexical (unchanged); Diagnostics.index_status carries the structural reason; Diagnostics.degraded stays the transient signal.

Error handling

• --pack on an incomplete store → loud refusal, exit code 2, a message naming the cached/total counts and pointing at index --full --confirm. Packing a partial store is forbidden: it would mispair rows against the corpus.
• Corrupt or truncated packed file (payload length ≠ n_rows × dim × 4), wrong magic, or a format_version newer than supported → IndexStatus::Mismatch(reason), prominent load warning, no index built, Mode Lexical serves. Not fatal — a structurally wrong index must not take down the lexical service (chosen mismatch posture).
• Corpus file unreadable for hashing → surfaces as the existing corpus::load error before any packed-load work (corpus load precedes it).

Testing strategy

• Round-trip (unit): write_packed then read_packed reproduces the header and every row bit-exactly (f32 little-endian preserved).
• Corpus hash (unit): corpus_sha256 is stable for identical bytes and differs when the corpus file changes by one byte.
• Must-fail binding (integration): a packed file built declaring one embed_model (or one corpus_sha256), loaded under a config that disagrees, yields vector == None and IndexStatus::Mismatch(_). Wrong bundle never builds an index.
• Positional alignment (integration): after read_packed, row i equals store.get(&entries[i].text) for sampled i — packing preserves corpus-order alignment.
• Observable degrade (integration): under a corpus-hash mismatch, a Mode Hybrid suggest returns degraded == true and index_status == Mismatch(_), distinguishable from a transient-IONOS degrade (which leaves index_status == Ok).
• Packed beats store (integration): with a valid packed file present and the per-file store absent, Pipeline::load builds the VectorIndex from the packed file alone (acceptance #3b).

Acceptance criteria

Issue #3 (pack):

• A single-file packed matrix is produced from a complete per-file store via alpha-id index --pack.
• Pipeline::load builds its VectorIndex from the packed file without reading the per-file store.
• Row order is positional in corpus order, documented here, and verifiably aligned to the corpus snapshot via the header's corpus_sha256.

Issue #4 (bind):

• A mismatch between the deployed index and the corpus+model is detectable at load time (the load-time warning and IndexStatus), not only via degraded: true after a request.
• The intended-mode-vs-actual-mode gap (Mode Hybrid requested, Mode Lexical served because the index is structurally wrong) is observable as a distinct index_status in diagnostics, separate from the transient degraded flag.

Out of scope:

• True mmap (a single fs::read is loaded; the brute-force full-scan cosine search touches every row per query, so memory-mapping yields no runtime benefit and is rejected to avoid an unsafe dependency).
• Incremental / delta re-pack (a corpus change triggers a full re-embed of the changed entries via the existing hash-keyed store, then a full re-pack; partial repack is not built).
• Removing or replacing the per-file store.