alpha-id/docs/specs/2026-05-31-packed-versioned-index-bundle.md

# Packed, Versioned Index Bundle — Design Spec

**Date:** 2026-05-31
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

## Goal

Turn the semantic index from a deployment liability into a single
self-describing, version-bound artifact, without disturbing the build path.

Two concerns, cross-linked by their natural shared artifact (issues #3, #4):

- **#3 — pack.** The index is stored as roughly one file per corpus entry
  (`index/embeddings/<safe-model>/<sha256>.f32`, ~90 896 files of 4096 B
  each, ~356 MB) — "roughly" because the store is keyed by
  `SHA256(model + text)`, so text-identical corpus entries collapse to a
  single file: the corpus has 90 898 entries
  (`tests/corpus_tests.rs::loads_full_real_corpus`) but ~90 896 unique store
  files. The file *count*, not the byte size, is hostile to container image
  layers, object storage, and Git-LFS. `Pipeline::load` opens every one of
  those files individually (`src/pipeline.rs:43-50`).
- **#4 — bind.** The index is keyed by `SHA256(model + text)`
  (`src/embed.rs:19-25`). A new corpus release or a changed `embed_model`
  silently invalidates every key; `Pipeline::load` then leaves the vector
  index `None` and Mode Hybrid degrades to Mode Lexical with only
  `degraded: true` to show for it — the same anonymous signal a transient
  IONOS outage raises. Nothing binds `{corpus snapshot, embed model, index}`
  as one unit.

This cycle delivers a single packed matrix file that carries a manifest in
its header, and a load-time check that makes a corpus/model mismatch a
distinct, observable state — loud but not fatal, so Mode Lexical (which never
needs the index) keeps serving.

The per-file store is **not** removed. It stays the build/resume path; the
packed file is an additive serve-path artifact.

## Architecture

Two iterations, one file format. The format (magic + manifest header +
payload) is born in iteration 1 and is stable from then on; iteration 2 adds
only the *checking* and the *observability*, never a format rewrite.

- **Iteration 1 (#3):** a new `src/packed.rs` module and an
  `alpha-id index --pack` subcommand. Packing reads a *complete* per-file
  store and writes `index/packed.<safe-model>.bin`. `Pipeline::load` prefers
  the packed file and builds the `VectorIndex` by **positional order**
  (row `i` = corpus entry `i`), falling back to the per-file store (today's
  path) when no packed file is present.
- **Iteration 2 (#4):** the corpus-hash binding in the header is now
  *enforced* at load: a load-time mismatch check, a distinct `index_status`
  carried in diagnostics, a prominent load-time stderr warning, and a Mode
  Hybrid degrade that names the reason instead of an anonymous `degraded`.

The packed-file path is derived, not configured: `<index_dir>` +
sanitized `embed_model`, exactly the sanitization `EmbeddingStore::open`
already applies (`src/embed.rs:27-33`). No new config field; `load` and
`--pack` compute the same path independently.

Row order is positional in corpus-load order (`corpus::load`,
`src/corpus.rs`): the packed matrix has exactly `n_rows = entries.len()`
rows (90 898 for the current corpus), one per corpus entry in load order.
This is the *entry* count, which exceeds the unique store-file count when
two entries share identical text — each such entry still gets its own row,
re-read from the one shared store file, so positional alignment holds.
Positional addressing is safe *because* the header binds the corpus
identity: if `corpus_sha256` matches the current corpus file, the
row-to-entry alignment is guaranteed; if it does not, the check fires and
the index is refused. The hash binding is what licenses dropping the
per-row SHA256 key.

## Concrete code shapes

### User-facing surface (the empirical evidence)

Producing the bundle, once, after a complete embedding run:

```text
$ alpha-id index --pack
packed 90898 rows × 1024 dims → index/packed.BAAI_bge-m3.bin (356 MB)
manifest: corpus=icd10gm2026_alphaidse_edvtxt_20250926.txt (sha256 4f3a…b1), model=BAAI/bge-m3
```

Packing an incomplete store is refused loudly:

```text
$ alpha-id index --pack
error: store incomplete: 41020 of 90898 entries embedded; run `index --full --confirm` first
$ echo $status
2
```

`suggest` and `eval` are unchanged at the CLI; `load` picks up the packed
file automatically. With a matching bundle:

```text
$ alpha-id suggest note.txt --mode hybrid
… suggestions …
[3 segments, mode Hybrid, 41 ms, degraded=false, index=ok]
```

With a corpus that no longer matches the deployed index — the formerly
silent downgrade is now named at load time and in the per-request line:

```text
$ alpha-id suggest note.txt --mode hybrid
warning: packed index index/packed.BAAI_bge-m3.bin does not match the current corpus
         (corpus sha256 mismatch); semantic index disabled, Mode Hybrid will degrade to Lexical
… suggestions …
[3 segments, mode Hybrid, 12 ms, degraded=true, index=mismatch: corpus sha256]
```

### Must-fail fixture (the criterion's evidence for the binding)

A packed file whose header model disagrees with the configured model MUST
NOT yield a vector index. The test pins exactly that rejection:

```rust
// A packed file built declaring embed_model "other/model", loaded under a
// config whose embed_model is "BAAI/bge-m3", must refuse the index.
let (vector, status) = load_packed_or_store(&cfg, &entries);
assert!(vector.is_none());
assert!(matches!(status, IndexStatus::Mismatch(_)));
```

### Header (written at pack, checked at load)

```rust
#[derive(serde::Serialize, serde::Deserialize)]
pub struct PackedHeader {
    pub n_rows: u32,
    pub dim: u32,
    pub embed_model: String,   // must equal cfg.embed_model
    pub corpus_sha256: String, // must equal corpus_sha256(cfg.alpha_id_path)
    pub corpus_name: String,   // human-readable; not part of the check
}
```

On-disk layout:

```text
0..4    magic  b"AIPK"
4..8    format_version  u32 LE      (fast reject without a JSON parse)
8..12   header_len      u32 LE
12..    header  (JSON, UTF-8, the PackedHeader above)
        pad to a 4-byte boundary
payload n_rows × dim × f32, little-endian, row-major, corpus order
```

### `Pipeline::load` — before → after

The load-bearing change replaces the per-entry collection
(`src/pipeline.rs:43-50`) with a packed-first helper.

```rust
// before
let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) {
    Ok(store) if entries.iter().all(|e| store.has(&e.text)) => {
        let vecs: Option<Vec<Vec<f32>>> =
            entries.iter().map(|e| store.get(&e.text)).collect();
        vecs.map(VectorIndex::from_vectors)
    }
    _ => None,
};
```

```rust
// after — packed file preferred; manifest enforced; per-file store is the fallback
let (vector, index_status) = load_packed_or_store(cfg, &entries);
if let IndexStatus::Mismatch(reason) = &index_status {
    eprintln!("warning: packed index {} does not match the current corpus/model \
               ({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical",
              packed_path(&cfg.index_dir, &cfg.embed_model).display());
}
// `index_status` is stored on the Pipeline and copied into Diagnostics per request.
```

### `IndexStatus` and the diagnostics surface

```rust
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
pub enum IndexStatus {
    Ok,               // packed file present and matches corpus + model
    Absent,           // no packed file and store incomplete — intentional lexical-only
    Mismatch(String), // packed file present but corpus/model/shape disagree; reason carried
}
```

`Diagnostics` (`src/model.rs:114-121`) gains an `index_status: IndexStatus`
field. The transient `degraded` flag keeps its meaning (a runtime IONOS step
failed); `index_status` answers the orthogonal question "is the deployed
index structurally the right one". A request can be `degraded: true` for
*either* reason, but `index=mismatch` names the structural one.

## Components

| File | Change |
|---|---|
| `src/packed.rs` *(new)* | `PackedHeader`; `write_packed(path, &PackedHeader, rows: &[Vec<f32>])`; `read_packed(path) -> Result<(PackedHeader, Vec<Vec<f32>>), AppError>`; `packed_path(index_dir, model) -> PathBuf`; `corpus_sha256(path) -> Result<String, AppError>` |
| `src/pipeline.rs` | `load_packed_or_store(cfg, entries) -> (Option<VectorIndex>, IndexStatus)` replacing the `:43-50` match; `Pipeline` stores `index_status`; `suggest` copies it into `Diagnostics` |
| `src/model.rs` | `enum IndexStatus`; `index_status` field on `Diagnostics` |
| `src/bin/alpha_id.rs` | `--pack` flag on the `Index` subcommand; the load-time mismatch warning; `index=…` in the human diagnostics line |
| `src/vector.rs` | **unchanged** — still consumes `Vec<Vec<f32>>` via `from_vectors` |
| `src/embed.rs` | **unchanged** — the per-file store stays the build/resume path |

## Data flow

- **pack** (`index --pack`): `corpus::load` → open the per-file store →
  require every entry cached (else refuse, exit 2) → collect rows in corpus
  order → `write_packed(header { embed_model, corpus_sha256, n_rows, dim },
  rows)`.
- **load** (`Pipeline::load`): if `packed_path` exists → `read_packed` →
  validate `header.embed_model == cfg.embed_model`,
  `header.corpus_sha256 == corpus_sha256(cfg.alpha_id_path)`,
  `header.n_rows as usize == entries.len()`, payload length consistent →
  on full match build `VectorIndex::from_vectors`, status `Ok`; on any
  disagreement status `Mismatch(reason)`, vector `None`. If no packed file →
  per-file store fallback (today's all-or-nothing build); complete ⇒ build,
  status `Ok`; incomplete ⇒ `None`, status `Absent`.
- **suggest**: Mode Hybrid with `vector == None` degrades to
  `collect_lexical` (unchanged); `Diagnostics.index_status` carries the
  structural reason; `Diagnostics.degraded` stays the transient signal.

## Error handling

- **`--pack` on an incomplete store** → loud refusal, exit code 2, a message
  naming the cached/total counts and pointing at `index --full --confirm`.
  Packing a partial store is forbidden: it would mispair rows against the
  corpus.
- **Corrupt or truncated packed file** (payload length ≠
  `n_rows × dim × 4`), **wrong magic**, or a **`format_version` newer than
  supported** → `IndexStatus::Mismatch(reason)`, prominent load warning, no
  index built, Mode Lexical serves. Not fatal — a structurally wrong index
  must not take down the lexical service (chosen mismatch posture).
- **Corpus file unreadable** for hashing → surfaces as the existing
  `corpus::load` error before any packed-load work (corpus load precedes it).

## Testing strategy

- **Round-trip (unit):** `write_packed` then `read_packed` reproduces the
  header and every row bit-exactly (f32 little-endian preserved).
- **Corpus hash (unit):** `corpus_sha256` is stable for identical bytes and
  differs when the corpus file changes by one byte.
- **Must-fail binding (integration):** a packed file built declaring one
  `embed_model` (or one `corpus_sha256`), loaded under a config that
  disagrees, yields `vector == None` and `IndexStatus::Mismatch(_)`. Wrong
  bundle never builds an index.
- **Positional alignment (integration):** after `read_packed`, row `i`
  equals `store.get(&entries[i].text)` for sampled `i` — packing preserves
  corpus-order alignment.
- **Observable degrade (integration):** under a corpus-hash mismatch, a Mode
  Hybrid `suggest` returns `degraded == true` *and* `index_status ==
  Mismatch(_)`, distinguishable from a transient-IONOS degrade (which leaves
  `index_status == Ok`).
- **Packed beats store (integration):** with a valid packed file present and
  the per-file store absent, `Pipeline::load` builds the `VectorIndex` from
  the packed file alone (acceptance #3b).

## Acceptance criteria

**Issue #3 (pack):**

- [ ] A single-file packed matrix is produced from a complete per-file store
  via `alpha-id index --pack`.
- [ ] `Pipeline::load` builds its `VectorIndex` from the packed file without
  reading the per-file store.
- [ ] Row order is positional in corpus order, documented here, and
  verifiably aligned to the corpus snapshot via the header's `corpus_sha256`.

**Issue #4 (bind):**

- [ ] A mismatch between the deployed index and the corpus+model is
  detectable at load time (the load-time warning and `IndexStatus`), not only
  via `degraded: true` after a request.
- [ ] The intended-mode-vs-actual-mode gap (Mode Hybrid requested, Mode
  Lexical served because the index is structurally wrong) is observable as a
  distinct `index_status` in diagnostics, separate from the transient
  `degraded` flag.

**Out of scope:**

- True `mmap` (a single `fs::read` is loaded; the brute-force full-scan
  cosine search touches every row per query, so memory-mapping yields no
  runtime benefit and is rejected to avoid an `unsafe` dependency).
- Incremental / delta re-pack (a corpus change triggers a full re-embed of
  the changed entries via the existing hash-keyed store, then a full
  re-pack; partial repack is not built).
- Removing or replacing the per-file store.