diff --git a/docs/plans/2026-06-01-packed-versioned-bundle-iter2.md b/docs/plans/2026-06-01-packed-versioned-bundle-iter2.md new file mode 100644 index 0000000..b574a26 --- /dev/null +++ b/docs/plans/2026-06-01-packed-versioned-bundle-iter2.md @@ -0,0 +1,396 @@ +# Index Binding Enforcement (Iteration 2) — Implementation Plan + +> **Parent spec:** `docs/specs/2026-05-31-packed-versioned-index-bundle.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` +> skill to run this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Enforce the packed header's corpus+model binding at load time +and make a mismatch observable — a load-time warning and a distinct +`index=` token in diagnostics — instead of an anonymous silent degrade. + +**Architecture:** Iteration 1 already delivers the format, `--pack`, the +`IndexStatus` enum, the `Diagnostics.index_status` field, and a +`load_packed_or_store` that validates structure only. This iteration adds +the *semantic* comparison (`header.embed_model` / `header.corpus_sha256` +vs the live config) inside `load_packed_or_store`, a `Display` for +`IndexStatus`, a load-time warning in `Pipeline::load`, and an `index=` +token on the human diagnostics line. + +**Tech Stack:** Rust; `src/pipeline.rs` (the load fn + the warning), +`src/model.rs` (`Display`), `src/bin/alpha_id.rs` (the diagnostics token); +`sha2` via the existing `packed::corpus_sha256`. + +**Iteration boundary:** This is iteration 2 (issue #4, acceptance d/e), +the enforcement + observability layer. Iteration 1 (#3) shipped the format +and structural validation. This plan does NOT touch the per-file store +build path, the embedding build, or `mmap`. No IONOS calls are made by any +test: under a mismatch the vector index is `None`, and `collect_hybrid` +short-circuits on `self.vector` being `None` *before* its first network +call (`src/pipeline.rs:176-177`, before the post at `:189`), so a Mode +Hybrid suggest degrades hermetically. + +--- + +**Files this plan creates or modifies:** + +- Modify: `src/model.rs` — add `impl std::fmt::Display for IndexStatus`. +- Modify: `src/pipeline.rs:27-35,51,84` — semantic checks in the + `load_packed_or_store` success branch; refresh the now-stale iteration-1 + boundary doc-comment; add the load-time mismatch warning in + `Pipeline::load`. +- Modify: `src/bin/alpha_id.rs` — append `, index={}` to the human + diagnostics `eprintln!`. +- Test: `tests/packed_load_tests.rs` — model-mismatch and + corpus-hash-mismatch yield `Mismatch`; a matching pair still yields + `Ok`; `Display` renders the spec tokens. +- Test: `tests/index_mismatch_tests.rs` *(new)* — a Mode Hybrid suggest + under a mismatched packed file is `degraded` with `index_status == + Mismatch`; the CLI prints the load warning and the `index=mismatch` + token to stderr. +- Test: `tests/pipeline_hybrid_tests.rs` — extend the existing + transient-degrade test with the `index_status == Ok` foil assertion. + +--- + +### Task 1: `Display` for `IndexStatus` + semantic enforcement in `load_packed_or_store` + +**Files:** +- Modify: `src/model.rs` +- Modify: `src/pipeline.rs:27-35,51` +- Test: `tests/packed_load_tests.rs` + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/packed_load_tests.rs` (the file already has helpers +`entry`, `cfg_in` and imports `Config`, `IndexStatus`, `packed_path`, +`write_packed`, `PackedHeader`, `load_packed_or_store`). Add one import line +at the top, `use std::fmt::Write as _;` is NOT needed — only add: + +```rust +use alpha_id::packed::corpus_sha256; +``` + +Then append these tests: + +```rust +fn header_for(cfg: &Config, n_rows: u32, model: &str, corpus_hash: &str) -> PackedHeader { + let _ = cfg; + PackedHeader { + n_rows, + dim: 2, + embed_model: model.to_string(), + corpus_sha256: corpus_hash.to_string(), + corpus_name: "c.txt".to_string(), + } +} + +#[test] +fn index_status_display_renders_spec_tokens() { + assert_eq!(format!("{}", IndexStatus::Ok), "ok"); + assert_eq!(format!("{}", IndexStatus::Absent), "absent"); + assert_eq!( + format!("{}", IndexStatus::Mismatch("corpus sha256".to_string())), + "mismatch: corpus sha256" + ); +} + +#[test] +fn model_mismatch_yields_mismatch_none() { + let dir = tempfile::tempdir().unwrap(); + let cfg = cfg_in(dir.path()); // cfg.embed_model == "test/model" + let entries = vec![entry("a"), entry("b")]; + let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]]; + // n_rows matches (2) so the structural check passes; the header declares a + // different model, so the semantic check must reject it. + let h = header_for(&cfg, 2, "other/model", "irrelevant"); + write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap(); + let (vector, status) = load_packed_or_store(&cfg, &entries); + assert!(vector.is_none()); + assert!(matches!(status, IndexStatus::Mismatch(_))); +} + +#[test] +fn corpus_hash_mismatch_yields_mismatch_none() { + let dir = tempfile::tempdir().unwrap(); + let mut cfg = cfg_in(dir.path()); + // Point the corpus at a tiny temp file so the hash is hermetic. + let corpus = dir.path().join("corpus.txt"); + std::fs::write(&corpus, b"some corpus bytes").unwrap(); + cfg.alpha_id_path = corpus.to_str().unwrap().to_string(); + let entries = vec![entry("a"), entry("b")]; + let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]]; + // Model matches, but the header's corpus hash does not match the file's. + let h = header_for(&cfg, 2, &cfg.embed_model, "deadbeef-not-the-real-hash"); + write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap(); + let (vector, status) = load_packed_or_store(&cfg, &entries); + assert!(vector.is_none()); + assert!(matches!(status, IndexStatus::Mismatch(_))); +} + +#[test] +fn matching_model_and_corpus_yields_ok() { + let dir = tempfile::tempdir().unwrap(); + let mut cfg = cfg_in(dir.path()); + let corpus = dir.path().join("corpus.txt"); + std::fs::write(&corpus, b"some corpus bytes").unwrap(); + cfg.alpha_id_path = corpus.to_str().unwrap().to_string(); + let entries = vec![entry("a"), entry("b")]; + let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]]; + let real_hash = corpus_sha256(&cfg.alpha_id_path).unwrap(); + let h = header_for(&cfg, 2, &cfg.embed_model, &real_hash); + write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap(); + let (vector, status) = load_packed_or_store(&cfg, &entries); + assert!(vector.is_some()); + assert_eq!(status, IndexStatus::Ok); +} +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cargo test --test packed_load_tests` +Expected: FAIL — `index_status_display_renders_spec_tokens` fails to compile +(`IndexStatus` doesn't implement `Display`, so `format!("{}", …)` errors), +which reddens the whole test binary. (Functionally, `model_mismatch_…` and +`corpus_hash_mismatch_…` would also fail once compiling, because iteration 1 +returns `(Some, Ok)` on an `n_rows` match without comparing model/corpus.) + +- [ ] **Step 3: Add `Display` for `IndexStatus` in `src/model.rs`** + +Immediately after the `IndexStatus` enum definition (the +`pub enum IndexStatus { … }` block), add: + +```rust +impl std::fmt::Display for IndexStatus { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + IndexStatus::Ok => write!(f, "ok"), + IndexStatus::Absent => write!(f, "absent"), + IndexStatus::Mismatch(reason) => write!(f, "mismatch: {reason}"), + } + } +} +``` + +- [ ] **Step 4: Add the semantic checks in `load_packed_or_store`** + +In `src/pipeline.rs`, replace the success-branch line (`src/pipeline.rs:51`): + +```rust + return (Some(VectorIndex::from_vectors(rows)), IndexStatus::Ok); +``` + +with the model-then-corpus comparison: + +```rust + if header.embed_model != cfg.embed_model { + return (None, IndexStatus::Mismatch("embed model".to_string())); + } + match packed::corpus_sha256(&cfg.alpha_id_path) { + Ok(h) if h == header.corpus_sha256 => { + return (Some(VectorIndex::from_vectors(rows)), IndexStatus::Ok); + } + Ok(_) => return (None, IndexStatus::Mismatch("corpus sha256".to_string())), + Err(e) => return (None, IndexStatus::Mismatch(format!("corpus hash: {e}"))), + } +``` + +- [ ] **Step 5: Refresh the now-stale iteration-1 boundary comment** + +In `src/pipeline.rs`, replace the doc-comment paragraph (`src/pipeline.rs:31-35`) +that currently reads: + +```rust +/// Iteration-1 scope: the packed file is validated *structurally* only +/// (`read_packed` checks magic/version/payload length; here we additionally +/// require `n_rows == entries.len()`). The header's `corpus_sha256` and +/// `embed_model` are NOT yet compared against the config — that enforcement +/// is iteration 2. +``` + +with: + +```rust +/// Validation: structural first (`read_packed` checks magic/version/payload +/// length; here we additionally require `n_rows == entries.len()`), then +/// semantic — the header's `embed_model` and `corpus_sha256` must match the +/// live config, else the packed file is refused as a `Mismatch`. A `Mismatch` +/// yields no index, so Mode Hybrid degrades to Lexical rather than serving a +/// matrix that does not belong to the deployed corpus/model. +``` + +- [ ] **Step 6: Run the tests to verify they pass** + +Run: `cargo test --test packed_load_tests` +Expected: PASS — all tests pass (the 4 from iteration 1 plus the 4 new ones). + +--- + +### Task 2: Load-time warning + `index=` diagnostics token + observable-degrade tests + +**Files:** +- Modify: `src/pipeline.rs:84` +- Modify: `src/bin/alpha_id.rs` +- Test: `tests/index_mismatch_tests.rs` *(new)* +- Test: `tests/pipeline_hybrid_tests.rs` + +- [ ] **Step 1: Write the failing tests** + +Create `tests/index_mismatch_tests.rs`: + +```rust +use alpha_id::model::{Config, Filter, IndexStatus, Mode}; +use alpha_id::packed::{packed_path, write_packed, PackedHeader}; +use alpha_id::pipeline::Pipeline; +use std::process::Command; + +// A structurally-mismatched packed file at the config's derived packed path. +// n_rows=1 will not equal the real corpus entry count, so load reports a +// Mismatch — enough to exercise the warning and the diagnostics token. Uses +// the real corpus/claml from config/default.toml with index_dir redirected to +// a temp dir, so the real index is never touched. +fn write_mismatched_packed(cfg: &Config) { + let header = PackedHeader { + n_rows: 1, + dim: 1, + embed_model: cfg.embed_model.clone(), + corpus_sha256: "x".to_string(), + corpus_name: "x".to_string(), + }; + write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &header, &[vec![0.0f32]]).unwrap(); +} + +#[test] +fn hybrid_suggest_under_mismatch_is_degraded_and_index_mismatch() { + let dir = tempfile::tempdir().unwrap(); + let mut cfg = Config::load("config/default.toml").unwrap(); + cfg.index_dir = dir.path().to_str().unwrap().to_string(); + cfg.token_path = "/nonexistent".to_string(); + write_mismatched_packed(&cfg); + + let p = Pipeline::load(&cfg).unwrap(); + let res = p.suggest("knee pain", Mode::Hybrid, &Filter::default(), 5); + assert!(res.diagnostics.degraded, "hybrid degrades when the index is mismatched"); + assert!( + matches!(res.diagnostics.index_status, IndexStatus::Mismatch(_)), + "index_status names the structural problem, not an anonymous degrade" + ); +} + +#[test] +fn cli_suggest_under_mismatch_warns_and_shows_index_token() { + let dir = tempfile::tempdir().unwrap(); + let mut cfg = Config::load("config/default.toml").unwrap(); + cfg.index_dir = dir.path().to_str().unwrap().to_string(); + write_mismatched_packed(&cfg); + + // Build a temp config TOML reusing the real data paths but the temp index dir. + let toml = format!( + r#"ionos_base_url = "http://127.0.0.1:1" +token_path = "/nonexistent" +embed_model = "{embed_model}" +rerank_model = "{rerank_model}" +alpha_id_path = "{alpha_id_path}" +claml_path = "{claml_path}" +index_dir = "{index_dir}" +pool_size = {pool_size} +top_k = {top_k} +"#, + embed_model = cfg.embed_model, + rerank_model = cfg.rerank_model, + alpha_id_path = cfg.alpha_id_path, + claml_path = cfg.claml_path, + index_dir = cfg.index_dir, + pool_size = cfg.pool_size, + top_k = cfg.top_k, + ); + let config_path = dir.path().join("config.toml"); + std::fs::write(&config_path, toml).unwrap(); + + // Mode Lexical needs no IONOS; the warning + token come from load, not the mode. + let out = Command::new(env!("CARGO_BIN_EXE_alpha-id")) + .args(["--config", config_path.to_str().unwrap(), "suggest", "-", "--mode", "lexical"]) + .env("ALPHA_ID_STDIN", "knee pain") + .output() + .unwrap(); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("does not match"), "load warning missing; stderr: {stderr}"); + assert!(stderr.contains("index=mismatch"), "diagnostics token missing; stderr: {stderr}"); +} +``` + +Append the foil assertion to the existing transient-degrade test in +`tests/pipeline_hybrid_tests.rs`. That test today degrades via a refused +IONOS URL while the real (complete) store yields a valid index, so its +`index_status` is `Ok` — the contrast to the mismatch case above. Add, after +its existing `assert!(… degraded …)` line, these two lines (and ensure +`IndexStatus` is imported at the top of the file — add +`use alpha_id::model::IndexStatus;` if absent): + +```rust + // Foil: a transient IONOS degrade leaves the index structurally Ok — + // distinct from a corpus/model mismatch, which would be IndexStatus::Mismatch. + assert_eq!(res.diagnostics.index_status, IndexStatus::Ok); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cargo test --test index_mismatch_tests` +Expected: FAIL — `cli_suggest_under_mismatch_warns_and_shows_index_token` +fails on `stderr.contains("index=mismatch")` (the diagnostics line has no +`index=` token yet) and on `stderr.contains("does not match")` (no load +warning is emitted yet). `hybrid_suggest_under_mismatch_is_degraded_and_index_mismatch` +already passes on the Task-1 enforcement (the mismatch is produced; this test +guards that the degrade path carries it), so the redness is in the CLI test. + +- [ ] **Step 3: Add the load-time warning in `Pipeline::load`** + +In `src/pipeline.rs`, the call site is `src/pipeline.rs:84`: + +```rust + let (vector, index_status) = load_packed_or_store(cfg, &entries); +``` + +Insert the warning immediately after it (before the `Ok(Self { … })`): + +```rust + let (vector, index_status) = load_packed_or_store(cfg, &entries); + if let IndexStatus::Mismatch(reason) = &index_status { + eprintln!( + "warning: packed index {} does not match the current corpus/model \ + ({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical", + packed::packed_path(&cfg.index_dir, &cfg.embed_model).display() + ); + } +``` + +- [ ] **Step 4: Append the `index=` token to the human diagnostics line** + +In `src/bin/alpha_id.rs`, the human diagnostics `eprintln!` currently reads: + +```rust + eprintln!("[{} segments, mode {}, {} ms, degraded={}]", + res.diagnostics.segment_count, res.diagnostics.mode, + res.diagnostics.millis, res.diagnostics.degraded); +``` + +Replace it with (adds `, index={}` and the `index_status` arg, which renders +via the `Display` from Task 1): + +```rust + eprintln!("[{} segments, mode {}, {} ms, degraded={}, index={}]", + res.diagnostics.segment_count, res.diagnostics.mode, + res.diagnostics.millis, res.diagnostics.degraded, + res.diagnostics.index_status); +``` + +- [ ] **Step 5: Run the new tests to verify they pass** + +Run: `cargo test --test index_mismatch_tests --test pipeline_hybrid_tests` +Expected: PASS — the CLI test now sees the warning and `index=mismatch`; the +hybrid lib test and the extended transient-degrade foil pass. + +- [ ] **Step 6: Run the full suite to verify no regression** + +Run: `cargo test` +Expected: PASS — the whole suite is green.