alpha-id/docs/plans/2026-06-01-packed-versioned-bundle-iter2.md

Index Binding Enforcement (Iteration 2) — Implementation Plan

Parent spec: docs/specs/2026-05-31-packed-versioned-index-bundle.md

For agentic workers: REQUIRED SUB-SKILL: use the implement skill to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Enforce the packed header's corpus+model binding at load time and make a mismatch observable — a load-time warning and a distinct index= token in diagnostics — instead of an anonymous silent degrade.

Architecture: Iteration 1 already delivers the format, --pack, the IndexStatus enum, the Diagnostics.index_status field, and a load_packed_or_store that validates structure only. This iteration adds the semantic comparison (header.embed_model / header.corpus_sha256 vs the live config) inside load_packed_or_store, a Display for IndexStatus, a load-time warning in Pipeline::load, and an index= token on the human diagnostics line.

Tech Stack: Rust; src/pipeline.rs (the load fn + the warning), src/model.rs (Display), src/bin/alpha_id.rs (the diagnostics token); sha2 via the existing packed::corpus_sha256.

Iteration boundary: This is iteration 2 (issue #4, acceptance d/e), the enforcement + observability layer. Iteration 1 (#3) shipped the format and structural validation. This plan does NOT touch the per-file store build path, the embedding build, or mmap. No IONOS calls are made by any test: under a mismatch the vector index is None, and collect_hybrid short-circuits on self.vector being None before its first network call (src/pipeline.rs:176-177, before the post at :189), so a Mode Hybrid suggest degrades hermetically.

---

Files this plan creates or modifies:

• Modify: src/model.rs — add impl std::fmt::Display for IndexStatus.
• Modify: src/pipeline.rs:27-35,51,84 — semantic checks in the load_packed_or_store success branch; refresh the now-stale iteration-1 boundary doc-comment; add the load-time mismatch warning in Pipeline::load.
• Modify: src/bin/alpha_id.rs — append , index={} to the human diagnostics eprintln!.
• Test: tests/packed_load_tests.rs — model-mismatch and corpus-hash-mismatch yield Mismatch; a matching pair still yields Ok; Display renders the spec tokens.
• Test: tests/index_mismatch_tests.rs (new) — a Mode Hybrid suggest under a mismatched packed file is degraded with index_status == Mismatch; the CLI prints the load warning and the index=mismatch token to stderr.
• Test: tests/pipeline_hybrid_tests.rs — extend the existing transient-degrade test with the index_status == Ok foil assertion.

---

Task 1: Display for IndexStatus + semantic enforcement in load_packed_or_store

Files:

• Modify: src/model.rs

• Modify: src/pipeline.rs:27-35,51

• Test: tests/packed_load_tests.rs

• Step 1: Write the failing tests

Append to tests/packed_load_tests.rs (the file already has helpers entry, cfg_in and imports Config, IndexStatus, packed_path, write_packed, PackedHeader, load_packed_or_store). Add one import line at the top, use std::fmt::Write as _; is NOT needed — only add:

use alpha_id::packed::corpus_sha256;

Then append these tests:

fn header_for(cfg: &Config, n_rows: u32, model: &str, corpus_hash: &str) -> PackedHeader {
    let _ = cfg;
    PackedHeader {
        n_rows,
        dim: 2,
        embed_model: model.to_string(),
        corpus_sha256: corpus_hash.to_string(),
        corpus_name: "c.txt".to_string(),
    }
}

#[test]
fn index_status_display_renders_spec_tokens() {
    assert_eq!(format!("{}", IndexStatus::Ok), "ok");
    assert_eq!(format!("{}", IndexStatus::Absent), "absent");
    assert_eq!(
        format!("{}", IndexStatus::Mismatch("corpus sha256".to_string())),
        "mismatch: corpus sha256"
    );
}

#[test]
fn model_mismatch_yields_mismatch_none() {
    let dir = tempfile::tempdir().unwrap();
    let cfg = cfg_in(dir.path()); // cfg.embed_model == "test/model"
    let entries = vec![entry("a"), entry("b")];
    let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]];
    // n_rows matches (2) so the structural check passes; the header declares a
    // different model, so the semantic check must reject it.
    let h = header_for(&cfg, 2, "other/model", "irrelevant");
    write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap();
    let (vector, status) = load_packed_or_store(&cfg, &entries);
    assert!(vector.is_none());
    assert!(matches!(status, IndexStatus::Mismatch(_)));
}

#[test]
fn corpus_hash_mismatch_yields_mismatch_none() {
    let dir = tempfile::tempdir().unwrap();
    let mut cfg = cfg_in(dir.path());
    // Point the corpus at a tiny temp file so the hash is hermetic.
    let corpus = dir.path().join("corpus.txt");
    std::fs::write(&corpus, b"some corpus bytes").unwrap();
    cfg.alpha_id_path = corpus.to_str().unwrap().to_string();
    let entries = vec![entry("a"), entry("b")];
    let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]];
    // Model matches, but the header's corpus hash does not match the file's.
    let h = header_for(&cfg, 2, &cfg.embed_model, "deadbeef-not-the-real-hash");
    write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap();
    let (vector, status) = load_packed_or_store(&cfg, &entries);
    assert!(vector.is_none());
    assert!(matches!(status, IndexStatus::Mismatch(_)));
}

#[test]
fn matching_model_and_corpus_yields_ok() {
    let dir = tempfile::tempdir().unwrap();
    let mut cfg = cfg_in(dir.path());
    let corpus = dir.path().join("corpus.txt");
    std::fs::write(&corpus, b"some corpus bytes").unwrap();
    cfg.alpha_id_path = corpus.to_str().unwrap().to_string();
    let entries = vec![entry("a"), entry("b")];
    let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]];
    let real_hash = corpus_sha256(&cfg.alpha_id_path).unwrap();
    let h = header_for(&cfg, 2, &cfg.embed_model, &real_hash);
    write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap();
    let (vector, status) = load_packed_or_store(&cfg, &entries);
    assert!(vector.is_some());
    assert_eq!(status, IndexStatus::Ok);
}

• Step 2: Run the tests to verify they fail

Run: cargo test --test packed_load_tests Expected: FAIL — index_status_display_renders_spec_tokens fails to compile (IndexStatus doesn't implement Display, so format!("{}", …) errors), which reddens the whole test binary. (Functionally, model_mismatch_… and corpus_hash_mismatch_… would also fail once compiling, because iteration 1 returns (Some, Ok) on an n_rows match without comparing model/corpus.)

• Step 3: Add Display for IndexStatus in src/model.rs

Immediately after the IndexStatus enum definition (the pub enum IndexStatus { … } block), add:

impl std::fmt::Display for IndexStatus {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            IndexStatus::Ok => write!(f, "ok"),
            IndexStatus::Absent => write!(f, "absent"),
            IndexStatus::Mismatch(reason) => write!(f, "mismatch: {reason}"),
        }
    }
}

• Step 4: Add the semantic checks in load_packed_or_store

In src/pipeline.rs, replace the success-branch line (src/pipeline.rs:51):

                return (Some(VectorIndex::from_vectors(rows)), IndexStatus::Ok);

with the model-then-corpus comparison:

                if header.embed_model != cfg.embed_model {
                    return (None, IndexStatus::Mismatch("embed model".to_string()));
                }
                match packed::corpus_sha256(&cfg.alpha_id_path) {
                    Ok(h) if h == header.corpus_sha256 => {
                        return (Some(VectorIndex::from_vectors(rows)), IndexStatus::Ok);
                    }
                    Ok(_) => return (None, IndexStatus::Mismatch("corpus sha256".to_string())),
                    Err(e) => return (None, IndexStatus::Mismatch(format!("corpus hash: {e}"))),
                }

• Step 5: Refresh the now-stale iteration-1 boundary comment

In src/pipeline.rs, replace the doc-comment paragraph (src/pipeline.rs:31-35) that currently reads:

/// Iteration-1 scope: the packed file is validated *structurally* only
/// (`read_packed` checks magic/version/payload length; here we additionally
/// require `n_rows == entries.len()`). The header's `corpus_sha256` and
/// `embed_model` are NOT yet compared against the config — that enforcement
/// is iteration 2.

with:

/// Validation: structural first (`read_packed` checks magic/version/payload
/// length; here we additionally require `n_rows == entries.len()`), then
/// semantic — the header's `embed_model` and `corpus_sha256` must match the
/// live config, else the packed file is refused as a `Mismatch`. A `Mismatch`
/// yields no index, so Mode Hybrid degrades to Lexical rather than serving a
/// matrix that does not belong to the deployed corpus/model.

• Step 6: Run the tests to verify they pass

Run: cargo test --test packed_load_tests Expected: PASS — all tests pass (the 4 from iteration 1 plus the 4 new ones).

---

Task 2: Load-time warning + index= diagnostics token + observable-degrade tests

Files:

• Modify: src/pipeline.rs:84

• Modify: src/bin/alpha_id.rs

• Test: tests/index_mismatch_tests.rs (new)

• Test: tests/pipeline_hybrid_tests.rs

• Step 1: Write the failing tests

Create tests/index_mismatch_tests.rs:

use alpha_id::model::{Config, Filter, IndexStatus, Mode};
use alpha_id::packed::{packed_path, write_packed, PackedHeader};
use alpha_id::pipeline::Pipeline;
use std::process::Command;

// A structurally-mismatched packed file at the config's derived packed path.
// n_rows=1 will not equal the real corpus entry count, so load reports a
// Mismatch — enough to exercise the warning and the diagnostics token. Uses
// the real corpus/claml from config/default.toml with index_dir redirected to
// a temp dir, so the real index is never touched.
fn write_mismatched_packed(cfg: &Config) {
    let header = PackedHeader {
        n_rows: 1,
        dim: 1,
        embed_model: cfg.embed_model.clone(),
        corpus_sha256: "x".to_string(),
        corpus_name: "x".to_string(),
    };
    write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &header, &[vec![0.0f32]]).unwrap();
}

#[test]
fn hybrid_suggest_under_mismatch_is_degraded_and_index_mismatch() {
    let dir = tempfile::tempdir().unwrap();
    let mut cfg = Config::load("config/default.toml").unwrap();
    cfg.index_dir = dir.path().to_str().unwrap().to_string();
    cfg.token_path = "/nonexistent".to_string();
    write_mismatched_packed(&cfg);

    let p = Pipeline::load(&cfg).unwrap();
    let res = p.suggest("knee pain", Mode::Hybrid, &Filter::default(), 5);
    assert!(res.diagnostics.degraded, "hybrid degrades when the index is mismatched");
    assert!(
        matches!(res.diagnostics.index_status, IndexStatus::Mismatch(_)),
        "index_status names the structural problem, not an anonymous degrade"
    );
}

#[test]
fn cli_suggest_under_mismatch_warns_and_shows_index_token() {
    let dir = tempfile::tempdir().unwrap();
    let mut cfg = Config::load("config/default.toml").unwrap();
    cfg.index_dir = dir.path().to_str().unwrap().to_string();
    write_mismatched_packed(&cfg);

    // Build a temp config TOML reusing the real data paths but the temp index dir.
    let toml = format!(
        r#"ionos_base_url = "http://127.0.0.1:1"
token_path = "/nonexistent"
embed_model = "{embed_model}"
rerank_model = "{rerank_model}"
alpha_id_path = "{alpha_id_path}"
claml_path = "{claml_path}"
index_dir = "{index_dir}"
pool_size = {pool_size}
top_k = {top_k}
"#,
        embed_model = cfg.embed_model,
        rerank_model = cfg.rerank_model,
        alpha_id_path = cfg.alpha_id_path,
        claml_path = cfg.claml_path,
        index_dir = cfg.index_dir,
        pool_size = cfg.pool_size,
        top_k = cfg.top_k,
    );
    let config_path = dir.path().join("config.toml");
    std::fs::write(&config_path, toml).unwrap();

    // Mode Lexical needs no IONOS; the warning + token come from load, not the mode.
    let out = Command::new(env!("CARGO_BIN_EXE_alpha-id"))
        .args(["--config", config_path.to_str().unwrap(), "suggest", "-", "--mode", "lexical"])
        .env("ALPHA_ID_STDIN", "knee pain")
        .output()
        .unwrap();
    let stderr = String::from_utf8_lossy(&out.stderr);
    assert!(stderr.contains("does not match"), "load warning missing; stderr: {stderr}");
    assert!(stderr.contains("index=mismatch"), "diagnostics token missing; stderr: {stderr}");
}

Append the foil assertion to the existing transient-degrade test in tests/pipeline_hybrid_tests.rs. That test today degrades via a refused IONOS URL while the real (complete) store yields a valid index, so its index_status is Ok — the contrast to the mismatch case above. Add, after its existing assert!(… degraded …) line, these two lines (and ensure IndexStatus is imported at the top of the file — add use alpha_id::model::IndexStatus; if absent):

    // Foil: a transient IONOS degrade leaves the index structurally Ok —
    // distinct from a corpus/model mismatch, which would be IndexStatus::Mismatch.
    assert_eq!(res.diagnostics.index_status, IndexStatus::Ok);

• Step 2: Run the tests to verify they fail

Run: cargo test --test index_mismatch_tests Expected: FAIL — cli_suggest_under_mismatch_warns_and_shows_index_token fails on stderr.contains("index=mismatch") (the diagnostics line has no index= token yet) and on stderr.contains("does not match") (no load warning is emitted yet). hybrid_suggest_under_mismatch_is_degraded_and_index_mismatch already passes on the Task-1 enforcement (the mismatch is produced; this test guards that the degrade path carries it), so the redness is in the CLI test.

• Step 3: Add the load-time warning in Pipeline::load

In src/pipeline.rs, the call site is src/pipeline.rs:84:

        let (vector, index_status) = load_packed_or_store(cfg, &entries);

Insert the warning immediately after it (before the Ok(Self { … })):

        let (vector, index_status) = load_packed_or_store(cfg, &entries);
        if let IndexStatus::Mismatch(reason) = &index_status {
            eprintln!(
                "warning: packed index {} does not match the current corpus/model \
                 ({reason}); semantic index disabled, Mode Hybrid will degrade to Lexical",
                packed::packed_path(&cfg.index_dir, &cfg.embed_model).display()
            );
        }

• Step 4: Append the index= token to the human diagnostics line

In src/bin/alpha_id.rs, the human diagnostics eprintln! currently reads:

                eprintln!("[{} segments, mode {}, {} ms, degraded={}]",
                    res.diagnostics.segment_count, res.diagnostics.mode,
                    res.diagnostics.millis, res.diagnostics.degraded);

Replace it with (adds , index={} and the index_status arg, which renders via the Display from Task 1):

                eprintln!("[{} segments, mode {}, {} ms, degraded={}, index={}]",
                    res.diagnostics.segment_count, res.diagnostics.mode,
                    res.diagnostics.millis, res.diagnostics.degraded,
                    res.diagnostics.index_status);

• Step 5: Run the new tests to verify they pass

Run: cargo test --test index_mismatch_tests --test pipeline_hybrid_tests Expected: PASS — the CLI test now sees the warning and index=mismatch; the hybrid lib test and the extended transient-degrade foil pass.

• Step 6: Run the full suite to verify no regression

Run: cargo test Expected: PASS — the whole suite is green.