# Packed Index Format (Iteration 1) — Implementation Plan > **Parent spec:** `docs/specs/2026-05-31-packed-versioned-index-bundle.md` > > **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` > skill to run this plan. Steps use `- [ ]` checkboxes for tracking. **Goal:** Pack a complete per-file embedding store into one self-describing matrix file and load the semantic index from it in a single read, with the per-file store kept as a fallback. **Architecture:** A new `src/packed.rs` owns the on-disk format (magic + JSON header + row-major f32 payload) and the corpus hash. `Pipeline::load` calls a new free fn `load_packed_or_store` that prefers the packed file (structural validation only) and falls back to today's per-file store, reporting which path it took via a new `IndexStatus`. An `index --pack` subcommand writes the file from a complete store. **Tech Stack:** Rust; `serde`/`serde_json` (header), `sha2` (corpus hash) — both already in `Cargo.toml`; `clap` (the `--pack` flag); `tempfile` (test fixtures). **Iteration boundary (load-bearing — do not exceed):** This plan is Iteration 1 (issue #3, acceptance a/b/c). The packed header *carries* `corpus_sha256` and `embed_model`, but `load_packed_or_store` validates only the file's *structure* (magic, supported version, `n_rows == entries.len()`, payload length) — it does **not** compare `corpus_sha256` or `embed_model` against the live config. The semantic mismatch check, the load-time warning, the `index=` token on the diagnostics line, and the Hybrid-degrade-names-reason wiring are Iteration 2 (issue #4) and are OUT OF SCOPE here. `IndexStatus::Mismatch` is produced in this iteration *only* for structural corruption. --- **Files this plan creates or modifies:** - Create: `src/packed.rs` — packed file format I/O: `PackedHeader`, `write_packed`, `read_packed`, `packed_path`, `corpus_sha256`. - Modify: `src/lib.rs:13` — add `pub mod packed;` to the module list. - Modify: `src/model.rs:80-81,114-121` — `IndexStatus` enum; `index_status` field on `Diagnostics`. - Modify: `src/pipeline.rs:15-23,26-60,229-235` — `load_packed_or_store` free fn replacing the `:43-50` match; `Pipeline.index_status` field; `suggest` copies it into `Diagnostics`. - Modify: `src/bin/alpha_id.rs:18,83-125` — `--pack` flag on `Index` and its handler branch. - Test: `tests/packed_tests.rs` — format round-trip, `corpus_sha256`, bad-magic rejection. - Test: `tests/packed_load_tests.rs` — `IndexStatus` default/serde, packed-beats-store, structural `n_rows` mismatch. - Test: `tests/pack_cli_tests.rs` — `--pack` refuses an incomplete store (exit 2) and writes a file from a complete store. --- ### Task 1: `src/packed.rs` — packed file format + corpus hash **Files:** - Create: `src/packed.rs` - Modify: `src/lib.rs:13` - Test: `tests/packed_tests.rs` - [ ] **Step 1: Write the failing tests** Create `tests/packed_tests.rs`: ```rust use alpha_id::packed::{corpus_sha256, read_packed, write_packed, PackedHeader}; use std::io::Write; fn header(n_rows: u32, dim: u32) -> PackedHeader { PackedHeader { n_rows, dim, embed_model: "test/model".to_string(), corpus_sha256: "deadbeef".to_string(), corpus_name: "corpus.txt".to_string(), } } #[test] fn write_read_roundtrip_preserves_header_and_rows() { let dir = tempfile::tempdir().unwrap(); let path = dir.path().join("packed.bin"); let rows = vec![vec![0.0f32, 1.5, -2.25], vec![3.0, 4.0, 5.0]]; let h = header(2, 3); write_packed(&path, &h, &rows).unwrap(); let (got_h, got_rows) = read_packed(&path).unwrap(); assert_eq!(got_h, h); assert_eq!(got_rows, rows); } #[test] fn corpus_sha256_stable_and_sensitive() { let dir = tempfile::tempdir().unwrap(); let p = dir.path().join("c.txt"); std::fs::write(&p, b"alpha|beta").unwrap(); let a = corpus_sha256(p.to_str().unwrap()).unwrap(); let b = corpus_sha256(p.to_str().unwrap()).unwrap(); assert_eq!(a, b, "same bytes hash identically"); std::fs::write(&p, b"alpha|betb").unwrap(); let c = corpus_sha256(p.to_str().unwrap()).unwrap(); assert_ne!(a, c, "one changed byte changes the hash"); } #[test] fn read_packed_rejects_bad_magic() { let dir = tempfile::tempdir().unwrap(); let path = dir.path().join("bad.bin"); let mut f = std::fs::File::create(&path).unwrap(); f.write_all(b"XXXX____________").unwrap(); assert!(read_packed(&path).is_err()); } ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cargo test --test packed_tests` Expected: FAIL — compile error, `unresolved import alpha_id::packed` (the module does not exist yet). - [ ] **Step 3: Create `src/packed.rs`** ```rust use crate::model::AppError; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::fs; use std::path::{Path, PathBuf}; const MAGIC: &[u8; 4] = b"AIPK"; pub const PACKED_FORMAT_VERSION: u32 = 1; /// Manifest written into the packed file header. `embed_model` and /// `corpus_sha256` bind the matrix to the corpus snapshot and model it was /// built from; iteration 1 records them, iteration 2 enforces them. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct PackedHeader { pub n_rows: u32, pub dim: u32, pub embed_model: String, pub corpus_sha256: String, pub corpus_name: String, } /// Derived packed-file path `/packed..bin`, using the /// same `/`→`_` model sanitization `EmbeddingStore::open` applies, so `--pack` /// and `load` compute the identical path from the same config. pub fn packed_path(index_dir: &str, model: &str) -> PathBuf { let safe = model.replace('/', "_"); PathBuf::from(index_dir).join(format!("packed.{safe}.bin")) } /// SHA-256 of the raw bytes of the corpus file at `path`, lower-case hex. pub fn corpus_sha256(path: &str) -> Result { let bytes = fs::read(path).map_err(|e| AppError::Io(format!("{path}: {e}")))?; let mut h = Sha256::new(); h.update(&bytes); Ok(h.finalize().iter().map(|x| format!("{:02x}", x)).collect()) } /// Write the packed matrix: magic + format_version + header_len + JSON header /// + pad-to-4 + row-major little-endian f32 payload. Atomic via temp+rename. pub fn write_packed(path: &Path, header: &PackedHeader, rows: &[Vec]) -> Result<(), AppError> { let json = serde_json::to_vec(header) .map_err(|e| AppError::Parse(format!("packed header: {e}")))?; let mut out = Vec::new(); out.extend_from_slice(MAGIC); out.extend_from_slice(&PACKED_FORMAT_VERSION.to_le_bytes()); out.extend_from_slice(&(json.len() as u32).to_le_bytes()); out.extend_from_slice(&json); while out.len() % 4 != 0 { out.push(0); } for row in rows { for f in row { out.extend_from_slice(&f.to_le_bytes()); } } let tmp = path.with_extension("tmp"); fs::write(&tmp, &out).map_err(|e| AppError::Io(format!("{}: {e}", tmp.display())))?; fs::rename(&tmp, path).map_err(|e| AppError::Io(format!("{}: {e}", path.display())))?; Ok(()) } /// Read a packed matrix back into its header and row vectors. Fails on a /// wrong magic, an unsupported (newer) format version, a malformed or /// truncated header, or a payload length inconsistent with `n_rows × dim`. pub fn read_packed(path: &Path) -> Result<(PackedHeader, Vec>), AppError> { let bytes = fs::read(path).map_err(|e| AppError::Io(format!("{}: {e}", path.display())))?; if bytes.len() < 12 || &bytes[0..4] != MAGIC { return Err(AppError::Parse(format!("{}: bad magic", path.display()))); } let version = u32::from_le_bytes([bytes[4], bytes[5], bytes[6], bytes[7]]); if version > PACKED_FORMAT_VERSION { return Err(AppError::Parse(format!( "{}: unsupported packed format version {version}", path.display() ))); } let header_len = u32::from_le_bytes([bytes[8], bytes[9], bytes[10], bytes[11]]) as usize; let header_end = 12 + header_len; if bytes.len() < header_end { return Err(AppError::Parse(format!("{}: truncated header", path.display()))); } let header: PackedHeader = serde_json::from_slice(&bytes[12..header_end]) .map_err(|e| AppError::Parse(format!("{}: header: {e}", path.display())))?; let mut payload_start = header_end; while payload_start % 4 != 0 { payload_start += 1; } let n = header.n_rows as usize; let dim = header.dim as usize; let expected = n * dim * 4; if bytes.len() - payload_start != expected { return Err(AppError::Parse(format!( "{}: payload length {} != n_rows×dim×4 {}", path.display(), bytes.len() - payload_start, expected ))); } let mut rows = Vec::with_capacity(n); let mut off = payload_start; for _ in 0..n { let mut row = Vec::with_capacity(dim); for _ in 0..dim { row.push(f32::from_le_bytes([ bytes[off], bytes[off + 1], bytes[off + 2], bytes[off + 3], ])); off += 4; } rows.push(row); } Ok((header, rows)) } ``` - [ ] **Step 4: Register the module in `src/lib.rs`** `src/lib.rs:13` currently ends the module list with `pub mod pipeline;` (line 13) and `pub mod eval;` (line 14). Add one line so the list reads: ```rust pub mod packed; pub mod pipeline; pub mod eval; ``` - [ ] **Step 5: Run tests to verify they pass** Run: `cargo test --test packed_tests` Expected: PASS — 3 passed. --- ### Task 2: `IndexStatus` + packed-first load path + diagnostics plumbing **Files:** - Modify: `src/model.rs:80-81,114-121` - Modify: `src/pipeline.rs:15-23,26-60,229-235` - Test: `tests/packed_load_tests.rs` - [ ] **Step 1: Write the failing tests** Create `tests/packed_load_tests.rs`: ```rust use alpha_id::model::{AlphaIdEntry, Config, IndexStatus}; use alpha_id::packed::{packed_path, write_packed, PackedHeader}; use alpha_id::pipeline::load_packed_or_store; fn entry(text: &str) -> AlphaIdEntry { AlphaIdEntry { alpha_id: "X".to_string(), valid: true, icd_primary: "A00".to_string(), icd_star: String::new(), icd_addon: String::new(), icd_primary2: String::new(), orpha: String::new(), text: text.to_string(), } } fn cfg_in(dir: &std::path::Path) -> Config { let mut cfg = Config::load("config/default.toml").unwrap(); cfg.index_dir = dir.to_str().unwrap().to_string(); cfg.embed_model = "test/model".to_string(); cfg } #[test] fn index_status_defaults_to_absent() { assert_eq!(IndexStatus::default(), IndexStatus::Absent); } #[test] fn index_status_serializes_externally_tagged() { let j = serde_json::to_string(&IndexStatus::Mismatch("x".to_string())).unwrap(); assert!(j.contains("Mismatch"), "got {j}"); } #[test] fn packed_beats_store_builds_without_store() { let dir = tempfile::tempdir().unwrap(); let cfg = cfg_in(dir.path()); let entries = vec![entry("a"), entry("b")]; let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]]; let h = PackedHeader { n_rows: 2, dim: 2, embed_model: cfg.embed_model.clone(), corpus_sha256: "irrelevant-in-iter1".to_string(), corpus_name: "c.txt".to_string(), }; write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap(); // No per-file store populated in `dir`; the index must come from the packed file. let (vector, status) = load_packed_or_store(&cfg, &entries); assert!(vector.is_some()); assert_eq!(status, IndexStatus::Ok); } #[test] fn n_rows_mismatch_yields_mismatch_none() { let dir = tempfile::tempdir().unwrap(); let cfg = cfg_in(dir.path()); let entries = vec![entry("a"), entry("b"), entry("c")]; // 3 entries let rows = vec![vec![1.0f32, 0.0], vec![0.0f32, 1.0]]; // packed claims 2 rows let h = PackedHeader { n_rows: 2, dim: 2, embed_model: cfg.embed_model.clone(), corpus_sha256: "x".to_string(), corpus_name: "c.txt".to_string(), }; write_packed(&packed_path(&cfg.index_dir, &cfg.embed_model), &h, &rows).unwrap(); let (vector, status) = load_packed_or_store(&cfg, &entries); assert!(vector.is_none()); assert!(matches!(status, IndexStatus::Mismatch(_))); } ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cargo test --test packed_load_tests` Expected: FAIL — compile error: `IndexStatus` not found in `alpha_id::model` and `load_packed_or_store` not found in `alpha_id::pipeline`. - [ ] **Step 3: Add `IndexStatus` and the `Diagnostics` field in `src/model.rs`** Insert the enum immediately after the `Mode` enum (`src/model.rs:80-81`): ```rust #[derive(Debug, Clone, PartialEq, Eq, Serialize, Default)] pub enum IndexStatus { Ok, // packed file present and structurally valid #[default] Absent, // no usable index — intentional lexical-only Mismatch(String), // packed file present but structurally invalid; reason carried } ``` (`Serialize` is already imported at `src/model.rs:1`.) Add the field to `Diagnostics` (`src/model.rs:114-121`), after `millis`: ```rust #[derive(Debug, Clone, Serialize, Default)] pub struct Diagnostics { pub mode: String, pub degraded: bool, pub segment_count: usize, pub ionos_calls: usize, pub millis: u128, pub index_status: IndexStatus, } ``` - [ ] **Step 4: Add `load_packed_or_store` and thread it through `Pipeline` in `src/pipeline.rs`** Add `use crate::packed;` to the imports (after `use crate::model::*;` at `src/pipeline.rs:7`; `IndexStatus` arrives via the `model::*` glob). Add the free fn (place it above `impl Pipeline`, after the struct at `src/pipeline.rs:23`): ```rust /// Build the semantic index, preferring the packed matrix file and falling /// back to the per-file store. Returns the index (`None` when none is usable) /// and an `IndexStatus` describing the path taken. /// /// Iteration-1 scope: the packed file is validated *structurally* only /// (`read_packed` checks magic/version/payload length; here we additionally /// require `n_rows == entries.len()`). The header's `corpus_sha256` and /// `embed_model` are NOT yet compared against the config — that enforcement /// is iteration 2. pub fn load_packed_or_store(cfg: &Config, entries: &[AlphaIdEntry]) -> (Option, IndexStatus) { let packed = packed::packed_path(&cfg.index_dir, &cfg.embed_model); if packed.exists() { match packed::read_packed(&packed) { Ok((header, rows)) => { if header.n_rows as usize != entries.len() { return ( None, IndexStatus::Mismatch(format!( "packed n_rows {} != corpus entries {}", header.n_rows, entries.len() )), ); } return (Some(VectorIndex::from_vectors(rows)), IndexStatus::Ok); } Err(e) => return (None, IndexStatus::Mismatch(format!("{e}"))), } } // Fallback: per-file store, all-or-nothing (today's behaviour). match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) { Ok(store) if entries.iter().all(|e| store.has(&e.text)) => { let vecs: Option>> = entries.iter().map(|e| store.get(&e.text)).collect(); match vecs { Some(v) => (Some(VectorIndex::from_vectors(v)), IndexStatus::Ok), None => (None, IndexStatus::Absent), } } _ => (None, IndexStatus::Absent), } } ``` Add the field to the `Pipeline` struct (`src/pipeline.rs:15-23`), after `vector: Option`: ```rust vector: Option, index_status: IndexStatus, } ``` Replace the `let vector = match ... { ... };` block (`src/pipeline.rs:43-50`) with: ```rust let (vector, index_status) = load_packed_or_store(cfg, &entries); ``` Add `index_status` to the `Self { ... }` constructor (`src/pipeline.rs:51-59`), after `vector,`: ```rust vector, index_status, }) ``` In `suggest`, add the field to the `Diagnostics { ... }` literal (`src/pipeline.rs:229-235`), after `millis: t0.elapsed().as_millis(),`: ```rust millis: t0.elapsed().as_millis(), index_status: self.index_status.clone(), ``` - [ ] **Step 5: Run the new tests to verify they pass** Run: `cargo test --test packed_load_tests` Expected: PASS — 4 passed. - [ ] **Step 6: Run the full suite to verify no regression** Run: `cargo test` Expected: PASS — the whole suite is green (the new `Diagnostics` field has a `Default`, so existing diagnostics-reading tests in `tests/pipeline_lexical_tests.rs`, `tests/pipeline_hybrid_tests.rs`, and `tests/cli_tests.rs` are unaffected). --- ### Task 3: `alpha-id index --pack` subcommand **Files:** - Modify: `src/bin/alpha_id.rs:18,83-125` - Test: `tests/pack_cli_tests.rs` - [ ] **Step 1: Write the failing tests** Create `tests/pack_cli_tests.rs`: ```rust use std::path::Path; use std::process::Command; // Write a minimal config TOML pointing the corpus + index dir at temp paths. // The `--pack` branch only reads `alpha_id_path` (corpus) and `index_dir` // (store); `claml_path` etc. are present for parse but unused on this path. fn write_config(dir: &Path, corpus: &Path, index_dir: &Path) -> std::path::PathBuf { let toml = format!( r#"ionos_base_url = "http://127.0.0.1:1" token_path = "/nonexistent" embed_model = "test/model" rerank_model = "test/rerank" alpha_id_path = "{corpus}" claml_path = "/nonexistent.xml" index_dir = "{index}" pool_size = 10 top_k = 10 "#, corpus = corpus.to_str().unwrap(), index = index_dir.to_str().unwrap(), ); let p = dir.join("config.toml"); std::fs::write(&p, toml).unwrap(); p } fn corpus_line(text: &str) -> String { // Alpha-ID format: valid|alpha_id|primary|star|addon|primary2|orpha|text format!("1|A0001|A00|||||{text}") } #[test] fn pack_refuses_incomplete_store_exit_2() { let dir = tempfile::tempdir().unwrap(); let corpus = dir.path().join("corpus.txt"); std::fs::write(&corpus, format!("{}\n{}\n", corpus_line("alpha"), corpus_line("beta"))).unwrap(); let index_dir = dir.path().join("index"); std::fs::create_dir_all(&index_dir).unwrap(); let config = write_config(dir.path(), &corpus, &index_dir); let out = Command::new(env!("CARGO_BIN_EXE_alpha-id")) .args(["--config", config.to_str().unwrap(), "index", "--pack"]) .output() .unwrap(); assert_eq!(out.status.code(), Some(2), "incomplete store must exit 2"); let stderr = String::from_utf8_lossy(&out.stderr); assert!(stderr.contains("store incomplete"), "stderr was: {stderr}"); } #[test] fn pack_writes_file_from_complete_store() { let dir = tempfile::tempdir().unwrap(); let corpus = dir.path().join("corpus.txt"); std::fs::write(&corpus, format!("{}\n{}\n", corpus_line("alpha"), corpus_line("beta"))).unwrap(); let index_dir = dir.path().join("index"); std::fs::create_dir_all(&index_dir).unwrap(); // Populate the per-file store for both entry texts so the pack succeeds. let store = alpha_id::embed::EmbeddingStore::open(index_dir.to_str().unwrap(), "test/model").unwrap(); store.put("alpha", &[1.0f32, 0.0]).unwrap(); store.put("beta", &[0.0f32, 1.0]).unwrap(); let config = write_config(dir.path(), &corpus, &index_dir); let out = Command::new(env!("CARGO_BIN_EXE_alpha-id")) .args(["--config", config.to_str().unwrap(), "index", "--pack"]) .output() .unwrap(); assert!(out.status.success(), "stderr: {}", String::from_utf8_lossy(&out.stderr)); let packed = index_dir.join("packed.test_model.bin"); assert!(packed.exists(), "packed file should be written at {}", packed.display()); } ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cargo test --test pack_cli_tests` Expected: FAIL — both tests fail. clap rejects the unknown `--pack` flag and prints `error: unexpected argument '--pack' found` to stderr, so `pack_writes_file_from_complete_store` fails on `out.status.success()` (clap exits non-zero) and `pack_refuses_incomplete_store_exit_2` fails on the `stderr.contains("store incomplete")` assertion (the stderr is clap's usage error, not our message). Note: clap's own usage-error exit code happens to be 2, so the exit-code assertion alone is not what reddens the first test — the `store incomplete` substring is. - [ ] **Step 3: Add the `--pack` flag to the `Index` subcommand** `src/bin/alpha_id.rs:18` currently reads: ```rust Index { #[arg(long)] sample: Option, #[arg(long)] full: bool, #[arg(long)] confirm: bool }, ``` Replace it with: ```rust Index { #[arg(long)] sample: Option, #[arg(long)] full: bool, #[arg(long)] confirm: bool, #[arg(long)] pack: bool }, ``` - [ ] **Step 4: Add the `--pack` branch in the `Index` handler** The handler arm starts at `src/bin/alpha_id.rs:83`: `Cmd::Index { sample, full, confirm } => {`. Update the pattern to bind `pack` and insert the pack branch right after the two `use` lines (`src/bin/alpha_id.rs:84-85`) and before `let p = Pipeline::load(&cfg)`: ```rust Cmd::Index { sample, full, confirm, pack } => { use alpha_id::embed::{EmbeddingStore, build_corpus_embeddings, estimate}; use alpha_id::ionos::IonosClient; if pack { use alpha_id::packed::{corpus_sha256, packed_path, write_packed, PackedHeader}; let entries = alpha_id::corpus::load(&cfg.alpha_id_path).expect("corpus"); let store = EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model).expect("store"); let cached = entries.iter().filter(|e| store.has(&e.text)).count(); if cached != entries.len() { eprintln!( "error: store incomplete: {} of {} entries embedded; \ run `index --full --confirm` first", cached, entries.len() ); std::process::exit(2); } let rows: Vec> = entries.iter().map(|e| store.get(&e.text).expect("cached vector")).collect(); let dim = rows.first().map(|r| r.len()).unwrap_or(0) as u32; let corpus_name = std::path::Path::new(&cfg.alpha_id_path) .file_name().and_then(|n| n.to_str()) .unwrap_or(&cfg.alpha_id_path).to_string(); let header = PackedHeader { n_rows: entries.len() as u32, dim, embed_model: cfg.embed_model.clone(), corpus_sha256: corpus_sha256(&cfg.alpha_id_path).expect("corpus hash"), corpus_name, }; let path = packed_path(&cfg.index_dir, &cfg.embed_model); write_packed(&path, &header, &rows).expect("write packed"); eprintln!("packed {} rows × {} dims → {}", header.n_rows, header.dim, path.display()); return; } let p = Pipeline::load(&cfg).expect("load"); ``` (The remainder of the arm — the `sample` and `full` branches — is unchanged.) - [ ] **Step 5: Run the new tests to verify they pass** Run: `cargo test --test pack_cli_tests` Expected: PASS — 2 passed. - [ ] **Step 6: Run the full suite to verify no regression** Run: `cargo test` Expected: PASS — the whole suite is green.