Cycle-close tidy for the packed-versioned-index-bundle cycle (issues #3 + #4,
both shipped and closed). Architect drift review over d83b10f..HEAD against the
spec + glossary (no paths.design_ledger / project CLAUDE.md configured, so spec
+ glossary are the only references; the regression gate is empty —
commands.regression: [] — so architect was the sole cycle-close gate).
Architect verdict: drift_found, minor — no spec contract violated. Format,
header schema, on-disk layout, and the load_packed_or_store validation order
(n_rows → model → corpus hash) match the spec exactly; both #3 and #4
acceptance criteria are test-evidenced; no glossary term drift; the
IndexStatus / degraded orthogonality is honored.
Resolutions:
- [medium] Missing positional-alignment test. The spec § Testing strategy
promised a "row i == store.get(entries[i].text)" guard, and it had not
shipped — the load-bearing invariant that licenses dropping the per-row
SHA256 key (row i = corpus entry i) was unprotected. FIXED: added
tests/pack_cli_tests.rs::cli_packed_rows_are_in_corpus_order, which packs a
store with order-revealing orthogonal vectors via the CLI, reads the packed
file back, and asserts the rows follow corpus-load order (swapped order
fails the test).
- [low] Stale IndexStatus doc comments still said "structurally valid/invalid"
though the variants now also carry semantic model/corpus mismatch. FIXED:
refreshed the Ok / Mismatch comments in src/model.rs.
- [low] The `index --pack` writer logic lives inline in main rather than in the
packed module, leaving its header construction outside library test reach.
CARRY-ON: filed as backlog issue #5 (idea/debt) — behaviour is correct and
CLI-test-covered; the refactor is a self-contained testability improvement,
not forced into this tidy.
- [note] No design ledger / project CLAUDE.md configured; drift is measurable
only against spec + glossary. Recorded as an infrastructure observation.
Regression: no scripts configured (no-op). Full suite green (68 tests, +1 for
the alignment guard). cycle packed-versioned-bundle tidy (drift-clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Iteration 1 shipped the packed format with the corpus_sha256 + embed_model
manifest in its header but only validated structure at load. A corpus release
or an embed_model change could still ship an index that loads "successfully"
yet belongs to a different corpus/model, and the only signal was the anonymous
degraded:true — indistinguishable from a transient IONOS outage. This closes
that gap.
load_packed_or_store now, after the structural checks pass, compares the
header's embed_model against cfg.embed_model and its corpus_sha256 against a
fresh hash of cfg.alpha_id_path. On either disagreement it returns
IndexStatus::Mismatch("embed model" | "corpus sha256") with no vector index,
so Mode Hybrid degrades to Lexical rather than serving a matrix that does not
belong to the deployed corpus/model. The check is ordered model-then-corpus so
a model mismatch never reads the corpus file.
The mismatch is now observable two ways, not just via degraded:
- Pipeline::load emits a prominent stderr warning naming the packed path and
the reason when the index is a Mismatch.
- IndexStatus gains a Display (ok / absent / mismatch: <reason>) and the human
diagnostics line carries an index=<status> token, e.g.
"[3 segments, mode Lexical, 12 ms, degraded=false, index=mismatch: corpus sha256]".
The index_status also rides in the JSON diagnostics (it was already a
serialized Diagnostics field from iteration 1). index_status is orthogonal to
degraded: a transient IONOS degrade leaves index_status=Ok, a structural
binding problem sets Mismatch.
Tests (67 green, +6 net): hermetic unit tests for the model- and
corpus-hash-mismatch rejections and a matching-pair acceptance, a Display
render test, a lib test that a Mode Hybrid suggest under a mismatch is
degraded with index_status=Mismatch (degrading before any IONOS post, so no
network), an end-to-end CLI test asserting the load warning and the
index=mismatch token reach stderr, and the existing transient-degrade test
extended with the index_status=Ok foil. The iteration-1
packed_beats_store_builds_without_store test was repaired (hermetic temp corpus
+ a real header hash) since the new semantic enforcement necessarily rejects
its previous structural-only-era placeholder hash; its intent (packed beats
store) is unchanged.
No IONOS calls are made by any test. The per-file store, embedding build, and
the VectorIndex are untouched. mmap remains rejected (spec § Out of scope).
Spec: docs/specs/2026-05-31-packed-versioned-index-bundle.md
Plan: docs/plans/2026-06-01-packed-versioned-bundle-iter2.md
closes#4
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The semantic index was ~90 896 per-entry files (index/embeddings/<safe-model>/
<sha256>.f32), a deployment liability: the file count — not the 356 MB — is
hostile to container layers, object storage, and Git-LFS, and Pipeline::load
opened every one individually. This packs a complete per-file store into one
self-describing matrix file loaded in a single read.
New src/packed.rs owns the format: 4-byte magic "AIPK", u32 LE format_version,
u32 LE header length, a JSON PackedHeader {n_rows, dim, embed_model,
corpus_sha256, corpus_name}, pad to a 4-byte boundary, then row-major
little-endian f32 payload in corpus order (row i = corpus entry i). write_packed
is atomic (temp+rename); read_packed rejects a wrong magic, a newer-than-
supported version, a malformed header, or a payload length inconsistent with
n_rows×dim. corpus_sha256 hashes the raw corpus file bytes.
Pipeline::load now calls a free fn load_packed_or_store(cfg, entries) that
prefers the packed file and falls back to today's all-or-nothing per-file store,
reporting which path it took via a new IndexStatus {Ok, Absent, Mismatch}
carried into Diagnostics. `alpha-id index --pack` writes the file from a
complete store (refusing an incomplete one with exit 2), recording corpus_sha256
+ embed_model in the header as the artifact-binding data.
Iteration boundary (deliberate): this is iteration 1 of the packed-versioned-
bundle cycle. load_packed_or_store validates the packed file STRUCTURALLY only
(magic/version/payload length via read_packed, plus n_rows == entries.len());
it does NOT yet compare the header's corpus_sha256/embed_model against the live
config, and IndexStatus::Mismatch fires only on structural corruption. The
load-time mismatch warning, the index= token on the diagnostics line, and the
Hybrid-degrade-names-reason wiring are iteration 2 (issue #4), which consumes
the binding data this commit writes. The per-file store is untouched — it stays
the build/resume path; the packed file is an additive serve-path artifact.
mmap was considered and rejected: the brute-force full-scan cosine search
touches every row per query, so memory-mapping yields no runtime benefit over a
single read and would add an unsafe dependency (spec § Out of scope).
Verification: cargo test green (61 tests; 9 new — 3 format round-trip/hash/bad-
magic, 4 load-path incl. packed-beats-store and structural n_rows mismatch, 2
--pack CLI incl. exit-2 refusal). An end-to-end seam test writes the packed
file via the CLI, removes the per-file store, and asserts the pipeline builds
its index from the packed file alone as IndexStatus::Ok (#3 acceptance b).
Spec: docs/specs/2026-05-31-packed-versioned-index-bundle.md
Plan: docs/plans/2026-06-01-packed-versioned-bundle-iter1.md
closes#3
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The retrieval-mode enum used opaque single letters `Mode { C, A }`;
neither the identifier nor the operator-facing `--mode a` token
conveyed what the mode does. Rename to self-describing names that map
directly onto the glossary definitions:
Mode::C -> Mode::Lexical (BM25 + tag filter, offline, no IONOS)
Mode::A -> Mode::Hybrid (lexical u semantic -> RRF -> rerank)
Decisions taken (the issue's open questions, resolved):
- CLI canonical tokens are now `lexical` / `hybrid`; the default is
`lexical`. The legacy `c` / `a` tokens stay accepted as
case-insensitive input aliases (zero cost; protects hand-typed
invocations). A unit test pins the alias contract.
- The diagnostics `mode` string (Debug-derived) now emits
"Lexical"/"Hybrid". Verified safe: no downstream consumer parses it.
doctate is the future integrator and embeds the library (the `Mode`
enum), not the JSON string (design spec §103).
- The reserved, out-of-scope generative tier (spec's "Mode B") keeps
conceptual room: a future `Mode::Generative` does not collide with
the retrieval-strategy names Lexical/Hybrid.
Surface: enum + FromStr, CLI defaults and `eval --mode both`
expansion, the private `collect_mode_c`/`collect_mode_a` methods (now
`collect_lexical`/`collect_hybrid`) and their comments, the two
pipeline mode test files (renamed), the eval CLI test, the glossary
(old letter forms demoted to Avoid lists), and the spec CLI synopsis.
closes#2
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pins the property that the `--chapters` CLI token list normalizes
to the zero-padded chapter-tag form ("01".."22") that Filter::accepts
compares against, so "4" filters identically to "04", and any token
mapping to no known chapter is surfaced as unrecognised (for a stderr
warning) rather than silently collapsing the result set to empty.
RED: alpha_id::model::normalize_chapters does not exist yet — the
test fails to compile. The GREEN side follows in the next commit.
refs #1
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
split_segments now treats each non-empty trimmed line as one retrieval
unit; blank lines are skipped, not paragraph delimiters. Name/signature
retained (minimal-churn decision). Full suite 45/0. End-to-end: feeding
the GP dictation as extracted phrases recovers I10.90 (rank 4) which was
entirely missing in both modes under the paragraph contract, and largely
dissolves the cross-segment diabetes crowding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue`
spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage,
"Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue,
ß→ss; it is applied only in lexical.rs (both corpus index and query),
so the path is symmetric under either spelling. The embedding path
(raw text for bge-m3) is intentionally untouched.
eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the
realistic ASCII-digraph variant (ä→ae) so the eval harness exercises —
and guards against regression of — exactly this failure class.
TDD: failing tests first; normalize_tests 5/5, full suite 44/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix 1: capture <ValidModifierClass> codes from <ModifiedBy all="false">
and filter each modifier level to only valid codes before cartesian
product; eliminates ~194 phantom codes (e.g. M07.01–03) while keeping
valid ones (M07.00/04/07/09). Split Start/Empty XML event arms to
correctly track children of <ModifiedBy>.
- Fix 2: synthesized para295/para301 inherit parent values; "V" is
promoted to "P" (parent "V" = needs subdivision, terminal is codeable).
Other types ("P", "O", "Z", "") pass through unchanged.
- Fix 3: debug_assert in ModifierClass handler guards against ClaML
document-order assumption changing silently in debug/test builds.
- Fix 4: add expands_single_modifier_chain regression test verifying
C88.00/C88.01 synthesis from a single all="true" ModifiedBy (corpus-
verified: Alpha-ID I30531, I31044).
ICD-10-GM models the 4th/5th sub-digits of many groups (incl. all E10-E14
diabetes) via <Modifier>/<ModifierClass> referenced by ordered <ModifiedBy>
on the parent <Class>, not as nested <Class> elements. load() previously
returned nothing for codes like E11.90/E11.72/I10.91, so ~1.7k corpus
codes (incl. ~1.2k billable) fell back to the non-billable 3-digit root
and were silently dropped under --billable-only.
Collect per-Class ModifiedBy refs and all ModifierClass defs during the
existing single stream, then expand: cartesian product of the referenced
modifiers' ModifierClass codes in ModifiedBy order, concatenated verbatim
onto the parent code (E11 + .9 + 0 = E11.90), matching the Alpha-ID
corpus format exactly. Terminal leaves are billable (Para295/Para301=P);
description = parent label + each ModifierClass label joined by ' - ';
chapter/group/exotic/ifsg/content inherited from parent. Purely additive:
explicit <Class> entries always win. Generic over every modifier group.
Map size 12334 -> 17249 (+4915 synthesized).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implement Pipeline::load/suggest for Mode C: lexical search per segment,
cross-segment dedup via fusion, tag derivation, filter application.
Add meta_lookup fallback (exact → strip trailing 0 → 3-char root) to
bridge the ICD code granularity gap between the corpus (E11.90-style
6-char codes) and ClaML (which only stores E11, E11.0-style entries).
Brummel