79de4e140491ddc512c9e66764c6f3d1c8312e2e
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
79de4e1404 |
feat: enforce + surface packed index binding (closes #4)
Iteration 1 shipped the packed format with the corpus_sha256 + embed_model
manifest in its header but only validated structure at load. A corpus release
or an embed_model change could still ship an index that loads "successfully"
yet belongs to a different corpus/model, and the only signal was the anonymous
degraded:true — indistinguishable from a transient IONOS outage. This closes
that gap.
load_packed_or_store now, after the structural checks pass, compares the
header's embed_model against cfg.embed_model and its corpus_sha256 against a
fresh hash of cfg.alpha_id_path. On either disagreement it returns
IndexStatus::Mismatch("embed model" | "corpus sha256") with no vector index,
so Mode Hybrid degrades to Lexical rather than serving a matrix that does not
belong to the deployed corpus/model. The check is ordered model-then-corpus so
a model mismatch never reads the corpus file.
The mismatch is now observable two ways, not just via degraded:
- Pipeline::load emits a prominent stderr warning naming the packed path and
the reason when the index is a Mismatch.
- IndexStatus gains a Display (ok / absent / mismatch: <reason>) and the human
diagnostics line carries an index=<status> token, e.g.
"[3 segments, mode Lexical, 12 ms, degraded=false, index=mismatch: corpus sha256]".
The index_status also rides in the JSON diagnostics (it was already a
serialized Diagnostics field from iteration 1). index_status is orthogonal to
degraded: a transient IONOS degrade leaves index_status=Ok, a structural
binding problem sets Mismatch.
Tests (67 green, +6 net): hermetic unit tests for the model- and
corpus-hash-mismatch rejections and a matching-pair acceptance, a Display
render test, a lib test that a Mode Hybrid suggest under a mismatch is
degraded with index_status=Mismatch (degrading before any IONOS post, so no
network), an end-to-end CLI test asserting the load warning and the
index=mismatch token reach stderr, and the existing transient-degrade test
extended with the index_status=Ok foil. The iteration-1
packed_beats_store_builds_without_store test was repaired (hermetic temp corpus
+ a real header hash) since the new semantic enforcement necessarily rejects
its previous structural-only-era placeholder hash; its intent (packed beats
store) is unchanged.
No IONOS calls are made by any test. The per-file store, embedding build, and
the VectorIndex are untouched. mmap remains rejected (spec § Out of scope).
Spec: docs/specs/2026-05-31-packed-versioned-index-bundle.md
Plan: docs/plans/2026-06-01-packed-versioned-bundle-iter2.md
closes #4
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Brummel
|
43b1b6fc25 |
feat: packed single-file embedding index (closes #3)
The semantic index was ~90 896 per-entry files (index/embeddings/<safe-model>/
<sha256>.f32), a deployment liability: the file count — not the 356 MB — is
hostile to container layers, object storage, and Git-LFS, and Pipeline::load
opened every one individually. This packs a complete per-file store into one
self-describing matrix file loaded in a single read.
New src/packed.rs owns the format: 4-byte magic "AIPK", u32 LE format_version,
u32 LE header length, a JSON PackedHeader {n_rows, dim, embed_model,
corpus_sha256, corpus_name}, pad to a 4-byte boundary, then row-major
little-endian f32 payload in corpus order (row i = corpus entry i). write_packed
is atomic (temp+rename); read_packed rejects a wrong magic, a newer-than-
supported version, a malformed header, or a payload length inconsistent with
n_rows×dim. corpus_sha256 hashes the raw corpus file bytes.
Pipeline::load now calls a free fn load_packed_or_store(cfg, entries) that
prefers the packed file and falls back to today's all-or-nothing per-file store,
reporting which path it took via a new IndexStatus {Ok, Absent, Mismatch}
carried into Diagnostics. `alpha-id index --pack` writes the file from a
complete store (refusing an incomplete one with exit 2), recording corpus_sha256
+ embed_model in the header as the artifact-binding data.
Iteration boundary (deliberate): this is iteration 1 of the packed-versioned-
bundle cycle. load_packed_or_store validates the packed file STRUCTURALLY only
(magic/version/payload length via read_packed, plus n_rows == entries.len());
it does NOT yet compare the header's corpus_sha256/embed_model against the live
config, and IndexStatus::Mismatch fires only on structural corruption. The
load-time mismatch warning, the index= token on the diagnostics line, and the
Hybrid-degrade-names-reason wiring are iteration 2 (issue #4), which consumes
the binding data this commit writes. The per-file store is untouched — it stays
the build/resume path; the packed file is an additive serve-path artifact.
mmap was considered and rejected: the brute-force full-scan cosine search
touches every row per query, so memory-mapping yields no runtime benefit over a
single read and would add an unsafe dependency (spec § Out of scope).
Verification: cargo test green (61 tests; 9 new — 3 format round-trip/hash/bad-
magic, 4 load-path incl. packed-beats-store and structural n_rows mismatch, 2
--pack CLI incl. exit-2 refusal). An end-to-end seam test writes the packed
file via the CLI, removes the per-file store, and asserts the pipeline builds
its index from the packed file alone as IndexStatus::Ok (#3 acceptance b).
Spec: docs/specs/2026-05-31-packed-versioned-index-bundle.md
Plan: docs/plans/2026-06-01-packed-versioned-bundle-iter1.md
closes #3
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|