Identity-ref resolution scans and re-loads the whole blueprint store per reference #191
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
aura_registry::find_blueprint_by_identity (the referential tier's identity-ref arm, cycle 0106/#189) resolves an identity id by reading EVERY blueprints/.json, loading it through blueprint_from_json, computing blueprint_identity_json, and hashing — O(store) with a full parse+serialize per entry, per identity reference, per validation call. Fine at current store sizes; grows linearly with research history.
Likely shape when it matters: an identity-id → content-id sidecar index maintained at put_blueprint time (or lazily memoized), so identity resolution is a lookup; invalidation is trivial since stored blueprints are immutable (content-addressed).
Design decisions (specify, autonomous run)
Derived decisions for this issue's spec; rationale inline. Reference: the fix replaces the O(store) scan in
find_blueprint_by_identitywith an identity-id → content-id index.blueprint_identity_index.jsonl, a sibling of the runs/families stores.Basis: derived — the growth term is the rescan per CLI invocation; an in-memory memo would leave every invocation O(store). Append-only JSONL is the established registry store pattern, and the registry-internal append lock (#276) already covers a new append path. No additional in-memory layer: with the per-blueprint roster-parse gone, a per-lookup read of the small index file is negligible, and file-mediated coherence stays correct across processes.
put_blueprintappends the mapping itself (no signature change: the identity id is a pure function of the stored canonical bytes, computed roster-free by a new doc-level helper in the engine's blueprint serde — parse doc, strip debug symbols, canonical-serialize, hash). A lookup that falls back to the scan performs a full-store repair pass, so a legacy store amortizes its entire backfill into one scan.Basis: derived — put-time keeps fresh entries O(1) forever; lazy full-repair covers stores written before the index existed without a migration step.
Basis: derived — resolution results must be byte-identical to the scan under roster drift, manual store surgery, or index corruption. The index is a pure cache; a blueprint the current roster cannot load stays unmatched exactly as the scan skips it today.
Basis: derived — write contexts propagate, read contexts stay reads semantically.
read_diryields first — an unspecified order. The index keeps one mapping per identity id (first seen); any candidate is equally valid, so the semantics class is unchanged.Status: design settled — ready for spec production.
Design revision after grounding-check (specify, autonomous run)
The grounding-check on the first spec draft returned BLOCK: the draft's put-time index maintenance relied on a roster-free doc-level identity function whose equivalence to the loaded-composite path is ratified by no green test — and the draft's stated basis for it was wrong. The composite load path canonicalizes structurally (sorting during doc construction), while a raw serde re-parse does not, so the equivalence would hold only for already-canonical stored bytes: an unstated precondition. Failure would be silent — correctness survives via verify-on-hit, but index keys would never match query keys and the speedup would collapse to a scan per lookup.
Resolution — the unratified mechanism is removed rather than test-backed:
put_blueprintstays untouched. The lookup's scan fallback becomes a full-store repair pass; every identity id the index ever stores is computed by the same roster-mediated path today's scan uses — no second identity-computation path exists.Basis: derived — the repair path is the load-bearing mechanism either way (a broken put-time key would be healed by it), so put-time maintenance was an optimization of an optimization, purchasable only with a new public engine API plus an unratified round-trip property. Cost of dropping it: a freshly stored blueprint costs exactly one scan at the next identity miss (today every lookup is that scan); absent-identity lookups must scan in any design, since only the scan proves absence.
Basis: derived — append-only store + healing requires later-wins semantics.
blueprint_identity_index.jsonl), verify-on-hit as the trust model, best-effort swallowed repair appends, index reads that never fail the lookup.Status: revised design settled — spec re-entering the grounding gate.
Spec auto-signed (autonomous run)
The revised spec (blueprint identity-id index: persistent JSONL sidecar, lazily repaired by the lookup's scan fallback, verify-on-hit, no write-path or engine changes) passed the grounding check: every load-bearing assumption about current codebase behaviour is ratified by a named, currently-green test. Under the autonomous-run contract that PASS is the signature — no human signed. The signed spec is a git-ignored working file; its durable design record is this issue's decision-log comments (issues/191#issuecomment-3724, issues/191#issuecomment-3726).
Status: spec signed — ready for implementation planning.
Cycle-close findings and fix decision (audit, autonomous run)
The architect drift review of the shipped index confirmed the load-bearing properties (results scan-identical via verify-on-hit; zero caller/write-path/engine changes; the registry append lock correctly extended to the fourth JSONL path) and surfaced two items:
Basis: derived — the walk's final mapping is the unique post-scan authoritative state, so diffing final-vs-snapshot is the only comparison that converges (per-entry comparison against a mid-walk-stale snapshot provably oscillates with twins); lookup answers (first read-dir match) and verify-on-hit are untouched, and the twin-corner index growth drops to zero once converged.