Aura/docs/design/contracts/c18-registry.history.md

C18 — Project management: one repo = one project, plus a run registry: history

FROZEN HISTORICAL RECORD. Each block below was true as of its cycle/date stamp and may be superseded; this file is NOT current truth and NOT a grounding surface. Current contract: c18-registry.md.

Realization (cycle 0029 — the flat run registry). The experiments-&-results plane shipped as aura-registry: an append-only JSONL store (runs/runs.jsonl), one serde_json RunReport (RunManifest{commit, params, window, seed, broker} + RunMetrics) per line, with a typed read-path (load) and best-first ranking (rank_by/optimize). C9 holds — the registry depends on aura-engine, never the reverse.

Realization (cycle 0045 — lineage as related records, #70). The lineage depth is now realized as a family store: a sweep / Monte-Carlo / walk-forward run (the C12 axes) is persisted as a set of related records — each a FamilyRunRecord (a RunReport stamped with its family + run + kind + ordinal) — in a sibling JSONL (families.jsonl), leaving the flat runs.jsonl path and its append/load/rank_by/optimize API byte-for-byte unchanged. the user-facing family_id = "{family}-{run}" handle is derived from the stored family name plus a per-name run index (assigned as a numeric max+1 — not a content hash; re-running the same family mints a fresh id). group_families is the round-trip that re-derives a family from the stored links (re-listable / rankable as a unit — C21). The manifest is the re-derivation recipe (#71): no input-stream blob / path / payload enters a record, and a member's window is producer-supplied via Source::bounds()/window_of (eager or streamed → byte-identical lineage), never a materialized-Vec scan at the call site. CLI surface: aura mc, aura runs families, aura runs family <id> [rank <metric>]; aura sweep / walkforward / mc persist via append_family with an optional --name.

Realization (cycle 0078 — cross-instrument family + instrument lineage, #146). The comparison axis (C12) is realized as a FamilyKind::CrossInstrument family: aura generalize runs one candidate across an instrument list and persists the M per-instrument runs via append_family, each member self-identifying through a new first-class RunManifest.instrument lineage field (serde-widened with skip_serializing_if, so legacy lines and every non-cross-instrument path stay byte-identical — C14/C23). The cross-instrument generalization score (worst-case R floor + sign-agreement + per-instrument breakdown) is a recomputable aggregate over those members, not a persisted family-level record — distinct from #144/#145's per-winner selection annotation on RunManifest.selection. Deferred (Non-goals): replay-dedup (content-addressed identity shipped — #158, cycle 0094, Realization below); the "run-diff" depth and ranking families against each other (cross-family, vs. within-family); and a live producer for the flat runs.jsonl standalone-run path — no CLI command writes it (sweep/walkforward persist to the family store; aura run does not persist). Resolved (#73, 2026-06): retired — aura runs list / rank dropped; families (C21) subsume standalone over-time comparison. The aura-registry flat lib API is retained: rank_by/optimize keep live consumers (optimize backs walk-forward's in-sample step, rank_by backs runs family … rank); append/ load (the flat-store half) remain public API with no in-tree caller after this retire — tested, available to external consumers, a latent dead-code surface a later sweep may revisit. Unknown-id contract (ratified, Runway fieldtest 2026-06). aura runs family <id> treats an unknown-but-well-formed id as an empty family (prints nothing, exit 0) — the same treat-as-empty discipline as Registry::load reading a missing store as Ok(empty), and deliberately distinct from the retired list/rank exit-2, which is argv-shape rejection before any store access, not a found-nothing lookup. Tightening to a non-zero no such family <id> exit (typo-safety) is an available future UX choice, not a current contract.

Refinement (2026-06-29 — a generated run's topology lives in / is addressed by its manifest, C24). The manifest identifies a run's topology today only via commit (this engine + a hand-coded harness) — sufficient while harnesses are a finite hand-coded menu. Once the World generates or structurally searches topologies (C21/C24), commit no longer identifies the graph, so the manifest must carry or content-address the topology-data to stay the re-derivation recipe (C18's "reproducible from a recorded manifest"). This pulls the previously-deferred content-addressed identity non-goal forward as the natural home for a generated topology's identity. The format and the carrier are C24's design; recorded here as the reproduction requirement it must meet.

Realization (2026-07-01, cycle 0094 — content-addressed reproduction of a generated run, #158). A blueprint sweep's topology is now content-addressed: the canonical blueprint_to_json bytes are stored once, keyed by the topology_hash the manifest already carries, in a dumb bytes-by-key store beside the run registry (runs/blueprints/<hash>.json — Registry::put_blueprint/get_blueprint, aura-registry; no sha2, no parse — the caller owns the hash, and reproduction's bit-identical compare is the integrity check). aura reproduce <family-id> re-derives every persisted member: load the member's blueprint by its topology_hash, reconstruct the sweep point from the recorded params, re-run through the same run_blueprint_member the live sweep uses (so bit-identity is by construction, C1), and compare metrics — refuse-don't-guess on an unknown id / missing stored blueprint (exit 1 — recorded state is missing, C14's runtime-failure class; this paragraph over-claimed exit 2 until #298 recorded the code's actual, C14-consistent behaviour), DIVERGED → exit 1. The id resolves first as the derived {family}-{run} handle; a bare enumeration name naming exactly one stored run resolves as fallback, an ambiguous name refuses listing the candidate handles (#298 — the list-then-reproduce seam). The manifest + the content-addressed store + the commit are the complete re-derivation recipe (C18). One blueprint is stored per family (all members share the signal topology_hash — C11/C12 dedup). aura graph introspect --content-id exposes the same id for an op-script, via the one content_id primitive topology_hash also uses (acc 1); a Tier-1 optional the blueprint does not use leaves the id byte-stable (acc 3, composing #156/#164). serde_json/float_roundtrip is enabled so stored f64 metrics round-trip exactly through families.jsonl — the precondition for a bit-identical compare (C1). Signal-only this cycle: the id covers the signal blueprint; the fixed r-sma scaffolding stays commit-identified (it is not yet blueprint-data, C24). Whole-harness / structural-axis content-addressing remains deferred. The debug-name-in-id question was settled additively (cycle 0104, #171): the identity id — the canonical form with every non-load-bearing debug symbol blanked (invariant 11), hashed through the same content_id primitive and surfaced as aura graph introspect --identity-id beside --content-id (combinable) — makes same-topology blueprints comparable across authoring paths, while the byte-exact topology_hash keeps every role untouched (introspection-only: no manifest field, no store key until a dedup consumer exists). Reproduction is proven on synthetic (deterministic) data; recorded-dataset reproduction rides the DataServer seam (#124). Cycle 0095 (#170): aura reproduce now also spans FamilyKind::MonteCarlo — branching on the family kind, it reconstructs each member's seed-driven synthetic walk from the recorded manifest.seed (the Sweep arm unchanged), so an aura mc family re-derives bit-identically through the same run_blueprint_member path. Cycle 0097 (#173): reproduce spans the third variant, FamilyKind::WalkForward — the same branch rebuilds each OOS member's windowed slice from manifest.window (winner params via the shared manifest→cells recovery), so an aura walkforward family re-derives bit-identically too. All three family kinds now persist and reproduce through the one shared topology_hash+put_blueprint hook.

Realization (cycle 0106, #189 — research-artifact document stores). The registry's content-addressed store family grew two siblings beside blueprints/: processes/ and campaigns/ hold the two research-artifact document types shipped by the #188 role-model pass — the process document (role 5: a named validation/eval methodology, a closed std stage vocabulary wrapping shipped primitives) and the campaign document (role 6b: persisted experiment intent — instruments × windows × strategy refs by content/identity id × param axes × process ref (content-id-only) × data-level presentation). Documents are canonical JSON (format_version envelope, omit-defaults, no trailing newline) keyed by the shared content-id primitive, now library-hosted (aura_research::content_id_of; aura-cli's content_id/topology_hash delegate byte-identically — the id goldens pin the move). Unlike put_blueprint (caller owns the hash), the document puts self-key from canonical bytes; gets are Ok(None) treat-as-empty. The referential tier (validate_campaign_refs) resolves process/strategy refs against the stores (identity refs by store scan in this cycle; index-first since #191, below) and checks each campaign axis — name AND declared ScalarKind, the axis carries its kind once — against the referenced blueprint's param_space. Campaign P1 control constructs (axes/gates/ladders per #188) carried intent only in this cycle: no executor existed yet — executor need was re-tested against the intent-persistence diagnosis (#189 triage, decided item 6; the cycle-0106 fieldtest F7 verdict was that evidence), and the v1 executor shipped the next cycle (below).

Realization (cycle 0107, #198/#196 — the campaign executor and its realization store). aura campaign run <file|content-id> executes a campaign (a file is register-then-run sugar; the content id is canonical): zero-fault referential gate, then the process pipeline — v1 executable shape std::sweep [std::gate]* [std::walk_forward]? (std::monte_carlo/std::generalize refused loudly at preflight in this cycle; they execute since cycle 0108, below) — runs once per (strategy, instrument, window) cell in doc order. Execution semantics live in the aura-campaign library crate (#198 home decision: reachable beyond the CLI; NOT C21's project-side World): grid odometer over the campaign axes, members through the engine sweep over a ListSpace (explicit point set beside GridSpace/RandomSpace — a gate's survivor subset has no cartesian structure), per-member gates via the 14-name member_metric roster (an R-predicate over a missing R block fails conservatively), walk-forward re-rolled in the doc's epoch-ms unit (WindowRoller; IS windows search ONLY the survivor points; OOS winner reports carry manifest.selection), deflation nulls seeded from the doc's seed (C1: realization is a pure function of doc + stores + data). Harness/data binding stays consumer-side behind the one-method MemberRunner seam — the CLI binds the shipped loaded-blueprint reduce convention with unique suffix-join of raw axis names onto the wrapped param_space. The registry grew the campaign_runs.jsonl JSONL sibling (beside runs.jsonl / families.jsonl): one thin CampaignRunRecord per run — campaign/process ids, seed, and per-cell realized stage prefixes linking family ids, gate survivor ordinals, and sweep selections — over untouched family records, run-counted per campaign id. Zero survivors truncate a cell's realized prefix and exit 0 (a null result is a valid research result); emit is honored (family_table/selection_report lines); persist_taps was deferred LOUDLY on stderr in this cycle (the wiring shipped in cycle 0109, below). The blueprint on-ramp (#196) closes the F5 authoring gap: aura graph register (store put keyed by content id == topology hash), aura graph introspect --params (the RAW param_space namespace campaign axes are validated against), and a blueprint-file mode on --content-id. The std::walk_forward vocabulary was corrected to machinery-true fields (in_sample_ms/out_of_sample_ms/step_ms/mode — WindowRoller's three lengths and both RollModes; the shipped folds slot mapped to nothing the machinery accepts), with a new ZeroWalkForwardLength intrinsic fault; wf-bearing process docs get new content ids by design (the 0106 fieldtest corpus stays as the historical record). Known debt: metric-roster triplication (still hand-maintained, but drift from the shipped aura-analysis types is now test-caught by a cross-crate guard, #190; single-source removal waits on #147), deflation-constant duplication (#199).

#300 (2026-07-21) closed the store's read loop: aura process|campaign show <content-id> prints a registered document's canonical bytes — the generate → retrieve → hand-extend → re-register cycle needs no direct store filesystem access.

Realization (cycle 0108, #200 — the annotator stages execute). The v2 executable shape is std::sweep [std::gate]* [std::walk_forward]? [std::monte_carlo]? [std::generalize]? — an ordered optional annotator suffix, each at most once, std::generalize strictly last; the executor preflight refuses what the intrinsic tier deliberately admits ([sweep, mc, walk_forward] is intrinsically valid — the tier boundary is test-pinned on both sides), plus three new static guards (single-instrument generalize, non-R generalize metric via the registry's check_r_metric, zero mc resamples/block_len). Both annotators are terminal (unanimous #200 triage): nothing flows out of them; filtering stays the gate's monopoly. std::monte_carlo bootstraps the stage's incoming R-evidence with one semantics, input-shaped by position — after a walk_forward, ONE r_bootstrap over the wf family's pooled per-window OOS net_trade_rs in roll order (PooledOos; #259 materialized this conduit as the cost-netted per-trade series r − cost_in_r, equal to the gross series bit-for-bit when no cost model is bound); after sweep/gates, one r_bootstrap per surviving member's fresh in-memory series (PerSurvivor, ordinals into the population family; a zero-trade member records the engine's defined all-zero degenerate) — seeded from the campaign doc's seed (C1; net_trade_rs is #[serde(skip)], so annotators run in-executor or not at all). std::generalize executes at campaign scope: after all cells, per (strategy, window) the per-cell nominees (last wf window's OOS report, else the sweep winner; none on gate truncation) across instruments feed the shipped generalization() when ≥ 2 exist — divergent per-instrument winners are exposed via their params, never averaged away; a shortfall is recorded, not computed around. Realization: StageRealization.bootstrap (StageBootstrap::PerSurvivor | PooledOos) and CampaignRunRecord.generalizations (CampaignGeneralization keyed strategy × window with winners/missing), both serde-default sparse — pre-0108 campaign_runs.jsonl lines parse unchanged (C14/C23). No document content id moved (introspection doc-strings dropped "in v1" only). Noted debt: the mc arm detects the wf family by the stringly block == "std::walk_forward" literal (a pre-existing lockstep pattern with the realization's block strings).

Realization (cycle 0109, #201 — persist_taps wired). Campaign presentation persists traces: the tap namespace is a closed vocabulary of the wrap convention's four sink names (equity/exposure/r_equity/net_r_equity; aura_research::tap_vocabulary, intrinsic DocFault::UnknownTap — the escalation for a new observable is a new vocabulary entry or an authored blueprint sink, never an open node-path namespace). Scope is the per-cell nominee only: after the pipeline settles, the CLI re-runs each nominee once in non-reduce mode (all four channels drained; windowed to the nominee manifest's own ns bounds) and asserts metrics equality against the recorded nominee — the C1 drift alarm, hard refusal on divergence (the reproduce precedent, enforced). Traces land in the existing TraceStore as traces/{campaign8}-{run}/{strategy8}-{instrument}-w{n}/{tap}.json (ensure_name_free(Family) once; window ordinal doc-derived), chartable by the unchanged viewer. The record carries ONE sparse pointer, CampaignRunRecord.trace_name (Some("{campaign8}-{run}") iff the doc requests taps — the claim-sentinel contract: execute claims, append_campaign_run composes the name via the single-sourced derive_trace_name, execute mirrors it onto the returned copy). Loud lines replace the retired deferral: per-cell no-nominee skip, per-run unproducible-tap skip (net_r_equity needs a cost leg; the campaign runner wires none), one summary. aura-campaign stays trace-agnostic (the MemberRunner seam is unchanged; the stamp is a pure name derivation). Noted debt: aura chart over the campaign family ROOT (cells spanning instruments) is untested/semantically undefined — only per-cell read-back is pinned.

Realization (#272, 2026-07-14 — per-cell fault isolation). A member fault (no-data, bind, run, or a caught panic) is a recorded per-cell outcome, never a global abort: run_cell returns a fault-annotated CellRealization (fault: Option<CellFault>, closed CellFaultKind) instead of Err, so execute's existing accumulate-then-append-once tail persists every healthy cell and the one run record. Containment granularity is the cell for a sweep stage (a grid hole compromises selection) and the fold for walk_forward (surviving folds pool; failed folds recorded as StageRealization. window_faults, the summary naming the ratio). ExecFault::Registry and doc-shape preflight faults stay global. The CLI declares holes (per-cell notes + a completion summary) and a run with ≥1 failed cell exits 3 ("completed with failed cells" — distinct from 0/1/2). A partially-covered window carries a CellCoverage annotation (effective bounds + interior gap months, from the #264 archive primitives). Generalize already treats a no-nominee cell as missing, so a failed cell surfaces there unchanged. Member panics are caught with catch_unwind(AssertUnwindSafe) at the three member-run sites and recorded as MemberFault::Panic; a ref-counted SilencedPanic guard (a process-global panic-hook save/no-op/restore behind a static Mutex, held only around each catch_unwind) suppresses the default crash backtrace so "recorded, campaign continues" is observably true on stderr, not merely true in the record. The guard's mutex serialises only the ref-count/hook-swap (O(1)), never the member computation, so C1 disjoint-parallel execution and determinism are preserved; ref-counting (save on 0→1, restore on 1→0) keeps concurrent sweep/walk-forward threads and any caller-installed hook correct.

Realization (#191, 2026-07-17 — identity-ref resolution is index-first). find_blueprint_by_identity no longer re-loads the whole blueprint store per reference: a fourth persistent JSONL sidecar, blueprint_identity_index.jsonl (identity id → content id; fixed-name sibling of the runs store, appends under the #276 lock), is consulted first, and every hit is verified by loading that one blueprint under the current resolver and recomputing its identity id — the index is a cache, never an oracle, so resolution results stay scan-identical under roster drift, store surgery, or index corruption (the one unspecified corner is unchanged in kind: which same-identity twin answers was read_dir-order-dependent before and is index-history-dependent now). Any miss or failed verification runs the old scan as a full-store repair pass, collect-then-diff-append: the walk's last-wins mapping is diffed against the pre-walk snapshot after the walk, so a converged index — twin stores included — appends nothing (the twin-convergence pin). Index reads never fail a lookup (missing/unreadable file → empty, unparseable lines skipped), repair appends are best-effort, and a pre-index store backfills itself on its first miss, no migration; a read-only store keeps scanning as before. Write paths, the engine, and both callers are untouched; maintenance is lazy-only — put-time indexing was rejected because it would need a roster-free doc-level identity function whose equivalence to the loaded-composite path no green test ratifies (decision log: #191 comments).