Aura/docs/design/contracts/c18-registry.md

C18 — Project management: one repo = one project, plus a run registry

Guarantee. Management has two planes. (1) Code & forward-queue: git (commit = identity; the frozen bot is a commit) + Gitea (ideas/hypotheses as the forward-queue, a research thrust = a milestone, the idea → experimental → validated → deployed label gradient). (2) Experiments & results: an Aura-native run registry — one record per run = a manifest (node-commit + params + data-window + seed + broker profile + instrument + topology_hash + selection) + metrics, queryable, with lineage (composite ← signals; run ← inputs). Determinism (C1/C12) makes a run reproducible from its tiny manifest, so the registry stores manifests + metrics and re-derives full results on demand. Because the World may generate or structurally search topologies, commit alone no longer identifies the graph: the manifest carries a topology_hash and the registry keeps the canonical topology bytes in a content-addressed store beside it, so the manifest + the content-addressed store + the commit are the complete re-derivation recipe (C24). Depth: structured (promotion/status, lineage, run-diff).

Forbids. Storing results not reproducible from a recorded manifest; duplicating git/Gitea inside aura; a multi-project workspace manager.

Why. Comparing experiments over time is the heart of the research loop and has no home in git/Gitea; determinism makes a structured registry cheap. Every run emits a manifest + metrics, and the registry/index is built over manifests that already exist. Content-addressed identity is the natural home for a generated topology's identity: once a graph is no longer fully named by its commit, the re-derivation recipe must carry or content-address the topology-data to remain "reproducible from a recorded manifest" — the format and carrier are C24's design, the reproduction requirement is C18's.

Current state

Two planes, one crate. The code/forward-queue plane is git + Gitea, unchanged by aura. The experiments-&-results plane is the run registry in aura-registry (C9: it depends on aura-engine, never the reverse).

The store family. All members are fixed-name siblings of the runs-store path (Registry::path.with_file_name(...), aura-registry):

• runs.jsonl — the append-only flat store, one RunReport per line (RunManifest + RunMetrics), with a typed read-path (load) and best-first ranking (rank_by/optimize). No live producer writes it today — sweep / walk-forward / mc persist to the family store and aura run does not persist — so the flat lib API is retained but selectively live: rank_by backs aura runs family … rank, optimize backs walk-forward's in-sample step (aura-campaign and the registry's own selection helper both call it), while append/load remain public API with no in-tree caller (a latent surface for external consumers).
• families.jsonl — the family store. A sweep / Monte-Carlo / walk-forward / cross-instrument run persists as a set of related records, each a FamilyRunRecord (a RunReport stamped with family + run + kind + ordinal; FamilyKind ∈ {Sweep, MonteCarlo, WalkForward, CrossInstrument}). group_families re-derives a family from the stored links (re-listable / rankable as a unit — C21). The user-facing family_id = "{family}-{run}" handle is derived from the stored family name plus a per-name run index (numeric max+1 — not a content hash, so re-running the same family mints a fresh id). CLI: aura runs families, aura runs family <id> [rank <metric>]; aura sweep/walkforward/mc persist via append_family with an optional --name.
• campaign_runs.jsonl — one thin CampaignRunRecord per campaign run (below), over untouched family records.
• blueprints/<hash>.json, processes/, campaigns/ — the content-addressed document stores (below).
• blueprint_identity_index.jsonl — the identity-ref resolution cache (below).

The manifest is the re-derivation recipe. No input-stream blob / path / payload enters a record; a member's data window is producer-supplied via Source::bounds()/window_of (eager or streamed → byte-identical lineage), never a materialized-Vec scan at the call site. New manifest fields (e.g. the first-class instrument lineage field) are serde-widened with skip_serializing_if, so legacy lines and every path that does not set them stay byte-identical (C14/C23). serde_json/float_roundtrip is enabled so stored f64 metrics round-trip exactly through families.jsonl — the precondition for a bit-identical compare (C1).

Content-addressed reproduction. A run's topology is content-addressed: the canonical blueprint_to_json bytes are stored once, keyed by the topology_hash the manifest carries, in a dumb bytes-by-key store (Registry::put_blueprint/get_blueprint → blueprints/<hash>.json; no sha2, no parse — the caller owns the hash and reproduction's bit-identical compare is the integrity check). One blueprint is stored per family (all members share the signal topology_hash — C11/C12 dedup). aura reproduce <id> re-derives every persisted member: load its blueprint by topology_hash, reconstruct the member (a Sweep point from the recorded params; a MonteCarlo member's seed-driven synthetic walk from manifest.seed; a WalkForward OOS member's windowed slice from manifest.window, winner params via the shared manifest→cells recovery), re-run through the same run_blueprint_member the live path uses (bit-identity by construction — C1), and compare metrics. All three family kinds persist and reproduce through the one shared topology_hash+put_blueprint hook. Refuse- don't-guess (aura_runner::reproduce): an unknown id, a missing stored blueprint, or a DIVERGED compare each exit 1 — recorded state missing or mismatched is C14's runtime-failure class. Id resolution: first as the derived {family}-{run} handle; a bare enumeration name naming exactly one stored run resolves as fallback; an ambiguous name refuses, listing the candidate handles (the list-then-reproduce seam, #298). The recipe's scope is signal-only today: content-addressing covers the signal blueprint, while the fixed scaffolding stays commit-identified (not yet blueprint-data, C24); whole-harness / structural-axis content-addressing is deferred. Reproduction is proven on synthetic (deterministic) data; recorded-dataset reproduction rides the DataServer seam (#124).

The one content_id primitive. topology_hash, the document store keys, and aura graph introspect --content-id/--identity-id all route through aura_research::content_id_of (aura-cli's content_id/topology_hash delegate byte-identically). The identity id is the canonical form with every non-load-bearing debug symbol blanked (invariant 11 / C23), hashed through the same primitive: same-topology blueprints become comparable across authoring paths, while the byte-exact topology_hash keeps every debug role untouched (introspection-only — no manifest field, no store key, until a dedup consumer exists). A Tier-1 optional the blueprint does not use leaves the id byte-stable. --content-id and --identity-id are combinable.

Cross-instrument generalization. FamilyKind::CrossInstrument: aura generalize runs one candidate across an instrument list and persists the M per-instrument runs via append_family, each member self-identifying through RunManifest.instrument. The generalization score (worst-case R floor + sign-agreement + per-instrument breakdown) is a recomputable aggregate over those members, not a persisted family-level record.

Research-artifact document stores. processes/ and campaigns/ hold two document types (C25 roles 5 / 6b): the process document (a named validation/eval methodology — a closed std stage vocabulary wrapping shipped primitives) and the campaign document (persisted experiment intent — instruments × windows × strategy refs by content/identity id × param axes × process ref (content-id-only) × data-level presentation). Documents are canonical JSON (format_version envelope, omit-defaults, no trailing newline) keyed by the shared content-id primitive; unlike put_blueprint (caller owns the hash) the document puts self-key from their canonical bytes, and gets are Ok(None) treat-as-empty. The referential tier (validate_campaign_refs) resolves process/strategy refs against the stores (identity refs index-first, below) and checks each campaign axis — name AND declared ScalarKind, the axis carrying its kind once — against the referenced blueprint's param_space. aura process|campaign show <content-id> prints a registered document's canonical bytes (#300), so the generate → retrieve → hand-extend → re-register loop needs no direct store filesystem access.

The campaign executor. aura campaign run <file|content-id> executes a campaign (a file is register-then-run sugar; the content id is canonical): a zero-fault referential gate, then the process pipeline. The executable shape is std::sweep [std::gate]* [std::walk_forward]? [std::monte_carlo]? [std::generalize]? — an ordered optional annotator suffix, each at most once, std::generalize strictly last. The executor preflight is deliberately stricter than the intrinsic tier ([sweep, mc, walk_forward] is intrinsically valid — the tier boundary is test-pinned on both sides), plus static guards: single-instrument generalize, a non-R generalize metric (via the registry's check_r_metric), zero mc resamples/block_len, and ZeroWalkForwardLength. Execution semantics live in the aura-campaign library crate (reachable beyond the CLI; NOT C21's project-side World): a grid odometer over the campaign axes, members through the engine sweep over a ListSpace (an explicit point set beside GridSpace/RandomSpace — a gate's survivor subset has no cartesian structure), per-member gates via the 14-name member_metric roster (an R-predicate over a missing R block fails conservatively), walk-forward re-rolled in the doc's epoch-ms unit (WindowRoller; IS windows search only the survivor points; OOS winner reports carry manifest.selection), deflation nulls seeded from the doc's seed — the whole realization is a pure function of doc + stores + data (C1). Harness/data binding stays consumer-side behind the one-method MemberRunner seam (the shipped implementation is aura_runner::DefaultMemberRunner; the CLI binds the loaded- blueprint reduce convention with a unique suffix-join of raw axis names onto the wrapped param_space). The campaign_runs.jsonl sibling records one thin CampaignRunRecord per run — campaign/process ids, seed, and per-cell realized stage prefixes linking family ids, gate survivor ordinals, and sweep selections — run-counted per campaign id. Zero survivors truncate a cell's realized prefix and exit 0 (a null result is a valid research result); emit is honored (family_table/selection_report lines). The blueprint on-ramp (#196): aura graph register (store put keyed by content id == topology hash), aura graph introspect --params (the raw param_space namespace axes validate against), and a blueprint-file mode on --content-id. std::walk_forward's machinery-true fields are in_sample_ms/out_of_sample_ms/step_ms/mode (WindowRoller's three lengths + both RollModes).

Annotators are terminal. Nothing flows out of an annotator; filtering stays the gate's monopoly. std::monte_carlo bootstraps the stage's incoming R-evidence with one semantics, input-shaped by position: after a walk_forward, one r_bootstrap over the wf family's pooled per-window OOS net_trade_rs in roll order (StageBootstrap::PooledOos; the conduit is the cost-netted per-trade series r − cost_in_r, equal to the gross series bit-for-bit when no cost model is bound, #259); after sweep/gates, one r_bootstrap per surviving member's fresh in-memory series (StageBootstrap::PerSurvivor, ordinals into the population family; a zero-trade member records the engine's defined all-zero degenerate) — seeded from the doc's seed (net_trade_rs is #[serde(skip)], so annotators run in-executor or not at all). std::generalize executes at campaign scope: after all cells, per (strategy, window) the per-cell nominees (last wf window's OOS report, else the sweep winner; none on gate truncation) across instruments feed the shipped generalization() when ≥ 2 exist — divergent per-instrument winners are exposed via their params, never averaged away; a shortfall is recorded, not computed around. StageRealization.bootstrap and CampaignRunRecord.generalizations (CampaignGeneralization keyed strategy × window with winners/missing) are serde-default sparse (C14/C23). Known debt: the mc arm detects the wf family by the stringly block == "std::walk_forward" literal.

Per-cell fault isolation (#272). A member fault (no-data, bind, run, or a caught panic) is a recorded per-cell outcome, never a global abort: run_cell returns a fault-annotated CellRealization (fault: Option<CellFault>, closed CellFaultKind) instead of Err, so execute's accumulate-then-append-once tail persists every healthy cell and the one run record. Containment granularity is the cell for a sweep stage (a grid hole compromises selection) and the fold for walk_forward (surviving folds pool; failed folds recorded as StageRealization.window_faults, the summary naming the ratio). ExecFault::Registry and doc-shape preflight faults stay global. The CLI declares holes (per-cell notes + a completion summary) and a run with ≥ 1 failed cell exits 3 ("completed with failed cells" — distinct from 0/1/2). A partially-covered window carries a CellCoverage annotation (effective bounds + interior gap months, #264). Generalize already treats a no-nominee cell as missing, so a failed cell surfaces there unchanged. Member panics are caught with catch_unwind(AssertUnwindSafe) at the three member-run sites and recorded as MemberFault::Panic; a ref-counted SilencedPanic guard (a process-global panic-hook save/no-op/restore behind a static Mutex, held only around each catch_unwind) suppresses the default crash backtrace so "recorded, campaign continues" is observably true on stderr. The guard's mutex serialises only the O(1) ref-count/hook-swap, never the member computation, so C1 disjoint-parallel execution and determinism hold; ref-counting (save on 0→1, restore on 1→0) keeps concurrent threads and any caller-installed hook correct.

Persisted taps (#201). Campaign presentation persists traces. The tap namespace is a closed vocabulary of the wrap convention's four sink names (equity/exposure/r_equity/net_r_equity; aura_research::tap_vocabulary, intrinsic DocFault::UnknownTap — the escalation for a new observable is a new vocabulary entry or an authored blueprint sink, never an open node-path namespace). Scope is the per-cell nominee only: after the pipeline settles the CLI re-runs each nominee once in non-reduce mode (all four channels drained, windowed to the nominee manifest's own ns bounds) and asserts metrics equality against the recorded nominee — the C1 drift alarm, a hard refusal on divergence (the reproduce precedent, enforced). Traces land in the existing TraceStore as traces/{campaign8}-{run}/{strategy8}-{instrument}-w{n}/{tap}.json, chartable by the unchanged viewer. The record carries one sparse pointer, CampaignRunRecord.trace_name (Some("{campaign8}-{run}") iff the doc requests taps — the claim-sentinel contract: execute claims, append_campaign_run composes the name via the single-sourced derive_trace_name, execute mirrors it onto the returned copy). aura-campaign stays trace-agnostic (the MemberRunner seam is unchanged; the stamp is a pure name derivation). Loud stderr lines cover a per-cell no-nominee skip and a per-run unproducible-tap skip (net_r_equity needs a cost leg the campaign runner wires none of). Known debt: aura chart over the campaign family ROOT (cells spanning instruments) is untested / semantically undefined — only per-cell read-back is pinned.

Identity-ref resolution is index-first (#191). find_blueprint_by_identity consults the persistent blueprint_identity_index.jsonl sidecar (identity id → content id; a fixed-name sibling of the runs store, appended under the #276 lock) first, and verifies every hit by loading that one blueprint under the current resolver and recomputing its identity id — the index is a cache, never an oracle, so resolution stays scan-identical under roster drift, store surgery, or index corruption (the one unspecified corner — which same-identity twin answers — is unchanged in kind: read_dir-order-dependent before, index-history-dependent now). Any miss or failed verification runs the old full-store scan as a repair pass, collect-then-diff- append: the walk's last-wins mapping is diffed against the pre-walk snapshot, so a converged index — twin stores included — appends nothing (the twin-convergence pin). Index reads never fail a lookup (missing/unreadable → empty, unparseable lines skipped); repair appends are best-effort; a pre-index store backfills on its first miss (no migration); a read-only store keeps scanning. Write paths, the engine, and both callers are untouched; maintenance is lazy-only — put-time indexing was rejected because it would need a roster-free doc-level identity function whose equivalence to the loaded-composite path no green test ratifies (decision log: #191).

Retired verbs and the unknown-id contract. Standalone aura runs list / rank are retired (#73): families (C21) subsume standalone over-time comparison. The unknown-id contract (ratified, Runway fieldtest 2026-06) is live law: aura runs family <id> treats an unknown-but-well-formed id as an empty family (prints nothing, exit 0) — the same treat-as-empty discipline as Registry::load reading a missing store as Ok(empty). This is deliberately distinct from aura reproduce <id>, which refuses an unknown id with exit 1 (reproduction of a named-but-absent family is missing recorded state, not a found-nothing lookup). Tightening runs family to a non-zero no such family <id> exit (typo-safety) is an available future UX choice, not a current contract.

Deferred. The run-diff depth and cross-family ranking (families against each other, vs. within-family) remain deferred; whole-harness / structural-axis content-addressing remains deferred (C24). Known debt across the campaign stack: metric-roster triplication (test-caught by the #190 cross-crate guard; single-source removal waits on #147) and deflation-constant duplication (#199).

See also

• C1 — determinism / bit-identity, the reproduction and drift-alarm correctness invariant
• C8 — sinks are the recording mechanism into the registry
• C9 — the registry depends on the engine, never the reverse
• C11 — record-then-replay; producer-supplied windows; C11/C12 dedup
• C12 — the four orchestration axes (sweep / MC / walk-forward / comparison) the family store persists
• C14 — additive serde back-compat and the runtime-failure exit class
• C21 — the World; families as the re-listable unit
• C23 — names non-load-bearing; the identity-id blanking
• C24 — the blueprint as serializable data; the topology data format the manifest content-addresses
• C25 — the role model; process/campaign documents as closed-vocabulary artifacts

History: c18-registry.history.md