spec: 0029 run-registry — index & compare a sweep family

Cycle D of the milestone "The World — parameter-space & sweep" (#33). A persistent, append-only run registry: each run's (manifest, metrics) is a record in runs/runs.jsonl; a read surface lists all records and ranks them best-first by a named metric — C18's "compare experiments over time", which has no home in git or Gitea. First cut = persist + list + rank; promotion/ status, lineage, run-diff, on-demand re-derivation, run_id, and Aura.toml runs-dir wiring are deferred. Contract-level change: C16's blanket "zero-external-dependency by commitment" is amended to a considered, per-case dependency policy (scrutinize what enters the frozen artifact; standard vetted crates like serde/rayon pass that review; never hand-roll what they do). The engine/project split + three-tier node reuse of C16 are unchanged. This admits serde/serde_json — the typed read-path ranking needs, and which closes #17 (typed RunReport handle). Shape: serde derives on RunManifest/RunMetrics/RunReport + Timestamp; SweepPoint carries a full RunReport (closing the manifest-per-point gap cycle C deferred to #33); new crate aura-registry (open/append/load/rank_by); CLI gains aura runs list / aura runs rank <metric> and aura sweep persists each point. Three iterations, first planner handoff = iteration 1 (foundation). Sign-off: spec_auto_sign was enabled for this cycle; the five-lens spec-skeptic panel returned 3 SOUND + 2 BLOCK (rank-ordering semantics; iteration-cut compile-ordering + Scalar->f64 coercion). Both BLOCKs were genuine defects, fixed before sign-off (rank is now best-first per-metric; the closure-type change lands with its callers in one green iteration; the coercion is shown). Non-unanimous panel routed the spec to human sign-off, granted by the user. refs #33
2026-06-10 19:59:27 +02:00
parent 728ca1a2d2
commit d309336a72
1 changed files with 353 additions and 0 deletions
@@ -0,0 +1,353 @@
+# Run Registry — index & compare a sweep family — Design Spec
+
+**Date:** 2026-06-10
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+## Goal
+
+Cycle D of the milestone *The World — parameter-space & sweep* (#33). A sweep
+(cycle C) already produces a family of disjoint runs. This cycle makes that
+family — and runs across separate invocations — **persistent, listable, and
+comparable over time**: the substance of C18's run registry, which has no home
+in git or Gitea.
+
+First cut (this cycle): **persist + list + rank**. Each run's `(manifest,
+metrics)` is appended as one record to a `runs/` store; a read surface lists all
+stored records and ranks them by a named metric. Determinism (C1/C12) makes the
+record a faithful, reproducible descriptor of its run.
+
+Deferred to later iterations / follow-up issues (named here so the cut is
+honest): promotion/status labels, the lineage graph (composite←signals,
+run←inputs), run-diff, on-demand re-derivation of full results, a `run_id`
+content digest, and `Aura.toml` runs-dir wiring.
+
+This cycle also makes a **contract-level change**: it amends C16 (see
+*Architecture → C16 amendment*).
+
+## Architecture
+
+### C16 amendment (load-bearing, user-ratified)
+
+C16 today carries a blanket clause: the engine workspace is
+*"zero-external-dependency by commitment"*, with `aura-ingest` as the sole
+*"external-dependency firewall"*. That blanket is **struck** and replaced by a
+**considered, per-case dependency policy**:
+
+> Dependencies are admitted by deliberate, per-case review — what a crate pulls
+> in is weighed against what it buys — with **particular scrutiny for anything
+> that enters the frozen deploy artifact** (C13: this bot = this commit).
+> Well-established standard crates (`serde`, `rayon`, …) pass that review and are
+> used wherever they do the job, including in the bot. There is **no blanket
+> zero-dependency commitment** and **no blanket admission**; hand-rolling what a
+> vetted standard crate already does is the anti-pattern, not the dependency.
+
+What of C16 is **unchanged**: the engine/project split and the three-tier node
+reuse (`aura-std` / shared node crates / project-local `nodes/`). `aura-ingest`
+remains the data-source ingestion edge — it is simply no longer framed as the
+*only* place an external crate may appear.
+
+C14's *"hand-rolled JSON"* rationale falls with the zero-dep clause. This cycle
+keeps the `RunReport::to_json` writer (it now drives **all** stdout); `serde` is
+added *alongside* it for the registry's storage/parse layer, and the
+stdout→serde unification (one shape for stored and printed records) is a
+follow-up. Two stdout consequences:
+
+- `aura run` stdout is **byte-stable** (still `run_sample().to_json()`).
+- `aura sweep` stdout **changes shape**: the per-point hand-rolled
+  `sweep_point_to_json` (`{params, metrics}`) is **retired** in favour of each
+  point's `RunReport::to_json` (`{manifest, metrics}`) — a consistency win
+  (`aura run`, `aura sweep`, `aura runs` now all print the same RunReport shape)
+  and the natural consequence of `SweepPoint` carrying a full report. The
+  cycle-C params-only golden is updated accordingly.
+
+**Per-case justification for the dependencies this cycle admits** (the policy
+requires one, sharper where it reaches the frozen artifact):
+
+- `serde` + `serde_json` (derive feature). *Buys:* a typed read-path for the
+  registry record (parse a stored line back into a `RunReport`) and idempotent
+  serialization — the exact capability ranking/comparison needs, and the thing
+  C14's writer-only JSON could never give. *Pulls in:* `serde`, `serde_derive`,
+  `serde_json`, `itoa`, `ryu`, `memchr` — a tiny, ubiquitous, deterministic,
+  heavily-audited closure. *Frozen-artifact reach:* the derives land on
+  `RunReport`/`RunManifest`/`RunMetrics` (in `aura-engine`) and `Timestamp` (in
+  `aura-core`), both linked by the frozen bot. Justified: `serde` is the
+  single most-vetted crate in the ecosystem and its output is deterministic
+  (C1-safe).
+
+### Where the registry lives
+
+A **new crate `aura-registry`** (recommended over an `aura-cli` module): C18
+calls the run registry a first-class concern. A dedicated crate keeps
+`aura-engine` pure compute (no filesystem I/O in the run-loop crate) and is the
+natural home as the registry grows (lineage, promotion/status, run-diff). It
+depends on `aura-engine` for `RunReport` and on `serde_json` for storage/parse.
+
+### The family becomes self-describing
+
+`SweepPoint` today carries only `RunMetrics`. To index a family the registry
+needs each point's full `(manifest, metrics)`. So `SweepPoint` carries a full
+**`RunReport`** — closing the manifest-per-point gap cycle C explicitly deferred
+to #33. The sweep's per-point closure returns `RunReport` instead of
+`RunMetrics`; the engine builds nothing new (the manifest is caller-supplied, as
+report.rs already documents).
+
+## Concrete code shapes
+
+### The user-facing program (acceptance-criterion evidence)
+
+```text
+$ aura sweep
+{"manifest":{"commit":"unknown","params":{"sma_cross.fast":2,"sma_cross.slow":4,"scale":0.5},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=0.0001)"},"metrics":{"total_pips":-0.13,"max_drawdown":0.17,"exposure_sign_flips":1}}
+… three more points …
+# each point now prints its full RunReport (manifest+metrics, like `aura run`)
+# — cycle C printed {params,metrics}; AND each point is appended to runs/runs.jsonl
+
+$ aura runs list
+# every record ever stored, across invocations, one per line (to_json shape)
+{"manifest":{…fast=2,slow=4…},"metrics":{"total_pips":-0.13,…}}
+…
+
+$ aura runs rank total_pips
+# the substance of #33 — the family compared, BEST first (total_pips: higher-is-better → desc)
+{"manifest":{…fast=3,slow=4…},"metrics":{"total_pips":-0.09,…}}
+{"manifest":{…fast=2,slow=5…},"metrics":{"total_pips":-0.112,…}}
+…
+
+$ aura runs rank max_drawdown
+# best-first is per-metric: max_drawdown is lower-is-better → ascending (shallowest drawdown first)
+{"manifest":{…fast=3,slow=4…},"metrics":{…"max_drawdown":0.09,…}}
+…
+
+$ aura runs rank bogus_metric
+aura: unknown metric 'bogus_metric' (known: total_pips, max_drawdown, exposure_sign_flips)
+# exit code 2 — strict, like every other aura arg error
+```
+
+### Implementation shapes (secondary)
+
+**serde derives (before → after).** In `crates/aura-core/src/scalar.rs`:
+
+```rust
+// before
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct Timestamp(pub i64);
+// after — newtype struct serializes transparently as its inner i64,
+// matching to_json's [from,to] window shape
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize)]
+pub struct Timestamp(pub i64);
+```
+
+In `crates/aura-engine/src/report.rs`, the same two derives are added to
+`RunMetrics`, `RunManifest`, and `RunReport` (no field changes). `Scalar` is
+**not** derived — the registry serializes `RunReport`, never a raw `Scalar`.
+
+**`SweepPoint` carries a report (before → after).** In
+`crates/aura-engine/src/sweep.rs`:
+
+```rust
+// before
+pub struct SweepPoint { pub params: Vec<Scalar>, pub metrics: RunMetrics }
+pub fn sweep<F>(space: &GridSpace, run_one: F) -> SweepFamily
+where F: Fn(&[Scalar]) -> RunMetrics + Sync { … }
+
+// after — the point keeps its typed enumeration coordinate AND the full result
+pub struct SweepPoint { pub params: Vec<Scalar>, pub report: RunReport }
+pub fn sweep<F>(space: &GridSpace, run_one: F) -> SweepFamily
+where F: Fn(&[Scalar]) -> RunReport + Sync { … }
+```
+
+The thread-fan-out internals are untouched except the collected element type
+(`(usize, RunMetrics)` → `(usize, RunReport)`). `SweepFamily` is unchanged in
+shape (`{ points: Vec<SweepPoint> }`).
+
+**The registry (new `crates/aura-registry/src/lib.rs`).**
+
+```rust
+use std::path::{Path, PathBuf};
+use aura_engine::RunReport;
+
+/// An append-only run registry over a JSONL file: one serde_json line per
+/// RunReport. The stored shape is serde's; display is the caller's concern
+/// (the CLI prints via RunReport::to_json for stdout consistency).
+pub struct Registry { path: PathBuf }
+
+impl Registry {
+    /// Bind to a JSONL path. No I/O; the file is created lazily on first append.
+    pub fn open(path: impl AsRef<Path>) -> Registry { Registry { path: path.as_ref().to_path_buf() } }
+
+    /// Append one record as a single JSON line. Creates the file (and parent
+    /// dir) if absent.
+    pub fn append(&self, report: &RunReport) -> std::io::Result<()> { /* serde_json::to_string + writeln in append mode */ }
+
+    /// Parse every non-empty line back into a typed RunReport, in file order.
+    /// A missing file is an empty registry (Ok(vec![])), not an error.
+    pub fn load(&self) -> Result<Vec<RunReport>, RegistryError> { /* read; serde_json::from_str per line */ }
+}
+
+/// Sort reports **best-first** by a named metric. "Best" is per-metric, fixed by
+/// each metric's meaning (spec 0009): `total_pips` higher-is-better → descending;
+/// `max_drawdown` (a positive magnitude) and `exposure_sign_flips` (a churn
+/// count) lower-is-better → ascending. So `aura runs rank <metric>` always puts
+/// the genuinely best run first, whichever metric. Ties keep file (insertion)
+/// order — a stable sort over `load()`'s file-order vector. `f64` keys use
+/// `partial_cmp` (total, since metrics are finite by construction). Unknown
+/// metric is an error the CLI maps to exit 2.
+pub fn rank_by(reports: Vec<RunReport>, metric: &str) -> Result<Vec<RunReport>, RegistryError> { … }
+
+#[derive(Debug)]
+pub enum RegistryError {
+    Io(std::io::Error),
+    Parse { line: usize, source: serde_json::Error },
+    UnknownMetric(String),
+}
+```
+
+**CLI dispatch (before → after)** in `crates/aura-cli/src/main.rs`:
+
+```rust
+// before
+["sweep"] => print!("{}", sweep_report()),
+// after — sweep_report now also persists each point; new read subcommands
+["sweep"]            => print!("{}", sweep_report()),               // persists as a side effect
+["runs", "list"]     => print!("{}", runs_list(&default_registry())),
+["runs", "rank", m]  => match runs_rank(&default_registry(), m) {
+    Ok(s)  => print!("{s}"),
+    Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
+},
+```
+
+The per-point sweep closure returns a `RunReport`. The manifest's
+`params: Vec<(String, f64)>` is built by zipping the `param_space()` names onto
+the point's `Vec<Scalar>` values, **coercing each `Scalar` to `f64`** — the
+manifest's params are `f64`-typed, but a grid point carries `Scalar::I64`
+(SMA lengths) and `Scalar::F64` (the scale):
+
+```rust
+fn scalar_as_param_f64(s: &Scalar) -> f64 {
+    match s {
+        Scalar::I64(n) => *n as f64,   // fast=2 -> 2.0
+        Scalar::F64(f) => *f,          // scale=0.5 -> 0.5
+        // numeric tuning params only; Bool/Ts never appear in a tuning grid
+        other => unreachable!("non-numeric sweep param: {other:?}"),
+    }
+}
+```
+
+The fixed manifest facts — `commit`/`window`/`seed`/`broker` — are exactly those
+`run_sample` already builds. `sweep_report` (after iteration 3) appends each
+`point.report` to `default_registry()` and prints it via `RunReport::to_json`
+(the hand-rolled `sweep_point_to_json` is retired — the report renders itself).
+`default_registry()` binds `runs/runs.jsonl` under cwd.
+
+## Components
+
+| Component | Crate / file | Change |
+|---|---|---|
+| C16 amendment | `docs/design/INDEX.md` | strike blanket zero-dep; record per-case policy |
+| workspace deps | `Cargo.toml` | add `[workspace.dependencies]` with `serde` (derive) + `serde_json` |
+| `Timestamp` serde | `crates/aura-core/src/scalar.rs` | add `Serialize`/`Deserialize` derives; `serde` dep |
+| report serde | `crates/aura-engine/src/report.rs` | derives on `RunMetrics`/`RunManifest`/`RunReport`; `serde` dep |
+| `SweepPoint` | `crates/aura-engine/src/sweep.rs` | `metrics: RunMetrics` → `report: RunReport`; closure `-> RunReport` |
+| `aura-registry` | `crates/aura-registry/` (new) | `Registry::{open, append, load}` + `rank_by` + `RegistryError` |
+| CLI subcommands | `crates/aura-cli/src/main.rs` | `sweep` persists; `runs list`; `runs rank <metric>`; USAGE |
+
+## Data flow
+
+```
+aura sweep
+  → GridSpace enumerates N points
+  → sweep(): per point, fresh blueprint → bootstrap_with_params → run
+             → drain sinks → summarize → RunReport{manifest, metrics}
+  → SweepFamily{ points: [SweepPoint{params, report}, …] }
+  → for each point: Registry::open("runs/runs.jsonl").append(&point.report)   [side effect]
+  → stdout: each point.report.to_json()                                       [full RunReport shape; cycle-C params-only golden updated]
+
+aura runs rank total_pips
+  → Registry::open("runs/runs.jsonl").load() -> Vec<RunReport>   (serde_json::from_str per line)
+  → rank_by(reports, "total_pips")  -> sorted desc
+  → stdout: each report.to_json()
+```
+
+## Error handling
+
+- **Append:** creates the file and its parent dir on first write; an I/O failure
+  surfaces as `RegistryError::Io` → the CLI prints `aura: <err>` and exits 2.
+- **Missing registry on read:** `load()` returns `Ok(vec![])` — an empty
+  registry is not an error; `aura runs list` on a fresh tree prints nothing and
+  exits 0.
+- **Malformed stored line:** `load()` returns `RegistryError::Parse { line, … }`
+  naming the 1-based line — a corrupt record is loud, never silently skipped.
+- **Unknown metric:** `aura runs rank bogus` → `RegistryError::UnknownMetric` →
+  `aura: unknown metric 'bogus' (known: …)` and exit 2.
+- **Non-finite f64:** `RunMetrics`/manifest params are finite by construction
+  (report.rs); `serde_json` would error on NaN/∞, which cannot occur here.
+- **Strict argv:** `aura runs`, `aura runs rank` (no metric), `aura sweep extra`,
+  etc. are usage errors → `aura: <USAGE>` and exit 2, consistent with the
+  existing whole-vector match.
+
+## Testing strategy
+
+- **`aura-core`:** a round-trip unit test — `Timestamp` serde survives
+  `to_string`→`from_str`.
+- **`aura-engine`:** `RunReport` serde round-trip (a built report → JSON → back,
+  equal). `sweep.rs` tests updated for `point.report` (the closure now returns
+  `RunReport`); the existing determinism/odometer/distinct-metrics properties are
+  preserved, reading `point.report.metrics`.
+- **`aura-registry`:** `append` then `load` round-trips N reports in order;
+  `load` on a missing path is `Ok(empty)`; a hand-written corrupt line makes
+  `load` return `Parse{line}`; `rank_by` orders by each known metric and errors
+  on an unknown one. Tests use a temp-dir path (no shared cwd state).
+- **`aura-cli`:** process-level goldens with cwd set to a temp dir —
+  `aura sweep` writes 4 lines to `runs/runs.jsonl` and prints 4 odometer lines,
+  each now a full `RunReport` (`{"manifest":…,"metrics":…}`); the cycle-C
+  params-only `aura sweep` golden is updated to the RunReport shape. A second
+  `aura sweep` then `aura runs list` shows 8 records; `aura runs rank
+  total_pips` prints them highest-first; `aura runs rank bogus` exits 2.
+
+## Acceptance criteria
+
+Against aura's feature-acceptance (CLAUDE.md): the audience naturally reaches for
+it, it measurably improves the loop, and it reintroduces no failure the core
+constraints forbid.
+
+1. **Audience reaches for it.** Comparing experiments over time is the heart of
+   the research loop (C18). `aura runs rank total_pips` is the first time a
+   researcher can ask "which point in my sweep — or across my sweeps — was
+   best?" without re-running anything. The worked program above is that ask.
+2. **Measurable improvement / removed redundancy.** Closes the manifest-per-point
+   gap cycle C deferred (the family becomes self-describing) and the typed
+   read-path gap (#17 — `serde` Deserialize *is* the typed handle). Retires the
+   hand-rolled `sweep_point_to_json`.
+3. **No reintroduced failure class.** Determinism (C1) holds — the registry
+   stores a reproducible manifest; `serde_json` output is deterministic. The
+   sweep stays disjoint-parallel. The new dependency is admitted under the
+   amended C16's per-case review, justified above, not by a blanket waiver.
+
+**Recommended iteration cut** (first planner handoff = iteration 1):
+
+- **Iteration 1 — foundation.** C16 amendment in the ledger; `[workspace.dependencies]`
+  serde/serde_json; serde derives on the report types + `Timestamp`; round-trip
+  tests. No behaviour change to the CLI. Workspace green.
+- **Iteration 2 — engine + registry + CLI closure rethread (lands together).**
+  `SweepPoint` carries `RunReport` (closure bound `-> RunReport`). Changing that
+  bound breaks its **only non-test caller** — the CLI's `sweep_report` closure
+  (main.rs:280-289) — and the `sweep.rs` test callers; per the planner's
+  caller-threading rule, **all of them are rethreaded in this same iteration** so
+  `cargo test --workspace` stays green: the CLI closure now returns `RunReport`
+  and prints via `RunReport::to_json` (retiring `sweep_point_to_json`), and the
+  cycle-C `aura sweep` golden is updated to the RunReport shape here. New
+  `aura-registry` crate (`open`/`append`/`load`/`rank_by`) with unit tests. No
+  persistence side-effect yet, no new subcommands. Workspace green.
+- **Iteration 3 — CLI persistence + read surface.** `aura sweep` appends each
+  point's report to `runs/runs.jsonl`; `aura runs list` / `aura runs rank
+  <metric>`; USAGE; process goldens (persist + list + rank + bogus-metric exit
+  2). Workspace green.
+
+Each iteration leaves `cargo build/test --workspace` green: the signature-change
+iteration (2) threads every caller it breaks, never deferring one past its own
+build gate.
+
+**Out of scope** (follow-up issues / later iterations): promotion/status,
+lineage graph, run-diff, on-demand re-derivation, `run_id` digest, `Aura.toml`
+runs-dir wiring, and the `aura run`/`aura sweep` stdout→serde unification.