From d309336a7210a6fc1c34f51d2c0479be9fa77a13 Mon Sep 17 00:00:00 2001 From: Brummel Date: Wed, 10 Jun 2026 19:59:27 +0200 Subject: [PATCH] =?UTF-8?q?spec:=200029=20run-registry=20=E2=80=94=20index?= =?UTF-8?q?=20&=20compare=20a=20sweep=20family?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cycle D of the milestone "The World — parameter-space & sweep" (#33). A persistent, append-only run registry: each run's (manifest, metrics) is a record in runs/runs.jsonl; a read surface lists all records and ranks them best-first by a named metric — C18's "compare experiments over time", which has no home in git or Gitea. First cut = persist + list + rank; promotion/ status, lineage, run-diff, on-demand re-derivation, run_id, and Aura.toml runs-dir wiring are deferred. Contract-level change: C16's blanket "zero-external-dependency by commitment" is amended to a considered, per-case dependency policy (scrutinize what enters the frozen artifact; standard vetted crates like serde/rayon pass that review; never hand-roll what they do). The engine/project split + three-tier node reuse of C16 are unchanged. This admits serde/serde_json — the typed read-path ranking needs, and which closes #17 (typed RunReport handle). Shape: serde derives on RunManifest/RunMetrics/RunReport + Timestamp; SweepPoint carries a full RunReport (closing the manifest-per-point gap cycle C deferred to #33); new crate aura-registry (open/append/load/rank_by); CLI gains aura runs list / aura runs rank and aura sweep persists each point. Three iterations, first planner handoff = iteration 1 (foundation). Sign-off: spec_auto_sign was enabled for this cycle; the five-lens spec-skeptic panel returned 3 SOUND + 2 BLOCK (rank-ordering semantics; iteration-cut compile-ordering + Scalar->f64 coercion). Both BLOCKs were genuine defects, fixed before sign-off (rank is now best-first per-metric; the closure-type change lands with its callers in one green iteration; the coercion is shown). Non-unanimous panel routed the spec to human sign-off, granted by the user. refs #33 --- docs/specs/0029-run-registry-sweep-family.md | 353 +++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 docs/specs/0029-run-registry-sweep-family.md diff --git a/docs/specs/0029-run-registry-sweep-family.md b/docs/specs/0029-run-registry-sweep-family.md new file mode 100644 index 0000000..9d0e6eb --- /dev/null +++ b/docs/specs/0029-run-registry-sweep-family.md @@ -0,0 +1,353 @@ +# Run Registry — index & compare a sweep family — Design Spec + +**Date:** 2026-06-10 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Cycle D of the milestone *The World — parameter-space & sweep* (#33). A sweep +(cycle C) already produces a family of disjoint runs. This cycle makes that +family — and runs across separate invocations — **persistent, listable, and +comparable over time**: the substance of C18's run registry, which has no home +in git or Gitea. + +First cut (this cycle): **persist + list + rank**. Each run's `(manifest, +metrics)` is appended as one record to a `runs/` store; a read surface lists all +stored records and ranks them by a named metric. Determinism (C1/C12) makes the +record a faithful, reproducible descriptor of its run. + +Deferred to later iterations / follow-up issues (named here so the cut is +honest): promotion/status labels, the lineage graph (composite←signals, +run←inputs), run-diff, on-demand re-derivation of full results, a `run_id` +content digest, and `Aura.toml` runs-dir wiring. + +This cycle also makes a **contract-level change**: it amends C16 (see +*Architecture → C16 amendment*). + +## Architecture + +### C16 amendment (load-bearing, user-ratified) + +C16 today carries a blanket clause: the engine workspace is +*"zero-external-dependency by commitment"*, with `aura-ingest` as the sole +*"external-dependency firewall"*. That blanket is **struck** and replaced by a +**considered, per-case dependency policy**: + +> Dependencies are admitted by deliberate, per-case review — what a crate pulls +> in is weighed against what it buys — with **particular scrutiny for anything +> that enters the frozen deploy artifact** (C13: this bot = this commit). +> Well-established standard crates (`serde`, `rayon`, …) pass that review and are +> used wherever they do the job, including in the bot. There is **no blanket +> zero-dependency commitment** and **no blanket admission**; hand-rolling what a +> vetted standard crate already does is the anti-pattern, not the dependency. + +What of C16 is **unchanged**: the engine/project split and the three-tier node +reuse (`aura-std` / shared node crates / project-local `nodes/`). `aura-ingest` +remains the data-source ingestion edge — it is simply no longer framed as the +*only* place an external crate may appear. + +C14's *"hand-rolled JSON"* rationale falls with the zero-dep clause. This cycle +keeps the `RunReport::to_json` writer (it now drives **all** stdout); `serde` is +added *alongside* it for the registry's storage/parse layer, and the +stdout→serde unification (one shape for stored and printed records) is a +follow-up. Two stdout consequences: + +- `aura run` stdout is **byte-stable** (still `run_sample().to_json()`). +- `aura sweep` stdout **changes shape**: the per-point hand-rolled + `sweep_point_to_json` (`{params, metrics}`) is **retired** in favour of each + point's `RunReport::to_json` (`{manifest, metrics}`) — a consistency win + (`aura run`, `aura sweep`, `aura runs` now all print the same RunReport shape) + and the natural consequence of `SweepPoint` carrying a full report. The + cycle-C params-only golden is updated accordingly. + +**Per-case justification for the dependencies this cycle admits** (the policy +requires one, sharper where it reaches the frozen artifact): + +- `serde` + `serde_json` (derive feature). *Buys:* a typed read-path for the + registry record (parse a stored line back into a `RunReport`) and idempotent + serialization — the exact capability ranking/comparison needs, and the thing + C14's writer-only JSON could never give. *Pulls in:* `serde`, `serde_derive`, + `serde_json`, `itoa`, `ryu`, `memchr` — a tiny, ubiquitous, deterministic, + heavily-audited closure. *Frozen-artifact reach:* the derives land on + `RunReport`/`RunManifest`/`RunMetrics` (in `aura-engine`) and `Timestamp` (in + `aura-core`), both linked by the frozen bot. Justified: `serde` is the + single most-vetted crate in the ecosystem and its output is deterministic + (C1-safe). + +### Where the registry lives + +A **new crate `aura-registry`** (recommended over an `aura-cli` module): C18 +calls the run registry a first-class concern. A dedicated crate keeps +`aura-engine` pure compute (no filesystem I/O in the run-loop crate) and is the +natural home as the registry grows (lineage, promotion/status, run-diff). It +depends on `aura-engine` for `RunReport` and on `serde_json` for storage/parse. + +### The family becomes self-describing + +`SweepPoint` today carries only `RunMetrics`. To index a family the registry +needs each point's full `(manifest, metrics)`. So `SweepPoint` carries a full +**`RunReport`** — closing the manifest-per-point gap cycle C explicitly deferred +to #33. The sweep's per-point closure returns `RunReport` instead of +`RunMetrics`; the engine builds nothing new (the manifest is caller-supplied, as +report.rs already documents). + +## Concrete code shapes + +### The user-facing program (acceptance-criterion evidence) + +```text +$ aura sweep +{"manifest":{"commit":"unknown","params":{"sma_cross.fast":2,"sma_cross.slow":4,"scale":0.5},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=0.0001)"},"metrics":{"total_pips":-0.13,"max_drawdown":0.17,"exposure_sign_flips":1}} +… three more points … +# each point now prints its full RunReport (manifest+metrics, like `aura run`) +# — cycle C printed {params,metrics}; AND each point is appended to runs/runs.jsonl + +$ aura runs list +# every record ever stored, across invocations, one per line (to_json shape) +{"manifest":{…fast=2,slow=4…},"metrics":{"total_pips":-0.13,…}} +… + +$ aura runs rank total_pips +# the substance of #33 — the family compared, BEST first (total_pips: higher-is-better → desc) +{"manifest":{…fast=3,slow=4…},"metrics":{"total_pips":-0.09,…}} +{"manifest":{…fast=2,slow=5…},"metrics":{"total_pips":-0.112,…}} +… + +$ aura runs rank max_drawdown +# best-first is per-metric: max_drawdown is lower-is-better → ascending (shallowest drawdown first) +{"manifest":{…fast=3,slow=4…},"metrics":{…"max_drawdown":0.09,…}} +… + +$ aura runs rank bogus_metric +aura: unknown metric 'bogus_metric' (known: total_pips, max_drawdown, exposure_sign_flips) +# exit code 2 — strict, like every other aura arg error +``` + +### Implementation shapes (secondary) + +**serde derives (before → after).** In `crates/aura-core/src/scalar.rs`: + +```rust +// before +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Timestamp(pub i64); +// after — newtype struct serializes transparently as its inner i64, +// matching to_json's [from,to] window shape +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize)] +pub struct Timestamp(pub i64); +``` + +In `crates/aura-engine/src/report.rs`, the same two derives are added to +`RunMetrics`, `RunManifest`, and `RunReport` (no field changes). `Scalar` is +**not** derived — the registry serializes `RunReport`, never a raw `Scalar`. + +**`SweepPoint` carries a report (before → after).** In +`crates/aura-engine/src/sweep.rs`: + +```rust +// before +pub struct SweepPoint { pub params: Vec, pub metrics: RunMetrics } +pub fn sweep(space: &GridSpace, run_one: F) -> SweepFamily +where F: Fn(&[Scalar]) -> RunMetrics + Sync { … } + +// after — the point keeps its typed enumeration coordinate AND the full result +pub struct SweepPoint { pub params: Vec, pub report: RunReport } +pub fn sweep(space: &GridSpace, run_one: F) -> SweepFamily +where F: Fn(&[Scalar]) -> RunReport + Sync { … } +``` + +The thread-fan-out internals are untouched except the collected element type +(`(usize, RunMetrics)` → `(usize, RunReport)`). `SweepFamily` is unchanged in +shape (`{ points: Vec }`). + +**The registry (new `crates/aura-registry/src/lib.rs`).** + +```rust +use std::path::{Path, PathBuf}; +use aura_engine::RunReport; + +/// An append-only run registry over a JSONL file: one serde_json line per +/// RunReport. The stored shape is serde's; display is the caller's concern +/// (the CLI prints via RunReport::to_json for stdout consistency). +pub struct Registry { path: PathBuf } + +impl Registry { + /// Bind to a JSONL path. No I/O; the file is created lazily on first append. + pub fn open(path: impl AsRef) -> Registry { Registry { path: path.as_ref().to_path_buf() } } + + /// Append one record as a single JSON line. Creates the file (and parent + /// dir) if absent. + pub fn append(&self, report: &RunReport) -> std::io::Result<()> { /* serde_json::to_string + writeln in append mode */ } + + /// Parse every non-empty line back into a typed RunReport, in file order. + /// A missing file is an empty registry (Ok(vec![])), not an error. + pub fn load(&self) -> Result, RegistryError> { /* read; serde_json::from_str per line */ } +} + +/// Sort reports **best-first** by a named metric. "Best" is per-metric, fixed by +/// each metric's meaning (spec 0009): `total_pips` higher-is-better → descending; +/// `max_drawdown` (a positive magnitude) and `exposure_sign_flips` (a churn +/// count) lower-is-better → ascending. So `aura runs rank ` always puts +/// the genuinely best run first, whichever metric. Ties keep file (insertion) +/// order — a stable sort over `load()`'s file-order vector. `f64` keys use +/// `partial_cmp` (total, since metrics are finite by construction). Unknown +/// metric is an error the CLI maps to exit 2. +pub fn rank_by(reports: Vec, metric: &str) -> Result, RegistryError> { … } + +#[derive(Debug)] +pub enum RegistryError { + Io(std::io::Error), + Parse { line: usize, source: serde_json::Error }, + UnknownMetric(String), +} +``` + +**CLI dispatch (before → after)** in `crates/aura-cli/src/main.rs`: + +```rust +// before +["sweep"] => print!("{}", sweep_report()), +// after — sweep_report now also persists each point; new read subcommands +["sweep"] => print!("{}", sweep_report()), // persists as a side effect +["runs", "list"] => print!("{}", runs_list(&default_registry())), +["runs", "rank", m] => match runs_rank(&default_registry(), m) { + Ok(s) => print!("{s}"), + Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); } +}, +``` + +The per-point sweep closure returns a `RunReport`. The manifest's +`params: Vec<(String, f64)>` is built by zipping the `param_space()` names onto +the point's `Vec` values, **coercing each `Scalar` to `f64`** — the +manifest's params are `f64`-typed, but a grid point carries `Scalar::I64` +(SMA lengths) and `Scalar::F64` (the scale): + +```rust +fn scalar_as_param_f64(s: &Scalar) -> f64 { + match s { + Scalar::I64(n) => *n as f64, // fast=2 -> 2.0 + Scalar::F64(f) => *f, // scale=0.5 -> 0.5 + // numeric tuning params only; Bool/Ts never appear in a tuning grid + other => unreachable!("non-numeric sweep param: {other:?}"), + } +} +``` + +The fixed manifest facts — `commit`/`window`/`seed`/`broker` — are exactly those +`run_sample` already builds. `sweep_report` (after iteration 3) appends each +`point.report` to `default_registry()` and prints it via `RunReport::to_json` +(the hand-rolled `sweep_point_to_json` is retired — the report renders itself). +`default_registry()` binds `runs/runs.jsonl` under cwd. + +## Components + +| Component | Crate / file | Change | +|---|---|---| +| C16 amendment | `docs/design/INDEX.md` | strike blanket zero-dep; record per-case policy | +| workspace deps | `Cargo.toml` | add `[workspace.dependencies]` with `serde` (derive) + `serde_json` | +| `Timestamp` serde | `crates/aura-core/src/scalar.rs` | add `Serialize`/`Deserialize` derives; `serde` dep | +| report serde | `crates/aura-engine/src/report.rs` | derives on `RunMetrics`/`RunManifest`/`RunReport`; `serde` dep | +| `SweepPoint` | `crates/aura-engine/src/sweep.rs` | `metrics: RunMetrics` → `report: RunReport`; closure `-> RunReport` | +| `aura-registry` | `crates/aura-registry/` (new) | `Registry::{open, append, load}` + `rank_by` + `RegistryError` | +| CLI subcommands | `crates/aura-cli/src/main.rs` | `sweep` persists; `runs list`; `runs rank `; USAGE | + +## Data flow + +``` +aura sweep + → GridSpace enumerates N points + → sweep(): per point, fresh blueprint → bootstrap_with_params → run + → drain sinks → summarize → RunReport{manifest, metrics} + → SweepFamily{ points: [SweepPoint{params, report}, …] } + → for each point: Registry::open("runs/runs.jsonl").append(&point.report) [side effect] + → stdout: each point.report.to_json() [full RunReport shape; cycle-C params-only golden updated] + +aura runs rank total_pips + → Registry::open("runs/runs.jsonl").load() -> Vec (serde_json::from_str per line) + → rank_by(reports, "total_pips") -> sorted desc + → stdout: each report.to_json() +``` + +## Error handling + +- **Append:** creates the file and its parent dir on first write; an I/O failure + surfaces as `RegistryError::Io` → the CLI prints `aura: ` and exits 2. +- **Missing registry on read:** `load()` returns `Ok(vec![])` — an empty + registry is not an error; `aura runs list` on a fresh tree prints nothing and + exits 0. +- **Malformed stored line:** `load()` returns `RegistryError::Parse { line, … }` + naming the 1-based line — a corrupt record is loud, never silently skipped. +- **Unknown metric:** `aura runs rank bogus` → `RegistryError::UnknownMetric` → + `aura: unknown metric 'bogus' (known: …)` and exit 2. +- **Non-finite f64:** `RunMetrics`/manifest params are finite by construction + (report.rs); `serde_json` would error on NaN/∞, which cannot occur here. +- **Strict argv:** `aura runs`, `aura runs rank` (no metric), `aura sweep extra`, + etc. are usage errors → `aura: ` and exit 2, consistent with the + existing whole-vector match. + +## Testing strategy + +- **`aura-core`:** a round-trip unit test — `Timestamp` serde survives + `to_string`→`from_str`. +- **`aura-engine`:** `RunReport` serde round-trip (a built report → JSON → back, + equal). `sweep.rs` tests updated for `point.report` (the closure now returns + `RunReport`); the existing determinism/odometer/distinct-metrics properties are + preserved, reading `point.report.metrics`. +- **`aura-registry`:** `append` then `load` round-trips N reports in order; + `load` on a missing path is `Ok(empty)`; a hand-written corrupt line makes + `load` return `Parse{line}`; `rank_by` orders by each known metric and errors + on an unknown one. Tests use a temp-dir path (no shared cwd state). +- **`aura-cli`:** process-level goldens with cwd set to a temp dir — + `aura sweep` writes 4 lines to `runs/runs.jsonl` and prints 4 odometer lines, + each now a full `RunReport` (`{"manifest":…,"metrics":…}`); the cycle-C + params-only `aura sweep` golden is updated to the RunReport shape. A second + `aura sweep` then `aura runs list` shows 8 records; `aura runs rank + total_pips` prints them highest-first; `aura runs rank bogus` exits 2. + +## Acceptance criteria + +Against aura's feature-acceptance (CLAUDE.md): the audience naturally reaches for +it, it measurably improves the loop, and it reintroduces no failure the core +constraints forbid. + +1. **Audience reaches for it.** Comparing experiments over time is the heart of + the research loop (C18). `aura runs rank total_pips` is the first time a + researcher can ask "which point in my sweep — or across my sweeps — was + best?" without re-running anything. The worked program above is that ask. +2. **Measurable improvement / removed redundancy.** Closes the manifest-per-point + gap cycle C deferred (the family becomes self-describing) and the typed + read-path gap (#17 — `serde` Deserialize *is* the typed handle). Retires the + hand-rolled `sweep_point_to_json`. +3. **No reintroduced failure class.** Determinism (C1) holds — the registry + stores a reproducible manifest; `serde_json` output is deterministic. The + sweep stays disjoint-parallel. The new dependency is admitted under the + amended C16's per-case review, justified above, not by a blanket waiver. + +**Recommended iteration cut** (first planner handoff = iteration 1): + +- **Iteration 1 — foundation.** C16 amendment in the ledger; `[workspace.dependencies]` + serde/serde_json; serde derives on the report types + `Timestamp`; round-trip + tests. No behaviour change to the CLI. Workspace green. +- **Iteration 2 — engine + registry + CLI closure rethread (lands together).** + `SweepPoint` carries `RunReport` (closure bound `-> RunReport`). Changing that + bound breaks its **only non-test caller** — the CLI's `sweep_report` closure + (main.rs:280-289) — and the `sweep.rs` test callers; per the planner's + caller-threading rule, **all of them are rethreaded in this same iteration** so + `cargo test --workspace` stays green: the CLI closure now returns `RunReport` + and prints via `RunReport::to_json` (retiring `sweep_point_to_json`), and the + cycle-C `aura sweep` golden is updated to the RunReport shape here. New + `aura-registry` crate (`open`/`append`/`load`/`rank_by`) with unit tests. No + persistence side-effect yet, no new subcommands. Workspace green. +- **Iteration 3 — CLI persistence + read surface.** `aura sweep` appends each + point's report to `runs/runs.jsonl`; `aura runs list` / `aura runs rank + `; USAGE; process goldens (persist + list + rank + bogus-metric exit + 2). Workspace green. + +Each iteration leaves `cargo build/test --workspace` green: the signature-change +iteration (2) threads every caller it breaks, never deferring one past its own +build gate. + +**Out of scope** (follow-up issues / later iterations): promotion/status, +lineage graph, run-diff, on-demand re-derivation, `run_id` digest, `Aura.toml` +runs-dir wiring, and the `aura run`/`aura sweep` stdout→serde unification.