diff --git a/docs/plans/0029-run-registry-cli.md b/docs/plans/0029-run-registry-cli.md new file mode 100644 index 0000000..8b78d49 --- /dev/null +++ b/docs/plans/0029-run-registry-cli.md @@ -0,0 +1,433 @@ +# Run Registry — Iteration 3 (CLI persistence + read surface) — Implementation Plan + +> **Parent spec:** `docs/specs/0029-run-registry-sweep-family.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Wire `aura-registry` into the CLI: `aura sweep` persists each point's +`RunReport` to `runs/runs.jsonl`; `aura runs list` prints every stored record; +`aura runs rank ` prints them best-first; with process-level goldens. + +**Architecture:** `sweep_report` is split — a production `sweep_family() -> +SweepFamily` (the pure run) and a `#[cfg(test)] sweep_report() -> String` +renderer kept for the in-bin tests. The `["sweep"]` arm becomes `run_sweep`, +which persists each point via `default_registry()` and prints it. Two new arms, +`["runs","list"]` and `["runs","rank", metric]`, read the registry back (typed, +via serde) and print via `RunReport::to_json`. Process tests run the binary in a +temp cwd so persistence never dirties the repo; `/runs/` is git-ignored as a +backstop. + +**Tech Stack:** `aura-cli`, `aura-registry` (Registry, rank_by). + +--- + +**Files this plan creates or modifies:** + +- Modify: `.gitignore` — ignore `/runs/` +- Modify: `crates/aura-cli/Cargo.toml:12-15` — add the `aura-registry` dependency +- Modify: `crates/aura-cli/src/main.rs:12-15` — import `SweepFamily` + `aura_registry::{rank_by, Registry}` +- Modify: `crates/aura-cli/src/main.rs:225-271` — split `sweep_report` into `sweep_family` + `#[cfg(test)] sweep_report`; add `default_registry`/`run_sweep`/`runs_list`/`runs_rank`/`load_runs_or_exit` +- Modify: `crates/aura-cli/src/main.rs:429` — extend `USAGE` +- Modify: `crates/aura-cli/src/main.rs:440` — replace the `["sweep"]` arm + add `runs` arms +- Test: `crates/aura-cli/tests/cli_run.rs` — temp-cwd helper, update the sweep golden, add the runs-flow / empty / strictness tests + +--- + +### Task 1: Prep — git-ignore `/runs/` and add the registry dependency + +**Files:** +- Modify: `.gitignore` +- Modify: `crates/aura-cli/Cargo.toml:12-15` + +- [ ] **Step 1: Ignore the default registry path** + +In `.gitignore`, append: + +``` +# `aura sweep` persists run records to ./runs/runs.jsonl by default; that store +# is local run telemetry, not a repo artifact. +/runs/ +``` + +- [ ] **Step 2: Add the aura-registry dependency** + +In `crates/aura-cli/Cargo.toml`, replace the `[dependencies]` block (lines +12-15): + +```toml +[dependencies] +aura-core = { path = "../aura-core" } +aura-engine = { path = "../aura-engine" } +aura-std = { path = "../aura-std" } +``` + +with: + +```toml +[dependencies] +aura-core = { path = "../aura-core" } +aura-engine = { path = "../aura-engine" } +aura-registry = { path = "../aura-registry" } +aura-std = { path = "../aura-std" } +``` + +- [ ] **Step 3: Verify the workspace still builds** + +Run: `cargo build --workspace` +Expected: builds green, exit 0 (the new dependency is not yet used — that is +fine; `unused_crate_dependencies` is allow-by-default). + +--- + +### Task 2: CLI wiring — persist on sweep, list + rank subcommands (RED-first) + +**Files:** +- Modify: `crates/aura-cli/tests/cli_run.rs` +- Modify: `crates/aura-cli/src/main.rs` + +- [ ] **Step 1: Write the failing process tests** + +In `crates/aura-cli/tests/cli_run.rs`, add the temp-cwd helper just after the +`BIN` constant (line 8): + +```rust +/// A fresh, unique working directory for a process test that persists run +/// records (so `aura sweep`'s `./runs/runs.jsonl` never dirties the repo). +/// Unique per test + per process; no external tempfile dependency. +fn temp_cwd(name: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!("aura-cli-{}-{}", std::process::id(), name)); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("create temp cwd"); + dir +} +``` + +Then replace `sweep_prints_four_json_lines_and_exits_zero` (the body up to and +including its closing `}`) — it must now run in a temp cwd, since `aura sweep` +persists: + +```rust +#[test] +fn sweep_prints_four_json_lines_and_exits_zero() { + let cwd = temp_cwd("sweep4"); + let out = Command::new(BIN).arg("sweep").current_dir(&cwd).output().expect("spawn aura sweep"); + assert!(out.status.success(), "exit status: {:?}", out.status); + + let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout"); + assert_eq!(stdout.lines().count(), 4, "stdout was: {stdout:?}"); + let first = stdout.lines().next().unwrap(); + // each line is a full RunReport; commit is the real git HEAD (volatile), so + // pin the first odometer point's manifest params, not the commit value. + assert!(first.starts_with("{\"manifest\":{\"commit\":\""), "got: {stdout}"); + assert!( + first.contains("\"params\":{\"sma_cross.fast\":2,\"sma_cross.slow\":4,\"scale\":0.5}"), + "got: {stdout}" + ); + let _ = std::fs::remove_dir_all(&cwd); +} +``` + +Then add the new tests at the end of the file: + +```rust +#[test] +fn runs_list_and_rank_across_invocations() { + let cwd = temp_cwd("runs-flow"); + // two sweeps accumulate 8 records over time + for _ in 0..2 { + let out = Command::new(BIN).arg("sweep").current_dir(&cwd).output().expect("spawn sweep"); + assert!(out.status.success(), "sweep exit: {:?}", out.status); + } + + // `runs list` shows all 8 stored records + let list = Command::new(BIN).args(["runs", "list"]).current_dir(&cwd).output().expect("spawn list"); + assert!(list.status.success(), "list exit: {:?}", list.status); + let list_out = String::from_utf8(list.stdout).expect("utf-8"); + assert_eq!(list_out.lines().count(), 8, "list: {list_out:?}"); + assert!(list_out.lines().all(|l| l.starts_with("{\"manifest\":{")), "list shape: {list_out:?}"); + + // `runs rank total_pips`: 8 lines, highest total_pips first + let rank = Command::new(BIN).args(["runs", "rank", "total_pips"]).current_dir(&cwd).output().expect("spawn rank"); + assert!(rank.status.success(), "rank exit: {:?}", rank.status); + let rank_out = String::from_utf8(rank.stdout).expect("utf-8"); + assert_eq!(rank_out.lines().count(), 8, "rank: {rank_out:?}"); + let pips: Vec = rank_out + .lines() + .map(|l| { + let key = "\"total_pips\":"; + let start = l.find(key).expect("line has total_pips") + key.len(); + let tail = &l[start..]; + let end = tail.find([',', '}']).expect("token end"); + tail[..end].parse().expect("total_pips is an f64") + }) + .collect(); + assert!(pips.windows(2).all(|w| w[0] >= w[1]), "rank not descending: {pips:?}"); + + // unknown metric is a usage error + let bogus = Command::new(BIN).args(["runs", "rank", "bogus"]).current_dir(&cwd).output().expect("spawn bogus"); + assert_eq!(bogus.status.code(), Some(2), "bogus exit: {:?}", bogus.status); + + let _ = std::fs::remove_dir_all(&cwd); +} + +#[test] +fn runs_list_on_empty_registry_is_zero_lines_exit_zero() { + let cwd = temp_cwd("runs-empty"); + let out = Command::new(BIN).args(["runs", "list"]).current_dir(&cwd).output().expect("spawn list"); + assert!(out.status.success(), "list exit: {:?}", out.status); + assert_eq!(out.stdout.len(), 0, "expected no output, got: {:?}", out.stdout); + let _ = std::fs::remove_dir_all(&cwd); +} + +#[test] +fn runs_bare_subcommand_is_strict_and_exits_two() { + // `aura runs` with no/unknown sub-subcommand is a usage error (#16 strict). + let out = Command::new(BIN).arg("runs").output().expect("spawn aura runs"); + assert_eq!(out.status.code(), Some(2), "exit status: {:?}", out.status); +} +``` + +- [ ] **Step 2: Run the new tests to verify they fail** + +Run: `cargo test -p aura-cli --test cli_run runs_` +Expected: FAIL — `runs_list_and_rank_across_invocations` and +`runs_list_on_empty_registry_is_zero_lines_exit_zero` fail because the current +binary has no `runs` arm, so `aura runs list` falls through to the usage-error +path and exits 2 (the success/line-count assertions fail). +(`runs_bare_subcommand_is_strict_and_exits_two` already passes — `aura runs` is +already a usage error today.) + +- [ ] **Step 3: Add the imports** + +In `crates/aura-cli/src/main.rs`, replace the import block (lines 12-15): + +```rust +use aura_engine::{ + f64_field, summarize, sweep, BlueprintNode, Composite, Edge, FlatGraph, GridSpace, Harness, + OutField, ParamAlias, Role, RunManifest, RunReport, SourceSpec, Target, +}; +use aura_std::{Ema, Exposure, Recorder, SimBroker, Sma, Sub}; +``` + +with (add `SweepFamily` to the engine list; add the registry import): + +```rust +use aura_engine::{ + f64_field, summarize, sweep, BlueprintNode, Composite, Edge, FlatGraph, GridSpace, Harness, + OutField, ParamAlias, Role, RunManifest, RunReport, SourceSpec, SweepFamily, Target, +}; +use aura_registry::{rank_by, Registry}; +use aura_std::{Ema, Exposure, Recorder, SimBroker, Sma, Sub}; +``` + +- [ ] **Step 4: Split `sweep_report` into `sweep_family` + a test renderer** + +In `crates/aura-cli/src/main.rs`, replace the whole `sweep_report` function +(from `fn sweep_report() -> String {` at line 225 through its closing `}` at +line 271 — keep the doc comment at line 224 above it): + +```rust +fn sweep_report() -> String { + let space = sample_blueprint_with_sinks().0.param_space(); + let grid = GridSpace::new( + &space, + vec![ + vec![Scalar::I64(2), Scalar::I64(3)], // fast ∈ {2, 3} + vec![Scalar::I64(4), Scalar::I64(5)], // slow ∈ {4, 5} + vec![Scalar::F64(0.5)], // scale ∈ {0.5} + ], + ) + .expect("the built-in grid matches the sample param-space"); + let family = sweep(&grid, |point| { + let (bp, rx_eq, rx_ex) = sample_blueprint_with_sinks(); + let mut h = bp + .bootstrap_with_params(point.to_vec()) + .expect("grid points are kind-checked against param_space"); + let prices = synthetic_prices(); + let window = ( + prices.first().expect("non-empty stream").0, + prices.last().expect("non-empty stream").0, + ); + h.run(vec![prices]); + let equity = f64_field(&rx_eq.try_iter().collect::>(), 0); + let exposure = f64_field(&rx_ex.try_iter().collect::>(), 0); + let params = space + .iter() + .zip(point) + .map(|(ps, v)| (ps.name.clone(), scalar_as_param_f64(v))) + .collect(); + RunReport { + manifest: RunManifest { + commit: option_env!("AURA_COMMIT").unwrap_or("unknown").to_string(), + params, + window, + seed: 0, + broker: "sim-optimal(pip_size=0.0001)".to_string(), + }, + metrics: summarize(&equity, &exposure), + } + }); + let mut out = String::new(); + for pt in &family.points { + out.push_str(&pt.report.to_json()); + out.push('\n'); + } + out +} +``` + +with (the run becomes the production `sweep_family`; the renderer stays as a +`#[cfg(test)]` helper so it is not dead code in a normal build): + +```rust +fn sweep_family() -> SweepFamily { + let space = sample_blueprint_with_sinks().0.param_space(); + let grid = GridSpace::new( + &space, + vec![ + vec![Scalar::I64(2), Scalar::I64(3)], // fast ∈ {2, 3} + vec![Scalar::I64(4), Scalar::I64(5)], // slow ∈ {4, 5} + vec![Scalar::F64(0.5)], // scale ∈ {0.5} + ], + ) + .expect("the built-in grid matches the sample param-space"); + sweep(&grid, |point| { + let (bp, rx_eq, rx_ex) = sample_blueprint_with_sinks(); + let mut h = bp + .bootstrap_with_params(point.to_vec()) + .expect("grid points are kind-checked against param_space"); + let prices = synthetic_prices(); + let window = ( + prices.first().expect("non-empty stream").0, + prices.last().expect("non-empty stream").0, + ); + h.run(vec![prices]); + let equity = f64_field(&rx_eq.try_iter().collect::>(), 0); + let exposure = f64_field(&rx_ex.try_iter().collect::>(), 0); + let params = space + .iter() + .zip(point) + .map(|(ps, v)| (ps.name.clone(), scalar_as_param_f64(v))) + .collect(); + RunReport { + manifest: RunManifest { + commit: option_env!("AURA_COMMIT").unwrap_or("unknown").to_string(), + params, + window, + seed: 0, + broker: "sim-optimal(pip_size=0.0001)".to_string(), + }, + metrics: summarize(&equity, &exposure), + } + }) +} + +/// Render a sweep family as one `RunReport` JSON line per point. Test helper: +/// production (`run_sweep`) renders *and* persists per point. +#[cfg(test)] +fn sweep_report() -> String { + let mut out = String::new(); + for pt in &sweep_family().points { + out.push_str(&pt.report.to_json()); + out.push('\n'); + } + out +} + +/// The default run registry: an append-only JSONL store under the current +/// working directory. (A project-configured runs-dir via `Aura.toml` is a later +/// refinement.) +fn default_registry() -> Registry { + Registry::open("runs/runs.jsonl") +} + +/// `aura sweep`: run the built-in sweep, persist each point's `RunReport` to the +/// registry (the run record, queryable over time — C18), and print each as one +/// JSON line. +fn run_sweep() { + let reg = default_registry(); + for pt in &sweep_family().points { + if let Err(e) = reg.append(&pt.report) { + eprintln!("aura: {e}"); + std::process::exit(2); + } + println!("{}", pt.report.to_json()); + } +} + +/// `aura runs list`: print every stored run record, in store (over-time) order. +fn runs_list() { + for report in &load_runs_or_exit() { + println!("{}", report.to_json()); + } +} + +/// `aura runs rank `: print the stored runs best-first by `metric`. +fn runs_rank(metric: &str) { + match rank_by(load_runs_or_exit(), metric) { + Ok(ranked) => { + for report in &ranked { + println!("{}", report.to_json()); + } + } + Err(e) => { + eprintln!("aura: {e}"); + std::process::exit(2); + } + } +} + +/// Load the registry or exit 2 with the error. A missing registry loads empty. +fn load_runs_or_exit() -> Vec { + default_registry().load().unwrap_or_else(|e| { + eprintln!("aura: {e}"); + std::process::exit(2); + }) +} +``` + +- [ ] **Step 5: Extend USAGE and the dispatch arms** + +In `crates/aura-cli/src/main.rs`, replace the `USAGE` constant (line 429): + +```rust +const USAGE: &str = "usage: aura run [--macd] | aura graph | aura sweep"; +``` + +with: + +```rust +const USAGE: &str = + "usage: aura run [--macd] | aura graph | aura sweep | aura runs list | aura runs rank "; +``` + +Then replace the `["sweep"]` dispatch arm (line 440): + +```rust + ["sweep"] => print!("{}", sweep_report()), +``` + +with: + +```rust + ["sweep"] => run_sweep(), + ["runs", "list"] => runs_list(), + ["runs", "rank", metric] => runs_rank(metric), +``` + +- [ ] **Step 6: Run the new tests to verify they pass** + +Run: `cargo test -p aura-cli --test cli_run runs_` +Expected: PASS — `runs_list_and_rank_across_invocations`, +`runs_list_on_empty_registry_is_zero_lines_exit_zero`, and +`runs_bare_subcommand_is_strict_and_exits_two` all green. + +- [ ] **Step 7: Full workspace gate** + +Run: `cargo test --workspace` +Expected: PASS — all crates green, including the updated `aura sweep` golden and +the in-bin `sweep_report_*` tests (still calling the `#[cfg(test)] sweep_report`). + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: no warnings (no dead `sweep_report`; no unused imports).