diff --git a/docs/plans/0075-walkforward-mc-r-validation.md b/docs/plans/0075-walkforward-mc-r-validation.md new file mode 100644 index 0000000..53470c3 --- /dev/null +++ b/docs/plans/0075-walkforward-mc-r-validation.md @@ -0,0 +1,768 @@ +# Walk-forward strategy-selectable + R-reporting (iteration 1) — Implementation Plan + +> **Parent spec:** `docs/specs/0075-walkforward-mc-r-validation.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Make `aura walkforward --strategy stage1-r --real ` roll IS→OOS +windows, sweep the stage1-r grid in-sample, pick the best by an R metric, run the +chosen params OOS, and report per-window + across-window-pooled OOS R-metrics — +while the bare SMA `walkforward` path stays byte-identical. + +**Architecture:** Two additive library changes in `aura-engine` (`RMetrics` carries +an in-memory `trade_rs` vector excluded from serde + equality; a new +`r_metrics_from_rs` pooled reducer), then `aura-cli` gains `--strategy` + grid +flags on `walkforward`, a strategy-dispatched `walkforward_family` (SMA arm +verbatim), windowed stage1-r helpers (`stage1_r_sweep_over` reduce-mode IS sweep, +`run_oos_r` non-reduce OOS run, `stage1_r_space`), and a pooled `oos_r` summary +block. The Monte-Carlo bootstrap half is iteration 2 (separate plan). + +**Tech Stack:** Rust workspace — `aura-engine` (`report.rs`), `aura-registry` +(`lib.rs` test helper), `aura-cli` (`main.rs`). Test command: +`cargo test --workspace` (one positional filter per invocation). + +--- + +**Files this plan creates or modifies:** + +- Modify: `crates/aura-engine/src/report.rs:44-62` — `RMetrics` gains + `#[serde(skip)] trade_rs: Vec`; drop `PartialEq` from the derive, hand-write + it (ignoring `trade_rs`). +- Modify: `crates/aura-engine/src/report.rs:89-219` — `summarize_r` sets `trade_rs` + in both return arms; add new `pub fn r_metrics_from_rs`. +- Modify: `crates/aura-registry/src/lib.rs:255` — `report_with_r` test helper + `RMetrics { .. }` literal gains `trade_rs: Vec::new()`. +- Modify: `crates/aura-cli/src/main.rs:1526-1546` — `parse_walkforward_args` 3-tuple + → 5-tuple (`--strategy` + four grid flags). +- Modify: `crates/aura-cli/src/main.rs:1615-1666` — `run_walkforward` + + `walkforward_family` gain `strategy`/`grid`; strategy dispatch. +- Modify: `crates/aura-cli/src/main.rs:1734-1744` — `walkforward_summary_json` gains + the `oos_r` pooled block. +- Modify: `crates/aura-cli/src/main.rs:2591-2592` — USAGE walkforward fragment. +- Modify: `crates/aura-cli/src/main.rs:2637-2645` — walkforward dispatch arm + destructures the 5-tuple. +- Create (new fns in `crates/aura-cli/src/main.rs`): `stage1_r_sweep_over`, + `run_oos_r`, `stage1_r_space`. +- Test: `crates/aura-engine/src/report.rs` (#cfg test) — `summarize_r` populates + `trade_rs`; `RMetrics` round-trip equal despite skipped `trade_rs`; + `r_metrics_from_rs` goldens. +- Test: `crates/aura-cli/src/main.rs` (#cfg test) — `parse_walkforward_args` + 5-tuple + `--strategy`/grid; update the two existing assertions at 3653-3656. +- Test: `crates/aura-cli/tests/cli_run.rs` — `aura walkforward --strategy stage1-r` + emits per-window `r` blocks + an `oos_r` summary (synthetic). + +--- + +### Task 1: `RMetrics.trade_rs` — in-memory per-trade R series + +**Files:** +- Modify: `crates/aura-engine/src/report.rs:44-62, 89-135, 206-219` +- Modify: `crates/aura-registry/src/lib.rs:255` +- Test: `crates/aura-engine/src/report.rs` (#cfg test) + +- [ ] **Step 1: Write the failing tests** — append to the `#[cfg(test)] mod tests` + in `crates/aura-engine/src/report.rs`: + +```rust +#[test] +fn summarize_r_populates_trade_rs_in_trade_order() { + // two closed trades (R = +2.0, then -1.0) over a minimal PositionManagement + // record; trade_rs must carry [2.0, -1.0] in trade order. + let rec = pm_record_two_closed_trades(); // helper below + let m = summarize_r(&rec, 0.0); + assert_eq!(m.trade_rs, vec![2.0, -1.0]); + assert_eq!(m.n_trades, 2); +} + +#[test] +fn summarize_r_empty_record_has_empty_trade_rs() { + let m = summarize_r(&[], 0.0); + assert!(m.trade_rs.is_empty()); + assert_eq!(m.n_trades, 0); +} + +#[test] +fn rmetrics_partial_eq_ignores_trade_rs() { + // two RMetrics equal in every metric but differing in trade_rs compare EQUAL + // (trade_rs is an in-memory conduit, excluded from equality) — this is what + // keeps serialize->deserialize round-trips equal (trade_rs is serde-skipped, + // so it deserializes empty). + let a = summarize_r(&pm_record_two_closed_trades(), 0.0); + let mut b = a.clone(); + b.trade_rs = Vec::new(); + assert_eq!(a, b); +} + +/// A minimal dense PositionManagement record with two closed trades at R = +2, -1. +/// Columns per `r_col` (CLOSED=0, REALIZED_R=1, DIRECTION=4, ENTRY_PRICE=6, +/// STOP_PRICE=7, CONVICTION_AT_ENTRY=9, SIZE=10, OPEN=11, UNREALIZED_R=12); width 13. +fn pm_record_two_closed_trades() -> Vec<(Timestamp, Vec)> { + let row = |closed: bool, r: f64, open: bool| { + let mut c = vec![Scalar::f64(0.0); 13]; + c[0] = Scalar::bool(closed); + c[1] = Scalar::f64(r); + c[6] = Scalar::f64(1.0); // entry + c[7] = Scalar::f64(0.5); // stop -> latched 0.5 + c[9] = Scalar::f64(0.3); // conviction + c[11] = Scalar::bool(open); + c + }; + vec![ + (Timestamp(1), row(true, 2.0, false)), + (Timestamp(2), row(true, -1.0, false)), + ] +} +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace summarize_r_populates_trade_rs_in_trade_order` +Expected: FAIL — compile error `no field `trade_rs` on type `RMetrics`` (and the +two siblings fail to compile likewise). + +- [ ] **Step 3: Add the field + hand-written `PartialEq`** — in + `crates/aura-engine/src/report.rs`, change the `RMetrics` derive (line 44) to drop + `PartialEq`, add the field after `conviction_terciles_r` (line 61), and add the + `impl PartialEq` directly after the struct: + +```rust +// line 44: drop PartialEq from the derive +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] +pub struct RMetrics { + // ... all existing fields unchanged ... + pub conviction_terciles_r: [f64; 3], + /// Realised R per closed trade, in trade order — an in-memory conduit for the + /// OOS R-series bootstrap (#139). Excluded from serde (`skip`) so the C18 wire + /// shape is unchanged, and from `PartialEq` (below) so every existing + /// `RMetrics`/`RunReport` equality assertion and round-trip stays green. + #[serde(skip)] + pub trade_rs: Vec, +} + +impl PartialEq for RMetrics { + fn eq(&self, o: &Self) -> bool { + self.expectancy_r == o.expectancy_r + && self.n_trades == o.n_trades + && self.win_rate == o.win_rate + && self.avg_win_r == o.avg_win_r + && self.avg_loss_r == o.avg_loss_r + && self.profit_factor == o.profit_factor + && self.max_r_drawdown == o.max_r_drawdown + && self.n_open_at_end == o.n_open_at_end + && self.sqn == o.sqn + && self.sqn_normalized == o.sqn_normalized + && self.net_expectancy_r == o.net_expectancy_r + && self.conviction_terciles_r == o.conviction_terciles_r + // trade_rs deliberately excluded + } +} +``` + +- [ ] **Step 4: Populate `trade_rs` in both `summarize_r` arms** — in + `crates/aura-engine/src/report.rs`: the empty-input arm (line 122-135 literal) adds + `trade_rs: Vec::new(),`, and the main return literal (line 206-219) adds + `trade_rs: rs,` (the `rs` vector built at line 137 is the per-trade realised R in + trade order; it is last-read by the sqn block before the return, so the move is + valid): + +```rust +// empty-input arm (line 122) — add as the last field: + return RMetrics { + // ... existing zero fields ... + conviction_terciles_r: [0.0; 3], + trade_rs: Vec::new(), + }; +// main return (line 206) — add as the last field: + RMetrics { + // ... existing fields ... + conviction_terciles_r, + trade_rs: rs, + } +``` + +- [ ] **Step 5: Thread the other two `RMetrics { .. }` literal sites** (the only + remaining struct literals workspace-wide — both test helpers): + +```rust +// crates/aura-engine/src/report.rs — the runmetrics_with_r_block_round_trips test +// literal (~line 1135): add `trade_rs: Vec::new(),` as the last field. +// crates/aura-registry/src/lib.rs:255 — the report_with_r helper literal: add +// `trade_rs: Vec::new(),` as the last field. +``` + +- [ ] **Step 6: Build gate (all literal sites threaded)** + +Run: `cargo build --workspace` +Expected: 0 errors (every `RMetrics { .. }` literal now carries `trade_rs`). + +- [ ] **Step 7: Run the new + back-compat tests** + +Run: `cargo test --workspace summarize_r_populates_trade_rs_in_trade_order` +Expected: PASS +Run: `cargo test --workspace rmetrics_partial_eq_ignores_trade_rs` +Expected: PASS +Run: `cargo test --workspace runmetrics_with_r_block_round_trips` +Expected: PASS (the serde-skip + hand-written eq keep the round-trip equal) + +### Task 2: `r_metrics_from_rs` — pooled reducer over a flat R series + +**Files:** +- Modify: `crates/aura-engine/src/report.rs` (new `pub fn` after `summarize_r`) +- Test: `crates/aura-engine/src/report.rs` (#cfg test) + +- [ ] **Step 1: Write the failing test** — append to the report.rs test module: + +```rust +#[test] +fn r_metrics_from_rs_folds_a_flat_series() { + // pooled across-window R series [2.0, -1.0, 1.0]: expectancy = 2/3, 2 wins of 3, + // profit_factor = (2+1)/1 = 3. At cost 0 (frictionless Stage-1) net == gross. + // conviction terciles are not pooled -> [0,0,0]; n_open_at_end is not a pooled + // concept -> 0. + let m = r_metrics_from_rs(&[2.0, -1.0, 1.0]); + assert_eq!(m.n_trades, 3); + assert!((m.expectancy_r - 2.0 / 3.0).abs() < 1e-12); + assert!((m.win_rate - 2.0 / 3.0).abs() < 1e-12); + assert!((m.profit_factor - 3.0).abs() < 1e-12); + assert_eq!(m.net_expectancy_r, m.expectancy_r); + assert_eq!(m.conviction_terciles_r, [0.0; 3]); + assert_eq!(m.n_open_at_end, 0); +} + +#[test] +fn r_metrics_from_rs_empty_is_all_zero() { + let m = r_metrics_from_rs(&[]); + assert_eq!(m.n_trades, 0); + assert_eq!(m.expectancy_r, 0.0); + assert_eq!(m.sqn, 0.0); +} +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace r_metrics_from_rs_folds_a_flat_series` +Expected: FAIL — `cannot find function `r_metrics_from_rs` in this scope`. + +- [ ] **Step 3: Add the reducer** — in `crates/aura-engine/src/report.rs`, directly + after `summarize_r` (line 220). The R-distribution arithmetic is copied + **verbatim** from `summarize_r` (lines 137-177) — the byte-pinned float + expressions must not be algebraically refactored (a shared factoring would shift + the golden by ~1 ULP); `summarize_r` itself is left untouched: + +```rust +/// Reduce a flat per-trade R series (e.g. the pooled across-window OOS series of a +/// walk-forward) into `RMetrics`. The R-distribution fields use the same arithmetic +/// as [`summarize_r`] (copied verbatim — the pinned floats are not refactored). The +/// fields a flat R series cannot carry are set honestly: `n_open_at_end = 0`, +/// `net_expectancy_r = expectancy_r` (exact under the Stage-1 cost = 0 invariant), +/// `conviction_terciles_r = [0,0,0]` (per-trade conviction is not pooled). Empty +/// input -> a well-defined all-zero `RMetrics`. +pub fn r_metrics_from_rs(rs: &[f64]) -> RMetrics { + let n = rs.len() as u64; + if n == 0 { + return RMetrics { + expectancy_r: 0.0, n_trades: 0, win_rate: 0.0, avg_win_r: 0.0, + avg_loss_r: 0.0, profit_factor: 0.0, max_r_drawdown: 0.0, + n_open_at_end: 0, sqn: 0.0, sqn_normalized: 0.0, net_expectancy_r: 0.0, + conviction_terciles_r: [0.0; 3], trade_rs: Vec::new(), + }; + } + let sum: f64 = rs.iter().sum(); + let mean = sum / n as f64; + let wins: Vec = rs.iter().copied().filter(|&r| r > 0.0).collect(); + let losses: Vec = rs.iter().copied().filter(|&r| r <= 0.0).collect(); + let sum_win: f64 = wins.iter().sum(); + let sum_loss: f64 = losses.iter().sum(); + let avg = |v: &[f64]| if v.is_empty() { 0.0 } else { v.iter().sum::() / v.len() as f64 }; + let mut peak = f64::NEG_INFINITY; + let mut cum = 0.0; + let mut max_dd = 0.0_f64; + for &r in rs { + cum += r; + if cum > peak { peak = cum; } + let dd = peak - cum; + if dd > max_dd { max_dd = dd; } + } + let (sqn, sqn_normalized) = if n < 2 { + (0.0, 0.0) + } else { + let var = rs.iter().map(|&r| (r - mean).powi(2)).sum::() / (n as f64 - 1.0); + let sd = var.sqrt(); + if sd > 0.0 { + ((n as f64).sqrt() * mean / sd, (n.min(SQN_CAP) as f64).sqrt() * mean / sd) + } else { + (0.0, 0.0) + } + }; + RMetrics { + expectancy_r: mean, + n_trades: n, + win_rate: wins.len() as f64 / n as f64, + avg_win_r: avg(&wins), + avg_loss_r: avg(&losses), + profit_factor: if sum_loss < 0.0 { sum_win / (-sum_loss) } else { 0.0 }, + max_r_drawdown: max_dd, + n_open_at_end: 0, + sqn, + sqn_normalized, + net_expectancy_r: mean, // cost = 0 -> net == gross (Stage-1 frictionless) + conviction_terciles_r: [0.0; 3], + trade_rs: Vec::new(), + } +} +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `cargo test --workspace r_metrics_from_rs_folds_a_flat_series` +Expected: PASS +Run: `cargo test --workspace r_metrics_from_rs_empty_is_all_zero` +Expected: PASS + +### Task 3: `walkforward` signature plumbing — `--strategy` + grid flags + +This task threads the strategy/grid through `parse_walkforward_args` → +`run_walkforward` → `walkforward_family` (SMA arm **verbatim**; stage1-r arm a +graceful error stub filled in Task 4) and every call site, so the workspace +compiles and all SMA goldens stay green. + +**Files:** +- Modify: `crates/aura-cli/src/main.rs:1526-1546, 1607-1666, 2591-2592, 2637-2645` +- Test: `crates/aura-cli/src/main.rs` (#cfg test, lines 3651-3660 updated + a new + case) + +- [ ] **Step 1: Write/adjust the failing parse test** — in the report-less + `crates/aura-cli/src/main.rs` test module, replace the body of + `parse_walkforward_args_defaults_and_accepts_real` (3652-3660) to expect the + 5-tuple, and add a stage1-r case: + +```rust +#[test] +fn parse_walkforward_args_defaults_and_accepts_real() { + assert_eq!( + parse_walkforward_args(&[]), + Ok((Strategy::SmaCross, "walkforward".to_string(), false, DataChoice::Synthetic, Stage1RGrid::default())) + ); + assert_eq!( + parse_walkforward_args(&["--real", "EURUSD", "--trace", "w"]), + Ok((Strategy::SmaCross, "w".to_string(), true, + DataChoice::Real { symbol: "EURUSD".to_string(), from_ms: None, to_ms: None }, + Stage1RGrid::default())) + ); + assert!(parse_walkforward_args(&["--name", "a", "--trace", "b"]).is_err()); + assert!(parse_walkforward_args(&["--real"]).is_err()); +} + +#[test] +fn parse_walkforward_args_accepts_strategy_and_grid_flags() { + let r = parse_walkforward_args(&["--strategy", "stage1-r", "--fast", "5,10", "--stop-k", "2.0,3.0"]); + let (strategy, _, _, _, grid) = r.expect("valid stage1-r walkforward args"); + assert_eq!(strategy, Strategy::Stage1R); + assert_eq!(grid.fast, vec![5, 10]); + assert_eq!(grid.stop_k, vec![2.0, 3.0]); + assert!(parse_walkforward_args(&["--strategy", "bogus"]).is_err()); +} +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace parse_walkforward_args_accepts_strategy_and_grid_flags` +Expected: FAIL — compile error (the 5-tuple / `Strategy` return type does not exist +yet; existing 3-tuple assertions also fail to compile). + +- [ ] **Step 3: Extend `parse_walkforward_args`** — replace lines 1523-1546 with the + 5-tuple grammar mirroring `parse_sweep_args` (1480-1521): + +```rust +/// Parse the `walkforward` tail: +/// `[--strategy ] [--real [--from ] [--to ]] [--name | --trace ] [--fast ] [--slow ] [--stop-length ] [--stop-k ] [--channel ] [--window ] [--band-k ]`. +/// Defaults: SMA-cross, synthetic, name "walkforward", no persist. `--name`/`--trace` +/// mutually exclusive; `--from`/`--to` require `--real`. Pure (no I/O / exit). +fn parse_walkforward_args( + rest: &[&str], +) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid), String> { + let usage = || "walkforward [--strategy ] [--real [--from ] [--to ]] [--name | --trace ] [--fast ] [--slow ] [--stop-length ] [--stop-k ] [--channel ] [--window ] [--band-k ]".to_string(); + let mut strategy = Strategy::SmaCross; + let mut name: Option<(String, bool)> = None; + let mut real = RealWindowGrammar::default(); + let mut grid = Stage1RGrid::default(); + let mut tail = rest; + while let Some((flag, t)) = tail.split_first() { + let (value, t) = t.split_first().ok_or_else(usage)?; + if real.accept(flag, value, &usage)? { + tail = t; + continue; + } + match *flag { + "--strategy" => { + strategy = match *value { + "sma" => Strategy::SmaCross, + "momentum" => Strategy::Momentum, + "stage1-r" => Strategy::Stage1R, + "stage1-breakout" => Strategy::Stage1Breakout, + "stage1-meanrev" => Strategy::Stage1MeanRev, + _ => return Err(usage()), + }; + } + "--name" if name.is_none() => name = Some(((*value).to_string(), false)), + "--trace" if name.is_none() => name = Some(((*value).to_string(), true)), + "--fast" => grid.fast = parse_csv_list(value).map_err(|()| usage())?, + "--slow" => grid.slow = parse_csv_list(value).map_err(|()| usage())?, + "--stop-length" => grid.stop_length = parse_csv_list(value).map_err(|()| usage())?, + "--stop-k" => grid.stop_k = parse_csv_list(value).map_err(|()| usage())?, + "--channel" => grid.channel = parse_csv_list(value).map_err(|()| usage())?, + "--window" => grid.window = parse_csv_list(value).map_err(|()| usage())?, + "--band-k" => grid.band_k = parse_csv_list(value).map_err(|()| usage())?, + _ => return Err(usage()), + } + tail = t; + } + let (name, persist) = name.unwrap_or_else(|| ("walkforward".to_string(), false)); + Ok((strategy, name, persist, real.finish(&usage)?, grid)) +} +``` + +- [ ] **Step 4: Thread `run_walkforward` + `walkforward_family`** — change + `run_walkforward` (1615) and `walkforward_family` (1643) signatures and add the + dispatch (SMA arm is today's body verbatim; stage1-r arm errors for now): + +```rust +fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid) { + if persist + && let Err(e) = TraceStore::open("runs").ensure_name_free(name, WriteKind::Family) + { + eprintln!("aura: {e}"); + std::process::exit(2); + } + let reg = default_registry(); + let result = walkforward_family(strategy, persist.then_some(name), &data, grid); + // ... rest of run_walkforward body unchanged (append_family / print loop / summary) ... +} + +fn walkforward_family( + strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid, +) -> WalkForwardResult { + let span = data.wf_full_span(); + let (is_len, oos_len, step) = data.wf_window_sizes(); + let roller = match WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling) { + Ok(r) => r, + Err(e) => { + eprintln!("aura: walk-forward window too short for one IS+OOS span: {e:?}"); + std::process::exit(2); + } + }; + match strategy { + Strategy::SmaCross => { + let space = sample_blueprint_with_sinks(data.pip_size()).0.param_space(); + walk_forward(roller, space, |w: WindowBounds| { + let is_family = sweep_over(w.is.0, w.is.1, data); + let best = optimize(&is_family, "total_pips").expect("total_pips is a known metric"); + let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) + } + Strategy::Stage1R => { + // filled in Task 4 (stage1_r_sweep_over / run_oos_r / stage1_r_space). + eprintln!("aura: walkforward --strategy stage1-r not yet wired"); + std::process::exit(2); + } + _ => { + eprintln!("aura: walkforward has no form for the selected strategy"); + std::process::exit(2); + } + } +} +``` + +- [ ] **Step 5: Update the dispatch arm + USAGE** + +```rust +// dispatch arm (main.rs:2637) +["walkforward", rest @ ..] => match parse_walkforward_args(rest) { + Ok((strategy, name, persist, choice, grid)) => { + run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid) + } + Err(msg) => { + eprintln!("aura: {msg}"); + std::process::exit(2); + } +}, +// USAGE (main.rs:2592) — replace the walkforward fragment with: +// aura walkforward [--strategy ] [--real [--from ] [--to ]] [--name |--trace ] [--fast ] [--slow ] [--stop-length ] [--stop-k ] +``` + +- [ ] **Step 6: Thread the remaining `walkforward_family` test callers** (1775, + 3151) so the workspace compiles — both pass the new args: + +```rust +// main.rs:1775 (walkforward_report helper) and :3151 (member-reports test): +// walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default()) +``` + +- [ ] **Step 7: Build gate** + +Run: `cargo build --workspace` +Expected: 0 errors (every `parse_walkforward_args` / `run_walkforward` / +`walkforward_family` caller threaded). + +- [ ] **Step 8: Run parse tests + the SMA walkforward goldens** + +Run: `cargo test --workspace parse_walkforward_args_accepts_strategy_and_grid_flags` +Expected: PASS +Run: `cargo test --workspace parse_walkforward_args_defaults_and_accepts_real` +Expected: PASS +Run: `cargo test --workspace walkforward_report_has_one_oos_line_per_window_plus_summary` +Expected: PASS (the SMA arm is byte-verbatim) + +### Task 4: stage1-r walk-forward arm — windowed helpers + dispatch + `oos_r` + +**Files:** +- Modify: `crates/aura-cli/src/main.rs` — add `stage1_r_space`, + `stage1_r_sweep_over`, `run_oos_r`; fill the `Strategy::Stage1R` arm in + `walkforward_family`; extend `walkforward_summary_json`. +- Test: `crates/aura-cli/tests/cli_run.rs` — synthetic stage1-r walkforward E2E. + +- [ ] **Step 1: Write the failing E2E test** — append to + `crates/aura-cli/tests/cli_run.rs` (model on the existing walkforward + stage1-r + shell tests; runs the built binary on synthetic data): + +```rust +#[test] +fn walkforward_strategy_stage1_r_reports_oos_r() { + // `aura walkforward --strategy stage1-r` on synthetic data: every per-window + // line carries an `r` block, and the summary line carries a pooled `oos_r` + // block. Deterministic: a second run is byte-identical. + let out = run_aura(&["walkforward", "--strategy", "stage1-r"]); + let lines: Vec<&str> = out.trim().lines().collect(); + let summary: serde_json::Value = serde_json::from_str(lines.last().unwrap()).unwrap(); + assert!(summary["walkforward"]["oos_r"].is_object(), "summary carries oos_r"); + assert!(summary["walkforward"]["oos_r"]["n_trades"].is_number()); + // each per-window member line carries metrics.r + for l in &lines[..lines.len() - 1] { + let v: serde_json::Value = serde_json::from_str(l).unwrap(); + assert!(v["report"]["metrics"]["r"].is_object(), "per-window r block present"); + } + let out2 = run_aura(&["walkforward", "--strategy", "stage1-r"]); + assert_eq!(out, out2, "stage1-r walkforward is deterministic"); +} +``` + +(If `run_aura` is not the existing harness name in `cli_run.rs`, use that file's +established binary-invocation helper — the same one +`walkforward_real_persists_one_oos_member_per_window` uses.) + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace walkforward_strategy_stage1_r_reports_oos_r` +Expected: FAIL — the process exits 2 ("not yet wired") so the output has no lines / +the assertion on `oos_r` fails. + +- [ ] **Step 3: Add `stage1_r_space`** — after `stage1_r_sweep_family` (main.rs:1275): + +```rust +/// The param-space of the OPEN stage1-r blueprint (all four knobs free) — the kinds +/// the per-window `WindowRun::chosen_params` are read against (C7). Mirrors the +/// throwaway-build param_space resolution inside `stage1_r_sweep_family`. +fn stage1_r_space() -> Vec { + let (tx_eq, _) = mpsc::channel(); + let (tx_ex, _) = mpsc::channel(); + let (tx_r, _) = mpsc::channel(); + let (tx_req, _) = mpsc::channel(); + stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true).param_space() +} +``` + +- [ ] **Step 4: Add `stage1_r_sweep_over`** — the windowed, reduce-mode IS sweep. It + is `stage1_r_sweep_family` (main.rs:1180-1275) with two changes: it always folds + (reduce = true, never persists), and it sources `data.windowed_sources(from, to)` + instead of `data.run_sources()`: + +```rust +/// Windowed reduce-mode stage1-r sweep over `[from, to]` — the in-sample leg of the +/// stage1-r walk-forward. Identical grid/axis/fold logic to `stage1_r_sweep_family`, +/// but windowed (`windowed_sources`) and always folded (O(trades)/member): each +/// member's RunReport carries `metrics.r = Some(summarize_r(..))`, so the family is +/// rankable by an R metric. +fn stage1_r_sweep_over(from: Timestamp, to: Timestamp, data: &DataSource, grid: &Stage1RGrid) -> SweepFamily { + let pip = data.pip_size(); + let (tx_eq, _) = mpsc::channel(); + let (tx_ex, _) = mpsc::channel(); + let (tx_r, _) = mpsc::channel(); + let (tx_req, _) = mpsc::channel(); + let bp = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true); + let space = bp.param_space(); + let stop_length_axis = space.iter().map(|p| p.name.clone()) + .find(|n| n.ends_with(STOP_LENGTH_SUFFIX)).expect("open stage1-r vol-stop exposes a stop_length axis"); + let stop_k_axis = space.iter().map(|p| p.name.clone()) + .find(|n| n.ends_with(STOP_K_SUFFIX)).expect("open stage1-r vol-stop exposes a stop_k axis"); + let binder = bp + .axis("fast.length", grid.fast.clone()) + .axis("slow.length", grid.slow.clone()) + .axis(&stop_length_axis, grid.stop_length.clone()) + .axis(&stop_k_axis, grid.stop_k.clone()); + let varying: HashSet = binder.varying_axes().into_iter() + .map(|n| stage1_r_friendly_name(&n)).collect(); + binder + .sweep(|point| { + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_ex, rx_ex) = mpsc::channel(); + let (tx_r, rx_r) = mpsc::channel(); + let (tx_req, _rx_req) = mpsc::channel(); + let mut h = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true) + .bootstrap_with_cells(point) + .expect("stage1-r grid points are kind-checked against param_space"); + let sources = data.windowed_sources(from, to); + let window = window_of(&sources).expect("non-empty in-sample window"); + h.run(sources); + let mut named: Vec<(String, Scalar)> = zip_params(&space, point).into_iter() + .map(|(n, v)| (stage1_r_friendly_name(&n), v)).collect(); + named.push(("bias_scale".to_string(), Scalar::f64(0.5))); + let key = member_key(&named, &varying); + let _ = &key; // member key not persisted here (in-memory IS family) + let mut manifest = sim_optimal_manifest(named, window, 0, pip); + manifest.broker = stage1_r_broker_label(pip); + let r_rows: Vec<(Timestamp, Vec)> = rx_r.try_iter().collect(); + let (total_pips, max_drawdown) = rx_eq.try_iter().next() + .map(|(_, row)| (row[0].as_f64(), row[1].as_f64())).unwrap_or((0.0, 0.0)); + let bias_sign_flips = rx_ex.try_iter().next().map(|(_, row)| row[2].as_i64() as u64).unwrap_or(0); + let mut m = RunMetrics { total_pips, max_drawdown, bias_sign_flips, r: None }; + m.r = Some(summarize_r(&r_rows, 0.0)); + RunReport { manifest, metrics: m } + }) + .expect("the stage1-r named grid matches the stage1-r param-space") +} +``` + +- [ ] **Step 5: Add `run_oos_r`** — the windowed, non-reduce OOS run (the pip-equity + curve is needed for `stitch`; folds `summarize` + `summarize_r`): + +```rust +/// Run the chosen stage1-r params over an OOS window; return the recorded pip-equity +/// segment (for stitching) and the OOS RunReport whose `metrics.r` carries both the +/// R metrics and the per-trade `trade_rs`. Non-reduce (raw recorders): one window's +/// curve is bounded, and `stitch` needs the full pip-equity series. +fn run_oos_r( + params: &[Cell], from: Timestamp, to: Timestamp, trace: Option<&str>, data: &DataSource, +) -> (Vec<(Timestamp, f64)>, RunReport) { + let pip = data.pip_size(); + let (tx_eq, rx_eq) = mpsc::channel(); + let (tx_ex, rx_ex) = mpsc::channel(); + let (tx_r, rx_r) = mpsc::channel(); + let (tx_req, rx_req) = mpsc::channel(); + let bp = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, false); + let space = bp.param_space(); + let mut h = bp.bootstrap_with_cells(params) + .expect("chosen params pre-validated by the in-sample GridSpace::new"); + let sources = data.windowed_sources(from, to); + let window = window_of(&sources).expect("non-empty out-of-sample window"); + h.run(sources); + let eq_rows = rx_eq.try_iter().collect::>(); + let ex_rows = rx_ex.try_iter().collect::>(); + let r_rows: Vec<(Timestamp, Vec)> = rx_r.try_iter().collect(); + let req_rows: Vec<(Timestamp, Vec)> = rx_req.try_iter().collect(); + let mut named: Vec<(String, Scalar)> = zip_params(&space, params).into_iter() + .map(|(n, v)| (stage1_r_friendly_name(&n), v)).collect(); + named.push(("bias_scale".to_string(), Scalar::f64(0.5))); + let mut manifest = sim_optimal_manifest(named, window, 0, pip); + manifest.broker = stage1_r_broker_label(pip); + if let Some(name) = trace { + persist_traces_r(&format!("{name}/oos{}", from.0), &manifest, &eq_rows, &ex_rows, &req_rows); + } + let equity = f64_field(&eq_rows, 0); + let exposure = f64_field(&ex_rows, 0); + let mut metrics = summarize(&equity, &exposure); + metrics.r = Some(summarize_r(&r_rows, 0.0)); + (equity, RunReport { manifest, metrics }) +} +``` + +- [ ] **Step 6: Fill the `Strategy::Stage1R` arm** in `walkforward_family` (replace + the Task-3 error stub): + +```rust + Strategy::Stage1R => { + let space = stage1_r_space(); + walk_forward(roller, space, |w: WindowBounds| { + let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); + let best = optimize(&is_family, "sqn_normalized").expect("sqn_normalized is a known metric"); + let (oos_equity, oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) + } +``` + +- [ ] **Step 7: Extend `walkforward_summary_json`** to add the pooled `oos_r` block + (only when at least one window carries an `r` block — the SMA path's windows have + `r: None`, so its summary line stays byte-identical): + +```rust +fn walkforward_summary_json(result: &WalkForwardResult) -> String { + let total = result.stitched_oos_equity.last().map(|&(_, v)| v).unwrap_or(0.0); + // pool the per-window OOS per-trade R series in roll order, then reduce. + let pooled_rs: Vec = result.windows.iter() + .flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default()) + .collect(); + let mut obj = serde_json::json!({ + "windows": result.windows.len(), + "stitched_total_pips": total, + "param_stability": param_stability(result), + }); + if result.windows.iter().any(|w| w.run.oos_report.metrics.r.is_some()) { + // RMetrics serializes its scalar fields (trade_rs is serde-skipped, so the + // oos_r block is the clean R-metric summary of the pooled series). + obj["oos_r"] = serde_json::to_value(r_metrics_from_rs(&pooled_rs)) + .expect("RMetrics serializes"); + } + serde_json::json!({ "walkforward": obj }).to_string() +} +``` + +(Import `r_metrics_from_rs` from `aura_engine` alongside the existing `summarize_r` +/ `summarize` imports at the top of `main.rs`.) + +- [ ] **Step 8: Build + run the E2E test + the determinism/golden guards** + +Run: `cargo build --workspace` +Expected: 0 errors +Run: `cargo test --workspace walkforward_strategy_stage1_r_reports_oos_r` +Expected: PASS +Run: `cargo test --workspace walkforward_report_has_one_oos_line_per_window_plus_summary` +Expected: PASS (SMA summary unchanged — no `oos_r` key, windows have `r: None`) + +- [ ] **Step 9: Full-suite regression gate** + +Run: `cargo test --workspace` +Expected: PASS — all pre-existing goldens (SMA walkforward, synthetic mc, stage1-r +single-run `stage1_r_single_run_output_golden`, C18 round-trips) green; the new +tests green. + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: 0 warnings. + +--- + +## Self-review + +- **Spec coverage:** Changes 1 (RMetrics.trade_rs / summarize_r — Task 1), the + pooled reducer (Task 2), Change 3 walkforward dispatch (Tasks 3+4), Change 5 USAGE + (Task 3). Monte-Carlo (spec Changes 2+4) is iteration 2 — out of this plan's scope + by design. ✓ +- **Placeholder scan:** no TBD/TODO; the Task-3 stage1-r error stub is an explicit, + compiling intermediate filled in Task 4 (named, not a placeholder). ✓ +- **Type/name consistency:** `RMetrics.trade_rs`, `r_metrics_from_rs`, + `stage1_r_sweep_over`, `run_oos_r`, `stage1_r_space`, `Strategy::Stage1R`, + `Stage1RGrid` used identically across tasks. ✓ +- **Step granularity:** each step is one edit or one command. ✓ +- **No commit steps.** ✓ +- **Compile-gate ordering (rule 7):** Task 1 threads all 4 `RMetrics { .. }` literal + sites before its build gate; Task 3 threads all `parse_walkforward_args` / + `walkforward_family` / `run_walkforward` callers (dispatch arm + 2 test helpers + + parse-test assertions) before its build gate, with the stage1-r arm a compiling + error-stub so the gate is satisfiable. ✓ +- **Filter strings (rule 8):** every `cargo test` filter names a real test — + pre-existing (`runmetrics_with_r_block_round_trips`, + `walkforward_report_has_one_oos_line_per_window_plus_summary`, + `parse_walkforward_args_defaults_and_accepts_real`, + `stage1_r_single_run_output_golden`) or introduced in the same task; the final + gate is the unfiltered `cargo test --workspace`. ✓