Files
Aura/docs/plans/0075-walkforward-mc-r-validation.md
T
Brummel 2593af86f8 plan: 0075 walk-forward strategy-selectable + R-reporting (iteration 1)
Iteration 1 of cycle 0075: RMetrics.trade_rs (in-memory, serde+eq-excluded)
+ summarize_r retains it; r_metrics_from_rs pooled reducer; walkforward
--strategy + grid flags; strategy-dispatched walkforward_family (SMA arm
verbatim); windowed stage1-r helpers (stage1_r_sweep_over reduce-mode IS
sweep, run_oos_r non-reduce OOS run, stage1_r_space); pooled oos_r summary.
Monte-Carlo bootstrap is iteration 2.

refs #139
2026-06-26 10:26:39 +02:00

34 KiB

Walk-forward strategy-selectable + R-reporting (iteration 1) — Implementation Plan

Parent spec: docs/specs/0075-walkforward-mc-r-validation.md

For agentic workers: REQUIRED SUB-SKILL: use the implement skill to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Make aura walkforward --strategy stage1-r --real <SYM> roll IS→OOS windows, sweep the stage1-r grid in-sample, pick the best by an R metric, run the chosen params OOS, and report per-window + across-window-pooled OOS R-metrics — while the bare SMA walkforward path stays byte-identical.

Architecture: Two additive library changes in aura-engine (RMetrics carries an in-memory trade_rs vector excluded from serde + equality; a new r_metrics_from_rs pooled reducer), then aura-cli gains --strategy + grid flags on walkforward, a strategy-dispatched walkforward_family (SMA arm verbatim), windowed stage1-r helpers (stage1_r_sweep_over reduce-mode IS sweep, run_oos_r non-reduce OOS run, stage1_r_space), and a pooled oos_r summary block. The Monte-Carlo bootstrap half is iteration 2 (separate plan).

Tech Stack: Rust workspace — aura-engine (report.rs), aura-registry (lib.rs test helper), aura-cli (main.rs). Test command: cargo test --workspace (one positional filter per invocation).


Files this plan creates or modifies:

  • Modify: crates/aura-engine/src/report.rs:44-62RMetrics gains #[serde(skip)] trade_rs: Vec<f64>; drop PartialEq from the derive, hand-write it (ignoring trade_rs).
  • Modify: crates/aura-engine/src/report.rs:89-219summarize_r sets trade_rs in both return arms; add new pub fn r_metrics_from_rs.
  • Modify: crates/aura-registry/src/lib.rs:255report_with_r test helper RMetrics { .. } literal gains trade_rs: Vec::new().
  • Modify: crates/aura-cli/src/main.rs:1526-1546parse_walkforward_args 3-tuple → 5-tuple (--strategy + four grid flags).
  • Modify: crates/aura-cli/src/main.rs:1615-1666run_walkforward + walkforward_family gain strategy/grid; strategy dispatch.
  • Modify: crates/aura-cli/src/main.rs:1734-1744walkforward_summary_json gains the oos_r pooled block.
  • Modify: crates/aura-cli/src/main.rs:2591-2592 — USAGE walkforward fragment.
  • Modify: crates/aura-cli/src/main.rs:2637-2645 — walkforward dispatch arm destructures the 5-tuple.
  • Create (new fns in crates/aura-cli/src/main.rs): stage1_r_sweep_over, run_oos_r, stage1_r_space.
  • Test: crates/aura-engine/src/report.rs (#cfg test) — summarize_r populates trade_rs; RMetrics round-trip equal despite skipped trade_rs; r_metrics_from_rs goldens.
  • Test: crates/aura-cli/src/main.rs (#cfg test) — parse_walkforward_args 5-tuple + --strategy/grid; update the two existing assertions at 3653-3656.
  • Test: crates/aura-cli/tests/cli_run.rsaura walkforward --strategy stage1-r emits per-window r blocks + an oos_r summary (synthetic).

Task 1: RMetrics.trade_rs — in-memory per-trade R series

Files:

  • Modify: crates/aura-engine/src/report.rs:44-62, 89-135, 206-219

  • Modify: crates/aura-registry/src/lib.rs:255

  • Test: crates/aura-engine/src/report.rs (#cfg test)

  • Step 1: Write the failing tests — append to the #[cfg(test)] mod tests in crates/aura-engine/src/report.rs:

#[test]
fn summarize_r_populates_trade_rs_in_trade_order() {
    // two closed trades (R = +2.0, then -1.0) over a minimal PositionManagement
    // record; trade_rs must carry [2.0, -1.0] in trade order.
    let rec = pm_record_two_closed_trades(); // helper below
    let m = summarize_r(&rec, 0.0);
    assert_eq!(m.trade_rs, vec![2.0, -1.0]);
    assert_eq!(m.n_trades, 2);
}

#[test]
fn summarize_r_empty_record_has_empty_trade_rs() {
    let m = summarize_r(&[], 0.0);
    assert!(m.trade_rs.is_empty());
    assert_eq!(m.n_trades, 0);
}

#[test]
fn rmetrics_partial_eq_ignores_trade_rs() {
    // two RMetrics equal in every metric but differing in trade_rs compare EQUAL
    // (trade_rs is an in-memory conduit, excluded from equality) — this is what
    // keeps serialize->deserialize round-trips equal (trade_rs is serde-skipped,
    // so it deserializes empty).
    let a = summarize_r(&pm_record_two_closed_trades(), 0.0);
    let mut b = a.clone();
    b.trade_rs = Vec::new();
    assert_eq!(a, b);
}

/// A minimal dense PositionManagement record with two closed trades at R = +2, -1.
/// Columns per `r_col` (CLOSED=0, REALIZED_R=1, DIRECTION=4, ENTRY_PRICE=6,
/// STOP_PRICE=7, CONVICTION_AT_ENTRY=9, SIZE=10, OPEN=11, UNREALIZED_R=12); width 13.
fn pm_record_two_closed_trades() -> Vec<(Timestamp, Vec<Scalar>)> {
    let row = |closed: bool, r: f64, open: bool| {
        let mut c = vec![Scalar::f64(0.0); 13];
        c[0] = Scalar::bool(closed);
        c[1] = Scalar::f64(r);
        c[6] = Scalar::f64(1.0); // entry
        c[7] = Scalar::f64(0.5); // stop -> latched 0.5
        c[9] = Scalar::f64(0.3); // conviction
        c[11] = Scalar::bool(open);
        c
    };
    vec![
        (Timestamp(1), row(true, 2.0, false)),
        (Timestamp(2), row(true, -1.0, false)),
    ]
}
  • Step 2: Run to verify it fails

Run: cargo test --workspace summarize_r_populates_trade_rs_in_trade_order Expected: FAIL — compile error no field trade_rson typeRMetrics`` (and the two siblings fail to compile likewise).

  • Step 3: Add the field + hand-written PartialEq — in crates/aura-engine/src/report.rs, change the RMetrics derive (line 44) to drop PartialEq, add the field after conviction_terciles_r (line 61), and add the impl PartialEq directly after the struct:
// line 44: drop PartialEq from the derive
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct RMetrics {
    // ... all existing fields unchanged ...
    pub conviction_terciles_r: [f64; 3],
    /// Realised R per closed trade, in trade order — an in-memory conduit for the
    /// OOS R-series bootstrap (#139). Excluded from serde (`skip`) so the C18 wire
    /// shape is unchanged, and from `PartialEq` (below) so every existing
    /// `RMetrics`/`RunReport` equality assertion and round-trip stays green.
    #[serde(skip)]
    pub trade_rs: Vec<f64>,
}

impl PartialEq for RMetrics {
    fn eq(&self, o: &Self) -> bool {
        self.expectancy_r == o.expectancy_r
            && self.n_trades == o.n_trades
            && self.win_rate == o.win_rate
            && self.avg_win_r == o.avg_win_r
            && self.avg_loss_r == o.avg_loss_r
            && self.profit_factor == o.profit_factor
            && self.max_r_drawdown == o.max_r_drawdown
            && self.n_open_at_end == o.n_open_at_end
            && self.sqn == o.sqn
            && self.sqn_normalized == o.sqn_normalized
            && self.net_expectancy_r == o.net_expectancy_r
            && self.conviction_terciles_r == o.conviction_terciles_r
        // trade_rs deliberately excluded
    }
}
  • Step 4: Populate trade_rs in both summarize_r arms — in crates/aura-engine/src/report.rs: the empty-input arm (line 122-135 literal) adds trade_rs: Vec::new(),, and the main return literal (line 206-219) adds trade_rs: rs, (the rs vector built at line 137 is the per-trade realised R in trade order; it is last-read by the sqn block before the return, so the move is valid):
// empty-input arm (line 122) — add as the last field:
        return RMetrics {
            // ... existing zero fields ...
            conviction_terciles_r: [0.0; 3],
            trade_rs: Vec::new(),
        };
// main return (line 206) — add as the last field:
    RMetrics {
        // ... existing fields ...
        conviction_terciles_r,
        trade_rs: rs,
    }
  • Step 5: Thread the other two RMetrics { .. } literal sites (the only remaining struct literals workspace-wide — both test helpers):
// crates/aura-engine/src/report.rs — the runmetrics_with_r_block_round_trips test
// literal (~line 1135): add `trade_rs: Vec::new(),` as the last field.
// crates/aura-registry/src/lib.rs:255 — the report_with_r helper literal: add
// `trade_rs: Vec::new(),` as the last field.
  • Step 6: Build gate (all literal sites threaded)

Run: cargo build --workspace Expected: 0 errors (every RMetrics { .. } literal now carries trade_rs).

  • Step 7: Run the new + back-compat tests

Run: cargo test --workspace summarize_r_populates_trade_rs_in_trade_order Expected: PASS Run: cargo test --workspace rmetrics_partial_eq_ignores_trade_rs Expected: PASS Run: cargo test --workspace runmetrics_with_r_block_round_trips Expected: PASS (the serde-skip + hand-written eq keep the round-trip equal)

Task 2: r_metrics_from_rs — pooled reducer over a flat R series

Files:

  • Modify: crates/aura-engine/src/report.rs (new pub fn after summarize_r)

  • Test: crates/aura-engine/src/report.rs (#cfg test)

  • Step 1: Write the failing test — append to the report.rs test module:

#[test]
fn r_metrics_from_rs_folds_a_flat_series() {
    // pooled across-window R series [2.0, -1.0, 1.0]: expectancy = 2/3, 2 wins of 3,
    // profit_factor = (2+1)/1 = 3. At cost 0 (frictionless Stage-1) net == gross.
    // conviction terciles are not pooled -> [0,0,0]; n_open_at_end is not a pooled
    // concept -> 0.
    let m = r_metrics_from_rs(&[2.0, -1.0, 1.0]);
    assert_eq!(m.n_trades, 3);
    assert!((m.expectancy_r - 2.0 / 3.0).abs() < 1e-12);
    assert!((m.win_rate - 2.0 / 3.0).abs() < 1e-12);
    assert!((m.profit_factor - 3.0).abs() < 1e-12);
    assert_eq!(m.net_expectancy_r, m.expectancy_r);
    assert_eq!(m.conviction_terciles_r, [0.0; 3]);
    assert_eq!(m.n_open_at_end, 0);
}

#[test]
fn r_metrics_from_rs_empty_is_all_zero() {
    let m = r_metrics_from_rs(&[]);
    assert_eq!(m.n_trades, 0);
    assert_eq!(m.expectancy_r, 0.0);
    assert_eq!(m.sqn, 0.0);
}
  • Step 2: Run to verify it fails

Run: cargo test --workspace r_metrics_from_rs_folds_a_flat_series Expected: FAIL — cannot find function r_metrics_from_rs in this scope.

  • Step 3: Add the reducer — in crates/aura-engine/src/report.rs, directly after summarize_r (line 220). The R-distribution arithmetic is copied verbatim from summarize_r (lines 137-177) — the byte-pinned float expressions must not be algebraically refactored (a shared factoring would shift the golden by ~1 ULP); summarize_r itself is left untouched:
/// Reduce a flat per-trade R series (e.g. the pooled across-window OOS series of a
/// walk-forward) into `RMetrics`. The R-distribution fields use the same arithmetic
/// as [`summarize_r`] (copied verbatim — the pinned floats are not refactored). The
/// fields a flat R series cannot carry are set honestly: `n_open_at_end = 0`,
/// `net_expectancy_r = expectancy_r` (exact under the Stage-1 cost = 0 invariant),
/// `conviction_terciles_r = [0,0,0]` (per-trade conviction is not pooled). Empty
/// input -> a well-defined all-zero `RMetrics`.
pub fn r_metrics_from_rs(rs: &[f64]) -> RMetrics {
    let n = rs.len() as u64;
    if n == 0 {
        return RMetrics {
            expectancy_r: 0.0, n_trades: 0, win_rate: 0.0, avg_win_r: 0.0,
            avg_loss_r: 0.0, profit_factor: 0.0, max_r_drawdown: 0.0,
            n_open_at_end: 0, sqn: 0.0, sqn_normalized: 0.0, net_expectancy_r: 0.0,
            conviction_terciles_r: [0.0; 3], trade_rs: Vec::new(),
        };
    }
    let sum: f64 = rs.iter().sum();
    let mean = sum / n as f64;
    let wins: Vec<f64> = rs.iter().copied().filter(|&r| r > 0.0).collect();
    let losses: Vec<f64> = rs.iter().copied().filter(|&r| r <= 0.0).collect();
    let sum_win: f64 = wins.iter().sum();
    let sum_loss: f64 = losses.iter().sum();
    let avg = |v: &[f64]| if v.is_empty() { 0.0 } else { v.iter().sum::<f64>() / v.len() as f64 };
    let mut peak = f64::NEG_INFINITY;
    let mut cum = 0.0;
    let mut max_dd = 0.0_f64;
    for &r in rs {
        cum += r;
        if cum > peak { peak = cum; }
        let dd = peak - cum;
        if dd > max_dd { max_dd = dd; }
    }
    let (sqn, sqn_normalized) = if n < 2 {
        (0.0, 0.0)
    } else {
        let var = rs.iter().map(|&r| (r - mean).powi(2)).sum::<f64>() / (n as f64 - 1.0);
        let sd = var.sqrt();
        if sd > 0.0 {
            ((n as f64).sqrt() * mean / sd, (n.min(SQN_CAP) as f64).sqrt() * mean / sd)
        } else {
            (0.0, 0.0)
        }
    };
    RMetrics {
        expectancy_r: mean,
        n_trades: n,
        win_rate: wins.len() as f64 / n as f64,
        avg_win_r: avg(&wins),
        avg_loss_r: avg(&losses),
        profit_factor: if sum_loss < 0.0 { sum_win / (-sum_loss) } else { 0.0 },
        max_r_drawdown: max_dd,
        n_open_at_end: 0,
        sqn,
        sqn_normalized,
        net_expectancy_r: mean, // cost = 0 -> net == gross (Stage-1 frictionless)
        conviction_terciles_r: [0.0; 3],
        trade_rs: Vec::new(),
    }
}
  • Step 4: Run to verify it passes

Run: cargo test --workspace r_metrics_from_rs_folds_a_flat_series Expected: PASS Run: cargo test --workspace r_metrics_from_rs_empty_is_all_zero Expected: PASS

Task 3: walkforward signature plumbing — --strategy + grid flags

This task threads the strategy/grid through parse_walkforward_argsrun_walkforwardwalkforward_family (SMA arm verbatim; stage1-r arm a graceful error stub filled in Task 4) and every call site, so the workspace compiles and all SMA goldens stay green.

Files:

  • Modify: crates/aura-cli/src/main.rs:1526-1546, 1607-1666, 2591-2592, 2637-2645

  • Test: crates/aura-cli/src/main.rs (#cfg test, lines 3651-3660 updated + a new case)

  • Step 1: Write/adjust the failing parse test — in the report-less crates/aura-cli/src/main.rs test module, replace the body of parse_walkforward_args_defaults_and_accepts_real (3652-3660) to expect the 5-tuple, and add a stage1-r case:

#[test]
fn parse_walkforward_args_defaults_and_accepts_real() {
    assert_eq!(
        parse_walkforward_args(&[]),
        Ok((Strategy::SmaCross, "walkforward".to_string(), false, DataChoice::Synthetic, Stage1RGrid::default()))
    );
    assert_eq!(
        parse_walkforward_args(&["--real", "EURUSD", "--trace", "w"]),
        Ok((Strategy::SmaCross, "w".to_string(), true,
            DataChoice::Real { symbol: "EURUSD".to_string(), from_ms: None, to_ms: None },
            Stage1RGrid::default()))
    );
    assert!(parse_walkforward_args(&["--name", "a", "--trace", "b"]).is_err());
    assert!(parse_walkforward_args(&["--real"]).is_err());
}

#[test]
fn parse_walkforward_args_accepts_strategy_and_grid_flags() {
    let r = parse_walkforward_args(&["--strategy", "stage1-r", "--fast", "5,10", "--stop-k", "2.0,3.0"]);
    let (strategy, _, _, _, grid) = r.expect("valid stage1-r walkforward args");
    assert_eq!(strategy, Strategy::Stage1R);
    assert_eq!(grid.fast, vec![5, 10]);
    assert_eq!(grid.stop_k, vec![2.0, 3.0]);
    assert!(parse_walkforward_args(&["--strategy", "bogus"]).is_err());
}
  • Step 2: Run to verify it fails

Run: cargo test --workspace parse_walkforward_args_accepts_strategy_and_grid_flags Expected: FAIL — compile error (the 5-tuple / Strategy return type does not exist yet; existing 3-tuple assertions also fail to compile).

  • Step 3: Extend parse_walkforward_args — replace lines 1523-1546 with the 5-tuple grammar mirroring parse_sweep_args (1480-1521):
/// Parse the `walkforward` tail:
/// `[--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n> | --trace <n>] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] [--channel <csv>] [--window <csv>] [--band-k <csv>]`.
/// Defaults: SMA-cross, synthetic, name "walkforward", no persist. `--name`/`--trace`
/// mutually exclusive; `--from`/`--to` require `--real`. Pure (no I/O / exit).
fn parse_walkforward_args(
    rest: &[&str],
) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid), String> {
    let usage = || "walkforward [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n> | --trace <n>] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] [--channel <csv>] [--window <csv>] [--band-k <csv>]".to_string();
    let mut strategy = Strategy::SmaCross;
    let mut name: Option<(String, bool)> = None;
    let mut real = RealWindowGrammar::default();
    let mut grid = Stage1RGrid::default();
    let mut tail = rest;
    while let Some((flag, t)) = tail.split_first() {
        let (value, t) = t.split_first().ok_or_else(usage)?;
        if real.accept(flag, value, &usage)? {
            tail = t;
            continue;
        }
        match *flag {
            "--strategy" => {
                strategy = match *value {
                    "sma" => Strategy::SmaCross,
                    "momentum" => Strategy::Momentum,
                    "stage1-r" => Strategy::Stage1R,
                    "stage1-breakout" => Strategy::Stage1Breakout,
                    "stage1-meanrev" => Strategy::Stage1MeanRev,
                    _ => return Err(usage()),
                };
            }
            "--name" if name.is_none() => name = Some(((*value).to_string(), false)),
            "--trace" if name.is_none() => name = Some(((*value).to_string(), true)),
            "--fast" => grid.fast = parse_csv_list(value).map_err(|()| usage())?,
            "--slow" => grid.slow = parse_csv_list(value).map_err(|()| usage())?,
            "--stop-length" => grid.stop_length = parse_csv_list(value).map_err(|()| usage())?,
            "--stop-k" => grid.stop_k = parse_csv_list(value).map_err(|()| usage())?,
            "--channel" => grid.channel = parse_csv_list(value).map_err(|()| usage())?,
            "--window" => grid.window = parse_csv_list(value).map_err(|()| usage())?,
            "--band-k" => grid.band_k = parse_csv_list(value).map_err(|()| usage())?,
            _ => return Err(usage()),
        }
        tail = t;
    }
    let (name, persist) = name.unwrap_or_else(|| ("walkforward".to_string(), false));
    Ok((strategy, name, persist, real.finish(&usage)?, grid))
}
  • Step 4: Thread run_walkforward + walkforward_family — change run_walkforward (1615) and walkforward_family (1643) signatures and add the dispatch (SMA arm is today's body verbatim; stage1-r arm errors for now):
fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid) {
    if persist
        && let Err(e) = TraceStore::open("runs").ensure_name_free(name, WriteKind::Family)
    {
        eprintln!("aura: {e}");
        std::process::exit(2);
    }
    let reg = default_registry();
    let result = walkforward_family(strategy, persist.then_some(name), &data, grid);
    // ... rest of run_walkforward body unchanged (append_family / print loop / summary) ...
}

fn walkforward_family(
    strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid,
) -> WalkForwardResult {
    let span = data.wf_full_span();
    let (is_len, oos_len, step) = data.wf_window_sizes();
    let roller = match WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling) {
        Ok(r) => r,
        Err(e) => {
            eprintln!("aura: walk-forward window too short for one IS+OOS span: {e:?}");
            std::process::exit(2);
        }
    };
    match strategy {
        Strategy::SmaCross => {
            let space = sample_blueprint_with_sinks(data.pip_size()).0.param_space();
            walk_forward(roller, space, |w: WindowBounds| {
                let is_family = sweep_over(w.is.0, w.is.1, data);
                let best = optimize(&is_family, "total_pips").expect("total_pips is a known metric");
                let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
                WindowRun { chosen_params: best.params, oos_equity, oos_report }
            })
        }
        Strategy::Stage1R => {
            // filled in Task 4 (stage1_r_sweep_over / run_oos_r / stage1_r_space).
            eprintln!("aura: walkforward --strategy stage1-r not yet wired");
            std::process::exit(2);
        }
        _ => {
            eprintln!("aura: walkforward has no form for the selected strategy");
            std::process::exit(2);
        }
    }
}
  • Step 5: Update the dispatch arm + USAGE
// dispatch arm (main.rs:2637)
["walkforward", rest @ ..] => match parse_walkforward_args(rest) {
    Ok((strategy, name, persist, choice, grid)) => {
        run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid)
    }
    Err(msg) => {
        eprintln!("aura: {msg}");
        std::process::exit(2);
    }
},
// USAGE (main.rs:2592) — replace the walkforward fragment with:
//   aura walkforward [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n>|--trace <n>] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>]
  • Step 6: Thread the remaining walkforward_family test callers (1775, 3151) so the workspace compiles — both pass the new args:
// main.rs:1775 (walkforward_report helper) and :3151 (member-reports test):
//   walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default())
  • Step 7: Build gate

Run: cargo build --workspace Expected: 0 errors (every parse_walkforward_args / run_walkforward / walkforward_family caller threaded).

  • Step 8: Run parse tests + the SMA walkforward goldens

Run: cargo test --workspace parse_walkforward_args_accepts_strategy_and_grid_flags Expected: PASS Run: cargo test --workspace parse_walkforward_args_defaults_and_accepts_real Expected: PASS Run: cargo test --workspace walkforward_report_has_one_oos_line_per_window_plus_summary Expected: PASS (the SMA arm is byte-verbatim)

Task 4: stage1-r walk-forward arm — windowed helpers + dispatch + oos_r

Files:

  • Modify: crates/aura-cli/src/main.rs — add stage1_r_space, stage1_r_sweep_over, run_oos_r; fill the Strategy::Stage1R arm in walkforward_family; extend walkforward_summary_json.

  • Test: crates/aura-cli/tests/cli_run.rs — synthetic stage1-r walkforward E2E.

  • Step 1: Write the failing E2E test — append to crates/aura-cli/tests/cli_run.rs (model on the existing walkforward + stage1-r shell tests; runs the built binary on synthetic data):

#[test]
fn walkforward_strategy_stage1_r_reports_oos_r() {
    // `aura walkforward --strategy stage1-r` on synthetic data: every per-window
    // line carries an `r` block, and the summary line carries a pooled `oos_r`
    // block. Deterministic: a second run is byte-identical.
    let out = run_aura(&["walkforward", "--strategy", "stage1-r"]);
    let lines: Vec<&str> = out.trim().lines().collect();
    let summary: serde_json::Value = serde_json::from_str(lines.last().unwrap()).unwrap();
    assert!(summary["walkforward"]["oos_r"].is_object(), "summary carries oos_r");
    assert!(summary["walkforward"]["oos_r"]["n_trades"].is_number());
    // each per-window member line carries metrics.r
    for l in &lines[..lines.len() - 1] {
        let v: serde_json::Value = serde_json::from_str(l).unwrap();
        assert!(v["report"]["metrics"]["r"].is_object(), "per-window r block present");
    }
    let out2 = run_aura(&["walkforward", "--strategy", "stage1-r"]);
    assert_eq!(out, out2, "stage1-r walkforward is deterministic");
}

(If run_aura is not the existing harness name in cli_run.rs, use that file's established binary-invocation helper — the same one walkforward_real_persists_one_oos_member_per_window uses.)

  • Step 2: Run to verify it fails

Run: cargo test --workspace walkforward_strategy_stage1_r_reports_oos_r Expected: FAIL — the process exits 2 ("not yet wired") so the output has no lines / the assertion on oos_r fails.

  • Step 3: Add stage1_r_space — after stage1_r_sweep_family (main.rs:1275):
/// The param-space of the OPEN stage1-r blueprint (all four knobs free) — the kinds
/// the per-window `WindowRun::chosen_params` are read against (C7). Mirrors the
/// throwaway-build param_space resolution inside `stage1_r_sweep_family`.
fn stage1_r_space() -> Vec<ParamSpec> {
    let (tx_eq, _) = mpsc::channel();
    let (tx_ex, _) = mpsc::channel();
    let (tx_r, _) = mpsc::channel();
    let (tx_req, _) = mpsc::channel();
    stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true).param_space()
}
  • Step 4: Add stage1_r_sweep_over — the windowed, reduce-mode IS sweep. It is stage1_r_sweep_family (main.rs:1180-1275) with two changes: it always folds (reduce = true, never persists), and it sources data.windowed_sources(from, to) instead of data.run_sources():
/// Windowed reduce-mode stage1-r sweep over `[from, to]` — the in-sample leg of the
/// stage1-r walk-forward. Identical grid/axis/fold logic to `stage1_r_sweep_family`,
/// but windowed (`windowed_sources`) and always folded (O(trades)/member): each
/// member's RunReport carries `metrics.r = Some(summarize_r(..))`, so the family is
/// rankable by an R metric.
fn stage1_r_sweep_over(from: Timestamp, to: Timestamp, data: &DataSource, grid: &Stage1RGrid) -> SweepFamily {
    let pip = data.pip_size();
    let (tx_eq, _) = mpsc::channel();
    let (tx_ex, _) = mpsc::channel();
    let (tx_r, _) = mpsc::channel();
    let (tx_req, _) = mpsc::channel();
    let bp = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true);
    let space = bp.param_space();
    let stop_length_axis = space.iter().map(|p| p.name.clone())
        .find(|n| n.ends_with(STOP_LENGTH_SUFFIX)).expect("open stage1-r vol-stop exposes a stop_length axis");
    let stop_k_axis = space.iter().map(|p| p.name.clone())
        .find(|n| n.ends_with(STOP_K_SUFFIX)).expect("open stage1-r vol-stop exposes a stop_k axis");
    let binder = bp
        .axis("fast.length", grid.fast.clone())
        .axis("slow.length", grid.slow.clone())
        .axis(&stop_length_axis, grid.stop_length.clone())
        .axis(&stop_k_axis, grid.stop_k.clone());
    let varying: HashSet<String> = binder.varying_axes().into_iter()
        .map(|n| stage1_r_friendly_name(&n)).collect();
    binder
        .sweep(|point| {
            let (tx_eq, rx_eq) = mpsc::channel();
            let (tx_ex, rx_ex) = mpsc::channel();
            let (tx_r, rx_r) = mpsc::channel();
            let (tx_req, _rx_req) = mpsc::channel();
            let mut h = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, true)
                .bootstrap_with_cells(point)
                .expect("stage1-r grid points are kind-checked against param_space");
            let sources = data.windowed_sources(from, to);
            let window = window_of(&sources).expect("non-empty in-sample window");
            h.run(sources);
            let mut named: Vec<(String, Scalar)> = zip_params(&space, point).into_iter()
                .map(|(n, v)| (stage1_r_friendly_name(&n), v)).collect();
            named.push(("bias_scale".to_string(), Scalar::f64(0.5)));
            let key = member_key(&named, &varying);
            let _ = &key; // member key not persisted here (in-memory IS family)
            let mut manifest = sim_optimal_manifest(named, window, 0, pip);
            manifest.broker = stage1_r_broker_label(pip);
            let r_rows: Vec<(Timestamp, Vec<Scalar>)> = rx_r.try_iter().collect();
            let (total_pips, max_drawdown) = rx_eq.try_iter().next()
                .map(|(_, row)| (row[0].as_f64(), row[1].as_f64())).unwrap_or((0.0, 0.0));
            let bias_sign_flips = rx_ex.try_iter().next().map(|(_, row)| row[2].as_i64() as u64).unwrap_or(0);
            let mut m = RunMetrics { total_pips, max_drawdown, bias_sign_flips, r: None };
            m.r = Some(summarize_r(&r_rows, 0.0));
            RunReport { manifest, metrics: m }
        })
        .expect("the stage1-r named grid matches the stage1-r param-space")
}
  • Step 5: Add run_oos_r — the windowed, non-reduce OOS run (the pip-equity curve is needed for stitch; folds summarize + summarize_r):
/// Run the chosen stage1-r params over an OOS window; return the recorded pip-equity
/// segment (for stitching) and the OOS RunReport whose `metrics.r` carries both the
/// R metrics and the per-trade `trade_rs`. Non-reduce (raw recorders): one window's
/// curve is bounded, and `stitch` needs the full pip-equity series.
fn run_oos_r(
    params: &[Cell], from: Timestamp, to: Timestamp, trace: Option<&str>, data: &DataSource,
) -> (Vec<(Timestamp, f64)>, RunReport) {
    let pip = data.pip_size();
    let (tx_eq, rx_eq) = mpsc::channel();
    let (tx_ex, rx_ex) = mpsc::channel();
    let (tx_r, rx_r) = mpsc::channel();
    let (tx_req, rx_req) = mpsc::channel();
    let bp = stage1_r_graph(tx_eq, tx_ex, tx_r, tx_req, None, None, true, false);
    let space = bp.param_space();
    let mut h = bp.bootstrap_with_cells(params)
        .expect("chosen params pre-validated by the in-sample GridSpace::new");
    let sources = data.windowed_sources(from, to);
    let window = window_of(&sources).expect("non-empty out-of-sample window");
    h.run(sources);
    let eq_rows = rx_eq.try_iter().collect::<Vec<_>>();
    let ex_rows = rx_ex.try_iter().collect::<Vec<_>>();
    let r_rows: Vec<(Timestamp, Vec<Scalar>)> = rx_r.try_iter().collect();
    let req_rows: Vec<(Timestamp, Vec<Scalar>)> = rx_req.try_iter().collect();
    let mut named: Vec<(String, Scalar)> = zip_params(&space, params).into_iter()
        .map(|(n, v)| (stage1_r_friendly_name(&n), v)).collect();
    named.push(("bias_scale".to_string(), Scalar::f64(0.5)));
    let mut manifest = sim_optimal_manifest(named, window, 0, pip);
    manifest.broker = stage1_r_broker_label(pip);
    if let Some(name) = trace {
        persist_traces_r(&format!("{name}/oos{}", from.0), &manifest, &eq_rows, &ex_rows, &req_rows);
    }
    let equity = f64_field(&eq_rows, 0);
    let exposure = f64_field(&ex_rows, 0);
    let mut metrics = summarize(&equity, &exposure);
    metrics.r = Some(summarize_r(&r_rows, 0.0));
    (equity, RunReport { manifest, metrics })
}
  • Step 6: Fill the Strategy::Stage1R arm in walkforward_family (replace the Task-3 error stub):
        Strategy::Stage1R => {
            let space = stage1_r_space();
            walk_forward(roller, space, |w: WindowBounds| {
                let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid);
                let best = optimize(&is_family, "sqn_normalized").expect("sqn_normalized is a known metric");
                let (oos_equity, oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
                WindowRun { chosen_params: best.params, oos_equity, oos_report }
            })
        }
  • Step 7: Extend walkforward_summary_json to add the pooled oos_r block (only when at least one window carries an r block — the SMA path's windows have r: None, so its summary line stays byte-identical):
fn walkforward_summary_json(result: &WalkForwardResult) -> String {
    let total = result.stitched_oos_equity.last().map(|&(_, v)| v).unwrap_or(0.0);
    // pool the per-window OOS per-trade R series in roll order, then reduce.
    let pooled_rs: Vec<f64> = result.windows.iter()
        .flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default())
        .collect();
    let mut obj = serde_json::json!({
        "windows": result.windows.len(),
        "stitched_total_pips": total,
        "param_stability": param_stability(result),
    });
    if result.windows.iter().any(|w| w.run.oos_report.metrics.r.is_some()) {
        // RMetrics serializes its scalar fields (trade_rs is serde-skipped, so the
        // oos_r block is the clean R-metric summary of the pooled series).
        obj["oos_r"] = serde_json::to_value(r_metrics_from_rs(&pooled_rs))
            .expect("RMetrics serializes");
    }
    serde_json::json!({ "walkforward": obj }).to_string()
}

(Import r_metrics_from_rs from aura_engine alongside the existing summarize_r / summarize imports at the top of main.rs.)

  • Step 8: Build + run the E2E test + the determinism/golden guards

Run: cargo build --workspace Expected: 0 errors Run: cargo test --workspace walkforward_strategy_stage1_r_reports_oos_r Expected: PASS Run: cargo test --workspace walkforward_report_has_one_oos_line_per_window_plus_summary Expected: PASS (SMA summary unchanged — no oos_r key, windows have r: None)

  • Step 9: Full-suite regression gate

Run: cargo test --workspace Expected: PASS — all pre-existing goldens (SMA walkforward, synthetic mc, stage1-r single-run stage1_r_single_run_output_golden, C18 round-trips) green; the new tests green.

Run: cargo clippy --workspace --all-targets -- -D warnings Expected: 0 warnings.


Self-review

  • Spec coverage: Changes 1 (RMetrics.trade_rs / summarize_r — Task 1), the pooled reducer (Task 2), Change 3 walkforward dispatch (Tasks 3+4), Change 5 USAGE (Task 3). Monte-Carlo (spec Changes 2+4) is iteration 2 — out of this plan's scope by design. ✓
  • Placeholder scan: no TBD/TODO; the Task-3 stage1-r error stub is an explicit, compiling intermediate filled in Task 4 (named, not a placeholder). ✓
  • Type/name consistency: RMetrics.trade_rs, r_metrics_from_rs, stage1_r_sweep_over, run_oos_r, stage1_r_space, Strategy::Stage1R, Stage1RGrid used identically across tasks. ✓
  • Step granularity: each step is one edit or one command. ✓
  • No commit steps.
  • Compile-gate ordering (rule 7): Task 1 threads all 4 RMetrics { .. } literal sites before its build gate; Task 3 threads all parse_walkforward_args / walkforward_family / run_walkforward callers (dispatch arm + 2 test helpers + parse-test assertions) before its build gate, with the stage1-r arm a compiling error-stub so the gate is satisfiable. ✓
  • Filter strings (rule 8): every cargo test filter names a real test — pre-existing (runmetrics_with_r_block_round_trips, walkforward_report_has_one_oos_line_per_window_plus_summary, parse_walkforward_args_defaults_and_accepts_real, stage1_r_single_run_output_golden) or introduced in the same task; the final gate is the unfiltered cargo test --workspace. ✓