Aura/docs/specs/0075-walkforward-mc-r-validation.md

Walk-forward + Monte-Carlo OOS validation for R candidates — Design Spec

Date: 2026-06-26 Status: Draft — awaiting user spec review Authors: orchestrator + Claude Issue: #139 (reference issue; design forks recorded in its comment thread)

Goal

Make the World layer's two out-of-sample validation verbs work on an R-based candidate on real data, not just the SMA sample blueprint:

1. aura walkforward --strategy stage1-r --real <SYM> — roll IS→OOS windows, sweep the stage1-r grid in-sample, pick the best by an R metric, run the chosen params out-of-sample, and report per-window and across-window-pooled OOS R-metrics (the r block), not pip-only metrics.
2. aura mc --strategy stage1-r --real <SYM> — bootstrap the walk-forward's pooled OOS per-trade R series to a confidence interval on E[R], so a positive OOS edge can be shown (un)likely under resampling.

Both are frictionless Stage-1 R (C7): costs are Stage-2, out of scope here. The existing SMA/total_pips walkforward path and the synthetic-seed mc path are preserved byte-for-byte (additive change; --strategy/--real select the new behaviour, their absence keeps today's).

Non-goals (recorded on #139): no genericisation of the orchestration over a metric type (that is #136, which absorbs these call sites later); no cross-symbol pooling (needs a multi-symbol CLI grammar — deferred follow-on); no costs.

Architecture

The library substrate is already candidate- and metric-generic and stays unchanged in signature:

• walk_forward(roller, space, run_window) (aura-engine/src/walkforward.rs:185) is generic over the run_window closure; it rolls bounds, runs each window disjointly (C1), and stitches the OOS pip-equity. No change.
• WindowRoller::new(span, is_len, oos_len, step, RollMode) (walkforward.rs:73). No change.
• optimize(family, metric) / metric_cmp (aura-registry/src/lib.rs:186/125) already resolve the R metric names (sqn, sqn_normalized, expectancy_r, net_expectancy_r), sorting an r: None member to NEG_INFINITY. No change.
• monte_carlo(base, seeds, run_one) (mc.rs:122) is MC-over-seeds — the wrong axis for this work; it is left untouched for the synthetic path.

Two library changes, one CLI surface change:

• RMetrics gains an in-memory-only per-trade R vector (report.rs:45), populated by summarize_r, excluded from serde and equality (fork A on #139).
• A new deterministic block-bootstrap primitive r_bootstrap in aura-engine/src/mc.rs, over a &[f64] R series (fork on #139).
• walkforward and mc become strategy-selectable in aura-cli/src/main.rs, dispatching the per-window/per-draw work onto the stage1-r blueprint and folding via summarize_r; the stage1-r IS sweep uses reduce-mode folded sinks, the OOS run uses raw recorders (fork B on #139).

The varying dimension of walk_forward is the data window (C12 axis 3); MC here is not axis 4 (seeds) but a post-hoc resample of an existing R sample.

Concrete code shapes

User-facing surface (the acceptance-criterion evidence)

A researcher validating a candidate runs exactly:

$ aura walkforward --strategy stage1-r --real USDJPY --from 1672531200000 --to 1735689599000
{"family_id":"wf-...","report":{...,"metrics":{"total_pips":...,"max_drawdown":...,
  "bias_sign_flips":...,"r":{"expectancy_r":0.12,"sqn_normalized":0.34,"n_trades":21,...}}}}
... one line per OOS window, each carrying the r block ...
{"walkforward":{"windows":21,"stitched_total_pips":-171.7,
  "oos_r":{"expectancy_r":-0.04,"sqn_normalized":-0.11,"n_trades":438,...},
  "param_stability":[...]}}

$ aura mc --strategy stage1-r --real USDJPY --resamples 2000 --block-len 5 --seed 1
{"mc_r_bootstrap":{"n_trades":438,"block_len":5,"n_resamples":2000,
  "e_r":{"mean":-0.04,"p5":-0.18,"p25":-0.10,"p50":-0.04,"p75":0.02,"p95":0.10},
  "prob_le_zero":0.71}}

The bare forms are unchanged:

$ aura walkforward                 # SMA sample, pip-only — byte-identical to today
$ aura mc                          # synthetic 3-seed resweep — byte-identical to today
$ aura mc --name foo               # unchanged

What the criterion demands and this shows: the intended audience (a researcher proving OOS edge) reaches for these two commands directly; the output is the instrument-agnostic R block + a resampled E[R] CI; no cost/friction enters (C7 Stage-1); determinism (C1) holds because the roller, the sweep, and the seeded bootstrap are all pure.

Change 1 — RMetrics carries the per-trade R series (in-memory only)

crates/aura-engine/src/report.rs:45

// BEFORE
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct RMetrics {
    pub expectancy_r: f64,
    // ... all existing fields ...
    pub conviction_terciles_r: [f64; 3],
}

// AFTER — `trade_rs` is excluded from serde (skip) AND equality, so the C18 wire
// shape and every existing RMetrics/RunReport equality assertion are unchanged.
// It is a pure in-memory conduit: summarize_r already builds this vector (the
// local `rs`, report.rs:137) and drops it; now it is retained for the bootstrap.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct RMetrics {
    pub expectancy_r: f64,
    // ... all existing fields ...
    pub conviction_terciles_r: [f64; 3],
    #[serde(skip)]
    pub trade_rs: Vec<f64>,   // realised R per closed trade, in trade order; empty after deserialize
}

// PartialEq hand-written to ignore `trade_rs` (so a serialize->deserialize
// round-trip still compares equal, and the all-zero `RMetrics` literal in
// summarize_r's empty-input arm needs no R-vector to match).
impl PartialEq for RMetrics {
    fn eq(&self, o: &Self) -> bool {
        self.expectancy_r == o.expectancy_r
            && /* ... all existing fields, trade_rs excluded ... */
            && self.conviction_terciles_r == o.conviction_terciles_r
    }
}

summarize_r (report.rs:89) sets trade_rs: rs.clone() (or moves rs) into the returned RMetrics on the non-empty path; the empty-input arm sets trade_rs: Vec::new().

Change 2 — the block-bootstrap primitive

crates/aura-engine/src/mc.rs (new function + result struct, reusing the existing MetricStats/quantile)

/// Distribution of E[R] under a moving-block bootstrap of an OOS per-trade R
/// series. `block_len == 1` is the i.i.d. trade-shuffle; `block_len > 1` preserves
/// the serial correlation of sequential trades (winner/loser streaks), which a
/// pure shuffle would erase and so overstate confidence. Deterministic (C1) given
/// `seed` via the existing SplitMix64.
#[derive(Clone, Debug, PartialEq)]
pub struct RBootstrap {
    pub e_r: MetricStats,   // mean + p5/p25/p50/p75/p95 of the resampled E[R]
    pub prob_le_zero: f64,  // fraction of resamples whose mean R <= 0
    pub n_trades: usize,
    pub block_len: usize,
    pub n_resamples: usize,
}

/// Moving-block bootstrap (non-circular; the final block is truncated so each
/// resample has exactly `n` trades). Empty `rs` -> all-zero RBootstrap. `block_len`
/// is clamped to `[1, n]`.
pub fn r_bootstrap(rs: &[f64], n_resamples: usize, block_len: usize, seed: u64) -> RBootstrap {
    let n = rs.len();
    // n == 0 guard -> all-zero result; block_len = block_len.clamp(1, n.max(1)).
    let mut rng = SplitMix64::new(seed);
    let mut means = Vec::with_capacity(n_resamples);
    for _ in 0..n_resamples {
        let mut sample = Vec::with_capacity(n);
        while sample.len() < n {
            let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize;
            for j in start..(start + block_len).min(start + (n - sample.len())) {
                sample.push(rs[j]);
            }
        }
        means.push(sample.iter().sum::<f64>() / n as f64);
    }
    RBootstrap {
        e_r: MetricStats::from_values(&means),
        prob_le_zero: means.iter().filter(|&&m| m <= 0.0).count() as f64 / n_resamples as f64,
        n_trades: n, block_len, n_resamples,
    }
}

SplitMix64 (harness.rs:139) is the existing deterministic RNG; if not already crate-visible, it is promoted to pub(crate) (the planner's mechanic).

Change 3 — walkforward strategy dispatch

crates/aura-cli/src/main.rs — parse_walkforward_args gains --strategy + the four grid flags (mirroring parse_sweep_args:1480); walkforward_family branches on the strategy.

// BEFORE  (main.rs:1526)
fn parse_walkforward_args(rest: &[&str]) -> Result<(String, bool, DataChoice), String> { ... }

// AFTER — same RealWindowGrammar + name/trace handling, plus --strategy and the
// stage1-r grid flags reusing parse_csv_list + Stage1RGrid (verbatim from sweep).
fn parse_walkforward_args(
    rest: &[&str],
) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid), String> { ... }

// BEFORE  (main.rs:1643) — hardwired SMA, optimize by total_pips, summarize (r:None)
fn walkforward_family(trace: Option<&str>, data: &DataSource) -> WalkForwardResult {
    let space = sample_blueprint_with_sinks(data.pip_size()).0.param_space();
    walk_forward(roller, space, |w| {
        let is_family = sweep_over(w.is.0, w.is.1, data);
        let best = optimize(&is_family, "total_pips").expect(...);
        let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
        WindowRun { chosen_params: best.params, oos_equity, oos_report }
    })
}

// AFTER — strategy-dispatched. SMA arm is verbatim today's code (goldens hold).
fn walkforward_family(
    strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid,
) -> WalkForwardResult {
    match strategy {
        Strategy::SmaCross => { /* exactly today's body */ }
        Strategy::Stage1R => {
            let space = stage1_r_space();   // param_space of the open stage1_r_graph
            walk_forward(roller, space, |w| {
                let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); // reduce-mode fold
                let best = optimize(&is_family, "sqn_normalized").expect("known R metric");
                let (oos_equity, oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
                WindowRun { chosen_params: best.params, oos_equity, oos_report }
            })
        }
        // other stage1 strategies: out of scope this cycle -> usage error or SmaCross-style guard
        _ => { /* return a clear "strategy has no walkforward form" error, like run_dispatch */ }
    }
}

New helpers (windowed siblings of the SMA sweep_over/run_oos, reusing the stage1-r machinery at main.rs:1180/2112):

• stage1_r_sweep_over(from, to, data, grid) -> SweepFamily — the in-sample stage1-r sweep over data.windowed_sources(from, to), reduce-mode (folded SeriesReducer/GatedRecorder, O(trades)/member, m.r = Some(summarize_r(..))) so the per-window sweep footprint stays bounded.
• run_oos_r(params, from, to, trace, data) -> (Vec<(Timestamp,f64)>, RunReport) — the chosen-params OOS run over the windowed sources, non-reduce raw recorders (the pip-equity curve is needed for stitch); folds summarize for the pip block and summarize_r for metrics.r (which now carries trade_rs).

walkforward_summary_json (main.rs:1734) gains an oos_r block: the pooled across-windows R-metrics, computed by concatenating each window's oos_report.metrics.r.trade_rs in roll order and folding (the pooled trade series → summarize-style R reduction, or a thin RMetrics-from-trade_rs reducer). The SMA path emits no oos_r (its windows have r: None), so its summary line is unchanged.

Change 4 — mc strategy dispatch + bootstrap

crates/aura-cli/src/main.rs — replace the three literal mc dispatch arms with a single ["mc", rest @ ..] routed through a new parse_mc_args.

// BEFORE  (main.rs:2646)
["mc"] => run_mc("mc", false),
["mc", "--name", n] => run_mc(n, false),
["mc", "--trace", n] => run_mc(n, true),

// AFTER
["mc", rest @ ..] => match parse_mc_args(rest) {
    Ok(McArgs::Synthetic { name, persist }) => run_mc(&name, persist),     // today's path, unchanged
    Ok(McArgs::RealR { strategy, choice, grid, block_len, n_resamples, seed }) =>
        run_mc_r_bootstrap(strategy, DataSource::from_choice(choice), &grid, block_len, n_resamples, seed),
    Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
},

run_mc_r_bootstrap reuses walkforward_family(strategy, None, &data, grid) to obtain the pooled OOS R series, then r_bootstrap(&pooled, n_resamples, block_len, seed), and prints one mc_r_bootstrap JSON line (shape above). Defaults: --resamples 1000, --block-len 1, --seed 1 (recorded; derivable).

Change 5 — USAGE

main.rs:2591 — extend the walkforward and mc fragments with [--strategy <...>] [--fast <csv>] ... and (for mc) [--block-len <n>] [--resamples <n>] [--seed <n>], matching the new grammars.

Components

Component	Location	Change
RMetrics.trade_rs + hand-written PartialEq	aura-engine/report.rs:45	new in-memory field, serde+eq excluded
summarize_r	aura-engine/report.rs:89	retain rs into trade_rs (was dropped)
r_bootstrap + RBootstrap	aura-engine/mc.rs	new deterministic moving-block bootstrap
SplitMix64 visibility	aura-engine/harness.rs:139	pub(crate) if needed
parse_walkforward_args	aura-cli/main.rs:1526	--strategy + grid flags
walkforward_family	aura-cli/main.rs:1643	strategy dispatch; SMA arm verbatim
stage1_r_sweep_over, run_oos_r, stage1_r_space	aura-cli/main.rs (new)	windowed reduce-mode IS sweep + non-reduce OOS run
walkforward_summary_json	aura-cli/main.rs:1734	oos_r pooled block (stage1-r only)
parse_mc_args + McArgs + run_mc_r_bootstrap	aura-cli/main.rs (new)	mc real-candidate path
mc dispatch arms, USAGE	aura-cli/main.rs:2646/2591	single ["mc", rest @ ..], usage text

Data flow

1. walk-forward (stage1-r): roller → per window: windowed IS sources → stage1-r reduce-mode sweep → optimize(.., "sqn_normalized") → chosen params → windowed OOS run (non-reduce) → (oos_equity, RunReport{metrics.r{..,trade_rs}}) → WindowRun. walk_forward stitches pip-equity; summary folds the pooled trade_rs into oos_r.
2. mc (stage1-r): run (1) → concat windows[*].oos_report.metrics.r.trade_rs in roll order → r_bootstrap → RBootstrap → one JSON line.

No look-ahead (C2: WindowRoller guarantees oos.0 > is.1). Disjoint windows run in parallel (C1). The bootstrap is a pure post-run reduction (C12).

Error handling

• Malformed grid list / unknown strategy / flag without value → the subcommand usage() string (strict, like parse_sweep_args).
• A strategy with no walk-forward/mc form (e.g. momentum) → a clear error string (mirroring run_dispatch's HarnessKind guards), not a panic.
• Span too short for one IS+OOS window → today's WindowRoller::new error path (main.rs:1646 exit 2), unchanged.
• Empty pooled R series (no OOS trades) → r_bootstrap returns an all-zero RBootstrap (well-defined), not a panic.

Testing strategy

• Goldens preserved (add, don't break): existing aura walkforward (SMA) and aura mc (synthetic) output goldens must stay byte-identical — covered by running the bare forms; the SMA walkforward_family arm and mc_family are untouched. C18 runs.jsonl shape unchanged (trade_rs is serde(skip)).
• r_bootstrap unit goldens: fixed rs + fixed seed → pinned RBootstrap quantiles; block_len = 1 vs > 1 both deterministic; block_len = n (one block) → every resample equals the full series mean; empty rs → all-zero.
• RMetrics round-trip: serialize → deserialize → PartialEq still equal (the hand-written eq ignores trade_rs), pinning fork-A back-compat.
• walk-forward stage1-r E2E: aura walkforward --strategy stage1-r --real <SYM> over a small fixed window emits per-window r blocks and an oos_r summary; determinism (same input → same output) pinned.
• mc stage1-r E2E: aura mc --strategy stage1-r --real <SYM> emits a mc_r_bootstrap line; same seed → same line.
• arg-parsing units: parse_walkforward_args / parse_mc_args accept the new flags and reject malformed input, mirroring parse_sweep_args unit tests.

Acceptance criteria

1. aura walkforward --strategy stage1-r --real <SYM> reports per-window and pooled OOS R-metrics (the r block + oos_r summary), not pip-only.
2. aura mc --strategy stage1-r --real <SYM> reports a bootstrap E[R] distribution / CI over the pooled OOS per-trade R series.
3. The four stage1-r grid flags (--fast/--slow/--stop-length/--stop-k) drive the per-window IS sweep, optimised by an R metric.
4. Bare aura walkforward and aura mc (and --name/--trace) are byte-identical to today; C18 wire shape unchanged.
5. Frictionless Stage-1 R: no cost enters the path. Determinism (C1) holds: same input + same seed → same output.
6. Per-member walk-forward footprint stays O(trades) — the IS sweep folds (reduce-mode), not O(cycles).