diff --git a/docs/specs/0075-walkforward-mc-r-validation.md b/docs/specs/0075-walkforward-mc-r-validation.md new file mode 100644 index 0000000..f20c8fa --- /dev/null +++ b/docs/specs/0075-walkforward-mc-r-validation.md @@ -0,0 +1,359 @@ +# Walk-forward + Monte-Carlo OOS validation for R candidates — Design Spec + +**Date:** 2026-06-26 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude +**Issue:** #139 (reference issue; design forks recorded in its comment thread) + +## Goal + +Make the World layer's two out-of-sample validation verbs work on an +**R-based candidate** on real data, not just the SMA sample blueprint: + +1. **`aura walkforward --strategy stage1-r --real `** — roll IS→OOS windows, + sweep the stage1-r grid in-sample, pick the best by an **R metric**, run the + chosen params out-of-sample, and report **per-window and across-window-pooled + OOS R-metrics** (the `r` block), not pip-only metrics. +2. **`aura mc --strategy stage1-r --real `** — bootstrap the walk-forward's + pooled OOS per-trade R series to a confidence interval on `E[R]`, so a positive + OOS edge can be shown (un)likely under resampling. + +Both are **frictionless Stage-1 R** (C7): costs are Stage-2, out of scope here. +The existing SMA/`total_pips` `walkforward` path and the synthetic-seed `mc` path +are **preserved byte-for-byte** (additive change; `--strategy`/`--real` select the +new behaviour, their absence keeps today's). + +Non-goals (recorded on #139): no genericisation of the orchestration over a metric +type (that is #136, which absorbs these call sites later); no cross-*symbol* +pooling (needs a multi-symbol CLI grammar — deferred follow-on); no costs. + +## Architecture + +The library substrate is already candidate- and metric-generic and stays +unchanged in signature: + +- `walk_forward(roller, space, run_window)` (`aura-engine/src/walkforward.rs:185`) + is generic over the `run_window` closure; it rolls bounds, runs each window + disjointly (C1), and stitches the OOS pip-equity. **No change.** +- `WindowRoller::new(span, is_len, oos_len, step, RollMode)` + (`walkforward.rs:73`). **No change.** +- `optimize(family, metric)` / `metric_cmp` (`aura-registry/src/lib.rs:186/125`) + already resolve the R metric names (`sqn`, `sqn_normalized`, `expectancy_r`, + `net_expectancy_r`), sorting an `r: None` member to `NEG_INFINITY`. **No change.** +- `monte_carlo(base, seeds, run_one)` (`mc.rs:122`) is MC-over-seeds — the **wrong + axis** for this work; it is left untouched for the synthetic path. + +Two library changes, one CLI surface change: + +- **`RMetrics` gains an in-memory-only per-trade R vector** (`report.rs:45`), + populated by `summarize_r`, excluded from serde and equality (fork A on #139). +- **A new deterministic block-bootstrap primitive** `r_bootstrap` in + `aura-engine/src/mc.rs`, over a `&[f64]` R series (fork on #139). +- **`walkforward` and `mc` become strategy-selectable** in `aura-cli/src/main.rs`, + dispatching the per-window/per-draw work onto the stage1-r blueprint and folding + via `summarize_r`; the stage1-r IS sweep uses reduce-mode folded sinks, the OOS + run uses raw recorders (fork B on #139). + +The varying dimension of `walk_forward` is the data window (C12 axis 3); MC here is +not axis 4 (seeds) but a post-hoc resample of an existing R sample. + +## Concrete code shapes + +### User-facing surface (the acceptance-criterion evidence) + +A researcher validating a candidate runs exactly: + +```console +$ aura walkforward --strategy stage1-r --real USDJPY --from 1672531200000 --to 1735689599000 +{"family_id":"wf-...","report":{...,"metrics":{"total_pips":...,"max_drawdown":..., + "bias_sign_flips":...,"r":{"expectancy_r":0.12,"sqn_normalized":0.34,"n_trades":21,...}}}} +... one line per OOS window, each carrying the r block ... +{"walkforward":{"windows":21,"stitched_total_pips":-171.7, + "oos_r":{"expectancy_r":-0.04,"sqn_normalized":-0.11,"n_trades":438,...}, + "param_stability":[...]}} +``` + +```console +$ aura mc --strategy stage1-r --real USDJPY --resamples 2000 --block-len 5 --seed 1 +{"mc_r_bootstrap":{"n_trades":438,"block_len":5,"n_resamples":2000, + "e_r":{"mean":-0.04,"p5":-0.18,"p25":-0.10,"p50":-0.04,"p75":0.02,"p95":0.10}, + "prob_le_zero":0.71}} +``` + +The bare forms are unchanged: + +```console +$ aura walkforward # SMA sample, pip-only — byte-identical to today +$ aura mc # synthetic 3-seed resweep — byte-identical to today +$ aura mc --name foo # unchanged +``` + +What the criterion demands and this shows: the intended audience (a researcher +proving OOS edge) reaches for these two commands directly; the output is the +instrument-agnostic R block + a resampled `E[R]` CI; no cost/friction enters +(C7 Stage-1); determinism (C1) holds because the roller, the sweep, and the +seeded bootstrap are all pure. + +### Change 1 — `RMetrics` carries the per-trade R series (in-memory only) + +`crates/aura-engine/src/report.rs:45` + +```rust +// BEFORE +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct RMetrics { + pub expectancy_r: f64, + // ... all existing fields ... + pub conviction_terciles_r: [f64; 3], +} + +// AFTER — `trade_rs` is excluded from serde (skip) AND equality, so the C18 wire +// shape and every existing RMetrics/RunReport equality assertion are unchanged. +// It is a pure in-memory conduit: summarize_r already builds this vector (the +// local `rs`, report.rs:137) and drops it; now it is retained for the bootstrap. +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] +pub struct RMetrics { + pub expectancy_r: f64, + // ... all existing fields ... + pub conviction_terciles_r: [f64; 3], + #[serde(skip)] + pub trade_rs: Vec, // realised R per closed trade, in trade order; empty after deserialize +} + +// PartialEq hand-written to ignore `trade_rs` (so a serialize->deserialize +// round-trip still compares equal, and the all-zero `RMetrics` literal in +// summarize_r's empty-input arm needs no R-vector to match). +impl PartialEq for RMetrics { + fn eq(&self, o: &Self) -> bool { + self.expectancy_r == o.expectancy_r + && /* ... all existing fields, trade_rs excluded ... */ + && self.conviction_terciles_r == o.conviction_terciles_r + } +} +``` + +`summarize_r` (`report.rs:89`) sets `trade_rs: rs.clone()` (or moves `rs`) into the +returned `RMetrics` on the non-empty path; the empty-input arm sets +`trade_rs: Vec::new()`. + +### Change 2 — the block-bootstrap primitive + +`crates/aura-engine/src/mc.rs` (new function + result struct, reusing the existing +`MetricStats`/`quantile`) + +```rust +/// Distribution of E[R] under a moving-block bootstrap of an OOS per-trade R +/// series. `block_len == 1` is the i.i.d. trade-shuffle; `block_len > 1` preserves +/// the serial correlation of sequential trades (winner/loser streaks), which a +/// pure shuffle would erase and so overstate confidence. Deterministic (C1) given +/// `seed` via the existing SplitMix64. +#[derive(Clone, Debug, PartialEq)] +pub struct RBootstrap { + pub e_r: MetricStats, // mean + p5/p25/p50/p75/p95 of the resampled E[R] + pub prob_le_zero: f64, // fraction of resamples whose mean R <= 0 + pub n_trades: usize, + pub block_len: usize, + pub n_resamples: usize, +} + +/// Moving-block bootstrap (non-circular; the final block is truncated so each +/// resample has exactly `n` trades). Empty `rs` -> all-zero RBootstrap. `block_len` +/// is clamped to `[1, n]`. +pub fn r_bootstrap(rs: &[f64], n_resamples: usize, block_len: usize, seed: u64) -> RBootstrap { + let n = rs.len(); + // n == 0 guard -> all-zero result; block_len = block_len.clamp(1, n.max(1)). + let mut rng = SplitMix64::new(seed); + let mut means = Vec::with_capacity(n_resamples); + for _ in 0..n_resamples { + let mut sample = Vec::with_capacity(n); + while sample.len() < n { + let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize; + for j in start..(start + block_len).min(start + (n - sample.len())) { + sample.push(rs[j]); + } + } + means.push(sample.iter().sum::() / n as f64); + } + RBootstrap { + e_r: MetricStats::from_values(&means), + prob_le_zero: means.iter().filter(|&&m| m <= 0.0).count() as f64 / n_resamples as f64, + n_trades: n, block_len, n_resamples, + } +} +``` + +`SplitMix64` (`harness.rs:139`) is the existing deterministic RNG; if not already +crate-visible, it is promoted to `pub(crate)` (the planner's mechanic). + +### Change 3 — `walkforward` strategy dispatch + +`crates/aura-cli/src/main.rs` — `parse_walkforward_args` gains `--strategy` + the +four grid flags (mirroring `parse_sweep_args:1480`); `walkforward_family` branches +on the strategy. + +```rust +// BEFORE (main.rs:1526) +fn parse_walkforward_args(rest: &[&str]) -> Result<(String, bool, DataChoice), String> { ... } + +// AFTER — same RealWindowGrammar + name/trace handling, plus --strategy and the +// stage1-r grid flags reusing parse_csv_list + Stage1RGrid (verbatim from sweep). +fn parse_walkforward_args( + rest: &[&str], +) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid), String> { ... } +``` + +```rust +// BEFORE (main.rs:1643) — hardwired SMA, optimize by total_pips, summarize (r:None) +fn walkforward_family(trace: Option<&str>, data: &DataSource) -> WalkForwardResult { + let space = sample_blueprint_with_sinks(data.pip_size()).0.param_space(); + walk_forward(roller, space, |w| { + let is_family = sweep_over(w.is.0, w.is.1, data); + let best = optimize(&is_family, "total_pips").expect(...); + let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) +} + +// AFTER — strategy-dispatched. SMA arm is verbatim today's code (goldens hold). +fn walkforward_family( + strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid, +) -> WalkForwardResult { + match strategy { + Strategy::SmaCross => { /* exactly today's body */ } + Strategy::Stage1R => { + let space = stage1_r_space(); // param_space of the open stage1_r_graph + walk_forward(roller, space, |w| { + let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); // reduce-mode fold + let best = optimize(&is_family, "sqn_normalized").expect("known R metric"); + let (oos_equity, oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) + } + // other stage1 strategies: out of scope this cycle -> usage error or SmaCross-style guard + _ => { /* return a clear "strategy has no walkforward form" error, like run_dispatch */ } + } +} +``` + +New helpers (windowed siblings of the SMA `sweep_over`/`run_oos`, reusing the +stage1-r machinery at `main.rs:1180`/`2112`): + +- `stage1_r_sweep_over(from, to, data, grid) -> SweepFamily` — the in-sample + stage1-r sweep over `data.windowed_sources(from, to)`, **reduce-mode** (folded + `SeriesReducer`/`GatedRecorder`, O(trades)/member, `m.r = Some(summarize_r(..))`) + so the per-window sweep footprint stays bounded. +- `run_oos_r(params, from, to, trace, data) -> (Vec<(Timestamp,f64)>, RunReport)` — + the chosen-params OOS run over the windowed sources, **non-reduce** raw recorders + (the pip-equity curve is needed for `stitch`); folds `summarize` for the pip + block and `summarize_r` for `metrics.r` (which now carries `trade_rs`). + +`walkforward_summary_json` (`main.rs:1734`) gains an `oos_r` block: the pooled +across-windows R-metrics, computed by concatenating each window's +`oos_report.metrics.r.trade_rs` in roll order and folding (the pooled trade series +→ `summarize`-style R reduction, or a thin `RMetrics`-from-`trade_rs` reducer). The +SMA path emits no `oos_r` (its windows have `r: None`), so its summary line is +unchanged. + +### Change 4 — `mc` strategy dispatch + bootstrap + +`crates/aura-cli/src/main.rs` — replace the three literal `mc` dispatch arms with a +single `["mc", rest @ ..]` routed through a new `parse_mc_args`. + +```rust +// BEFORE (main.rs:2646) +["mc"] => run_mc("mc", false), +["mc", "--name", n] => run_mc(n, false), +["mc", "--trace", n] => run_mc(n, true), + +// AFTER +["mc", rest @ ..] => match parse_mc_args(rest) { + Ok(McArgs::Synthetic { name, persist }) => run_mc(&name, persist), // today's path, unchanged + Ok(McArgs::RealR { strategy, choice, grid, block_len, n_resamples, seed }) => + run_mc_r_bootstrap(strategy, DataSource::from_choice(choice), &grid, block_len, n_resamples, seed), + Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } +}, +``` + +`run_mc_r_bootstrap` reuses `walkforward_family(strategy, None, &data, grid)` to +obtain the pooled OOS R series, then `r_bootstrap(&pooled, n_resamples, block_len, +seed)`, and prints one `mc_r_bootstrap` JSON line (shape above). Defaults: +`--resamples 1000`, `--block-len 1`, `--seed 1` (recorded; derivable). + +### Change 5 — USAGE + +`main.rs:2591` — extend the `walkforward` and `mc` fragments with +`[--strategy <...>] [--fast ] ...` and (for mc) `[--block-len ] +[--resamples ] [--seed ]`, matching the new grammars. + +## Components + +| Component | Location | Change | +|---|---|---| +| `RMetrics.trade_rs` + hand-written `PartialEq` | `aura-engine/report.rs:45` | new in-memory field, serde+eq excluded | +| `summarize_r` | `aura-engine/report.rs:89` | retain `rs` into `trade_rs` (was dropped) | +| `r_bootstrap` + `RBootstrap` | `aura-engine/mc.rs` | new deterministic moving-block bootstrap | +| `SplitMix64` visibility | `aura-engine/harness.rs:139` | `pub(crate)` if needed | +| `parse_walkforward_args` | `aura-cli/main.rs:1526` | `--strategy` + grid flags | +| `walkforward_family` | `aura-cli/main.rs:1643` | strategy dispatch; SMA arm verbatim | +| `stage1_r_sweep_over`, `run_oos_r`, `stage1_r_space` | `aura-cli/main.rs` (new) | windowed reduce-mode IS sweep + non-reduce OOS run | +| `walkforward_summary_json` | `aura-cli/main.rs:1734` | `oos_r` pooled block (stage1-r only) | +| `parse_mc_args` + `McArgs` + `run_mc_r_bootstrap` | `aura-cli/main.rs` (new) | mc real-candidate path | +| `mc` dispatch arms, USAGE | `aura-cli/main.rs:2646/2591` | single `["mc", rest @ ..]`, usage text | + +## Data flow + +1. **walk-forward (stage1-r):** roller → per window: windowed IS sources → + stage1-r reduce-mode sweep → `optimize(.., "sqn_normalized")` → chosen params → + windowed OOS run (non-reduce) → `(oos_equity, RunReport{metrics.r{..,trade_rs}})` + → `WindowRun`. `walk_forward` stitches pip-equity; summary folds the pooled + `trade_rs` into `oos_r`. +2. **mc (stage1-r):** run (1) → concat `windows[*].oos_report.metrics.r.trade_rs` + in roll order → `r_bootstrap` → `RBootstrap` → one JSON line. + +No look-ahead (C2: `WindowRoller` guarantees `oos.0 > is.1`). Disjoint windows run +in parallel (C1). The bootstrap is a pure post-run reduction (C12). + +## Error handling + +- Malformed grid list / unknown strategy / flag without value → the subcommand + `usage()` string (strict, like `parse_sweep_args`). +- A strategy with no walk-forward/mc form (e.g. `momentum`) → a clear error string + (mirroring `run_dispatch`'s `HarnessKind` guards), not a panic. +- Span too short for one IS+OOS window → today's `WindowRoller::new` error path + (`main.rs:1646` exit 2), unchanged. +- Empty pooled R series (no OOS trades) → `r_bootstrap` returns an all-zero + `RBootstrap` (well-defined), not a panic. + +## Testing strategy + +- **Goldens preserved (add, don't break):** existing `aura walkforward` (SMA) and + `aura mc` (synthetic) output goldens must stay byte-identical — covered by + running the bare forms; the SMA `walkforward_family` arm and `mc_family` are + untouched. C18 `runs.jsonl` shape unchanged (`trade_rs` is `serde(skip)`). +- **`r_bootstrap` unit goldens:** fixed `rs` + fixed seed → pinned `RBootstrap` + quantiles; `block_len = 1` vs `> 1` both deterministic; `block_len = n` (one + block) → every resample equals the full series mean; empty `rs` → all-zero. +- **`RMetrics` round-trip:** serialize → deserialize → `PartialEq` still equal + (the hand-written `eq` ignores `trade_rs`), pinning fork-A back-compat. +- **walk-forward stage1-r E2E:** `aura walkforward --strategy stage1-r --real ` + over a small fixed window emits per-window `r` blocks and an `oos_r` summary; + determinism (same input → same output) pinned. +- **mc stage1-r E2E:** `aura mc --strategy stage1-r --real ` emits a + `mc_r_bootstrap` line; same seed → same line. +- **arg-parsing units:** `parse_walkforward_args` / `parse_mc_args` accept the new + flags and reject malformed input, mirroring `parse_sweep_args` unit tests. + +## Acceptance criteria + +1. `aura walkforward --strategy stage1-r --real ` reports per-window and + pooled OOS **R-metrics** (the `r` block + `oos_r` summary), not pip-only. +2. `aura mc --strategy stage1-r --real ` reports a bootstrap `E[R]` + distribution / CI over the pooled OOS per-trade R series. +3. The four stage1-r grid flags (`--fast/--slow/--stop-length/--stop-k`) drive the + per-window IS sweep, optimised by an R metric. +4. Bare `aura walkforward` and `aura mc` (and `--name`/`--trace`) are + byte-identical to today; C18 wire shape unchanged. +5. Frictionless Stage-1 R: no cost enters the path. Determinism (C1) holds: + same input + same seed → same output. +6. Per-member walk-forward footprint stays O(trades) — the IS sweep folds + (reduce-mode), not O(cycles).