spec: 0075 walk-forward + Monte-Carlo OOS validation for R candidates (boss-signed)

Make the World layer's two OOS validation verbs candidate-aware and R-reporting instead of locked to the SMA sample blueprint: - aura walkforward --strategy stage1-r --real <SYM>: roll IS->OOS, sweep the stage1-r grid in-sample, pick by an R metric, run OOS, report per-window + across-window-pooled OOS R-metrics. - aura mc --strategy stage1-r --real <SYM>: moving-block bootstrap of the pooled OOS per-trade R series -> E[R] distribution / CI. Frictionless Stage-1 R (costs are Stage-2). Bare walkforward/mc paths and their goldens preserved byte-for-byte. Grounding-check PASS; signed under /boss (design forks recorded on the reference issue). refs #139
2026-06-26 10:12:40 +02:00
parent c99e7b8f7f
commit 3a1de05ef3
1 changed files with 359 additions and 0 deletions
@@ -0,0 +1,359 @@
+# Walk-forward + Monte-Carlo OOS validation for R candidates — Design Spec
+
+**Date:** 2026-06-26
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+**Issue:** #139 (reference issue; design forks recorded in its comment thread)
+
+## Goal
+
+Make the World layer's two out-of-sample validation verbs work on an
+**R-based candidate** on real data, not just the SMA sample blueprint:
+
+1. **`aura walkforward --strategy stage1-r --real <SYM>`** — roll IS→OOS windows,
+   sweep the stage1-r grid in-sample, pick the best by an **R metric**, run the
+   chosen params out-of-sample, and report **per-window and across-window-pooled
+   OOS R-metrics** (the `r` block), not pip-only metrics.
+2. **`aura mc --strategy stage1-r --real <SYM>`** — bootstrap the walk-forward's
+   pooled OOS per-trade R series to a confidence interval on `E[R]`, so a positive
+   OOS edge can be shown (un)likely under resampling.
+
+Both are **frictionless Stage-1 R** (C7): costs are Stage-2, out of scope here.
+The existing SMA/`total_pips` `walkforward` path and the synthetic-seed `mc` path
+are **preserved byte-for-byte** (additive change; `--strategy`/`--real` select the
+new behaviour, their absence keeps today's).
+
+Non-goals (recorded on #139): no genericisation of the orchestration over a metric
+type (that is #136, which absorbs these call sites later); no cross-*symbol*
+pooling (needs a multi-symbol CLI grammar — deferred follow-on); no costs.
+
+## Architecture
+
+The library substrate is already candidate- and metric-generic and stays
+unchanged in signature:
+
+- `walk_forward(roller, space, run_window)` (`aura-engine/src/walkforward.rs:185`)
+  is generic over the `run_window` closure; it rolls bounds, runs each window
+  disjointly (C1), and stitches the OOS pip-equity. **No change.**
+- `WindowRoller::new(span, is_len, oos_len, step, RollMode)`
+  (`walkforward.rs:73`). **No change.**
+- `optimize(family, metric)` / `metric_cmp` (`aura-registry/src/lib.rs:186/125`)
+  already resolve the R metric names (`sqn`, `sqn_normalized`, `expectancy_r`,
+  `net_expectancy_r`), sorting an `r: None` member to `NEG_INFINITY`. **No change.**
+- `monte_carlo(base, seeds, run_one)` (`mc.rs:122`) is MC-over-seeds — the **wrong
+  axis** for this work; it is left untouched for the synthetic path.
+
+Two library changes, one CLI surface change:
+
+- **`RMetrics` gains an in-memory-only per-trade R vector** (`report.rs:45`),
+  populated by `summarize_r`, excluded from serde and equality (fork A on #139).
+- **A new deterministic block-bootstrap primitive** `r_bootstrap` in
+  `aura-engine/src/mc.rs`, over a `&[f64]` R series (fork on #139).
+- **`walkforward` and `mc` become strategy-selectable** in `aura-cli/src/main.rs`,
+  dispatching the per-window/per-draw work onto the stage1-r blueprint and folding
+  via `summarize_r`; the stage1-r IS sweep uses reduce-mode folded sinks, the OOS
+  run uses raw recorders (fork B on #139).
+
+The varying dimension of `walk_forward` is the data window (C12 axis 3); MC here is
+not axis 4 (seeds) but a post-hoc resample of an existing R sample.
+
+## Concrete code shapes
+
+### User-facing surface (the acceptance-criterion evidence)
+
+A researcher validating a candidate runs exactly:
+
+```console
+$ aura walkforward --strategy stage1-r --real USDJPY --from 1672531200000 --to 1735689599000
+{"family_id":"wf-...","report":{...,"metrics":{"total_pips":...,"max_drawdown":...,
+  "bias_sign_flips":...,"r":{"expectancy_r":0.12,"sqn_normalized":0.34,"n_trades":21,...}}}}
+... one line per OOS window, each carrying the r block ...
+{"walkforward":{"windows":21,"stitched_total_pips":-171.7,
+  "oos_r":{"expectancy_r":-0.04,"sqn_normalized":-0.11,"n_trades":438,...},
+  "param_stability":[...]}}
+```
+
+```console
+$ aura mc --strategy stage1-r --real USDJPY --resamples 2000 --block-len 5 --seed 1
+{"mc_r_bootstrap":{"n_trades":438,"block_len":5,"n_resamples":2000,
+  "e_r":{"mean":-0.04,"p5":-0.18,"p25":-0.10,"p50":-0.04,"p75":0.02,"p95":0.10},
+  "prob_le_zero":0.71}}
+```
+
+The bare forms are unchanged:
+
+```console
+$ aura walkforward                 # SMA sample, pip-only — byte-identical to today
+$ aura mc                          # synthetic 3-seed resweep — byte-identical to today
+$ aura mc --name foo               # unchanged
+```
+
+What the criterion demands and this shows: the intended audience (a researcher
+proving OOS edge) reaches for these two commands directly; the output is the
+instrument-agnostic R block + a resampled `E[R]` CI; no cost/friction enters
+(C7 Stage-1); determinism (C1) holds because the roller, the sweep, and the
+seeded bootstrap are all pure.
+
+### Change 1 — `RMetrics` carries the per-trade R series (in-memory only)
+
+`crates/aura-engine/src/report.rs:45`
+
+```rust
+// BEFORE
+#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
+pub struct RMetrics {
+    pub expectancy_r: f64,
+    // ... all existing fields ...
+    pub conviction_terciles_r: [f64; 3],
+}
+
+// AFTER — `trade_rs` is excluded from serde (skip) AND equality, so the C18 wire
+// shape and every existing RMetrics/RunReport equality assertion are unchanged.
+// It is a pure in-memory conduit: summarize_r already builds this vector (the
+// local `rs`, report.rs:137) and drops it; now it is retained for the bootstrap.
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+pub struct RMetrics {
+    pub expectancy_r: f64,
+    // ... all existing fields ...
+    pub conviction_terciles_r: [f64; 3],
+    #[serde(skip)]
+    pub trade_rs: Vec<f64>,   // realised R per closed trade, in trade order; empty after deserialize
+}
+
+// PartialEq hand-written to ignore `trade_rs` (so a serialize->deserialize
+// round-trip still compares equal, and the all-zero `RMetrics` literal in
+// summarize_r's empty-input arm needs no R-vector to match).
+impl PartialEq for RMetrics {
+    fn eq(&self, o: &Self) -> bool {
+        self.expectancy_r == o.expectancy_r
+            && /* ... all existing fields, trade_rs excluded ... */
+            && self.conviction_terciles_r == o.conviction_terciles_r
+    }
+}
+```
+
+`summarize_r` (`report.rs:89`) sets `trade_rs: rs.clone()` (or moves `rs`) into the
+returned `RMetrics` on the non-empty path; the empty-input arm sets
+`trade_rs: Vec::new()`.
+
+### Change 2 — the block-bootstrap primitive
+
+`crates/aura-engine/src/mc.rs` (new function + result struct, reusing the existing
+`MetricStats`/`quantile`)
+
+```rust
+/// Distribution of E[R] under a moving-block bootstrap of an OOS per-trade R
+/// series. `block_len == 1` is the i.i.d. trade-shuffle; `block_len > 1` preserves
+/// the serial correlation of sequential trades (winner/loser streaks), which a
+/// pure shuffle would erase and so overstate confidence. Deterministic (C1) given
+/// `seed` via the existing SplitMix64.
+#[derive(Clone, Debug, PartialEq)]
+pub struct RBootstrap {
+    pub e_r: MetricStats,   // mean + p5/p25/p50/p75/p95 of the resampled E[R]
+    pub prob_le_zero: f64,  // fraction of resamples whose mean R <= 0
+    pub n_trades: usize,
+    pub block_len: usize,
+    pub n_resamples: usize,
+}
+
+/// Moving-block bootstrap (non-circular; the final block is truncated so each
+/// resample has exactly `n` trades). Empty `rs` -> all-zero RBootstrap. `block_len`
+/// is clamped to `[1, n]`.
+pub fn r_bootstrap(rs: &[f64], n_resamples: usize, block_len: usize, seed: u64) -> RBootstrap {
+    let n = rs.len();
+    // n == 0 guard -> all-zero result; block_len = block_len.clamp(1, n.max(1)).
+    let mut rng = SplitMix64::new(seed);
+    let mut means = Vec::with_capacity(n_resamples);
+    for _ in 0..n_resamples {
+        let mut sample = Vec::with_capacity(n);
+        while sample.len() < n {
+            let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize;
+            for j in start..(start + block_len).min(start + (n - sample.len())) {
+                sample.push(rs[j]);
+            }
+        }
+        means.push(sample.iter().sum::<f64>() / n as f64);
+    }
+    RBootstrap {
+        e_r: MetricStats::from_values(&means),
+        prob_le_zero: means.iter().filter(|&&m| m <= 0.0).count() as f64 / n_resamples as f64,
+        n_trades: n, block_len, n_resamples,
+    }
+}
+```
+
+`SplitMix64` (`harness.rs:139`) is the existing deterministic RNG; if not already
+crate-visible, it is promoted to `pub(crate)` (the planner's mechanic).
+
+### Change 3 — `walkforward` strategy dispatch
+
+`crates/aura-cli/src/main.rs` — `parse_walkforward_args` gains `--strategy` + the
+four grid flags (mirroring `parse_sweep_args:1480`); `walkforward_family` branches
+on the strategy.
+
+```rust
+// BEFORE  (main.rs:1526)
+fn parse_walkforward_args(rest: &[&str]) -> Result<(String, bool, DataChoice), String> { ... }
+
+// AFTER — same RealWindowGrammar + name/trace handling, plus --strategy and the
+// stage1-r grid flags reusing parse_csv_list + Stage1RGrid (verbatim from sweep).
+fn parse_walkforward_args(
+    rest: &[&str],
+) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid), String> { ... }
+```
+
+```rust
+// BEFORE  (main.rs:1643) — hardwired SMA, optimize by total_pips, summarize (r:None)
+fn walkforward_family(trace: Option<&str>, data: &DataSource) -> WalkForwardResult {
+    let space = sample_blueprint_with_sinks(data.pip_size()).0.param_space();
+    walk_forward(roller, space, |w| {
+        let is_family = sweep_over(w.is.0, w.is.1, data);
+        let best = optimize(&is_family, "total_pips").expect(...);
+        let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
+        WindowRun { chosen_params: best.params, oos_equity, oos_report }
+    })
+}
+
+// AFTER — strategy-dispatched. SMA arm is verbatim today's code (goldens hold).
+fn walkforward_family(
+    strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid,
+) -> WalkForwardResult {
+    match strategy {
+        Strategy::SmaCross => { /* exactly today's body */ }
+        Strategy::Stage1R => {
+            let space = stage1_r_space();   // param_space of the open stage1_r_graph
+            walk_forward(roller, space, |w| {
+                let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); // reduce-mode fold
+                let best = optimize(&is_family, "sqn_normalized").expect("known R metric");
+                let (oos_equity, oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
+                WindowRun { chosen_params: best.params, oos_equity, oos_report }
+            })
+        }
+        // other stage1 strategies: out of scope this cycle -> usage error or SmaCross-style guard
+        _ => { /* return a clear "strategy has no walkforward form" error, like run_dispatch */ }
+    }
+}
+```
+
+New helpers (windowed siblings of the SMA `sweep_over`/`run_oos`, reusing the
+stage1-r machinery at `main.rs:1180`/`2112`):
+
+- `stage1_r_sweep_over(from, to, data, grid) -> SweepFamily` — the in-sample
+  stage1-r sweep over `data.windowed_sources(from, to)`, **reduce-mode** (folded
+  `SeriesReducer`/`GatedRecorder`, O(trades)/member, `m.r = Some(summarize_r(..))`)
+  so the per-window sweep footprint stays bounded.
+- `run_oos_r(params, from, to, trace, data) -> (Vec<(Timestamp,f64)>, RunReport)` —
+  the chosen-params OOS run over the windowed sources, **non-reduce** raw recorders
+  (the pip-equity curve is needed for `stitch`); folds `summarize` for the pip
+  block and `summarize_r` for `metrics.r` (which now carries `trade_rs`).
+
+`walkforward_summary_json` (`main.rs:1734`) gains an `oos_r` block: the pooled
+across-windows R-metrics, computed by concatenating each window's
+`oos_report.metrics.r.trade_rs` in roll order and folding (the pooled trade series
+→ `summarize`-style R reduction, or a thin `RMetrics`-from-`trade_rs` reducer). The
+SMA path emits no `oos_r` (its windows have `r: None`), so its summary line is
+unchanged.
+
+### Change 4 — `mc` strategy dispatch + bootstrap
+
+`crates/aura-cli/src/main.rs` — replace the three literal `mc` dispatch arms with a
+single `["mc", rest @ ..]` routed through a new `parse_mc_args`.
+
+```rust
+// BEFORE  (main.rs:2646)
+["mc"] => run_mc("mc", false),
+["mc", "--name", n] => run_mc(n, false),
+["mc", "--trace", n] => run_mc(n, true),
+
+// AFTER
+["mc", rest @ ..] => match parse_mc_args(rest) {
+    Ok(McArgs::Synthetic { name, persist }) => run_mc(&name, persist),     // today's path, unchanged
+    Ok(McArgs::RealR { strategy, choice, grid, block_len, n_resamples, seed }) =>
+        run_mc_r_bootstrap(strategy, DataSource::from_choice(choice), &grid, block_len, n_resamples, seed),
+    Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
+},
+```
+
+`run_mc_r_bootstrap` reuses `walkforward_family(strategy, None, &data, grid)` to
+obtain the pooled OOS R series, then `r_bootstrap(&pooled, n_resamples, block_len,
+seed)`, and prints one `mc_r_bootstrap` JSON line (shape above). Defaults:
+`--resamples 1000`, `--block-len 1`, `--seed 1` (recorded; derivable).
+
+### Change 5 — USAGE
+
+`main.rs:2591` — extend the `walkforward` and `mc` fragments with
+`[--strategy <...>] [--fast <csv>] ...` and (for mc) `[--block-len <n>]
+[--resamples <n>] [--seed <n>]`, matching the new grammars.
+
+## Components
+
+| Component | Location | Change |
+|---|---|---|
+| `RMetrics.trade_rs` + hand-written `PartialEq` | `aura-engine/report.rs:45` | new in-memory field, serde+eq excluded |
+| `summarize_r` | `aura-engine/report.rs:89` | retain `rs` into `trade_rs` (was dropped) |
+| `r_bootstrap` + `RBootstrap` | `aura-engine/mc.rs` | new deterministic moving-block bootstrap |
+| `SplitMix64` visibility | `aura-engine/harness.rs:139` | `pub(crate)` if needed |
+| `parse_walkforward_args` | `aura-cli/main.rs:1526` | `--strategy` + grid flags |
+| `walkforward_family` | `aura-cli/main.rs:1643` | strategy dispatch; SMA arm verbatim |
+| `stage1_r_sweep_over`, `run_oos_r`, `stage1_r_space` | `aura-cli/main.rs` (new) | windowed reduce-mode IS sweep + non-reduce OOS run |
+| `walkforward_summary_json` | `aura-cli/main.rs:1734` | `oos_r` pooled block (stage1-r only) |
+| `parse_mc_args` + `McArgs` + `run_mc_r_bootstrap` | `aura-cli/main.rs` (new) | mc real-candidate path |
+| `mc` dispatch arms, USAGE | `aura-cli/main.rs:2646/2591` | single `["mc", rest @ ..]`, usage text |
+
+## Data flow
+
+1. **walk-forward (stage1-r):** roller → per window: windowed IS sources →
+   stage1-r reduce-mode sweep → `optimize(.., "sqn_normalized")` → chosen params →
+   windowed OOS run (non-reduce) → `(oos_equity, RunReport{metrics.r{..,trade_rs}})`
+   → `WindowRun`. `walk_forward` stitches pip-equity; summary folds the pooled
+   `trade_rs` into `oos_r`.
+2. **mc (stage1-r):** run (1) → concat `windows[*].oos_report.metrics.r.trade_rs`
+   in roll order → `r_bootstrap` → `RBootstrap` → one JSON line.
+
+No look-ahead (C2: `WindowRoller` guarantees `oos.0 > is.1`). Disjoint windows run
+in parallel (C1). The bootstrap is a pure post-run reduction (C12).
+
+## Error handling
+
+- Malformed grid list / unknown strategy / flag without value → the subcommand
+  `usage()` string (strict, like `parse_sweep_args`).
+- A strategy with no walk-forward/mc form (e.g. `momentum`) → a clear error string
+  (mirroring `run_dispatch`'s `HarnessKind` guards), not a panic.
+- Span too short for one IS+OOS window → today's `WindowRoller::new` error path
+  (`main.rs:1646` exit 2), unchanged.
+- Empty pooled R series (no OOS trades) → `r_bootstrap` returns an all-zero
+  `RBootstrap` (well-defined), not a panic.
+
+## Testing strategy
+
+- **Goldens preserved (add, don't break):** existing `aura walkforward` (SMA) and
+  `aura mc` (synthetic) output goldens must stay byte-identical — covered by
+  running the bare forms; the SMA `walkforward_family` arm and `mc_family` are
+  untouched. C18 `runs.jsonl` shape unchanged (`trade_rs` is `serde(skip)`).
+- **`r_bootstrap` unit goldens:** fixed `rs` + fixed seed → pinned `RBootstrap`
+  quantiles; `block_len = 1` vs `> 1` both deterministic; `block_len = n` (one
+  block) → every resample equals the full series mean; empty `rs` → all-zero.
+- **`RMetrics` round-trip:** serialize → deserialize → `PartialEq` still equal
+  (the hand-written `eq` ignores `trade_rs`), pinning fork-A back-compat.
+- **walk-forward stage1-r E2E:** `aura walkforward --strategy stage1-r --real <SYM>`
+  over a small fixed window emits per-window `r` blocks and an `oos_r` summary;
+  determinism (same input → same output) pinned.
+- **mc stage1-r E2E:** `aura mc --strategy stage1-r --real <SYM>` emits a
+  `mc_r_bootstrap` line; same seed → same line.
+- **arg-parsing units:** `parse_walkforward_args` / `parse_mc_args` accept the new
+  flags and reject malformed input, mirroring `parse_sweep_args` unit tests.
+
+## Acceptance criteria
+
+1. `aura walkforward --strategy stage1-r --real <SYM>` reports per-window and
+   pooled OOS **R-metrics** (the `r` block + `oos_r` summary), not pip-only.
+2. `aura mc --strategy stage1-r --real <SYM>` reports a bootstrap `E[R]`
+   distribution / CI over the pooled OOS per-trade R series.
+3. The four stage1-r grid flags (`--fast/--slow/--stop-length/--stop-k`) drive the
+   per-window IS sweep, optimised by an R metric.
+4. Bare `aura walkforward` and `aura mc` (and `--name`/`--trace`) are
+   byte-identical to today; C18 wire shape unchanged.
+5. Frictionless Stage-1 R: no cost enters the path. Determinism (C1) holds:
+   same input + same seed → same output.
+6. Per-member walk-forward footprint stays O(trades) — the IS sweep folds
+   (reduce-mode), not O(cycles).