diff --git a/docs/plans/0076-walkforward-mc-r-bootstrap.md b/docs/plans/0076-walkforward-mc-r-bootstrap.md new file mode 100644 index 0000000..2825938 --- /dev/null +++ b/docs/plans/0076-walkforward-mc-r-bootstrap.md @@ -0,0 +1,487 @@ +# Monte-Carlo R-bootstrap (cycle 0075, iteration 2) — Implementation Plan + +> **Parent spec:** `docs/specs/0075-walkforward-mc-r-validation.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** `aura mc --strategy stage1-r [--real ]` runs the stage1-r +walk-forward, pools its OOS per-trade R series, and reports a moving-block +bootstrap `E[R]` distribution / confidence interval (`mc_r_bootstrap` line) — while +the bare synthetic `aura mc` path stays byte-identical. + +**Architecture:** A new deterministic engine primitive `r_bootstrap` (moving-block +bootstrap over a flat R series, reusing `MetricStats`/`SplitMix64`), then a CLI +`parse_mc_args`/`McArgs` split routing the bare/`--name`/`--trace` forms to today's +`run_mc` (unchanged) and the `--strategy stage1-r` form to a new +`run_mc_r_bootstrap` that reuses iteration 1's `walkforward_family` + +`RMetrics.trade_rs`. The mc real-R path accepts only `--strategy stage1-r` (the +sole R-reporting walk-forward strategy) — a bare `--real` without it stays a usage +error. + +**Tech Stack:** Rust workspace — `aura-engine` (`mc.rs`, `lib.rs`), `aura-cli` +(`main.rs`, `tests/cli_run.rs`). Test command: `cargo test --workspace` (one +positional filter per invocation). + +--- + +**Files this plan creates or modifies:** + +- Modify: `crates/aura-engine/src/mc.rs:111` — add `pub struct RBootstrap` + + `pub fn r_bootstrap`; `use crate::harness::SplitMix64;` at the module head. +- Modify: `crates/aura-engine/src/lib.rs:72` — re-export `RBootstrap`, `r_bootstrap` + from the `pub use mc::{..}` block. +- Modify: `crates/aura-cli/src/main.rs:2811-2813` — replace the three literal `mc` + dispatch arms with one `["mc", rest @ ..]` arm. +- Modify: `crates/aura-cli/src/main.rs` (new fns near the mc cluster ~2045) — + `parse_mc_args`, `enum McArgs`, `run_mc_r_bootstrap`, `mc_r_bootstrap_json`. +- Modify: `crates/aura-cli/src/main.rs:2756-2757` — USAGE mc fragment. +- Test: `crates/aura-engine/src/mc.rs` (#cfg test ~155) — `r_bootstrap` goldens. +- Test: `crates/aura-cli/src/main.rs` (#cfg test) — `parse_mc_args` units. +- Test: `crates/aura-cli/tests/cli_run.rs` — `aura mc --strategy stage1-r` E2E; the + `mc_rejects_real_flag_with_usage_exit_2` doc-comment refresh (assertions stay). + +--- + +### Task 1: `r_bootstrap` — moving-block bootstrap over a flat R series + +**Files:** +- Modify: `crates/aura-engine/src/mc.rs:1-13` (module-head `use`), `:111` (new + struct + fn) +- Modify: `crates/aura-engine/src/lib.rs:72` +- Test: `crates/aura-engine/src/mc.rs` (#cfg test, ~155) + +- [ ] **Step 1: Write the failing tests** — append to the `mod tests` in + `crates/aura-engine/src/mc.rs` (`use super::*` is already in scope): + +```rust +#[test] +fn r_bootstrap_empty_series_is_all_zero() { + let b = r_bootstrap(&[], 100, 1, 7); + assert_eq!(b.n_trades, 0); + assert_eq!(b.e_r.mean, 0.0); + assert_eq!(b.prob_le_zero, 0.0); +} + +#[test] +fn r_bootstrap_single_block_equals_full_series_mean() { + // block_len == n: the only valid start is 0, so every resample IS the full + // series -> every resample mean == the series mean -> zero spread. + let rs = [1.0, -2.0, 3.0]; // mean = 2/3 + let b = r_bootstrap(&rs, 500, rs.len(), 7); + let mean = 2.0 / 3.0; + assert!((b.e_r.mean - mean).abs() < 1e-12); + assert!((b.e_r.p5 - mean).abs() < 1e-12); + assert!((b.e_r.p95 - mean).abs() < 1e-12); + assert_eq!(b.prob_le_zero, 0.0); // mean > 0 + assert_eq!(b.n_trades, 3); + assert_eq!(b.block_len, 3); + assert_eq!(b.n_resamples, 500); +} + +#[test] +fn r_bootstrap_is_deterministic_given_seed() { + let rs = [0.5, -1.0, 2.0, -0.5, 1.5, -2.0, 0.25]; + let a = r_bootstrap(&rs, 1000, 1, 42); + let b = r_bootstrap(&rs, 1000, 1, 42); + assert_eq!(a, b, "same rs+seed+params -> identical RBootstrap (C1)"); + let c = r_bootstrap(&rs, 1000, 1, 43); + assert_ne!(a.e_r.p5, c.e_r.p5, "a different seed reshuffles differently"); +} + +#[test] +fn r_bootstrap_block_len_is_clamped_to_series_len() { + // block_len > n clamps to n -> single-block behaviour (no panic, no OOB). + let rs = [1.0, 2.0]; + let b = r_bootstrap(&rs, 10, 99, 1); + assert_eq!(b.block_len, 2); + assert!((b.e_r.mean - 1.5).abs() < 1e-12); +} +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace r_bootstrap_single_block_equals_full_series_mean` +Expected: FAIL — `cannot find function `r_bootstrap` in this scope`. + +- [ ] **Step 3: Add the module-head import** — in `crates/aura-engine/src/mc.rs`, + after the existing `use crate::{RunMetrics, RunReport, Scalar};` (line 13): + +```rust +use crate::harness::SplitMix64; +``` + +- [ ] **Step 4: Add `RBootstrap` + `r_bootstrap`** — in + `crates/aura-engine/src/mc.rs`, after the `quantile` fn (line 110), before + `#[cfg(test)] mod tests`: + +```rust +/// Distribution of E[R] under a moving-block bootstrap of an OOS per-trade R series +/// (#139). `block_len == 1` is the i.i.d. trade-shuffle; `block_len > 1` resamples +/// contiguous runs, preserving the serial correlation of sequential trades a pure +/// shuffle would erase. Deterministic (C1) given `seed`. +#[derive(Clone, Debug, PartialEq)] +pub struct RBootstrap { + pub e_r: MetricStats, // mean + p5/p25/p50/p75/p95 of the resampled E[R] + pub prob_le_zero: f64, // fraction of resamples whose mean R <= 0 + pub n_trades: usize, + pub block_len: usize, + pub n_resamples: usize, +} + +/// Moving-block bootstrap of `rs` (non-circular; the final block of each resample is +/// truncated so the resample has exactly `n` values). `block_len` is clamped to +/// `[1, n]`. Empty `rs` or zero resamples -> an all-zero `RBootstrap`. Pure given +/// `seed` (drives the existing `SplitMix64`). +pub fn r_bootstrap(rs: &[f64], n_resamples: usize, block_len: usize, seed: u64) -> RBootstrap { + let n = rs.len(); + if n == 0 || n_resamples == 0 { + return RBootstrap { + e_r: MetricStats { mean: 0.0, p5: 0.0, p25: 0.0, p50: 0.0, p75: 0.0, p95: 0.0 }, + prob_le_zero: 0.0, + n_trades: n, + block_len: block_len.clamp(1, n.max(1)), + n_resamples, + }; + } + let block_len = block_len.clamp(1, n); + let mut rng = SplitMix64::new(seed); + let mut means: Vec = Vec::with_capacity(n_resamples); + for _ in 0..n_resamples { + let mut sample: Vec = Vec::with_capacity(n); + while sample.len() < n { + let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize; + let take = block_len.min(n - sample.len()); + sample.extend_from_slice(&rs[start..start + take]); + } + means.push(sample.iter().sum::() / n as f64); + } + let e_r = MetricStats::from_values(&means); + let prob_le_zero = means.iter().filter(|&&m| m <= 0.0).count() as f64 / n_resamples as f64; + RBootstrap { e_r, prob_le_zero, n_trades: n, block_len, n_resamples } +} +``` + +- [ ] **Step 5: Re-export from the crate root** — in + `crates/aura-engine/src/lib.rs:72`, add `RBootstrap` and `r_bootstrap` to the + `pub use mc::{..}` list: + +```rust +pub use mc::{monte_carlo, r_bootstrap, McAggregate, McDraw, McFamily, MetricStats, RBootstrap}; +``` + +- [ ] **Step 6: Run to verify they pass** + +Run: `cargo test --workspace r_bootstrap_single_block_equals_full_series_mean` +Expected: PASS +Run: `cargo test --workspace r_bootstrap_is_deterministic_given_seed` +Expected: PASS +Run: `cargo test --workspace r_bootstrap_empty_series_is_all_zero` +Expected: PASS +Run: `cargo test --workspace r_bootstrap_block_len_is_clamped_to_series_len` +Expected: PASS + +### Task 2: `parse_mc_args` + `McArgs` + `run_mc_r_bootstrap` + dispatch + +**Files:** +- Modify: `crates/aura-cli/src/main.rs` — new `enum McArgs`, `parse_mc_args`, + `run_mc_r_bootstrap`, `mc_r_bootstrap_json` (near the mc cluster); dispatch arm + (2811-2813); USAGE (2757). +- Test: `crates/aura-cli/src/main.rs` (#cfg test) — `parse_mc_args` units. + +- [ ] **Step 1: Write the failing parse tests** — append to the `mod tests` in + `crates/aura-cli/src/main.rs`: + +```rust +#[test] +fn parse_mc_args_bare_and_name_route_to_synthetic() { + assert_eq!(parse_mc_args(&[]), Ok(McArgs::Synthetic { name: "mc".to_string(), persist: false })); + assert_eq!(parse_mc_args(&["--name", "m"]), Ok(McArgs::Synthetic { name: "m".to_string(), persist: false })); + assert_eq!(parse_mc_args(&["--trace", "m"]), Ok(McArgs::Synthetic { name: "m".to_string(), persist: true })); +} + +#[test] +fn parse_mc_args_stage1_r_routes_to_real_r_with_defaults_and_overrides() { + assert_eq!( + parse_mc_args(&["--strategy", "stage1-r"]), + Ok(McArgs::RealR { + choice: DataChoice::Synthetic, grid: Stage1RGrid::default(), + block_len: 1, n_resamples: 1000, seed: 1, + }) + ); + let r = parse_mc_args(&["--strategy", "stage1-r", "--real", "USDJPY", + "--block-len", "5", "--resamples", "200", "--seed", "9", "--fast", "5,10"]); + let McArgs::RealR { choice, grid, block_len, n_resamples, seed } = r.expect("valid real-R mc args") + else { panic!("expected RealR") }; + assert_eq!(choice, DataChoice::Real { symbol: "USDJPY".to_string(), from_ms: None, to_ms: None }); + assert_eq!((block_len, n_resamples, seed), (5, 200, 9)); + assert_eq!(grid.fast, vec![5, 10]); +} + +#[test] +fn parse_mc_args_real_without_stage1_r_is_usage_error() { + // a bare --real (no R candidate) is still rejected — the synthetic seed-resweep + // is undefined over real bars; the R path needs --strategy stage1-r. + assert!(parse_mc_args(&["--real", "EURUSD"]).is_err()); + assert!(parse_mc_args(&["--strategy", "sma"]).is_err()); + assert!(parse_mc_args(&["--strategy", "stage1-r", "--name", "x"]).is_err()); // name invalid on R path +} +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test --workspace parse_mc_args_bare_and_name_route_to_synthetic` +Expected: FAIL — `cannot find function `parse_mc_args` / type `McArgs``. + +- [ ] **Step 3: Add `McArgs` + `parse_mc_args`** — in `crates/aura-cli/src/main.rs`, + after `run_mc` (~line 2045): + +```rust +/// Parsed form of the `mc` tail: either the synthetic seed-resweep (today's path, +/// preserved byte-for-byte) or the real-candidate R-bootstrap. The R path accepts +/// ONLY `--strategy stage1-r` (the sole R-reporting walk-forward strategy); a bare +/// `--real` without it is a usage error (the synthetic seed-resweep is undefined +/// over real bars). +#[derive(Clone, Debug, PartialEq)] +enum McArgs { + Synthetic { name: String, persist: bool }, + RealR { choice: DataChoice, grid: Stage1RGrid, block_len: usize, n_resamples: usize, seed: u64 }, +} + +fn parse_mc_args(rest: &[&str]) -> Result { + let usage = || "mc [--name |--trace ] | mc --strategy stage1-r [--real [--from ] [--to ]] [--fast ] [--slow ] [--stop-length ] [--stop-k ] [--block-len ] [--resamples ] [--seed ]".to_string(); + let mut strategy: Option = None; + let mut name: Option<(String, bool)> = None; + let mut real = RealWindowGrammar::default(); + let mut grid = Stage1RGrid::default(); + let mut block_len: usize = 1; + let mut n_resamples: usize = 1000; + let mut seed: u64 = 1; + let mut r_path = false; + let mut tail = rest; + while let Some((flag, t)) = tail.split_first() { + let (value, t) = t.split_first().ok_or_else(usage)?; + if real.accept(flag, value, &usage)? { + r_path = true; + tail = t; + continue; + } + match *flag { + "--strategy" => { + strategy = Some(match *value { + "stage1-r" => Strategy::Stage1R, + _ => return Err(usage()), + }); + r_path = true; + } + "--name" if name.is_none() => name = Some(((*value).to_string(), false)), + "--trace" if name.is_none() => name = Some(((*value).to_string(), true)), + "--fast" => { grid.fast = parse_csv_list(value).map_err(|()| usage())?; r_path = true; } + "--slow" => { grid.slow = parse_csv_list(value).map_err(|()| usage())?; r_path = true; } + "--stop-length" => { grid.stop_length = parse_csv_list(value).map_err(|()| usage())?; r_path = true; } + "--stop-k" => { grid.stop_k = parse_csv_list(value).map_err(|()| usage())?; r_path = true; } + "--block-len" => { block_len = value.parse().map_err(|_| usage())?; r_path = true; } + "--resamples" => { n_resamples = value.parse().map_err(|_| usage())?; r_path = true; } + "--seed" => { seed = value.parse().map_err(|_| usage())?; r_path = true; } + _ => return Err(usage()), + } + tail = t; + } + if r_path { + if strategy != Some(Strategy::Stage1R) { + return Err(usage()); + } + if name.is_some() { + return Err(usage()); + } + let choice = real.finish(&usage)?; + Ok(McArgs::RealR { choice, grid, block_len, n_resamples, seed }) + } else { + let (name, persist) = name.unwrap_or_else(|| ("mc".to_string(), false)); + Ok(McArgs::Synthetic { name, persist }) + } +} +``` + +- [ ] **Step 4: Add `run_mc_r_bootstrap` + `mc_r_bootstrap_json`** — after + `parse_mc_args`: + +```rust +/// `aura mc --strategy stage1-r [--real ]`: run the stage1-r walk-forward, pool +/// every OOS window's per-trade R series in roll order, and print one moving-block +/// bootstrap `mc_r_bootstrap` line (E[R] distribution + P(E[R] <= 0)). Frictionless +/// Stage-1 (no costs); deterministic given `seed` (C1). +fn run_mc_r_bootstrap(data: DataSource, grid: &Stage1RGrid, block_len: usize, n_resamples: usize, seed: u64) { + let result = walkforward_family(Strategy::Stage1R, None, &data, grid); + let pooled: Vec = result + .windows + .iter() + .flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default()) + .collect(); + let boot = r_bootstrap(&pooled, n_resamples, block_len, seed); + println!("{}", mc_r_bootstrap_json(&boot)); +} + +/// Render an `RBootstrap` as one canonical JSON line (`MetricStats` serializes; the +/// scalar fields are spliced in), mirroring `mc_aggregate_json`. +fn mc_r_bootstrap_json(b: &RBootstrap) -> String { + serde_json::json!({ + "mc_r_bootstrap": { + "n_trades": b.n_trades, + "block_len": b.block_len, + "n_resamples": b.n_resamples, + "e_r": b.e_r, + "prob_le_zero": b.prob_le_zero, + } + }) + .to_string() +} +``` + +(Add `r_bootstrap`, `RBootstrap` to the `use aura_engine::{..}` import block at the +top of `main.rs`, alongside the existing `monte_carlo` / `McFamily` imports.) + +- [ ] **Step 5: Replace the dispatch arms** — in `crates/aura-cli/src/main.rs`, + replace the three literal arms (2811-2813): + +```rust +["mc", rest @ ..] => match parse_mc_args(rest) { + Ok(McArgs::Synthetic { name, persist }) => run_mc(&name, persist), + Ok(McArgs::RealR { choice, grid, block_len, n_resamples, seed }) => { + run_mc_r_bootstrap(DataSource::from_choice(choice), &grid, block_len, n_resamples, seed) + } + Err(msg) => { + eprintln!("aura: {msg}"); + std::process::exit(2); + } +}, +``` + +- [ ] **Step 6: Extend USAGE** — in `crates/aura-cli/src/main.rs:2757`, replace the + `aura mc [--name |--trace ]` fragment with: + +``` +aura mc [--name |--trace ] | aura mc --strategy stage1-r [--real [--from ] [--to ]] [--fast ] [--slow ] [--stop-length ] [--stop-k ] [--block-len ] [--resamples ] [--seed ] +``` + +- [ ] **Step 7: Build gate** + +Run: `cargo build --workspace` +Expected: 0 errors. + +- [ ] **Step 8: Run the parse units + the preserved synthetic mc goldens** + +Run: `cargo test --workspace parse_mc_args_bare_and_name_route_to_synthetic` +Expected: PASS +Run: `cargo test --workspace parse_mc_args_stage1_r_routes_to_real_r_with_defaults_and_overrides` +Expected: PASS +Run: `cargo test --workspace parse_mc_args_real_without_stage1_r_is_usage_error` +Expected: PASS +Run: `cargo test --workspace mc_runs_persists_a_monte_carlo_family_and_lists_it` +Expected: PASS (bare `aura mc` unchanged — routed to `run_mc` byte-for-byte) +Run: `cargo test --workspace mc_rejects_real_flag_with_usage_exit_2` +Expected: PASS (a bare `--real` without `--strategy stage1-r` still exits 2 + usage) + +### Task 3: stage1-r mc E2E + doc-comment refresh + +**Files:** +- Test: `crates/aura-cli/tests/cli_run.rs` — new `aura mc --strategy stage1-r` E2E; + refresh the `mc_rejects_real_flag_with_usage_exit_2` doc-comment (1298-1304). + +- [ ] **Step 1: Write the failing E2E test** — append to + `crates/aura-cli/tests/cli_run.rs` (uses the established `Command::new(BIN)` + + `temp_cwd` harness): + +```rust +/// Property: `aura mc --strategy stage1-r` runs the stage1-r walk-forward on +/// synthetic data, bootstraps the pooled OOS per-trade R series, and prints exactly +/// one canonical `mc_r_bootstrap` line carrying an `e_r` distribution block and +/// `prob_le_zero`. Deterministic: a second run with the same (default) seed is +/// byte-identical (C1). Frictionless Stage-1 — no costs, no `runs/` family write. +#[test] +fn mc_strategy_stage1_r_prints_a_bootstrap_line_deterministically() { + let dir = temp_cwd("mc_stage1r_boot"); + let run = || { + Command::new(BIN) + .args(["mc", "--strategy", "stage1-r", "--resamples", "200"]) + .current_dir(&dir) + .output() + .expect("spawn aura mc --strategy stage1-r") + }; + let out = run(); + assert!(out.status.success(), "exit: {:?} stderr: {}", out.status, String::from_utf8_lossy(&out.stderr)); + let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout"); + assert_eq!(stdout.lines().count(), 1, "exactly one mc_r_bootstrap line: {stdout:?}"); + let v: serde_json::Value = serde_json::from_str(stdout.trim()).expect("canonical JSON"); + assert!(v["mc_r_bootstrap"]["e_r"]["mean"].is_number(), "e_r block present: {stdout}"); + assert!(v["mc_r_bootstrap"]["prob_le_zero"].is_number(), "prob_le_zero present: {stdout}"); + assert_eq!(v["mc_r_bootstrap"]["n_resamples"], 200); + + let out2 = run(); + assert_eq!(out.stdout, out2.stdout, "same seed -> byte-identical line (C1)"); + let _ = std::fs::remove_dir_all(&dir); +} +``` + +- [ ] **Step 2: Run to verify it fails** + +(Without Task 2 this would fail; with Tasks 1-2 already applied in the same iter, +this verifies the wired path.) + +Run: `cargo test --workspace mc_strategy_stage1_r_prints_a_bootstrap_line_deterministically` +Expected: PASS once Tasks 1-2 are in the tree (the implementer applies tasks in +order; if run before, FAIL with exit 2 / no `mc_r_bootstrap` line). + +- [ ] **Step 3: Refresh the stale doc-comment** — the + `mc_rejects_real_flag_with_usage_exit_2` assertions stay (a bare `--real` is still + rejected), but its rationale prose (1298-1304) overclaims that mc carves out the + real path entirely. Replace that doc-comment block with: + +```rust +/// The synthetic seed-resweep `mc` is undefined over real bars (one realization -> +/// identical members), so a bare `aura mc --real EURUSD` — with no R candidate — is +/// REJECTED (usage on stderr, exit 2) at the binary boundary. The real path is +/// reachable ONLY via `--strategy stage1-r` (the R-bootstrap over the pooled OOS R +/// series, `mc_strategy_stage1_r_prints_a_bootstrap_line_deterministically`); a +/// `--real` without it stays a usage error. NOT gated — the refusal precedes any +/// data access, so it is CI-safe on every machine. +``` + +- [ ] **Step 4: Full-suite regression gate** + +Run: `cargo test --workspace` +Expected: PASS — the new engine + parse + E2E tests green; all pre-existing goldens +(bare `aura mc`, `aura walkforward` SMA, stage1-r single-run, C18 round-trips) +green. + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: 0 warnings. + +--- + +## Self-review + +- **Spec coverage:** spec Change 2 (`RBootstrap`/`r_bootstrap` — Task 1) and Change 4 + (`parse_mc_args`/`McArgs`/`run_mc_r_bootstrap`/dispatch/USAGE — Task 2; E2E + + doc-refresh — Task 3). Both in-scope sections covered. ✓ +- **Placeholder scan:** no TBD/TODO/"similar to". ✓ +- **Type/name consistency:** `RBootstrap`, `r_bootstrap`, `McArgs`, + `parse_mc_args`, `run_mc_r_bootstrap`, `mc_r_bootstrap_json`, `Strategy::Stage1R`, + `Stage1RGrid`, `DataChoice` used identically across tasks; the pooled field path + is `w.run.oos_report.metrics.r.trade_rs` (per recon, not the spec's abbreviated + `w.oos_report...`). ✓ +- **Step granularity:** each step is one edit or one command. ✓ +- **No commit steps.** ✓ +- **Compile-gate ordering (rule 7):** Task 1 is purely additive (new symbol + its + re-export + tests) — no caller breakage. Task 2 replaces the dispatch arms and + adds `parse_mc_args`/`McArgs`/`run_mc_r_bootstrap` together (the arm references + both, so they land in one task) before its build gate; the only `run_mc` caller + becomes the `McArgs::Synthetic` branch, threaded in the same task. ✓ +- **Filter strings (rule 8):** every `cargo test` filter names a real test — + pre-existing (`mc_runs_persists_a_monte_carlo_family_and_lists_it`, + `mc_rejects_real_flag_with_usage_exit_2`) or introduced in the same task; the + final gate is the unfiltered `cargo test --workspace`. ✓ +- **mc-R restriction (derived, recorded on #139):** the R path accepts only + `--strategy stage1-r`; a bare `--real` stays rejected, so + `mc_rejects_real_flag_with_usage_exit_2`'s assertions are preserved (only its + prose is refreshed). ✓