Brummel/Aura
1
0
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(0075): Monte-Carlo R-bootstrap for OOS validation (iter 2)

Browse Source 
Add `aura mc --strategy stage1-r [--real <SYM>]`: run the stage1-r
walk-forward, pool its OOS per-trade R series, and report a moving-block
bootstrap E[R] distribution / CI (one `mc_r_bootstrap` line with an `e_r`
quantile block + P(E[R]<=0)). Completes spec 0075's Monte-Carlo half on top
of iter 1's walk-forward R-reporting.

Engine: `r_bootstrap(rs, n_resamples, block_len, seed) -> RBootstrap` — a
deterministic moving-block bootstrap (non-circular, final block truncated;
block_len=1 = i.i.d. trade-shuffle; clamped to [1,n]; empty -> all-zero),
reusing the existing MetricStats/quantile and SplitMix64.

CLI: parse_mc_args/McArgs split — bare/`--name`/`--trace` route to the
unchanged synthetic seed-resweep `run_mc` (byte-for-byte preserved); the R
path accepts ONLY `--strategy stage1-r`, so a bare `--real` stays a usage
error (mc_rejects_real_flag_with_usage_exit_2 preserved, doc-comment
refreshed). run_mc_r_bootstrap reuses walkforward_family + RMetrics.trade_rs.

Corrective fixes over the plan snippets, both verified: the mc usage string
carries a `usage:` token (the new `["mc", rest @ ..]` arm now handles the
`--real` rejection, and the preserved golden asserts `stderr.contains
("usage")`); `#[allow(clippy::large_enum_variant)]` on McArgs keeps the
unboxed Stage1RGrid field layout.

Scope: single `--real <SYM>` per invocation; cross-symbol pooling is a
deferred follow-on (needs a multi-symbol grammar). Verified: cargo build
clean, full `cargo test --workspace` green (609 passed), clippy -D warnings
clean; bare `aura mc`, SMA walkforward, stage1-r single-run, and C18
round-trip goldens all preserved.

refs #139
This commit is contained in:
Brummel
2026-06-26 12:12:06 +02:00
parent b3505c7609
commit 2e77ab1dda
4 changed files with 471 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/aura-cli/src/main.rs
+216 -13
View File
@@ -17,11 +17,11 @@ use render::{ChartData, ChartMeta, ChartMode, ReduceKind, Series};
use aura_core::{zip_params, Cell, Firing, ParamSpec, Scalar, ScalarKind, Timestamp};
use aura_composites::{risk_executor, risk_executor_vol_open, StopRule};
use aura_engine::{
f64_field, join_on_ts, monte_carlo, param_stability, r_metrics_from_rs, summarize, summarize_r,
walk_forward, window_of, ColumnarTrace, Composite, Edge, FlatGraph, GraphBuilder, Harness,
JoinedRow, McAggregate, McFamily, RollMode, RunManifest, RunMetrics, RunReport, SourceSpec,
SweepFamily, SweepPoint, SyntheticSpec, Target, VecSource, WalkForwardResult, WindowBounds,
WindowRoller, WindowRun,
f64_field, join_on_ts, monte_carlo, param_stability, r_bootstrap, r_metrics_from_rs, summarize,
summarize_r, walk_forward, window_of, ColumnarTrace, Composite, Edge, FlatGraph, GraphBuilder,
Harness, JoinedRow, McAggregate, McFamily, RBootstrap, RollMode, RunManifest, RunMetrics,
RunReport, SourceSpec, SweepFamily, SweepPoint, SyntheticSpec, Target, VecSource,
WalkForwardResult, WindowBounds, WindowRoller, WindowRun,
};
use aura_registry::{
group_families, mc_member_reports, optimize, rank_by, sweep_member_reports,
@@ -1885,15 +1885,24 @@ fn run_oos(
(equity, report)
}
/// Pool every OOS window's per-trade R series into one flat vector, in roll order
/// (window order, then within-window trade order). Windows with no `r` block
/// contribute nothing. The single home of the pooling-in-roll-order semantics —
/// both the walk-forward `oos_r` summary and the `mc` R-bootstrap reduce this.
fn pooled_oos_trade_rs(result: &WalkForwardResult) -> Vec<f64> {
result
.windows
.iter()
.flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default())
.collect()
}
/// The walk-forward summary line: window count, stitched OOS total pips (the last
/// stitched-curve value), and the on-demand per-param stability. Canonical JSON
/// (C14).
fn walkforward_summary_json(result: &WalkForwardResult) -> String {
let total = result.stitched_oos_equity.last().map(|&(_, v)| v).unwrap_or(0.0);
// pool the per-window OOS per-trade R series in roll order, then reduce.
let pooled_rs: Vec<f64> = result.windows.iter()
.flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default())
.collect();
let pooled_rs = pooled_oos_trade_rs(result);
let mut obj = serde_json::json!({
"windows": result.windows.len(),
"stitched_total_pips": total,
@@ -2022,6 +2031,110 @@ fn run_mc(name: &str, persist: bool) {
println!("{}", mc_aggregate_json(&family.aggregate));
}
/// Parsed form of the `mc` tail: either the synthetic seed-resweep (today's path,
/// preserved byte-for-byte) or the real-candidate R-bootstrap. The R path accepts
/// ONLY `--strategy stage1-r` (the sole R-reporting walk-forward strategy); a bare
/// `--real` without it is a usage error (the synthetic seed-resweep is undefined
/// over real bars).
// Parsed once at the dispatch boundary and immediately destructured into the run
// call; never stored or collected, so the inter-variant size gap is free here.
#[allow(clippy::large_enum_variant)]
#[derive(Clone, Debug, PartialEq)]
enum McArgs {
Synthetic { name: String, persist: bool },
RealR { choice: DataChoice, grid: Stage1RGrid, block_len: usize, n_resamples: usize, seed: u64 },
}
fn parse_mc_args(rest: &[&str]) -> Result<McArgs, String> {
let usage = || "usage: mc [--name <n>|--trace <n>] | mc --strategy stage1-r [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] [--block-len <n>] [--resamples <n>] [--seed <n>]".to_string();
let mut strategy: Option<Strategy> = None;
let mut name: Option<(String, bool)> = None;
let mut real = RealWindowGrammar::default();
let mut grid = Stage1RGrid::default();
let mut block_len: usize = 1;
let mut n_resamples: usize = 1000;
let mut seed: u64 = 1;
let mut r_path = false;
let mut tail = rest;
while let Some((flag, t)) = tail.split_first() {
let (value, t) = t.split_first().ok_or_else(usage)?;
if real.accept(flag, value, &usage)? {
r_path = true;
tail = t;
continue;
}
match *flag {
"--strategy" => {
strategy = Some(match *value {
"stage1-r" => Strategy::Stage1R,
_ => return Err(usage()),
});
r_path = true;
}
"--name" if name.is_none() => name = Some(((*value).to_string(), false)),
"--trace" if name.is_none() => name = Some(((*value).to_string(), true)),
"--fast" => { grid.fast = parse_csv_list(value).map_err(|()| usage())?; r_path = true; }
"--slow" => { grid.slow = parse_csv_list(value).map_err(|()| usage())?; r_path = true; }
"--stop-length" => { grid.stop_length = parse_csv_list(value).map_err(|()| usage())?; r_path = true; }
"--stop-k" => { grid.stop_k = parse_csv_list(value).map_err(|()| usage())?; r_path = true; }
"--block-len" => { block_len = value.parse().map_err(|_| usage())?; r_path = true; }
"--resamples" => { n_resamples = value.parse().map_err(|_| usage())?; r_path = true; }
"--seed" => { seed = value.parse().map_err(|_| usage())?; r_path = true; }
_ => return Err(usage()),
}
tail = t;
}
if r_path {
if strategy != Some(Strategy::Stage1R) {
return Err(usage());
}
if name.is_some() {
return Err(usage());
}
let choice = real.finish(&usage)?;
Ok(McArgs::RealR { choice, grid, block_len, n_resamples, seed })
} else {
let (name, persist) = name.unwrap_or_else(|| ("mc".to_string(), false));
Ok(McArgs::Synthetic { name, persist })
}
}
/// `aura mc --strategy stage1-r [--real <SYM>]`: run the stage1-r walk-forward, pool
/// every OOS window's per-trade R series in roll order, and print one moving-block
/// bootstrap `mc_r_bootstrap` line (E[R] distribution + P(E[R] <= 0)). Frictionless
/// Stage-1 (no costs); deterministic given `seed` (C1).
fn run_mc_r_bootstrap(data: DataSource, grid: &Stage1RGrid, block_len: usize, n_resamples: usize, seed: u64) {
println!("{}", mc_r_bootstrap_report(&data, grid, block_len, n_resamples, seed));
}
/// Assemble the `mc` R-bootstrap line: run the stage1-r walk-forward, pool the OOS
/// per-trade R series in roll order, bootstrap E[R], and render the `mc_r_bootstrap`
/// line. The body of `run_mc_r_bootstrap` minus the `println!`, so the full real-R
/// wiring (walk-forward -> non-empty pooling -> bootstrap -> render) is reachable
/// over synthetic data in a `#[cfg(test)]` unit, mirroring `mc_report` /
/// `walkforward_report` / `sweep_report`.
fn mc_r_bootstrap_report(data: &DataSource, grid: &Stage1RGrid, block_len: usize, n_resamples: usize, seed: u64) -> String {
let result = walkforward_family(Strategy::Stage1R, None, data, grid);
let pooled = pooled_oos_trade_rs(&result);
let boot = r_bootstrap(&pooled, n_resamples, block_len, seed);
mc_r_bootstrap_json(&boot)
}
/// Render an `RBootstrap` as one canonical JSON line (`MetricStats` serializes; the
/// scalar fields are spliced in), mirroring `mc_aggregate_json`.
fn mc_r_bootstrap_json(b: &RBootstrap) -> String {
serde_json::json!({
"mc_r_bootstrap": {
"n_trades": b.n_trades,
"block_len": b.block_len,
"n_resamples": b.n_resamples,
"e_r": b.e_r,
"prob_le_zero": b.prob_le_zero,
}
})
.to_string()
}
/// Render the built-in Monte-Carlo family as the per-draw `RunReport` lines plus
/// the aggregate line — the `run_mc` shape minus registry persistence (no
/// `family_id`, which is store-assigned). Test helper, mirroring `sweep_report` /
@@ -2754,7 +2867,7 @@ fn run_dispatch(args: RunArgs) -> Result<RunReport, String> {
}
const USAGE: &str =
"usage: aura run [--harness <sma|macd|stage1-r>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--trace <name>] | aura chart <name> [--tap <t>] [--panels] | aura graph | aura sweep [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n>|--trace <n>] | aura mc [--name <n>|--trace <n>] | aura walkforward [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n>|--trace <n>] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] | aura runs families | aura runs family <id> [rank <metric>]";
"usage: aura run [--harness <sma|macd|stage1-r>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--trace <name>] | aura chart <name> [--tap <t>] [--panels] | aura graph | aura sweep [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n>|--trace <n>] | aura mc [--name <n>|--trace <n>] | aura mc --strategy stage1-r [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] [--block-len <n>] [--resamples <n>] [--seed <n>] | aura walkforward [--strategy <sma|momentum|stage1-r|stage1-breakout|stage1-meanrev>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n>|--trace <n>] [--fast <csv>] [--slow <csv>] [--stop-length <csv>] [--stop-k <csv>] | aura runs families | aura runs family <id> [rank <metric>]";
fn main() {
// Collect argv and match the whole vector: every accepted form is exhaustive,
@@ -2808,9 +2921,16 @@ fn main() {
std::process::exit(2);
}
},
["mc"] => run_mc("mc", false),
["mc", "--name", n] => run_mc(n, false),
["mc", "--trace", n] => run_mc(n, true),
["mc", rest @ ..] => match parse_mc_args(rest) {
Ok(McArgs::Synthetic { name, persist }) => run_mc(&name, persist),
Ok(McArgs::RealR { choice, grid, block_len, n_resamples, seed }) => {
run_mc_r_bootstrap(DataSource::from_choice(choice), &grid, block_len, n_resamples, seed)
}
Err(msg) => {
eprintln!("aura: {msg}");
std::process::exit(2);
}
},
["runs", "families"] => runs_families(),
["runs", "family", id] => runs_family(id, None),
["runs", "family", id, "rank", metric] => runs_family(id, Some(metric)),
@@ -3925,4 +4045,87 @@ mod tests {
assert!(parse_run_args(&["--real", "GER40", "--from"]).is_err()); // flag missing its value
assert!(parse_run_args(&["bogus"]).is_err()); // unknown trailing token
}
#[test]
fn parse_mc_args_bare_and_name_route_to_synthetic() {
assert_eq!(parse_mc_args(&[]), Ok(McArgs::Synthetic { name: "mc".to_string(), persist: false }));
assert_eq!(parse_mc_args(&["--name", "m"]), Ok(McArgs::Synthetic { name: "m".to_string(), persist: false }));
assert_eq!(parse_mc_args(&["--trace", "m"]), Ok(McArgs::Synthetic { name: "m".to_string(), persist: true }));
}
#[test]
fn parse_mc_args_stage1_r_routes_to_real_r_with_defaults_and_overrides() {
assert_eq!(
parse_mc_args(&["--strategy", "stage1-r"]),
Ok(McArgs::RealR {
choice: DataChoice::Synthetic, grid: Stage1RGrid::default(),
block_len: 1, n_resamples: 1000, seed: 1,
})
);
let r = parse_mc_args(&["--strategy", "stage1-r", "--real", "USDJPY",
"--block-len", "5", "--resamples", "200", "--seed", "9", "--fast", "5,10"]);
let McArgs::RealR { choice, grid, block_len, n_resamples, seed } = r.expect("valid real-R mc args")
else { panic!("expected RealR") };
assert_eq!(choice, DataChoice::Real { symbol: "USDJPY".to_string(), from_ms: None, to_ms: None });
assert_eq!((block_len, n_resamples, seed), (5, 200, 9));
assert_eq!(grid.fast, vec![5, 10]);
}
#[test]
fn parse_mc_args_real_without_stage1_r_is_usage_error() {
// a bare --real (no R candidate) is still rejected — the synthetic seed-resweep
// is undefined over real bars; the R path needs --strategy stage1-r.
assert!(parse_mc_args(&["--real", "EURUSD"]).is_err());
assert!(parse_mc_args(&["--strategy", "sma"]).is_err());
assert!(parse_mc_args(&["--strategy", "stage1-r", "--name", "x"]).is_err()); // name invalid on R path
}
#[test]
fn mc_r_bootstrap_json_carries_every_bootstrap_field_under_the_mc_r_bootstrap_key() {
// Property: the `mc_r_bootstrap` output line is the full RBootstrap shape —
// each field is named and value-faithful, so a renamed/dropped field here
// breaks a test instead of silently shipping. Pins the user-visible wire
// shape of the mc R-bootstrap render (the parser + engine primitive are
// covered elsewhere; this is the output-shape layer).
let boot = r_bootstrap(&[1.0, -0.5, 2.0, -1.0], 64, 2, 7);
let line = mc_r_bootstrap_json(&boot);
let v: serde_json::Value = serde_json::from_str(&line).expect("canonical json line");
let obj = &v["mc_r_bootstrap"];
assert_eq!(obj["n_trades"], serde_json::json!(boot.n_trades));
assert_eq!(obj["block_len"], serde_json::json!(boot.block_len));
assert_eq!(obj["n_resamples"], serde_json::json!(boot.n_resamples));
assert_eq!(obj["prob_le_zero"], serde_json::json!(boot.prob_le_zero));
// e_r is the nested MetricStats block (mean + quantiles), not a flat scalar.
assert_eq!(obj["e_r"], serde_json::to_value(&boot.e_r).expect("MetricStats serializes"));
assert!(obj["e_r"]["mean"].is_number(), "e_r should nest the MetricStats block: {line}");
}
#[test]
fn mc_r_bootstrap_report_pools_a_non_empty_oos_r_series_over_synthetic() {
// Property: the full real-R assembly path — walkforward_family(Stage1R) ->
// pooled_oos_trade_rs -> r_bootstrap -> mc_r_bootstrap_json — wires up and
// reduces a NON-EMPTY pooled OOS R series (the synthetic stage1-r walk-forward
// closes >= 1 trade across its windows). Guards the wiring + the non-empty
// pooling branch the parser/primitive unit tests cannot reach; mirrors
// `mc_report` / `walkforward_report`. Deterministic (C1).
let result = walkforward_family(Strategy::Stage1R, None, &DataSource::Synthetic, &Stage1RGrid::default());
let pooled = pooled_oos_trade_rs(&result);
assert!(!pooled.is_empty(), "synthetic stage1-r walk-forward must pool >= 1 OOS trade R");
let line = mc_r_bootstrap_report(&DataSource::Synthetic, &Stage1RGrid::default(), 1, 256, 1);
let v: serde_json::Value = serde_json::from_str(&line).expect("canonical json line");
let obj = &v["mc_r_bootstrap"];
// n_trades is the pooled-series length the bootstrap actually saw — non-zero
// proves the assembled pooling fed the primitive, not an empty fallback.
assert_eq!(obj["n_trades"], serde_json::json!(pooled.len()));
assert!(obj["n_trades"].as_u64().expect("n_trades is an integer") >= 1);
assert_eq!(obj["n_resamples"], serde_json::json!(256));
assert!(obj["e_r"]["mean"].as_f64().expect("e_r.mean is a number").is_finite());
// C1 at the CLI edge: the assembled real-R line is byte-identical on re-run.
assert_eq!(
line,
mc_r_bootstrap_report(&DataSource::Synthetic, &Stage1RGrid::default(), 1, 256, 1),
);
}
}