Walk-forward + Monte-Carlo OOS validation for R candidates (not just the SMA sample) #139
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Brummel/Aura#139
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
The edge research needs honest out-of-sample R-metrics for an arbitrary
candidate strategy on real data. Today the validation machinery exists as a
clean library API but is only half-reachable from the CLI, and not at all for
R-based candidates:
aura walkforwardis hardwired to the SMA sample blueprint with fixed90 / 30 / 30-day windows and pip metrics only — it cannot run stage1-r
(or any new candidate) and reports no R.
aura mcis synthetic-only, 3 hardcoded seeds — useless on a realcandidate.
walk_forward(WindowRoller, space, run_window)withWindowRoller::new(span, is_len, oos_len, step, RollMode::{Rolling,Anchored}),monte_carlo(base, seeds, run_one),summarize_r(record, cost).What to build
A candidate-aware OOS validation path (CLI subcommand and/or extended
walkforward/mc) that, for a chosen strategy (stage1-r first; the griddableflags already exist) on
--real <SYM>:candidate's param grid and pick the best by an R metric (e.g.
sqn_normalizedorexpectancy_r); measure that param set on the held-outOOS window; stitch the OOS trades. Configurable IS/OOS length + step + mode.
(trade-shuffle and/or block) to get a distribution / confidence interval on
E[R]— so a positive OOS edge is shown to be unlikely under reshuffling.Dependencies / notes
it multiplies the per-member footprint that already nearly exhausts RAM. Fix
that first.
walkforward/mcSMA paths and their goldens must be preserved (add,don't break).
refs #137
Surviving the edge-research milestone wind-down (#137). This is a genuine engine-capability gap — the World layer's walk-forward + Monte-Carlo are locked to the SMA sample blueprint and carry no R; they should validate ANY harness. Independent of the abandoned edge hunt. Detached from the (now closed) milestone; kept open as standalone engine work.
Empirical confirmation (2026-06-26, post-0074)
Ran the existing
aura walkforward --real {USDJPY,GER40}(2023-01..2024-12,90/30/30) to confirm the gap is live at HEAD:
sample_blueprint, optimizeson
total_pips, and the OOS report carries pip metrics only(
total_pips/max_drawdown/bias_sign_flips, norblock). No--strategyflag, so stage1-r is unreachable here.symbols): USDJPY stitched −171.7 pips (8/21 OOS windows positive); GER40
−271.1 pips (11/21). Param choice unstable window-to-window (USDJPY trend fast
50↔100, slow 200↔400). Both negative OOS — the in-sample stage1-r best-of-grid
looked +0.1..0.2 E[R], so this is exactly the overfit-exposure the candidate-
aware OOS path is for.
Scope narrowing — the sweep half already works.
aura sweep --strategy stage1-r --real <SYM>reports the R block (expectancy_r,sqn_normalized) andis pip-agnostic across instruments (verified cross-instrument over Gold/USDJPY/
GER40). So the remaining work is specifically the walkforward + mc side:
make them strategy-selectable and R-reporting, reusing the existing stage1-r grid
(
--fast/--slow/--stop-length/--stop-k) +summarize_r.RAM note. This modest run (2y, default sample grid, no per-window candidate
grid) completed fast with no OOM. The cited RAM dependency likely bites at the
larger scale (per-window candidate grid × all symbols × MC bootstrap), not a
modest walk-forward — worth re-measuring before treating it as a hard blocker.
/bossrun — decision log (design settled in-context with the user, 2026-06-26)1. #139 is NOT blocked by #136. It builds correct today against the
already-generic
walk_forward/monte_carlo/optimize/summarize_rwithconcrete
RunMetrics. The work arrow is #139 → #136: #136 later genericises thatorchestration surface over a metric type
Mand absorbs #139's added callsites. This is a merge-collision on the sweep/mc/walkforward files, not a logical
dependency. Sequence #139 first (smaller, value-bearing).
2. The cited "memory BLOCKER" is discharged. It pointed at #138
(
stage1-r sweep ~2 GiB/member), CLOSED in cycle 0070 with the explicit note"unblocks #139"; per-member RAM dropped from O(cycles) to O(trades) via the
finalizelifecycle + folding sinks. Not a gate — a constraint: the newper-window / per-MC runs MUST use the folded reduce-mode sinks (as the stage1-r
sweep already does), otherwise O(cycles) returns per window.
3. Per-trade R series placement: carry it inside the R-metric block
(
RMetrics),#[serde(skip)].summarize_ralready computes the per-trade Rvector internally and drops it (
report.rs:137) — stop dropping it.serde(skip)keeps it in-memory and off the C18 wire (no
runs.jsonlbloat). Adds no newconcrete series field to the engine's
WindowRun(the R block already ridesthrough
oos_report). Rationale is today-substantive (the natural home for thedistribution's own sample; the smallest change), explicitly not #136-driven —
chosen for clarity now, #136 relocates the whole
RMetricstype later regardless.4. MC bootstrap shape: block bootstrap with a configurable block length;
trade-shuffle is the
block_len = 1special case. R-multiples from sequentialtrades are not i.i.d. (winner/loser streaks in a trend follower), so a pure
trade-shuffle overstates the confidence on E[R]. One block-aware primitive
subsumes both. Build it as a new deterministic primitive (on
SplitMix64) over aVec<f64>of R, golden-testable on fixed quantiles. This is the one genuinely newlibrary component; everything else is CLI wiring onto already-generic APIs.
5. Reuse map (substrate already green at HEAD).
aura sweep --strategy stage1-r --real <SYM>is fully R-reporting today (R block viasummarize_r, allfour grid flags
--fast/--slow/--stop-length/--stop-k, pip-agnostic acrossinstruments). #139 reuses:
parse_csv_list+Stage1RGrid+ theStrategyenum(arg parsing),
stage1_r_sweep_family/stage1_r_graph(blueprint),summarize_r(R),optimize(.., "sqn_normalized" | "expectancy_r")(ISobjective). Walk-forward half = wire this into the already-generic
walk_forward+WindowRoller; MC half = the new R-series bootstrap above.6. Scope & invariants. Frictionless Stage-1 R (costs are Stage-2). Report
per-symbol and pooled OOS R-metrics. Preserve the existing SMA/
total_pipswalkforward goldens and the synthetic-mc goldens — add, don't break. Rough
size: walk-forward strategy-selectable + R-reporting S–M (mostly CLI wiring +
small R-series exposure); MC real-candidate R-bootstrap M (the new primitive).
/bossrun — derived sub-fork decisions (specify, grounding read)Four finer forks surfaced while reading the code to write spec 0075. All
derived (sources / code / consistency / risk), none a user-preference fork.
A. The OOS per-trade R series rides in
RMetrics, excluded from BOTH serdeand equality.
walk_forwardmust stay unchanged, so the series can onlysurvive into
result.windowsvia the already-carriedoos_report→ so it liveson
RMetrics(decision #3 in the prior comment). To keep the C18 wire shape andevery existing
RunReport/RMetricsequality assertion byte- and value-unchanged("add, don't break"), the new field participates in neither
serde(skip) norPartialEq— it is a pure in-memory conduit for the bootstrap. Basis: derived —forced by
walk_forward-unchanged + C18/golden stability.B. In-sample sweep folds (reduce-mode); the single OOS run uses raw recorders.
The RAM constraint bites on the per-window sweep (many members × O(cycles)) — so
the IS stage1-r sweep uses the folded reduce-mode sinks (
GatedRecorder→O(trades)/member), optimised by an R metric. The one OOS run per window stays
non-reduce: stitching needs the full pip-equity curve, and one window's curve is
bounded. Basis: derived — the decision-log RAM constraint applied at the right
granularity; mirrors the existing SMA
sweep_over/run_oossplit.C.
aura mc --strategy ... --real ...runs the walk-forward internally, thenbootstraps its pooled OOS R series. The MC bootstrap operates on "the stitched
OOS per-trade R series" — which the walk-forward produces. So the real-candidate
mcpath reuseswalkforward_familyto obtain the pooled OOS R series, thenapplies the new bootstrap — no new on-disk artifact, and
aura mcstays a singlecommand. The synthetic seed-resweep
mc_familyis untouched (its goldens hold).Basis: derived — reuse over a new persistence format; least surface.
D. Scope: single
--real <SYM>per invocation; "pooled" = pooled across theOOS windows of that symbol. The issue's "per-symbol and pooled" is delivered
as: each invocation reports one symbol's OOS R-metrics over its stitched
(across-windows) OOS trade series. Cross-symbol pooling needs a multi-symbol CLI
grammar (
RealWindowGrammaraccepts one symbol today) — a separable expansion,deferred to a follow-on, not built this cycle. Basis: derived scope narrowing;
flagged here so it is vetoable.
/bossrun — pooledoos_rreducer (planner, derived)The spec left the pooled-
oos_rmechanism as "a new reducer or an inlinefold". Decided: a new self-contained
pub fn r_metrics_from_rs(rs: &[f64]) -> RMetricsinaura-engine/report.rs, not a refactor ofsummarize_r. Itcopies
summarize_r's rs-derived arithmetic verbatim (byte-pinned floats mustnot be algebraically refactored), leaving
summarize_rand its goldens untouched.The two fields a flat R series cannot carry are set honestly:
net_expectancy_r = expectancy_r(exact at the Stage-1 cost=0 invariant) andconviction_terciles_r = [0,0,0](per-trade conviction is not pooled). So the pooled OOS block reports thewell-defined R-distribution metrics (E[R], SQN, n_trades, win-rate, profit-factor,
R-drawdown); the per-window
rblocks keep full fidelity. Basis: derived — leastrisk to pinned arithmetic; honest field semantics under frictionless Stage-1.