From a4712eb336a0b4757e1803003378700371967320 Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 15 Jun 2026 14:56:12 +0200 Subject: [PATCH] spec: 0044 walk-forward family (C12 axis 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample) splits over a time span, walk_forward runs a disjoint harness per split via the shared run_indexed core (C1), and stitches the OOS pip-equity segments into one continuous curve. The varying dimension is the data window (axis 3), realized eager-agnostically (#71): the public surface carries WindowBounds + a per-window closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds invariant (oos.0 > is.1), checkable with zero ticks. The in-sample optimize (axis 2) is closure-supplied, not called by the engine: aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search policy into the primitive. The CLI bridges both crates in run_walkforward. Param-stability shape (the load-bearing fork) resolved with the user as R2: on-demand, not stored. WalkForwardResult stores only raw per-window outcomes + the stitched curve; param_stability(&result) -> Vec is a public on-demand reduction over the retained per-window chosen params (sweep-precedent: summaries computed on demand, not cached, vs McFamily's stored aggregate). A stored summary would be recomputable from the raw windows (redundant) and would force one statistic canonical when "stability" admits several. Recorded with provenance in the #69 reconciliation comment. MetricStats::from_values is extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests guard) so the MC aggregate and the helper share one reduction. Empty OOS segment contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)). Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean, grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated on the scope-fork design lens (param-stability shape) to human sign-off; the user resolved it (R2), so this is a user-signed spec, not boss-signed. refs #69 --- docs/specs/0044-walk-forward-family.md | 430 +++++++++++++++++++++++++ 1 file changed, 430 insertions(+) create mode 100644 docs/specs/0044-walk-forward-family.md diff --git a/docs/specs/0044-walk-forward-family.md b/docs/specs/0044-walk-forward-family.md new file mode 100644 index 0000000..7006913 --- /dev/null +++ b/docs/specs/0044-walk-forward-family.md @@ -0,0 +1,430 @@ +# Walk-forward orchestration family (C12 axis 3) — Design Spec + +**Date:** 2026-06-15 +**Status:** Approved — user sign-off (R2, 2026-06-15) +**Authors:** orchestrator + Claude + +## Goal + +Deliver C12's third orchestration axis: **walk-forward**. Unlike the grid sweep +(axis 1, varies a param) and Monte-Carlo (axis 4, varies the seed), walk-forward +varies the **data window** — it rolls (in-sample, out-of-sample) splits over a +time span, runs a disjoint harness per split (optionally optimizing params on the +in-sample slice, reusing axis 2), and stitches the out-of-sample pip-equity +segments into one continuous out-of-sample curve. The result carries the +per-window **chosen params** as the stability substrate; a per-parameter +stability summary is an **on-demand reduction** over them (a provided helper), +**not** a stored field — see Architecture. + +The axis is realized **eager-agnostically** (the #71 firewall): the public +surface carries **time-bounds, never tick data**. A 20-year history is the +~190 GB eager object; the walk-forward surface must not be able to encode a +materialized history. The window roller is a pure iterator of `WindowBounds`; +per-window data comes from a bounds-keyed producer closure. The C2 no-look-ahead +guarantee is then a pure bounds invariant (`oos.0 > is.1`), checkable with zero +ticks materialized. + +## Architecture + +A new `walkforward` module in `aura-engine`, alongside `sweep` and `mc`. It owns +three things the other two axes' modules already model: **enumeration** (the +`WindowRoller`, analog to `GridSpace`), **execution** (the `walk_forward` +function, reusing the shared `run_indexed` disjoint-parallel core — C1), and +**collection** (`WalkForwardResult`, analog to `SweepFamily`/`McFamily`). + +The orchestrator owns rolling + disjoint execution + stitching. It +delegates the per-window run — build/bootstrap/run, and the *optional* in-sample +param optimization — to a caller-supplied closure, exactly as `sweep` and `mc` +delegate the per-unit run to `run_one`. This is forced, not chosen: + +- **Crate separation (C9) + dependency direction.** `aura-engine` is the engine; + `aura-registry` depends on `aura-engine` and owns `optimize` (axis 2). The + walk-forward orchestrator lives in `aura-engine` (it reuses the `pub(crate)` + `run_indexed`), so it **cannot** call `optimize` directly without inverting the + dependency. The in-sample selection is therefore a closure concern: the caller + (the CLI / a World, both of which depend on both crates) supplies `optimize` as + the in-sample step. +- **C12 forbids baking search policy into the primitive.** "Optimization" is a + pluggable policy atop the atomic unit, not part of the orchestration primitive. + A closure-supplied in-sample step is the contract C12 already mandates. + +The in-sample/out-of-sample data both flow through a bounds-keyed producer +`Fn(Timestamp, Timestamp) -> impl Source` living **inside** the caller's closure +— which maps onto the existing `aura-ingest` method +`DataServer::stream_m1_windowed(symbol, from: Option, to: Option)` +(`aura-ingest/src/lib.rs`; per-bound `Option` windows) for real-data production. +No materialized stream `Vec` crosses the `walk_forward` API boundary (#71). The +engine code this iteration adds never calls that method — it is the production +mapping the closure body fulfils; the built-in CLI demo uses an **in-memory +windowed source** instead (mirroring `run_sweep`'s `showcase_prices`), so the +iteration adds no new `aura-ingest` dependency. + +**Param-stability is on-demand, not a stored field — and this is the load-bearing +shape decision (resolved with the user, see the #69 reconciliation comment).** +The result stores only the raw per-window outcomes + the stitched curve, exactly +as a `SweepFamily` stores only raw points and `optimize`/`rank_by` are computed on +demand (never cached). The raw per-window `chosen_params` *are* the stability +substrate; a stored summary would be a pure reduction over them — recomputable, +hence redundant — and would force one statistic to be canonical when "stability" +admits several (the MC `MetricStats` distribution, a moment std/CV, IQR, a +distinct-value count). So the result caches none of them. A convenience reduction +ships as the **public on-demand fn `param_stability(&WalkForwardResult) -> +Vec`**, which reuses the Monte-Carlo family's `MetricStats` (mean + +type-7 quantiles) per param slot; the caller is free to compute any other measure +from the same `windows[*].run.chosen_params`. The `MetricStats` reduction is +extracted from `McAggregate::from_draws` into a shared +`MetricStats::from_values(&[f64])` (a behaviour-preserving refactor; the seven +existing `mc` tests are the green guard) so both the MC aggregate and this helper +share one reduction. + +## Concrete code shapes + +### Worked author example (the user-facing program — acceptance evidence) + +A World author validates a strategy out-of-sample by composing the roller, the +axis-2 optimizer, and a per-window run inside one `walk_forward` call: + +```rust +use aura_engine::{param_stability, walk_forward, RollMode, WindowBounds, WindowRoller, WindowRun, Timestamp}; +use aura_registry::optimize; + +// A bounds-keyed producer: for real data, maps onto the method +// DataServer::stream_m1_windowed(symbol, Option, Option). The SURFACE is +// bounds; a first cut MAY collect (from..to) internally (#71 firewall: the *type* +// cannot encode a materialized history, the impl may be eager for now). +let make_source = |from: Timestamp, to: Timestamp| -> MyWindowedSource { /* ... */ }; + +// 1 year in-sample, 3 months out-of-sample, stepping 3 months, rolling +// (fixed-length IS). Anchored (growing IS) is RollMode::Anchored. +let roller = WindowRoller::new(span, IS_LEN, OOS_LEN, STEP, RollMode::Rolling)?; + +let result = walk_forward(roller, |w: WindowBounds| -> WindowRun { + // in-sample: sweep a grid, pick the best params (axis 2 INSIDE axis 3) + let is_family = grid.sweep(|pt| run_point(pt, make_source(w.is.0, w.is.1))); + let best = optimize(&is_family, "total_pips").expect("known metric"); + // out-of-sample: run the chosen params on the held-out window, record equity + let (oos_equity, oos_report) = run_and_record(&best.params, make_source(w.oos.0, w.oos.1)); + WindowRun { chosen_params: best.params, oos_equity, oos_report } +}); +// Runs are disjoint across windows (C1); no look-ahead by construction +// (C2: every w.oos.0 > w.is.1, guaranteed by the roller). + +result.stitched_oos_equity; // one continuous OOS pip curve across all windows +result.windows[0].run.chosen_params; // raw per-window chosen params (the stability substrate) +param_stability(&result)[0]; // on-demand: MetricStats of param 0 across windows +``` + +CLI surface (mirrors `aura sweep`): + +``` +$ aura walkforward +{"manifest":{...,"window":[,],...},"metrics":{...}} # window 0 OOS record +{"manifest":{...,"window":[,],...},"metrics":{...}} # window 1 OOS record +... +{"walkforward":{"windows":N,"stitched_total_pips":X.0,"param_stability":[{"mean":..,"p5":..,...}]}} +``` + +Each per-window OOS `RunReport` is persisted to the run registry (C18) and printed +as one JSON line; a final summary line carries the window count, the stitched +total, and the per-param stability. + +### Implementation shapes (secondary) + +New, in `crates/aura-engine/src/walkforward.rs`: + +```rust +/// One rolling split's time bounds: an in-sample window and the out-of-sample +/// window that follows it. Carries ONLY bounds — never tick data (#71). C2 +/// no-look-ahead is a pure invariant on these: oos.0 > is.1. Bounds are +/// inclusive (from, to) epoch-unit timestamps, matching RunManifest.window. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct WindowBounds { + pub is: (Timestamp, Timestamp), + pub oos: (Timestamp, Timestamp), +} + +/// Rolling (fixed-length IS, start advances by `step`) vs anchored (IS always +/// starts at the span origin and grows; OOS positions identical to rolling). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RollMode { Rolling, Anchored } + +/// A pure iterator of WindowBounds over a [origin, end] span — never holds or +/// indexes tick data (#71). Window k (k = 0, 1, ...): +/// oos_start_k = origin + is_len + k*step +/// is = (origin + k*step (Rolling) | origin (Anchored), oos_start_k - 1) +/// oos = (oos_start_k, oos_start_k + oos_len - 1) +/// Emitted while oos.1 <= end. Every split satisfies oos.0 == is.1 + 1 > is.1 (C2). +pub struct WindowRoller { /* span, is_len, oos_len, step, mode, k */ } + +impl WindowRoller { + /// Validate config: is_len/oos_len/step must be > 0; the span must fit at + /// least window 0 (origin + is_len + oos_len - 1 <= end). Else WalkForwardError. + pub fn new( + span: (Timestamp, Timestamp), + is_len: i64, + oos_len: i64, + step: i64, + mode: RollMode, + ) -> Result; +} + +impl Iterator for WindowRoller { + type Item = WindowBounds; + fn next(&mut self) -> Option; +} + +/// A structural fault configuring a walk-forward — the typed gate before any run, +/// analog to SweepError. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum WalkForwardError { + /// is_len / oos_len / step was <= 0. + NonPositiveLength { field: &'static str, value: i64 }, + /// The span is too short to fit even window 0. + SpanTooShort { span: (Timestamp, Timestamp), need: i64 }, +} + +/// What a per-window run yields: the params chosen on the in-sample window, the +/// recorded OOS pip-equity segment (for stitching), and the OOS RunReport (the +/// per-window C18 record). Self-describing, analog to SweepPoint/McDraw. +#[derive(Clone, Debug, PartialEq)] +pub struct WindowRun { + pub chosen_params: Vec, + pub oos_equity: Vec<(Timestamp, f64)>, + pub oos_report: RunReport, +} + +/// One completed window: its bounds + the run the closure produced, in roll order. +#[derive(Clone, Debug, PartialEq)] +pub struct WindowOutcome { + pub bounds: WindowBounds, + pub run: WindowRun, +} + +/// The result family of a walk-forward. Analog to SweepFamily — it stores ONLY +/// the raw per-window outcomes plus the stitched curve; any summary over the +/// windows (param stability, etc.) is an on-demand reduction, NOT a stored field +/// (just as SweepFamily stores raw points and optimize/rank_by are computed on +/// demand). Non-redundant by construction. +#[derive(Clone, Debug, PartialEq)] +pub struct WalkForwardResult { + /// Per-window outcomes in roll order (window 0 first), independent of thread + /// completion (C1). Each carries its bounds + the chosen params (the stability + /// substrate) + the OOS equity segment + the OOS RunReport. + pub windows: Vec, + /// The stitched continuous OOS pip-equity curve: each window's OOS segment + /// offset by the running sum of all prior segments' final cumulative values, + /// so equity carries forward across the IS/OOS boundaries (continuous, not a + /// per-window sawtooth). An empty OOS segment contributes 0.0 to the running + /// offset (mirroring summarize's unwrap_or(0.0) for an empty curve) and adds + /// no points, so an equity-less window leaves the curve unbroken. For the + /// canonical step == oos_len tiling the segment timestamps are also contiguous. + pub stitched_oos_equity: Vec<(Timestamp, f64)>, +} + +/// On-demand param-stability summary: one MetricStats per param slot, over the +/// chosen values (Scalar coerced to f64) across all windows — reuses the MC +/// family's MetricStats + type-7 quantile via MetricStats::from_values. A pure +/// reduction over result.windows[*].run.chosen_params, NOT stored on the result +/// (sweep-precedent: a summary is computed on demand, not cached), so it stays +/// non-redundant and the caller can compute any other measure (std, IQR, +/// distinct-count) from the same raw substrate. One entry per param slot; empty +/// if the strategy has no params. Slot count is the first window's arity (all +/// windows of one blueprint share the param-space arity, C11). +pub fn param_stability(result: &WalkForwardResult) -> Vec; + +/// Roll the windows, run `run_window` on each disjointly in parallel (C1, via the +/// shared run_indexed core), then stitch the OOS equity into one continuous curve. +/// The varying dimension is the DATA WINDOW (C12 axis 3). Eager-agnostic (#71): +/// run_window receives only WindowBounds; its data comes from a bounds-keyed +/// producer in the closure body, never a materialized stream Vec in this API. +/// Precondition: the roller yields >= 1 window (WindowRoller::new rejects an empty +/// roll). Param stability is a separate on-demand reduction — param_stability(&result) +/// — never computed here. +pub fn walk_forward(roller: WindowRoller, run_window: F) -> WalkForwardResult +where + F: Fn(WindowBounds) -> WindowRun + Sync; + +// Module-private thread-count-explicit core (walk_forward_with_threads), as in +// sweep/mc, so tests pin determinism at 1 and at N threads (C1). +``` + +Refactor, in `crates/aura-engine/src/mc.rs` (behaviour-preserving): + +```rust +impl MetricStats { + /// Mean + the fixed type-7 quantile set over a value set. Sorts a copy; + /// values must be finite and non-empty. The shared reduction behind both the + /// MC aggregate (a metric across draws) and the walk-forward param-stability + /// summary (a param across windows). + pub fn from_values(values: &[f64]) -> MetricStats { + let mut xs = values.to_vec(); + xs.sort_by(|a, b| a.partial_cmp(b).expect("values are finite")); + MetricStats { + mean: xs.iter().sum::() / xs.len() as f64, + p5: quantile(&xs, 0.05), + p25: quantile(&xs, 0.25), + p50: quantile(&xs, 0.50), + p75: quantile(&xs, 0.75), + p95: quantile(&xs, 0.95), + } + } +} +// McAggregate::from_draws' per-metric `pick` collects the metric's values, then +// calls MetricStats::from_values(&xs) — same numbers, one reduction. quantile +// stays module-private (from_values is in mc.rs). + +// MetricStats also gains serde::{Serialize, Deserialize} (consistent with +// RunMetrics/RunManifest/RunReport) so the CLI can render the stability summary +// (C14) and a later registry-lineage cycle (#70) can persist it. +``` + +Exports, in `crates/aura-engine/src/lib.rs`: + +```rust +mod walkforward; +pub use walkforward::{ + param_stability, walk_forward, RollMode, WalkForwardError, WalkForwardResult, + WindowBounds, WindowOutcome, WindowRoller, WindowRun, +}; +``` + +CLI, in `crates/aura-cli/src/main.rs`: + +```rust +// dispatch (the strict full-vector match): +["walkforward"] => run_walkforward(), +// USAGE gains "| aura walkforward". + +/// `aura walkforward`: run a built-in rolling walk-forward over the sample +/// blueprint + a synthetic windowed source; per window, sweep the built-in grid +/// on the in-sample slice, optimize by total_pips (axis 2 inside axis 3, where +/// aura-cli bridges engine + registry), run the chosen params out-of-sample, +/// persist the OOS RunReport (C18), print it, and finally print the stitched +/// summary. Deterministic (C1). +fn run_walkforward() { /* mirrors run_sweep */ } +``` + +## Components + +- **`WindowRoller`** (new, `walkforward.rs`) — bounds-only iterator; the C12 + axis-3 enumeration. Validated at construction (`WalkForwardError`). Pure: no + tick data, no I/O. +- **`walk_forward` / `walk_forward_with_threads`** (new, `walkforward.rs`) — + collects the roller's bounds (tiny: 4 timestamps each), runs `run_window` over + them via `run_indexed` (C1 disjoint parallel, roll-order output), then stitches + and summarizes. +- **`stitch` helper** (new, private, `walkforward.rs`) — pure reduction folding + the per-window OOS segments into `stitched_oos_equity` (running cumulative + offset; an empty segment adds +0.0 and no points). +- **`param_stability`** (new, **public on-demand fn**, `walkforward.rs`) — pure + reduction over `result.windows[*].run.chosen_params`; coerces `Scalar` to `f64` + (mirroring the CLI's existing `scalar_as_param_f64`) and calls + `MetricStats::from_values` per slot. NOT stored on the result. +- **`MetricStats::from_values`** (extracted, `mc.rs`) — the shared sort + mean + + type-7-quantile reduction; `McAggregate::from_draws` is rewired to it. +- **`run_walkforward`** (new, `main.rs`) — the CLI surface; the place the + engine→registry layering is bridged (it calls `aura_registry::optimize`). + +## Data flow + +1. `WindowRoller::new` validates config → an iterator of `WindowBounds`. +2. `walk_forward` collects the bounds (firewall-safe: bounds only), then + `run_indexed(n, nthreads, |i| run_window(bounds[i]))` runs each window + disjointly, results sorted back to roll order (C1). +3. Inside `run_window` (caller's): `make_source(is.0, is.1)` → in-sample run(s) → + `optimize` picks params → `make_source(oos.0, oos.1)` → OOS run → records the + OOS equity stream + builds the OOS `RunReport`. +4. `walk_forward` assembles `WindowOutcome`s and stitches the OOS segments into + `stitched_oos_equity` (running cumulative offset; empty segment → +0.0). Param + stability is **not** computed here — it is the on-demand `param_stability(&result)` + reduction a caller invokes when wanted. +5. CLI: each OOS `RunReport` → registry append (C18) + stdout line; the final + summary line calls `param_stability(&result)` for its per-param block. + +## Error handling + +- **Config faults** are typed and caught before any run: `WalkForwardError` + (`NonPositiveLength`, `SpanTooShort`) from `WindowRoller::new` — the same + "typed gate before execution" shape as `SweepError`. No look-ahead is not an + error path: it is structurally impossible (the roller only ever emits + `oos.0 = is.1 + 1`). +- **`walk_forward` returns the family directly** (no `Result`), matching + `sweep`/`mc`: the roller is pre-validated (non-empty by construction) and the + per-window closure owns its own fallibility. A `debug_assert!(!bounds.is_empty())` + guards the precondition, mirroring `monte_carlo`. +- **CLI registry I/O** failure → `eprintln!` + `exit(2)`, identical to `run_sweep`. + +## Testing strategy + +RED-first. New tests pin each load-bearing property; the `mc` refactor is guarded +by its seven existing tests staying green. + +`walkforward.rs`: + +1. `roller_rolling_emits_consecutive_oos_windows` — known span/lengths: window + count and `is`/`oos` bounds match the rolling formula; OOS windows tile + contiguously when `step == oos_len`. +2. `roller_anchored_fixes_is_start_grows_is_end` — Anchored: every `is.0 == origin`, + `is.1` grows by `step`; OOS positions identical to Rolling. +3. `roller_every_split_has_no_lookahead` — C2 bounds invariant: for every emitted + `WindowBounds`, `oos.0 > is.1` (zero ticks materialized). +4. `roller_rejects_nonpositive_lengths_and_short_span` — `WalkForwardError` on + `is_len<=0`, `oos_len<=0`, `step<=0`, and a span too short for window 0. +5. `walk_forward_runs_one_window_per_split_in_roll_order` — N bounds → N outcomes, + `windows[k].bounds` in roll order (window 0 first). +6. `walk_forward_is_deterministic_across_thread_counts` — `walk_forward_with_threads` + at 1 and at 8 are identical, and equal the public `walk_forward` (C1: order is + roll order, not completion order). +7. `stitched_curve_carries_equity_forward` — known per-window OOS segments stitch + with a running offset: segs `[(t0,2),(t1,5)]` then `[(t2,1),(t3,3)]` → + `[(t0,2),(t1,5),(t2,6),(t3,8)]` (continuous; no reset to 1 at t2). +7b. `stitched_curve_passes_through_empty_segment` — an empty OOS segment adds 0.0 + to the running offset and no points: segs `[(t0,2),(t1,5)]`, `[]`, `[(t2,1)]` + → `[(t0,2),(t1,5),(t2,6)]` (offset after the empty window stays 5, curve + unbroken). +8. `param_stability_reduces_chosen_params_per_slot` — the **on-demand** + `param_stability(&result)` returns one `MetricStats` per slot over the coerced + chosen values across windows (computed from `result.windows`, not a stored + field); known fixture (e.g. slot 0 values `[2,2,3,3]` → `mean 2.5`, `p50 2.5`). + +`mc.rs`: + +9. `metric_stats_from_values_matches_known_fixture` — `MetricStats::from_values(&[0,1,2,3,4])` + → `mean 2.0`, `p50 2.0`, `p5 ≈ 0.2`, `p95 ≈ 3.8` (the same numbers the existing + aggregate test pins, now directly on the extracted fn). Existing 7 tests stay green. + +`main.rs`: + +10. `walkforward_report_is_deterministic` — the built-in WFO render is byte-identical + across two calls (C1). +11. `walkforward_report_has_one_oos_line_per_window_plus_summary` — output is + `N` per-window OOS `RunReport` JSON lines followed by one `{"walkforward":...}` + summary line, for the built-in `N`-window roll. + +Gates: `cargo test --workspace`, `cargo clippy --workspace --all-targets -D warnings`, +`cargo doc -p aura-engine --no-deps`. + +## Acceptance criteria + +Applying aura's feature-acceptance criterion (CLAUDE.md), prospectively: + +- **The audience naturally reaches for it.** Walk-forward is a named C12 axis and + the canonical out-of-sample robustness validation; the worked example shows a + World author composing roller + `optimize` + run in one call. ✓ +- **Measurably improves correctness / removes redundancy.** It adds the only + axis that varies the data window (no current way to do rolling IS/OOS), reuses + `run_indexed`, `optimize`, and `MetricStats` rather than duplicating, and the + shared `MetricStats::from_values` extraction removes the duplicated reduction. + The result stores only raw per-window outcomes + the stitched curve — every + summary (param stability, std, IQR) is an on-demand reduction over the retained + substrate (sweep-precedent), so nothing recomputable is cached. ✓ +- **Reintroduces no eliminated failure class.** C1 (disjoint parallel via + `run_indexed`, roll-order output, 1-vs-N determinism pinned); C2 (no look-ahead + is structural — the roller only emits `oos.0 > is.1`, checkable with zero + ticks); #71 firewall (the API carries bounds + a closure, never a materialized + stream `Vec`); C9 (engine stays free of a registry dependency — `optimize` is + closure-supplied). ✓ + +Concretely, the issue's acceptance: **a known multi-window split produces +deterministic per-window results (tests 5, 6, 10) and a correctly-stitched OOS +curve (test 7), with no look-ahead across the IS/OOS boundary (test 3, C2).**