spec: 0044 walk-forward family (C12 axis 3)

Walk-forward orchestration: a WindowRoller rolls (in-sample, out-of-sample)
splits over a time span, walk_forward runs a disjoint harness per split via the
shared run_indexed core (C1), and stitches the OOS pip-equity segments into one
continuous curve. The varying dimension is the data window (axis 3), realized
eager-agnostically (#71): the public surface carries WindowBounds + a per-window
closure, never a materialized stream Vec; C2 no-look-ahead is a pure bounds
invariant (oos.0 > is.1), checkable with zero ticks.

The in-sample optimize (axis 2) is closure-supplied, not called by the engine:
aura-engine cannot depend on aura-registry (C9), and C12 forbids baking search
policy into the primitive. The CLI bridges both crates in run_walkforward.

Param-stability shape (the load-bearing fork) resolved with the user as R2:
on-demand, not stored. WalkForwardResult stores only raw per-window outcomes +
the stitched curve; param_stability(&result) -> Vec<MetricStats> is a public
on-demand reduction over the retained per-window chosen params (sweep-precedent:
summaries computed on demand, not cached, vs McFamily's stored aggregate). A
stored summary would be recomputable from the raw windows (redundant) and would
force one statistic canonical when "stability" admits several. Recorded with
provenance in the #69 reconciliation comment. MetricStats::from_values is
extracted from McAggregate::from_draws (behaviour-preserving; the 7 mc tests
guard) so the MC aggregate and the helper share one reduction. Empty OOS segment
contributes 0.0 to the stitch offset (mirrors summarize's unwrap_or(0.0)).

Gates: Step-1.5 precondition clean (no fork silently picked), self-review clean,
grounding-check PASS (twice — re-run after each edit). Auto-sign panel escalated
on the scope-fork design lens (param-stability shape) to human sign-off; the user
resolved it (R2), so this is a user-signed spec, not boss-signed.

refs #69
This commit is contained in:
2026-06-15 14:56:12 +02:00
parent 61a8436c2c
commit a4712eb336
+430
View File
@@ -0,0 +1,430 @@
# Walk-forward orchestration family (C12 axis 3) — Design Spec
**Date:** 2026-06-15
**Status:** Approved — user sign-off (R2, 2026-06-15)
**Authors:** orchestrator + Claude
## Goal
Deliver C12's third orchestration axis: **walk-forward**. Unlike the grid sweep
(axis 1, varies a param) and Monte-Carlo (axis 4, varies the seed), walk-forward
varies the **data window** — it rolls (in-sample, out-of-sample) splits over a
time span, runs a disjoint harness per split (optionally optimizing params on the
in-sample slice, reusing axis 2), and stitches the out-of-sample pip-equity
segments into one continuous out-of-sample curve. The result carries the
per-window **chosen params** as the stability substrate; a per-parameter
stability summary is an **on-demand reduction** over them (a provided helper),
**not** a stored field — see Architecture.
The axis is realized **eager-agnostically** (the #71 firewall): the public
surface carries **time-bounds, never tick data**. A 20-year history is the
~190 GB eager object; the walk-forward surface must not be able to encode a
materialized history. The window roller is a pure iterator of `WindowBounds`;
per-window data comes from a bounds-keyed producer closure. The C2 no-look-ahead
guarantee is then a pure bounds invariant (`oos.0 > is.1`), checkable with zero
ticks materialized.
## Architecture
A new `walkforward` module in `aura-engine`, alongside `sweep` and `mc`. It owns
three things the other two axes' modules already model: **enumeration** (the
`WindowRoller`, analog to `GridSpace`), **execution** (the `walk_forward`
function, reusing the shared `run_indexed` disjoint-parallel core — C1), and
**collection** (`WalkForwardResult`, analog to `SweepFamily`/`McFamily`).
The orchestrator owns rolling + disjoint execution + stitching. It
delegates the per-window run — build/bootstrap/run, and the *optional* in-sample
param optimization — to a caller-supplied closure, exactly as `sweep` and `mc`
delegate the per-unit run to `run_one`. This is forced, not chosen:
- **Crate separation (C9) + dependency direction.** `aura-engine` is the engine;
`aura-registry` depends on `aura-engine` and owns `optimize` (axis 2). The
walk-forward orchestrator lives in `aura-engine` (it reuses the `pub(crate)`
`run_indexed`), so it **cannot** call `optimize` directly without inverting the
dependency. The in-sample selection is therefore a closure concern: the caller
(the CLI / a World, both of which depend on both crates) supplies `optimize` as
the in-sample step.
- **C12 forbids baking search policy into the primitive.** "Optimization" is a
pluggable policy atop the atomic unit, not part of the orchestration primitive.
A closure-supplied in-sample step is the contract C12 already mandates.
The in-sample/out-of-sample data both flow through a bounds-keyed producer
`Fn(Timestamp, Timestamp) -> impl Source` living **inside** the caller's closure
— which maps onto the existing `aura-ingest` method
`DataServer::stream_m1_windowed(symbol, from: Option<i64>, to: Option<i64>)`
(`aura-ingest/src/lib.rs`; per-bound `Option` windows) for real-data production.
No materialized stream `Vec` crosses the `walk_forward` API boundary (#71). The
engine code this iteration adds never calls that method — it is the production
mapping the closure body fulfils; the built-in CLI demo uses an **in-memory
windowed source** instead (mirroring `run_sweep`'s `showcase_prices`), so the
iteration adds no new `aura-ingest` dependency.
**Param-stability is on-demand, not a stored field — and this is the load-bearing
shape decision (resolved with the user, see the #69 reconciliation comment).**
The result stores only the raw per-window outcomes + the stitched curve, exactly
as a `SweepFamily` stores only raw points and `optimize`/`rank_by` are computed on
demand (never cached). The raw per-window `chosen_params` *are* the stability
substrate; a stored summary would be a pure reduction over them — recomputable,
hence redundant — and would force one statistic to be canonical when "stability"
admits several (the MC `MetricStats` distribution, a moment std/CV, IQR, a
distinct-value count). So the result caches none of them. A convenience reduction
ships as the **public on-demand fn `param_stability(&WalkForwardResult) ->
Vec<MetricStats>`**, which reuses the Monte-Carlo family's `MetricStats` (mean +
type-7 quantiles) per param slot; the caller is free to compute any other measure
from the same `windows[*].run.chosen_params`. The `MetricStats` reduction is
extracted from `McAggregate::from_draws` into a shared
`MetricStats::from_values(&[f64])` (a behaviour-preserving refactor; the seven
existing `mc` tests are the green guard) so both the MC aggregate and this helper
share one reduction.
## Concrete code shapes
### Worked author example (the user-facing program — acceptance evidence)
A World author validates a strategy out-of-sample by composing the roller, the
axis-2 optimizer, and a per-window run inside one `walk_forward` call:
```rust
use aura_engine::{param_stability, walk_forward, RollMode, WindowBounds, WindowRoller, WindowRun, Timestamp};
use aura_registry::optimize;
// A bounds-keyed producer: for real data, maps onto the method
// DataServer::stream_m1_windowed(symbol, Option<i64>, Option<i64>). The SURFACE is
// bounds; a first cut MAY collect (from..to) internally (#71 firewall: the *type*
// cannot encode a materialized history, the impl may be eager for now).
let make_source = |from: Timestamp, to: Timestamp| -> MyWindowedSource { /* ... */ };
// 1 year in-sample, 3 months out-of-sample, stepping 3 months, rolling
// (fixed-length IS). Anchored (growing IS) is RollMode::Anchored.
let roller = WindowRoller::new(span, IS_LEN, OOS_LEN, STEP, RollMode::Rolling)?;
let result = walk_forward(roller, |w: WindowBounds| -> WindowRun {
// in-sample: sweep a grid, pick the best params (axis 2 INSIDE axis 3)
let is_family = grid.sweep(|pt| run_point(pt, make_source(w.is.0, w.is.1)));
let best = optimize(&is_family, "total_pips").expect("known metric");
// out-of-sample: run the chosen params on the held-out window, record equity
let (oos_equity, oos_report) = run_and_record(&best.params, make_source(w.oos.0, w.oos.1));
WindowRun { chosen_params: best.params, oos_equity, oos_report }
});
// Runs are disjoint across windows (C1); no look-ahead by construction
// (C2: every w.oos.0 > w.is.1, guaranteed by the roller).
result.stitched_oos_equity; // one continuous OOS pip curve across all windows
result.windows[0].run.chosen_params; // raw per-window chosen params (the stability substrate)
param_stability(&result)[0]; // on-demand: MetricStats of param 0 across windows
```
CLI surface (mirrors `aura sweep`):
```
$ aura walkforward
{"manifest":{...,"window":[<oos0_from>,<oos0_to>],...},"metrics":{...}} # window 0 OOS record
{"manifest":{...,"window":[<oos1_from>,<oos1_to>],...},"metrics":{...}} # window 1 OOS record
...
{"walkforward":{"windows":N,"stitched_total_pips":X.0,"param_stability":[{"mean":..,"p5":..,...}]}}
```
Each per-window OOS `RunReport` is persisted to the run registry (C18) and printed
as one JSON line; a final summary line carries the window count, the stitched
total, and the per-param stability.
### Implementation shapes (secondary)
New, in `crates/aura-engine/src/walkforward.rs`:
```rust
/// One rolling split's time bounds: an in-sample window and the out-of-sample
/// window that follows it. Carries ONLY bounds — never tick data (#71). C2
/// no-look-ahead is a pure invariant on these: oos.0 > is.1. Bounds are
/// inclusive (from, to) epoch-unit timestamps, matching RunManifest.window.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct WindowBounds {
pub is: (Timestamp, Timestamp),
pub oos: (Timestamp, Timestamp),
}
/// Rolling (fixed-length IS, start advances by `step`) vs anchored (IS always
/// starts at the span origin and grows; OOS positions identical to rolling).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum RollMode { Rolling, Anchored }
/// A pure iterator of WindowBounds over a [origin, end] span — never holds or
/// indexes tick data (#71). Window k (k = 0, 1, ...):
/// oos_start_k = origin + is_len + k*step
/// is = (origin + k*step (Rolling) | origin (Anchored), oos_start_k - 1)
/// oos = (oos_start_k, oos_start_k + oos_len - 1)
/// Emitted while oos.1 <= end. Every split satisfies oos.0 == is.1 + 1 > is.1 (C2).
pub struct WindowRoller { /* span, is_len, oos_len, step, mode, k */ }
impl WindowRoller {
/// Validate config: is_len/oos_len/step must be > 0; the span must fit at
/// least window 0 (origin + is_len + oos_len - 1 <= end). Else WalkForwardError.
pub fn new(
span: (Timestamp, Timestamp),
is_len: i64,
oos_len: i64,
step: i64,
mode: RollMode,
) -> Result<WindowRoller, WalkForwardError>;
}
impl Iterator for WindowRoller {
type Item = WindowBounds;
fn next(&mut self) -> Option<WindowBounds>;
}
/// A structural fault configuring a walk-forward — the typed gate before any run,
/// analog to SweepError.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum WalkForwardError {
/// is_len / oos_len / step was <= 0.
NonPositiveLength { field: &'static str, value: i64 },
/// The span is too short to fit even window 0.
SpanTooShort { span: (Timestamp, Timestamp), need: i64 },
}
/// What a per-window run yields: the params chosen on the in-sample window, the
/// recorded OOS pip-equity segment (for stitching), and the OOS RunReport (the
/// per-window C18 record). Self-describing, analog to SweepPoint/McDraw.
#[derive(Clone, Debug, PartialEq)]
pub struct WindowRun {
pub chosen_params: Vec<Scalar>,
pub oos_equity: Vec<(Timestamp, f64)>,
pub oos_report: RunReport,
}
/// One completed window: its bounds + the run the closure produced, in roll order.
#[derive(Clone, Debug, PartialEq)]
pub struct WindowOutcome {
pub bounds: WindowBounds,
pub run: WindowRun,
}
/// The result family of a walk-forward. Analog to SweepFamily — it stores ONLY
/// the raw per-window outcomes plus the stitched curve; any summary over the
/// windows (param stability, etc.) is an on-demand reduction, NOT a stored field
/// (just as SweepFamily stores raw points and optimize/rank_by are computed on
/// demand). Non-redundant by construction.
#[derive(Clone, Debug, PartialEq)]
pub struct WalkForwardResult {
/// Per-window outcomes in roll order (window 0 first), independent of thread
/// completion (C1). Each carries its bounds + the chosen params (the stability
/// substrate) + the OOS equity segment + the OOS RunReport.
pub windows: Vec<WindowOutcome>,
/// The stitched continuous OOS pip-equity curve: each window's OOS segment
/// offset by the running sum of all prior segments' final cumulative values,
/// so equity carries forward across the IS/OOS boundaries (continuous, not a
/// per-window sawtooth). An empty OOS segment contributes 0.0 to the running
/// offset (mirroring summarize's unwrap_or(0.0) for an empty curve) and adds
/// no points, so an equity-less window leaves the curve unbroken. For the
/// canonical step == oos_len tiling the segment timestamps are also contiguous.
pub stitched_oos_equity: Vec<(Timestamp, f64)>,
}
/// On-demand param-stability summary: one MetricStats per param slot, over the
/// chosen values (Scalar coerced to f64) across all windows — reuses the MC
/// family's MetricStats + type-7 quantile via MetricStats::from_values. A pure
/// reduction over result.windows[*].run.chosen_params, NOT stored on the result
/// (sweep-precedent: a summary is computed on demand, not cached), so it stays
/// non-redundant and the caller can compute any other measure (std, IQR,
/// distinct-count) from the same raw substrate. One entry per param slot; empty
/// if the strategy has no params. Slot count is the first window's arity (all
/// windows of one blueprint share the param-space arity, C11).
pub fn param_stability(result: &WalkForwardResult) -> Vec<MetricStats>;
/// Roll the windows, run `run_window` on each disjointly in parallel (C1, via the
/// shared run_indexed core), then stitch the OOS equity into one continuous curve.
/// The varying dimension is the DATA WINDOW (C12 axis 3). Eager-agnostic (#71):
/// run_window receives only WindowBounds; its data comes from a bounds-keyed
/// producer in the closure body, never a materialized stream Vec in this API.
/// Precondition: the roller yields >= 1 window (WindowRoller::new rejects an empty
/// roll). Param stability is a separate on-demand reduction — param_stability(&result)
/// — never computed here.
pub fn walk_forward<F>(roller: WindowRoller, run_window: F) -> WalkForwardResult
where
F: Fn(WindowBounds) -> WindowRun + Sync;
// Module-private thread-count-explicit core (walk_forward_with_threads), as in
// sweep/mc, so tests pin determinism at 1 and at N threads (C1).
```
Refactor, in `crates/aura-engine/src/mc.rs` (behaviour-preserving):
```rust
impl MetricStats {
/// Mean + the fixed type-7 quantile set over a value set. Sorts a copy;
/// values must be finite and non-empty. The shared reduction behind both the
/// MC aggregate (a metric across draws) and the walk-forward param-stability
/// summary (a param across windows).
pub fn from_values(values: &[f64]) -> MetricStats {
let mut xs = values.to_vec();
xs.sort_by(|a, b| a.partial_cmp(b).expect("values are finite"));
MetricStats {
mean: xs.iter().sum::<f64>() / xs.len() as f64,
p5: quantile(&xs, 0.05),
p25: quantile(&xs, 0.25),
p50: quantile(&xs, 0.50),
p75: quantile(&xs, 0.75),
p95: quantile(&xs, 0.95),
}
}
}
// McAggregate::from_draws' per-metric `pick` collects the metric's values, then
// calls MetricStats::from_values(&xs) — same numbers, one reduction. quantile
// stays module-private (from_values is in mc.rs).
// MetricStats also gains serde::{Serialize, Deserialize} (consistent with
// RunMetrics/RunManifest/RunReport) so the CLI can render the stability summary
// (C14) and a later registry-lineage cycle (#70) can persist it.
```
Exports, in `crates/aura-engine/src/lib.rs`:
```rust
mod walkforward;
pub use walkforward::{
param_stability, walk_forward, RollMode, WalkForwardError, WalkForwardResult,
WindowBounds, WindowOutcome, WindowRoller, WindowRun,
};
```
CLI, in `crates/aura-cli/src/main.rs`:
```rust
// dispatch (the strict full-vector match):
["walkforward"] => run_walkforward(),
// USAGE gains "| aura walkforward".
/// `aura walkforward`: run a built-in rolling walk-forward over the sample
/// blueprint + a synthetic windowed source; per window, sweep the built-in grid
/// on the in-sample slice, optimize by total_pips (axis 2 inside axis 3, where
/// aura-cli bridges engine + registry), run the chosen params out-of-sample,
/// persist the OOS RunReport (C18), print it, and finally print the stitched
/// summary. Deterministic (C1).
fn run_walkforward() { /* mirrors run_sweep */ }
```
## Components
- **`WindowRoller`** (new, `walkforward.rs`) — bounds-only iterator; the C12
axis-3 enumeration. Validated at construction (`WalkForwardError`). Pure: no
tick data, no I/O.
- **`walk_forward` / `walk_forward_with_threads`** (new, `walkforward.rs`) —
collects the roller's bounds (tiny: 4 timestamps each), runs `run_window` over
them via `run_indexed` (C1 disjoint parallel, roll-order output), then stitches
and summarizes.
- **`stitch` helper** (new, private, `walkforward.rs`) — pure reduction folding
the per-window OOS segments into `stitched_oos_equity` (running cumulative
offset; an empty segment adds +0.0 and no points).
- **`param_stability`** (new, **public on-demand fn**, `walkforward.rs`) — pure
reduction over `result.windows[*].run.chosen_params`; coerces `Scalar` to `f64`
(mirroring the CLI's existing `scalar_as_param_f64`) and calls
`MetricStats::from_values` per slot. NOT stored on the result.
- **`MetricStats::from_values`** (extracted, `mc.rs`) — the shared sort + mean +
type-7-quantile reduction; `McAggregate::from_draws` is rewired to it.
- **`run_walkforward`** (new, `main.rs`) — the CLI surface; the place the
engine→registry layering is bridged (it calls `aura_registry::optimize`).
## Data flow
1. `WindowRoller::new` validates config → an iterator of `WindowBounds`.
2. `walk_forward` collects the bounds (firewall-safe: bounds only), then
`run_indexed(n, nthreads, |i| run_window(bounds[i]))` runs each window
disjointly, results sorted back to roll order (C1).
3. Inside `run_window` (caller's): `make_source(is.0, is.1)` → in-sample run(s) →
`optimize` picks params → `make_source(oos.0, oos.1)` → OOS run → records the
OOS equity stream + builds the OOS `RunReport`.
4. `walk_forward` assembles `WindowOutcome`s and stitches the OOS segments into
`stitched_oos_equity` (running cumulative offset; empty segment → +0.0). Param
stability is **not** computed here — it is the on-demand `param_stability(&result)`
reduction a caller invokes when wanted.
5. CLI: each OOS `RunReport` → registry append (C18) + stdout line; the final
summary line calls `param_stability(&result)` for its per-param block.
## Error handling
- **Config faults** are typed and caught before any run: `WalkForwardError`
(`NonPositiveLength`, `SpanTooShort`) from `WindowRoller::new` — the same
"typed gate before execution" shape as `SweepError`. No look-ahead is not an
error path: it is structurally impossible (the roller only ever emits
`oos.0 = is.1 + 1`).
- **`walk_forward` returns the family directly** (no `Result`), matching
`sweep`/`mc`: the roller is pre-validated (non-empty by construction) and the
per-window closure owns its own fallibility. A `debug_assert!(!bounds.is_empty())`
guards the precondition, mirroring `monte_carlo`.
- **CLI registry I/O** failure → `eprintln!` + `exit(2)`, identical to `run_sweep`.
## Testing strategy
RED-first. New tests pin each load-bearing property; the `mc` refactor is guarded
by its seven existing tests staying green.
`walkforward.rs`:
1. `roller_rolling_emits_consecutive_oos_windows` — known span/lengths: window
count and `is`/`oos` bounds match the rolling formula; OOS windows tile
contiguously when `step == oos_len`.
2. `roller_anchored_fixes_is_start_grows_is_end` — Anchored: every `is.0 == origin`,
`is.1` grows by `step`; OOS positions identical to Rolling.
3. `roller_every_split_has_no_lookahead` — C2 bounds invariant: for every emitted
`WindowBounds`, `oos.0 > is.1` (zero ticks materialized).
4. `roller_rejects_nonpositive_lengths_and_short_span``WalkForwardError` on
`is_len<=0`, `oos_len<=0`, `step<=0`, and a span too short for window 0.
5. `walk_forward_runs_one_window_per_split_in_roll_order` — N bounds → N outcomes,
`windows[k].bounds` in roll order (window 0 first).
6. `walk_forward_is_deterministic_across_thread_counts``walk_forward_with_threads`
at 1 and at 8 are identical, and equal the public `walk_forward` (C1: order is
roll order, not completion order).
7. `stitched_curve_carries_equity_forward` — known per-window OOS segments stitch
with a running offset: segs `[(t0,2),(t1,5)]` then `[(t2,1),(t3,3)]`
`[(t0,2),(t1,5),(t2,6),(t3,8)]` (continuous; no reset to 1 at t2).
7b. `stitched_curve_passes_through_empty_segment` — an empty OOS segment adds 0.0
to the running offset and no points: segs `[(t0,2),(t1,5)]`, `[]`, `[(t2,1)]`
`[(t0,2),(t1,5),(t2,6)]` (offset after the empty window stays 5, curve
unbroken).
8. `param_stability_reduces_chosen_params_per_slot` — the **on-demand**
`param_stability(&result)` returns one `MetricStats` per slot over the coerced
chosen values across windows (computed from `result.windows`, not a stored
field); known fixture (e.g. slot 0 values `[2,2,3,3]``mean 2.5`, `p50 2.5`).
`mc.rs`:
9. `metric_stats_from_values_matches_known_fixture``MetricStats::from_values(&[0,1,2,3,4])`
`mean 2.0`, `p50 2.0`, `p5 ≈ 0.2`, `p95 ≈ 3.8` (the same numbers the existing
aggregate test pins, now directly on the extracted fn). Existing 7 tests stay green.
`main.rs`:
10. `walkforward_report_is_deterministic` — the built-in WFO render is byte-identical
across two calls (C1).
11. `walkforward_report_has_one_oos_line_per_window_plus_summary` — output is
`N` per-window OOS `RunReport` JSON lines followed by one `{"walkforward":...}`
summary line, for the built-in `N`-window roll.
Gates: `cargo test --workspace`, `cargo clippy --workspace --all-targets -D warnings`,
`cargo doc -p aura-engine --no-deps`.
## Acceptance criteria
Applying aura's feature-acceptance criterion (CLAUDE.md), prospectively:
- **The audience naturally reaches for it.** Walk-forward is a named C12 axis and
the canonical out-of-sample robustness validation; the worked example shows a
World author composing roller + `optimize` + run in one call. ✓
- **Measurably improves correctness / removes redundancy.** It adds the only
axis that varies the data window (no current way to do rolling IS/OOS), reuses
`run_indexed`, `optimize`, and `MetricStats` rather than duplicating, and the
shared `MetricStats::from_values` extraction removes the duplicated reduction.
The result stores only raw per-window outcomes + the stitched curve — every
summary (param stability, std, IQR) is an on-demand reduction over the retained
substrate (sweep-precedent), so nothing recomputable is cached. ✓
- **Reintroduces no eliminated failure class.** C1 (disjoint parallel via
`run_indexed`, roll-order output, 1-vs-N determinism pinned); C2 (no look-ahead
is structural — the roller only emits `oos.0 > is.1`, checkable with zero
ticks); #71 firewall (the API carries bounds + a closure, never a materialized
stream `Vec`); C9 (engine stays free of a registry dependency — `optimize` is
closure-supplied). ✓
Concretely, the issue's acceptance: **a known multi-window split produces
deterministic per-window results (tests 5, 6, 10) and a correctly-stitched OOS
curve (test 7), with no look-ahead across the IS/OOS boundary (test 3, C2).**