diff --git a/docs/specs/0097-is-refit-walk-forward-over-a-loaded-blueprint.md b/docs/specs/0097-is-refit-walk-forward-over-a-loaded-blueprint.md new file mode 100644 index 0000000..7570f61 --- /dev/null +++ b/docs/specs/0097-is-refit-walk-forward-over-a-loaded-blueprint.md @@ -0,0 +1,314 @@ +# In-sample-refit walk-forward over a loaded blueprint — Design Spec + +**Date:** 2026-07-01 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Ship the World/C21 milestone's last piece: a **true in-sample-refit walk-forward +over a loaded blueprint**. `aura walkforward --axis = +[--axis …]` rolls IS/OOS windows over a synthetic span; on each IS window it +**re-optimizes** the blueprint's params over the user's `--axis` grid, runs the +winner **out-of-sample**, and aggregates the per-window OOS reports into a +`FamilyKind::WalkForward` family that is content-addressed and `aura reproduce`s +bit-identically. This is Arm A (the user's decision, #173) — the honest OOS +validation the milestone exists to enable, and the form that legitimately carries +the "walk-forward" name (matching the hard-wired verb's IS-optimizing guarantee). + +## Architecture + +The hard-wired `aura walkforward --strategy stage1-r` ALREADY does IS-refit +(`walkforward_family`'s `Stage1R` arm, `crates/aura-cli/src/main.rs`): for each +window it `stage1_r_sweep_over(is) → select_winner("sqn_normalized") → +run_oos_r(winner, oos)`. The loaded-blueprint form is the **same shape with one +substitution** — the per-window IS-sweep and OOS-run source the **loaded +blueprint** (wrapped by `wrap_stage1r` via cycle-0096's `blueprint_axis_probe`) +instead of the built-in strategy. Everything else is reused verbatim: the generic +engine driver `walk_forward(roller, space, run_window)` (strategy-agnostic, +`crates/aura-engine/src/walkforward.rs`), `select_winner` (Argmax-deflated / +Plateau), the `24/12/12`-over-60-bar synthetic schedule (`wf_full_span` / +`wf_window_sizes`), and `FamilyKind::WalkForward` + `walkforward_member_reports` + +`walkforward_summary_json`. + +Reused windowing seam: `data.windowed_sources(from, to)` (the same seam +`stage1_r_sweep_over` and `run_oos_r` use). No look-ahead: IS strictly precedes +OOS in every window (`WindowRoller`), and the loaded-member path (`run_blueprint_member`) +already only sees its window's stream (invariants 1/2). + +Persistence + reproduce are shipped **together** (they are coupled — a persisted +family with no store hook reports members as "blueprint missing", the #170 +hazard). The family is stored once by the shared `topology_hash` +(`put_blueprint`, the C18 hook the sweep/mc use) + `append_family`; `aura +reproduce` gains a `WalkForward` branch that reconstructs each OOS member's +windowed source (from its manifest window) and re-runs it with the member's +stored winner params — mirroring the `MonteCarlo` reproduce branch from cycle +0095. One signal topology → one shared `topology_hash` across all windows; each +member manifest carries `(topology_hash, window, chosen params, selection)`. + +Scope: `crates/aura-cli` only (+ read-only reuse of the engine/registry WF +machinery). No engine/registry change — invariants 8/9 hold. + +## Concrete code shapes + +### Worked user program (the acceptance evidence) + +A researcher has an OPEN strategy blueprint and wants an honest walk-forward that +re-fits its two SMA lengths each window (the axis names come from `--list-axes`, +cycle 0096): + +```console +$ aura sweep crates/aura-cli/tests/fixtures/stage1_signal_open.json --list-axes +stage1_signal.fast.length:I64 +stage1_signal.slow.length:I64 + +$ aura walkforward crates/aura-cli/tests/fixtures/stage1_signal_open.json \ + --axis stage1_signal.fast.length=2,3 --axis stage1_signal.slow.length=4,6 --name wf1 +walkforward-0 +walkforward-1 +walkforward-2 +{"walkforward":{"windows":3,"stitched_total_pips":,"param_stability":<…>,"oos_r":{…}}} + +$ # content-addressed: the family reproduces bit-identically +$ aura reproduce walkforward-0 +reproduced 3/3 members bit-identically +``` + +Per window the IS grid (2×2 here) is re-optimized by `sqn_normalized`; the winner +runs OOS; the three OOS reports are the family members (line shape identical to +the hard-wired `aura walkforward`, which reuses `family_member_line`). Exact +metric numbers are run-determined; the *shape* — three re-fit windows, OOS +members, a `{"walkforward":…}` summary, and a bit-identical reproduce — is the +criterion. + +An OPEN blueprint with **no** `--axis` (nothing to re-fit) is a usage error: + +```console +$ aura walkforward crates/aura-cli/tests/fixtures/stage1_signal_open.json +aura: walkforward requires >= 1 --axis to re-fit per window +$ echo $? +2 +``` + +### Implementation shape (secondary) + +**The family builder** — the loaded-blueprint analog of `walkforward_family`'s +`Stage1R` arm; reuses the generic `walk_forward` driver, substituting the loaded +blueprint into the per-window closure: + +```rust +fn blueprint_walkforward_family( + doc: &str, axes: &[(String, Vec)], data: &DataSource, select: Selection, +) -> WalkForwardResult { + let span = data.wf_full_span(); + let (is_len, oos_len, step) = data.wf_window_sizes(); + let roller = WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling) + .unwrap_or_else(|e| { eprintln!("aura: walk-forward window too short: {e:?}"); std::process::exit(2); }); + let space = blueprint_axis_probe(doc).param_space(); // the wrapped namespace (cycle 0096) + let pip = data.pip_size(); + let topo = topology_hash(&blueprint_from_json(doc, &|t| std_vocabulary(t)) + .expect("doc parse-validated at the dispatch boundary")); + walk_forward(roller, space.clone(), |w: WindowBounds| { + // IS: re-fit the loaded blueprint over the user grid, windowed to [is] + let (is_family, lattice) = blueprint_sweep_over(doc, axes, w.is.0, w.is.1, data); + let (best, selection) = select_winner(&is_family, "sqn_normalized", select, lattice.as_deref()) + .unwrap_or_else(|msg| { eprintln!("aura: {msg}"); std::process::exit(2); }); + // OOS: run the winner over [oos] via the loaded-member path + let (oos_equity, mut oos_report) = + run_oos_blueprint(doc, &best.params, &space, w.oos.0, w.oos.1, pip, &topo, data); + oos_report.manifest.selection = Some(selection); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) +} +``` + +**The windowed IS-sweep** — `blueprint_sweep_over`: the structural twin of +`stage1_r_sweep_over` (uses `sweep_with_lattice` so `--select plateau` works) and +of `blueprint_sweep_family` (loaded blueprint, `--axis` grid), but windowed to +`[from,to]` via `data.windowed_sources`: + +```rust +fn blueprint_sweep_over(doc: &str, axes: &[(String, Vec)], + from: Timestamp, to: Timestamp, data: &DataSource) + -> (SweepFamily, Option>) { + let pip = data.pip_size(); + let topo = topology_hash(&reload(doc)); + let probe = blueprint_axis_probe(doc); + let space = probe.param_space(); + // seed the user axes verbatim (same as blueprint_sweep_family), then sweep WITH lattice + let mut binder = /* probe.axis(first) then SweepBinder::axis(rest) */; + binder.sweep_with_lattice(|point| { + let sources = data.windowed_sources(from, to); + let window = window_of(&sources).expect("non-empty in-sample window"); + run_blueprint_member(reload(doc), point, &space, sources, window, 0, pip, &topo) + }).map(|(fam, lat)| (fam, Some(lat))) + .expect("the user axes are name/kind-checked at the sweep terminal") +} +``` + +**The windowed OOS-run** — `run_oos_blueprint`: the loaded-member analog of +`run_oos_r`, running the winner params over the OOS window. The loaded-member +path (`run_blueprint_member`) is **reduce-mode**: it folds to R-metrics +(`summarize_r`) and retains the per-trade R series, but NOT the raw per-cycle pip +curve the hard-wired non-reduce `run_oos_r` returns. So `oos_equity` is **empty** +— the loaded walk-forward is measured in **R** (the C10-reworked yardstick), and +the meaningful per-window record is the OOS `RunReport`'s R-metrics: + +```rust +fn run_oos_blueprint(doc: &str, params: &[Cell], space: &[ParamSpec], + from: Timestamp, to: Timestamp, pip: f64, topo: &str, data: &DataSource) + -> (Vec<(Timestamp, f64)>, RunReport) { + let sources = data.windowed_sources(from, to); + let window = window_of(&sources).expect("non-empty out-of-sample window"); + let report = run_blueprint_member(reload(doc), params, space, sources, window, 0, pip, topo); + // reduce-mode: no raw pip curve retained -> empty stitching segment. The OOS + // record is R-metrics (report.metrics.r); an empty segment leaves the stitched + // curve unbroken (stitch treats it as a 0-offset, no-point contribution). + (Vec::new(), report) +} +``` + +Consequence for the summary: `walkforward_summary_json`'s `stitched_total_pips` +is `0.0` (no retained pip curve — the loaded path is R-measured, not pip-measured), +and its `oos_r` block (pooled per-trade Rs across the OOS windows) is the +meaningful line. This matches the R-first world of the C10 rework; a raw pip-equity +curve would need a non-reduce OOS run (`--trace`-style), out of scope here. + +**The IO wrapper + dispatch** — `run_blueprint_walkforward` mirrors +`run_walkforward` (persist via `put_blueprint` + `append_family`, print members + +summary); the `["walkforward", rest @ ..]` dispatch gains the `.json` +discriminator BEFORE `parse_walkforward_args`, mirroring the sweep/mc arms: + +```rust +["walkforward", rest @ ..] => { + if let Some(path) = rest.first().filter(|a| a.ends_with(".json") && Path::new(a).is_file()) { + let doc = std::fs::read_to_string(path).unwrap_or_else(|e| { eprintln!("aura: {path}: {e}"); exit(2) }); + if let Err(e) = blueprint_from_json(&doc, &|t| std_vocabulary(t)) { + eprintln!("aura: {path}: {e:?}"); exit(2); + } + let WfBlueprintArgs { axes, name, persist, select } = + parse_walkforward_blueprint_args(&rest[1..]).unwrap_or_else(|m| { eprintln!("aura: {m}"); exit(2) }); + run_blueprint_walkforward(&doc, &axes, &name, persist, DataSource::Synthetic, select); + return; + } + match parse_walkforward_args(rest) { /* unchanged hard-wired path */ } +} +``` + +**The reproduce branch** — `reproduce_family_in` already re-runs each member via +`run_blueprint_member` with a per-member `(sources, window, params)` triple chosen +by `family.kind` (cycle 0094 recovers the member's params from its manifest +against the reloaded space; cycle 0095 added the `MonteCarlo` arm that rebuilds +the seed-driven walk). The new `WalkForward` arm keeps the **same per-member param +recovery** (the winner params are stored in each OOS member's manifest, exactly as +a sweep member's are) and differs only in the **source**: the OOS window's slice, +rebuilt from the member manifest's stored window bounds: + +```rust +match family.kind { + FamilyKind::MonteCarlo => { /* cycle 0095: manifest.seed -> synthetic walk */ } + FamilyKind::WalkForward => { + // the OOS slice from the member manifest's window; params via the SAME + // manifest->cells recovery the Sweep arm already uses (winner params). + let (from, to) = stored.manifest.window; + let sources = data.windowed_sources(from, to); + (sources, window_of(&sources)/* the member window */) + } + _ => (data.run_sources(), data.full_window()), // Sweep: the showcase series +} +``` + +The member's params are recovered by the existing shared path (not re-invented +here); only the `(sources, window)` pair is realization-specific — `WalkForward` +supplies the windowed OOS slice, exactly as `MonteCarlo` supplies the seeded walk. + +## Components + +- `blueprint_walkforward_family(doc, axes, data, select) -> WalkForwardResult` + (new) — the per-window re-fit driver over the loaded blueprint. +- `blueprint_sweep_over(doc, axes, from, to, data)` (new) — windowed IS-sweep with + lattice (twin of `stage1_r_sweep_over` + `blueprint_sweep_family`). +- `run_oos_blueprint(doc, params, space, from, to, pip, topo, data)` (new) — + windowed OOS run over the loaded-member path. +- `run_blueprint_walkforward(doc, axes, name, persist, data, select)` (new) — IO + wrapper: `put_blueprint` + `append_family(FamilyKind::WalkForward)` + print + members + `walkforward_summary_json`. +- `WfBlueprintArgs` + `parse_walkforward_blueprint_args` (new) — `--axis` + (≥1 required) + `--select` + `--name|--trace`; rejects a no-axis invocation. +- `["walkforward", ..]` dispatch (modified) — `.json` discriminator. +- `reproduce_family_in` (modified) — `WalkForward` realization branch. + +## Data flow + +`aura walkforward --axis …` → boundary read + `blueprint_from_json` +validate → `parse_walkforward_blueprint_args` → `blueprint_walkforward_family`: +`WindowRoller` rolls 3 windows → per window `blueprint_sweep_over(is)` re-fits the +grid (each member a windowed loaded-blueprint run) → `select_winner` picks by +`sqn_normalized` → `run_oos_blueprint(oos)` runs the winner → `WindowRun` → +`WalkForwardResult` → `put_blueprint(topo)` + `append_family(WalkForward)` → print +per-window OOS members + summary. `aura reproduce ` → reload blueprint by +`topology_hash` → per member reconstruct `windowed_sources(manifest.window)` + +`manifest.params` → `run_blueprint_member` → assert bit-identical. + +## Error handling + +- **OPEN blueprint with no `--axis`** → usage error (exit 2): a walk-forward with + nothing to re-fit is meaningless; mirror the loaded-blueprint sweep's "≥1 axis" + contract. Message names that ≥1 `--axis` is required. +- **Unknown / kind-mismatched `--axis` name** → the sweep terminal's + `UnknownKnob`/kind error, rendered `aura: …` + exit 2 (reused from the + loaded-blueprint sweep; the axis names come from `--list-axes`). +- **Malformed / unreadable blueprint** → rejected at the dispatch boundary + (`blueprint_from_json`, exit 2, `UnknownNodeType`), before any run — pinned by + an E2E (as cycle 0096 did), so the internal `.expect` reloads are unreachable. +- **`--select plateau` without a lattice** → `select_winner` already returns the + named refusal; the loaded IS-sweep uses `sweep_with_lattice`, so a grid sweep + always has a lattice (the refusal is unreachable here but preserved). +- **Window too short for one IS+OOS span** → `WindowRoller::new` error, exit 2 + (reused). +- No panics on any user-reachable input; `walk_forward` runs windows disjointly + in parallel, deterministic in roll order (C1). + +## Testing strategy + +- **Unit — `blueprint_walkforward_family`** (`cargo test -p aura-cli --bin aura + blueprint_walkforward_family`): over `stage1_signal_open.json` with a 2×2 + `--axis` grid → a `WalkForwardResult` with 3 windows, each `chosen_params` of + arity 2 (both axes re-fit), and each window's OOS `RunReport` carrying R-metrics + (`metrics.r`, incl. `sqn_normalized`). The stitched OOS pip-equity curve is + **empty** (reduce-mode retains no raw pip curve — the empty segments leave the + curve unbroken), so the per-window R-metrics, not a pip curve, are the OOS + record. Pins the per-window re-fit shape. +- **Unit — `parse_walkforward_blueprint_args`** (`cargo test -p aura-cli --bin + aura parse_walkforward_blueprint_args`): `--axis a=1,2 --select plateau:mean` + parses; a no-`--axis` tail → `Err`; `--select bogus` → `Err`. +- **E2E — the walk-forward runs + persists** (`cargo test -p aura-cli --test + cli_run `): `aura walkforward --axis + stage1_signal.fast.length=2,3 --axis stage1_signal.slow.length=4,6 --name wfx` + → exit 0, 3 `walkforward-*` OOS member lines, a `{"walkforward":…}` summary, + and exactly one `runs/blueprints/.json`. +- **E2E — reproduce bit-identical** (`--test cli_run`): after the above, `aura + reproduce wfx-0` → "reproduced 3/3 members bit-identically", exit 0. +- **E2E — no-axis rejection** (`--test cli_run`): `aura walkforward ` + (no `--axis`) → exit 2 + "requires >= 1 --axis" on stderr. +- **E2E — malformed-blueprint safety** (`--test cli_run`): `aura walkforward + crates/aura-cli/tests/fixtures/unknown_node.json --axis a=1,2` → exit 2 + + `UnknownNodeType` on stderr + no `panicked`. + +## Acceptance criteria + +1. `aura walkforward --axis = [--axis …]` re-optimizes + the blueprint's params on each IS window and runs the winner OOS, aggregating + to a `FamilyKind::WalkForward` family (per-window OOS member lines + a + `{"walkforward":…}` summary), exit 0. +2. The family is content-addressed (one `runs/blueprints/.json`) + and `aura reproduce`s every OOS member bit-identically. +3. An OPEN blueprint with no `--axis` is a usage error (exit 2), as is an unknown + axis name; a malformed blueprint is rejected at the boundary (exit 2, no panic). +4. `--select argmax` (default) and `--select plateau:mean|plateau:worst` both work + (the IS-sweep carries a lattice). +5. No look-ahead: IS strictly precedes OOS per window; runs are deterministic + (C1), parallelism across windows only. +6. No `aura-engine` / `aura-registry` change; invariants 1, 2, 8, 9 preserved. +7. `cargo build --workspace`, `cargo test --workspace`, `cargo clippy --workspace + --all-targets -- -D warnings` all green.