spec: 0097 IS-refit walk-forward over a loaded blueprint (boss-signed)

Add `aura walkforward <blueprint.json> --axis <name>=<csv> [--axis …]`: a true in-sample-refit walk-forward over a loaded blueprint (Arm A, #173). Per IS window it re-optimizes the blueprint's params over the user's --axis grid, selects the winner by sqn_normalized, runs it out-of-sample, and aggregates the per-window OOS reports into a FamilyKind::WalkForward family that is content-addressed (put_blueprint) and aura-reproduce-aware. Closes the World/C21 milestone's last piece (the walk-forward half of #170). Design (Arm A settled by the user; sub-forks derived and recorded on #173): the loaded-blueprint WF mirrors the hard-wired stage1-r walk-forward arm one substitution deep — the generic engine walk_forward driver + select_winner + the 24/12/12 synthetic schedule are reused verbatim; only the per-window IS-sweep and OOS-run source the loaded blueprint (wrap_stage1r via cycle-0096's blueprint_axis_probe) over the user --axis grid. Persist + reproduce ship together (coupled — a persisted family with no store hook is unreproducible, the #170 hazard); the reproduce WalkForward branch reconstructs each OOS window's windowed source + winner params from the member manifest, mirroring the 0095 MonteCarlo branch. Loaded members are reduce-mode (R-metrics, no raw pip curve), so the WF is measured in R (oos_r), stitched pip-equity empty — consistent with the C10 R-first yardstick. Boss-signed on grounding-check PASS (independent, fresh-context; all 8 load-bearing current-behaviour assumptions ratified by named green tests — incl. an empirically-confirmed one: a loaded-blueprint member carries and ranks by sqn_normalized. One spec-internal tension flagged + fixed before sign). Engine/registry untouched (invariants 1/2/8/9). refs #173
2026-07-01 15:16:35 +02:00
parent 485f3c8c89
commit 1e2ac82473
1 changed files with 314 additions and 0 deletions
@@ -0,0 +1,314 @@
+# In-sample-refit walk-forward over a loaded blueprint — Design Spec
+
+**Date:** 2026-07-01
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+## Goal
+
+Ship the World/C21 milestone's last piece: a **true in-sample-refit walk-forward
+over a loaded blueprint**. `aura walkforward <blueprint.json> --axis <name>=<csv>
+[--axis …]` rolls IS/OOS windows over a synthetic span; on each IS window it
+**re-optimizes** the blueprint's params over the user's `--axis` grid, runs the
+winner **out-of-sample**, and aggregates the per-window OOS reports into a
+`FamilyKind::WalkForward` family that is content-addressed and `aura reproduce`s
+bit-identically. This is Arm A (the user's decision, #173) — the honest OOS
+validation the milestone exists to enable, and the form that legitimately carries
+the "walk-forward" name (matching the hard-wired verb's IS-optimizing guarantee).
+
+## Architecture
+
+The hard-wired `aura walkforward --strategy stage1-r` ALREADY does IS-refit
+(`walkforward_family`'s `Stage1R` arm, `crates/aura-cli/src/main.rs`): for each
+window it `stage1_r_sweep_over(is) → select_winner("sqn_normalized") →
+run_oos_r(winner, oos)`. The loaded-blueprint form is the **same shape with one
+substitution** — the per-window IS-sweep and OOS-run source the **loaded
+blueprint** (wrapped by `wrap_stage1r` via cycle-0096's `blueprint_axis_probe`)
+instead of the built-in strategy. Everything else is reused verbatim: the generic
+engine driver `walk_forward(roller, space, run_window)` (strategy-agnostic,
+`crates/aura-engine/src/walkforward.rs`), `select_winner` (Argmax-deflated /
+Plateau), the `24/12/12`-over-60-bar synthetic schedule (`wf_full_span` /
+`wf_window_sizes`), and `FamilyKind::WalkForward` + `walkforward_member_reports` +
+`walkforward_summary_json`.
+
+Reused windowing seam: `data.windowed_sources(from, to)` (the same seam
+`stage1_r_sweep_over` and `run_oos_r` use). No look-ahead: IS strictly precedes
+OOS in every window (`WindowRoller`), and the loaded-member path (`run_blueprint_member`)
+already only sees its window's stream (invariants 1/2).
+
+Persistence + reproduce are shipped **together** (they are coupled — a persisted
+family with no store hook reports members as "blueprint missing", the #170
+hazard). The family is stored once by the shared `topology_hash`
+(`put_blueprint`, the C18 hook the sweep/mc use) + `append_family`; `aura
+reproduce` gains a `WalkForward` branch that reconstructs each OOS member's
+windowed source (from its manifest window) and re-runs it with the member's
+stored winner params — mirroring the `MonteCarlo` reproduce branch from cycle
+0095. One signal topology → one shared `topology_hash` across all windows; each
+member manifest carries `(topology_hash, window, chosen params, selection)`.
+
+Scope: `crates/aura-cli` only (+ read-only reuse of the engine/registry WF
+machinery). No engine/registry change — invariants 8/9 hold.
+
+## Concrete code shapes
+
+### Worked user program (the acceptance evidence)
+
+A researcher has an OPEN strategy blueprint and wants an honest walk-forward that
+re-fits its two SMA lengths each window (the axis names come from `--list-axes`,
+cycle 0096):
+
+```console
+$ aura sweep crates/aura-cli/tests/fixtures/stage1_signal_open.json --list-axes
+stage1_signal.fast.length:I64
+stage1_signal.slow.length:I64
+
+$ aura walkforward crates/aura-cli/tests/fixtures/stage1_signal_open.json \
+    --axis stage1_signal.fast.length=2,3 --axis stage1_signal.slow.length=4,6 --name wf1
+walkforward-0  <oos window 0 metric line>
+walkforward-1  <oos window 1 metric line>
+walkforward-2  <oos window 2 metric line>
+{"walkforward":{"windows":3,"stitched_total_pips":<f>,"param_stability":<…>,"oos_r":{…}}}
+
+$ # content-addressed: the family reproduces bit-identically
+$ aura reproduce walkforward-0
+reproduced 3/3 members bit-identically
+```
+
+Per window the IS grid (2×2 here) is re-optimized by `sqn_normalized`; the winner
+runs OOS; the three OOS reports are the family members (line shape identical to
+the hard-wired `aura walkforward`, which reuses `family_member_line`). Exact
+metric numbers are run-determined; the *shape* — three re-fit windows, OOS
+members, a `{"walkforward":…}` summary, and a bit-identical reproduce — is the
+criterion.
+
+An OPEN blueprint with **no** `--axis` (nothing to re-fit) is a usage error:
+
+```console
+$ aura walkforward crates/aura-cli/tests/fixtures/stage1_signal_open.json
+aura: walkforward <blueprint.json> requires >= 1 --axis to re-fit per window
+$ echo $?
+2
+```
+
+### Implementation shape (secondary)
+
+**The family builder** — the loaded-blueprint analog of `walkforward_family`'s
+`Stage1R` arm; reuses the generic `walk_forward` driver, substituting the loaded
+blueprint into the per-window closure:
+
+```rust
+fn blueprint_walkforward_family(
+    doc: &str, axes: &[(String, Vec<Scalar>)], data: &DataSource, select: Selection,
+) -> WalkForwardResult {
+    let span = data.wf_full_span();
+    let (is_len, oos_len, step) = data.wf_window_sizes();
+    let roller = WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling)
+        .unwrap_or_else(|e| { eprintln!("aura: walk-forward window too short: {e:?}"); std::process::exit(2); });
+    let space = blueprint_axis_probe(doc).param_space();      // the wrapped namespace (cycle 0096)
+    let pip = data.pip_size();
+    let topo = topology_hash(&blueprint_from_json(doc, &|t| std_vocabulary(t))
+        .expect("doc parse-validated at the dispatch boundary"));
+    walk_forward(roller, space.clone(), |w: WindowBounds| {
+        // IS: re-fit the loaded blueprint over the user grid, windowed to [is]
+        let (is_family, lattice) = blueprint_sweep_over(doc, axes, w.is.0, w.is.1, data);
+        let (best, selection) = select_winner(&is_family, "sqn_normalized", select, lattice.as_deref())
+            .unwrap_or_else(|msg| { eprintln!("aura: {msg}"); std::process::exit(2); });
+        // OOS: run the winner over [oos] via the loaded-member path
+        let (oos_equity, mut oos_report) =
+            run_oos_blueprint(doc, &best.params, &space, w.oos.0, w.oos.1, pip, &topo, data);
+        oos_report.manifest.selection = Some(selection);
+        WindowRun { chosen_params: best.params, oos_equity, oos_report }
+    })
+}
+```
+
+**The windowed IS-sweep** — `blueprint_sweep_over`: the structural twin of
+`stage1_r_sweep_over` (uses `sweep_with_lattice` so `--select plateau` works) and
+of `blueprint_sweep_family` (loaded blueprint, `--axis` grid), but windowed to
+`[from,to]` via `data.windowed_sources`:
+
+```rust
+fn blueprint_sweep_over(doc: &str, axes: &[(String, Vec<Scalar>)],
+                        from: Timestamp, to: Timestamp, data: &DataSource)
+    -> (SweepFamily, Option<Vec<usize>>) {
+    let pip = data.pip_size();
+    let topo = topology_hash(&reload(doc));
+    let probe = blueprint_axis_probe(doc);
+    let space = probe.param_space();
+    // seed the user axes verbatim (same as blueprint_sweep_family), then sweep WITH lattice
+    let mut binder = /* probe.axis(first) then SweepBinder::axis(rest) */;
+    binder.sweep_with_lattice(|point| {
+        let sources = data.windowed_sources(from, to);
+        let window = window_of(&sources).expect("non-empty in-sample window");
+        run_blueprint_member(reload(doc), point, &space, sources, window, 0, pip, &topo)
+    }).map(|(fam, lat)| (fam, Some(lat)))
+      .expect("the user axes are name/kind-checked at the sweep terminal")
+}
+```
+
+**The windowed OOS-run** — `run_oos_blueprint`: the loaded-member analog of
+`run_oos_r`, running the winner params over the OOS window. The loaded-member
+path (`run_blueprint_member`) is **reduce-mode**: it folds to R-metrics
+(`summarize_r`) and retains the per-trade R series, but NOT the raw per-cycle pip
+curve the hard-wired non-reduce `run_oos_r` returns. So `oos_equity` is **empty**
+— the loaded walk-forward is measured in **R** (the C10-reworked yardstick), and
+the meaningful per-window record is the OOS `RunReport`'s R-metrics:
+
+```rust
+fn run_oos_blueprint(doc: &str, params: &[Cell], space: &[ParamSpec],
+                     from: Timestamp, to: Timestamp, pip: f64, topo: &str, data: &DataSource)
+    -> (Vec<(Timestamp, f64)>, RunReport) {
+    let sources = data.windowed_sources(from, to);
+    let window = window_of(&sources).expect("non-empty out-of-sample window");
+    let report = run_blueprint_member(reload(doc), params, space, sources, window, 0, pip, topo);
+    // reduce-mode: no raw pip curve retained -> empty stitching segment. The OOS
+    // record is R-metrics (report.metrics.r); an empty segment leaves the stitched
+    // curve unbroken (stitch treats it as a 0-offset, no-point contribution).
+    (Vec::new(), report)
+}
+```
+
+Consequence for the summary: `walkforward_summary_json`'s `stitched_total_pips`
+is `0.0` (no retained pip curve — the loaded path is R-measured, not pip-measured),
+and its `oos_r` block (pooled per-trade Rs across the OOS windows) is the
+meaningful line. This matches the R-first world of the C10 rework; a raw pip-equity
+curve would need a non-reduce OOS run (`--trace`-style), out of scope here.
+
+**The IO wrapper + dispatch** — `run_blueprint_walkforward` mirrors
+`run_walkforward` (persist via `put_blueprint` + `append_family`, print members +
+summary); the `["walkforward", rest @ ..]` dispatch gains the `.json`
+discriminator BEFORE `parse_walkforward_args`, mirroring the sweep/mc arms:
+
+```rust
+["walkforward", rest @ ..] => {
+    if let Some(path) = rest.first().filter(|a| a.ends_with(".json") && Path::new(a).is_file()) {
+        let doc = std::fs::read_to_string(path).unwrap_or_else(|e| { eprintln!("aura: {path}: {e}"); exit(2) });
+        if let Err(e) = blueprint_from_json(&doc, &|t| std_vocabulary(t)) {
+            eprintln!("aura: {path}: {e:?}"); exit(2);
+        }
+        let WfBlueprintArgs { axes, name, persist, select } =
+            parse_walkforward_blueprint_args(&rest[1..]).unwrap_or_else(|m| { eprintln!("aura: {m}"); exit(2) });
+        run_blueprint_walkforward(&doc, &axes, &name, persist, DataSource::Synthetic, select);
+        return;
+    }
+    match parse_walkforward_args(rest) { /* unchanged hard-wired path */ }
+}
+```
+
+**The reproduce branch** — `reproduce_family_in` already re-runs each member via
+`run_blueprint_member` with a per-member `(sources, window, params)` triple chosen
+by `family.kind` (cycle 0094 recovers the member's params from its manifest
+against the reloaded space; cycle 0095 added the `MonteCarlo` arm that rebuilds
+the seed-driven walk). The new `WalkForward` arm keeps the **same per-member param
+recovery** (the winner params are stored in each OOS member's manifest, exactly as
+a sweep member's are) and differs only in the **source**: the OOS window's slice,
+rebuilt from the member manifest's stored window bounds:
+
+```rust
+match family.kind {
+    FamilyKind::MonteCarlo => { /* cycle 0095: manifest.seed -> synthetic walk */ }
+    FamilyKind::WalkForward => {
+        // the OOS slice from the member manifest's window; params via the SAME
+        // manifest->cells recovery the Sweep arm already uses (winner params).
+        let (from, to) = stored.manifest.window;
+        let sources = data.windowed_sources(from, to);
+        (sources, window_of(&sources)/* the member window */)
+    }
+    _ => (data.run_sources(), data.full_window()),   // Sweep: the showcase series
+}
+```
+
+The member's params are recovered by the existing shared path (not re-invented
+here); only the `(sources, window)` pair is realization-specific — `WalkForward`
+supplies the windowed OOS slice, exactly as `MonteCarlo` supplies the seeded walk.
+
+## Components
+
+- `blueprint_walkforward_family(doc, axes, data, select) -> WalkForwardResult`
+  (new) — the per-window re-fit driver over the loaded blueprint.
+- `blueprint_sweep_over(doc, axes, from, to, data)` (new) — windowed IS-sweep with
+  lattice (twin of `stage1_r_sweep_over` + `blueprint_sweep_family`).
+- `run_oos_blueprint(doc, params, space, from, to, pip, topo, data)` (new) —
+  windowed OOS run over the loaded-member path.
+- `run_blueprint_walkforward(doc, axes, name, persist, data, select)` (new) — IO
+  wrapper: `put_blueprint` + `append_family(FamilyKind::WalkForward)` + print
+  members + `walkforward_summary_json`.
+- `WfBlueprintArgs` + `parse_walkforward_blueprint_args` (new) — `--axis`
+  (≥1 required) + `--select` + `--name|--trace`; rejects a no-axis invocation.
+- `["walkforward", ..]` dispatch (modified) — `.json` discriminator.
+- `reproduce_family_in` (modified) — `WalkForward` realization branch.
+
+## Data flow
+
+`aura walkforward <bp.json> --axis …` → boundary read + `blueprint_from_json`
+validate → `parse_walkforward_blueprint_args` → `blueprint_walkforward_family`:
+`WindowRoller` rolls 3 windows → per window `blueprint_sweep_over(is)` re-fits the
+grid (each member a windowed loaded-blueprint run) → `select_winner` picks by
+`sqn_normalized` → `run_oos_blueprint(oos)` runs the winner → `WindowRun` →
+`WalkForwardResult` → `put_blueprint(topo)` + `append_family(WalkForward)` → print
+per-window OOS members + summary. `aura reproduce <id>` → reload blueprint by
+`topology_hash` → per member reconstruct `windowed_sources(manifest.window)` +
+`manifest.params` → `run_blueprint_member` → assert bit-identical.
+
+## Error handling
+
+- **OPEN blueprint with no `--axis`** → usage error (exit 2): a walk-forward with
+  nothing to re-fit is meaningless; mirror the loaded-blueprint sweep's "≥1 axis"
+  contract. Message names that ≥1 `--axis` is required.
+- **Unknown / kind-mismatched `--axis` name** → the sweep terminal's
+  `UnknownKnob`/kind error, rendered `aura: …` + exit 2 (reused from the
+  loaded-blueprint sweep; the axis names come from `--list-axes`).
+- **Malformed / unreadable blueprint** → rejected at the dispatch boundary
+  (`blueprint_from_json`, exit 2, `UnknownNodeType`), before any run — pinned by
+  an E2E (as cycle 0096 did), so the internal `.expect` reloads are unreachable.
+- **`--select plateau` without a lattice** → `select_winner` already returns the
+  named refusal; the loaded IS-sweep uses `sweep_with_lattice`, so a grid sweep
+  always has a lattice (the refusal is unreachable here but preserved).
+- **Window too short for one IS+OOS span** → `WindowRoller::new` error, exit 2
+  (reused).
+- No panics on any user-reachable input; `walk_forward` runs windows disjointly
+  in parallel, deterministic in roll order (C1).
+
+## Testing strategy
+
+- **Unit — `blueprint_walkforward_family`** (`cargo test -p aura-cli --bin aura
+  blueprint_walkforward_family`): over `stage1_signal_open.json` with a 2×2
+  `--axis` grid → a `WalkForwardResult` with 3 windows, each `chosen_params` of
+  arity 2 (both axes re-fit), and each window's OOS `RunReport` carrying R-metrics
+  (`metrics.r`, incl. `sqn_normalized`). The stitched OOS pip-equity curve is
+  **empty** (reduce-mode retains no raw pip curve — the empty segments leave the
+  curve unbroken), so the per-window R-metrics, not a pip curve, are the OOS
+  record. Pins the per-window re-fit shape.
+- **Unit — `parse_walkforward_blueprint_args`** (`cargo test -p aura-cli --bin
+  aura parse_walkforward_blueprint_args`): `--axis a=1,2 --select plateau:mean`
+  parses; a no-`--axis` tail → `Err`; `--select bogus` → `Err`.
+- **E2E — the walk-forward runs + persists** (`cargo test -p aura-cli --test
+  cli_run <filter>`): `aura walkforward <open fixture> --axis
+  stage1_signal.fast.length=2,3 --axis stage1_signal.slow.length=4,6 --name wfx`
+  → exit 0, 3 `walkforward-*` OOS member lines, a `{"walkforward":…}` summary,
+  and exactly one `runs/blueprints/<hash>.json`.
+- **E2E — reproduce bit-identical** (`--test cli_run`): after the above, `aura
+  reproduce wfx-0` → "reproduced 3/3 members bit-identically", exit 0.
+- **E2E — no-axis rejection** (`--test cli_run`): `aura walkforward <open fixture>`
+  (no `--axis`) → exit 2 + "requires >= 1 --axis" on stderr.
+- **E2E — malformed-blueprint safety** (`--test cli_run`): `aura walkforward
+  crates/aura-cli/tests/fixtures/unknown_node.json --axis a=1,2` → exit 2 +
+  `UnknownNodeType` on stderr + no `panicked`.
+
+## Acceptance criteria
+
+1. `aura walkforward <blueprint.json> --axis <name>=<csv> [--axis …]` re-optimizes
+   the blueprint's params on each IS window and runs the winner OOS, aggregating
+   to a `FamilyKind::WalkForward` family (per-window OOS member lines + a
+   `{"walkforward":…}` summary), exit 0.
+2. The family is content-addressed (one `runs/blueprints/<topology_hash>.json`)
+   and `aura reproduce`s every OOS member bit-identically.
+3. An OPEN blueprint with no `--axis` is a usage error (exit 2), as is an unknown
+   axis name; a malformed blueprint is rejected at the boundary (exit 2, no panic).
+4. `--select argmax` (default) and `--select plateau:mean|plateau:worst` both work
+   (the IS-sweep carries a lattice).
+5. No look-ahead: IS strictly precedes OOS per window; runs are deterministic
+   (C1), parallelism across windows only.
+6. No `aura-engine` / `aura-registry` change; invariants 1, 2, 8, 9 preserved.
+7. `cargo build --workspace`, `cargo test --workspace`, `cargo clippy --workspace
+   --all-targets -- -D warnings` all green.