plan: 0097 IS-refit walk-forward over a loaded blueprint
Four tasks, one iteration: (1) the windowed IS-sweep + OOS-run helpers + blueprint_walkforward_family (mirrors the hard-wired Stage1R walkforward arm one substitution deep — loaded blueprint via blueprint_axis_probe); (2) the --axis/--select parse grammar + IO wrapper (persist via put_blueprint + append_family, content-addressed like aura mc) + the .json walkforward dispatch arm; (3) the reproduce_family_in WalkForward branch (persist+reproduce land together); (4) the E2Es (persist shape, reproduce bit-identical, no-axis reject, malformed-safety) + the full clippy/suite gate. Panic-safety: blueprint_sweep_over returns Result<_, BindError> (user --axis fallible); a bad axis → clean in-closure exit 2 (mirrors the hard-wired arm's select_winner handling), never a panic. reduce-mode members carry R-metrics (no raw pip curve) → empty stitched equity, oos_r is the meaningful summary (C10 R-first). Closes the World/C21 milestone's last piece. refs #173
This commit is contained in:
@@ -0,0 +1,572 @@
|
||||
# IS-refit walk-forward over a loaded blueprint — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0097-is-refit-walk-forward-over-a-loaded-blueprint.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to
|
||||
> run this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Add `aura walkforward <blueprint.json> --axis <name>=<csv> [--axis …]` —
|
||||
an in-sample-refit walk-forward over a loaded blueprint (re-optimize per IS
|
||||
window, run the winner OOS), aggregated to a persisted, content-addressed,
|
||||
reproduce-aware `FamilyKind::WalkForward` family.
|
||||
|
||||
**Architecture:** Mirror `walkforward_family`'s `Stage1R` arm one substitution
|
||||
deep — reuse the generic engine `walk_forward` driver + `select_winner` +
|
||||
the 24/12/12 synthetic schedule; the per-window IS-sweep and OOS-run source the
|
||||
LOADED blueprint (via cycle-0096's `blueprint_axis_probe`) over the user `--axis`
|
||||
grid, windowed by `data.windowed_sources`. Persist via `put_blueprint` +
|
||||
`append_family` (like the loaded sweep/mc), and add the `reproduce_family_in`
|
||||
`WalkForward` branch so the family reproduces bit-identically. All changes in
|
||||
`crates/aura-cli`; engine/registry are read-only callees (invariants 1/2/8/9).
|
||||
|
||||
**Tech Stack:** `crates/aura-cli/src/main.rs` (six new fns + two modified sites),
|
||||
`crates/aura-cli/tests/cli_run.rs` (E2Es). Read-only reuse: `walk_forward` /
|
||||
`WindowRun` / `WalkForwardResult` (aura-engine), `FamilyKind::WalkForward` /
|
||||
`walkforward_member_reports` / `put_blueprint` / `append_family` (aura-registry).
|
||||
|
||||
---
|
||||
|
||||
**Files this plan creates or modifies:**
|
||||
|
||||
- Modify: `crates/aura-cli/src/main.rs` — new `blueprint_sweep_over`,
|
||||
`run_oos_blueprint`, `blueprint_walkforward_family`, `WfBlueprintArgs`,
|
||||
`parse_walkforward_blueprint_args`, `run_blueprint_walkforward`; modified
|
||||
`["walkforward", ..]` dispatch (~4175) + `reproduce_family_in` match (~2566).
|
||||
- Test: `crates/aura-cli/src/main.rs` (`#[cfg(test)] mod tests` ~4238) — unit
|
||||
`blueprint_walkforward_family` + unit `parse_walkforward_blueprint_args`.
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — 4 new E2Es beside the blueprint-family
|
||||
block (~3269-3700).
|
||||
|
||||
Line numbers are current-tree recon anchors; match on surrounding code, not the
|
||||
number. **Gate note:** Tasks 1-3 use `cargo build` + scoped `cargo test` (early
|
||||
tasks add not-yet-wired fns, which are a build *warning*, not an error); the full
|
||||
`cargo clippy -- -D warnings` + `cargo test --workspace` gate runs in Task 4,
|
||||
after every fn is wired.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: The family builder + its two windowed helpers
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` (new fns near `blueprint_sweep_family` ~3197 and `stage1_r_sweep_over` ~1329)
|
||||
- Test: `crates/aura-cli/src/main.rs` (`mod tests`)
|
||||
|
||||
- [ ] **Step 1: Write the failing unit test**
|
||||
|
||||
Add to the `#[cfg(test)] mod tests` block (near `blueprint_axis_probe_lists_prefixed_open_knobs`):
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn blueprint_walkforward_family_refits_each_window() {
|
||||
// The open fixture's two SMA lengths are re-fit per IS window over a 2x2 grid.
|
||||
let doc = include_str!("../tests/fixtures/stage1_signal_open.json");
|
||||
let axes = vec![
|
||||
("stage1_signal.fast.length".to_string(), vec![Scalar::i64(2), Scalar::i64(3)]),
|
||||
("stage1_signal.slow.length".to_string(), vec![Scalar::i64(4), Scalar::i64(6)]),
|
||||
];
|
||||
let result = blueprint_walkforward_family(doc, &axes, &DataSource::Synthetic, Selection::Argmax);
|
||||
// 24/12/12 over the 60-bar synthetic span -> 3 rolling windows.
|
||||
assert_eq!(result.windows.len(), 3, "three rolling IS/OOS windows");
|
||||
for w in &result.windows {
|
||||
assert_eq!(w.run.chosen_params.len(), 2, "both axes re-fit each window");
|
||||
assert!(w.run.oos_report.metrics.r.is_some(), "OOS record is R-metrics");
|
||||
}
|
||||
// reduce-mode retains no raw pip curve -> the stitched pip-equity is empty.
|
||||
assert!(result.stitched_oos_equity.is_empty(), "no raw pip curve in reduce-mode");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test to verify it fails (RED — unresolved name)**
|
||||
|
||||
Run: `cargo test -p aura-cli --bin aura blueprint_walkforward_family`
|
||||
Expected: FAIL to compile — `cannot find function 'blueprint_walkforward_family'`.
|
||||
|
||||
- [ ] **Step 3: Add `blueprint_sweep_over` (windowed IS-sweep, with lattice)**
|
||||
|
||||
Insert near `blueprint_sweep_family` (~3232). Twin of `stage1_r_sweep_over`
|
||||
(uses `sweep_with_lattice`) crossed with `blueprint_sweep_family`'s loaded-member
|
||||
seed-axes pattern, windowed by `[from,to]`. Returns `Result<_, BindError>` (user
|
||||
axes are fallible — an unknown/kind-mismatched `--axis` returns `BindError`, not a
|
||||
panic; the sweep terminal name/kind-checks):
|
||||
|
||||
```rust
|
||||
/// Sweep the LOADED blueprint over the user `--axis` grid on an in-sample window
|
||||
/// `[from,to]` — the windowed, lattice-carrying twin of `stage1_r_sweep_over` and
|
||||
/// `blueprint_sweep_family`. `sweep_with_lattice` gives the grid lattice `--select
|
||||
/// plateau` needs. An unknown/kind-mismatched axis surfaces as `BindError` at the
|
||||
/// sweep terminal (no panic, no hidden exit) for the caller to render.
|
||||
fn blueprint_sweep_over(
|
||||
doc: &str, axes: &[(String, Vec<Scalar>)], from: Timestamp, to: Timestamp, data: &DataSource,
|
||||
) -> Result<(SweepFamily, Vec<usize>), BindError> {
|
||||
let reload = |d: &str| {
|
||||
blueprint_from_json(d, &|t| std_vocabulary(t))
|
||||
.expect("doc parse-validated at the dispatch boundary; reload is infallible")
|
||||
};
|
||||
let pip = data.pip_size();
|
||||
let topo = topology_hash(&reload(doc));
|
||||
let probe = blueprint_axis_probe(doc);
|
||||
let space = probe.param_space();
|
||||
let mut iter = axes.iter();
|
||||
let (first_name, first_vals) = iter.next().expect("a blueprint walk-forward declares >= 1 axis");
|
||||
let mut binder = probe.axis(first_name, first_vals.clone());
|
||||
for (n, vals) in iter {
|
||||
binder = binder.axis(n, vals.clone());
|
||||
}
|
||||
binder.sweep_with_lattice(|point| {
|
||||
let sources = data.windowed_sources(from, to);
|
||||
let window = window_of(&sources).expect("non-empty in-sample window");
|
||||
run_blueprint_member(reload(doc), point, &space, sources, window, 0, pip, &topo)
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add `run_oos_blueprint` (windowed OOS-run)**
|
||||
|
||||
Insert next to `blueprint_sweep_over`. Loaded-member analog of `run_oos_r`;
|
||||
reduce-mode retains no raw pip curve, so the pip segment is empty:
|
||||
|
||||
```rust
|
||||
/// Run the winner params over an out-of-sample window `[from,to]` on the loaded
|
||||
/// blueprint — the loaded-member analog of `run_oos_r`. The reduce-mode member
|
||||
/// (`run_blueprint_member`) retains R-metrics, not a raw pip curve, so the stitching
|
||||
/// segment is empty (an empty segment leaves the stitched curve unbroken).
|
||||
fn run_oos_blueprint(
|
||||
doc: &str, params: &[Cell], space: &[ParamSpec], from: Timestamp, to: Timestamp,
|
||||
pip: f64, topo: &str, data: &DataSource,
|
||||
) -> (Vec<(Timestamp, f64)>, RunReport) {
|
||||
let reload = blueprint_from_json(doc, &|t| std_vocabulary(t))
|
||||
.expect("doc parse-validated at the dispatch boundary; reload is infallible");
|
||||
let sources = data.windowed_sources(from, to);
|
||||
let window = window_of(&sources).expect("non-empty out-of-sample window");
|
||||
let report = run_blueprint_member(reload, params, space, sources, window, 0, pip, topo);
|
||||
(Vec::new(), report)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add `blueprint_walkforward_family` (the per-window re-fit driver)**
|
||||
|
||||
Insert next to the two helpers. Structural twin of `walkforward_family`'s
|
||||
`Stage1R` arm — reuses the generic `walk_forward` driver; the per-window closure
|
||||
re-fits the loaded blueprint. In-closure errors `exit(2)` exactly as the Stage1R
|
||||
arm does for `select_winner` (clean process exit, never a panic):
|
||||
|
||||
```rust
|
||||
/// The loaded-blueprint IS-refit walk-forward: per IS window, re-optimize the
|
||||
/// blueprint over the user `--axis` grid, select by `sqn_normalized`, run the
|
||||
/// winner OOS. Structural twin of `walkforward_family`'s `Stage1R` arm — the
|
||||
/// generic `walk_forward` driver + `select_winner` reused; only the per-window
|
||||
/// sweep/OOS source the loaded blueprint. In-closure errors (a bad `--axis`)
|
||||
/// `exit(2)` with the sweep terminal's message, as the hard-wired arm does.
|
||||
fn blueprint_walkforward_family(
|
||||
doc: &str, axes: &[(String, Vec<Scalar>)], data: &DataSource, select: Selection,
|
||||
) -> WalkForwardResult {
|
||||
let span = data.wf_full_span();
|
||||
let (is_len, oos_len, step) = data.wf_window_sizes();
|
||||
let roller = match WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling) {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
eprintln!("aura: walk-forward window too short for one IS+OOS span: {e:?}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
};
|
||||
let space = blueprint_axis_probe(doc).param_space();
|
||||
let pip = data.pip_size();
|
||||
let topo = topology_hash(&blueprint_from_json(doc, &|t| std_vocabulary(t))
|
||||
.expect("doc parse-validated at the dispatch boundary; reload is infallible"));
|
||||
walk_forward(roller, space.clone(), |w: WindowBounds| {
|
||||
let (is_family, lattice) = blueprint_sweep_over(doc, axes, w.is.0, w.is.1, data)
|
||||
.unwrap_or_else(|e| { eprintln!("aura: {e}"); std::process::exit(2); });
|
||||
let (best, selection) = match select_winner(&is_family, "sqn_normalized", select, Some(&lattice)) {
|
||||
Ok(v) => v,
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
};
|
||||
let (oos_equity, mut oos_report) =
|
||||
run_oos_blueprint(doc, &best.params, &space, w.oos.0, w.oos.1, pip, &topo, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
WindowRun { chosen_params: best.params, oos_equity, oos_report }
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Verify the unit test passes + the workspace builds**
|
||||
|
||||
Run: `cargo test -p aura-cli --bin aura blueprint_walkforward_family`
|
||||
Expected: PASS (1 test — 3 windows, both axes re-fit, R-metrics present, empty curve).
|
||||
|
||||
Run: `cargo build --workspace`
|
||||
Expected: `Finished` — 0 errors (unused-fn *warnings* for the not-yet-wired fns
|
||||
are expected until Task 2 wires them; warnings are not errors).
|
||||
|
||||
---
|
||||
|
||||
## Task 2: The parse grammar, IO wrapper, and dispatch (the persisting verb)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` (new `WfBlueprintArgs` + `parse_walkforward_blueprint_args` near `parse_sweep_blueprint_args` ~3939; new `run_blueprint_walkforward` near `run_blueprint_sweep` ~3301; dispatch ~4175)
|
||||
- Test: `crates/aura-cli/src/main.rs` (`mod tests`)
|
||||
|
||||
- [ ] **Step 1: Write the failing unit test**
|
||||
|
||||
Add beside `parse_sweep_blueprint_args_grammar`:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn parse_walkforward_blueprint_args_grammar() {
|
||||
let a = parse_walkforward_blueprint_args(&[
|
||||
"--axis", "stage1_signal.fast.length=2,3", "--select", "plateau:mean", "--name", "wf",
|
||||
]).expect("valid tail parses");
|
||||
assert_eq!(a.axes.len(), 1);
|
||||
assert_eq!(a.name, "wf");
|
||||
assert!(matches!(a.select, Selection::Plateau(_)));
|
||||
// a walk-forward with nothing to re-fit is rejected:
|
||||
assert!(parse_walkforward_blueprint_args(&[]).is_err());
|
||||
assert!(parse_walkforward_blueprint_args(&["--name", "wf"]).is_err());
|
||||
// an unknown --select value is rejected:
|
||||
assert!(parse_walkforward_blueprint_args(&["--axis", "a=1,2", "--select", "bogus"]).is_err());
|
||||
// default name + default argmax:
|
||||
let b = parse_walkforward_blueprint_args(&["--axis", "a=1,2"]).expect("minimal tail parses");
|
||||
assert_eq!(b.name, "walkforward");
|
||||
assert!(matches!(b.select, Selection::Argmax));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test to verify it fails (RED)**
|
||||
|
||||
Run: `cargo test -p aura-cli --bin aura parse_walkforward_blueprint_args`
|
||||
Expected: FAIL to compile — `cannot find function 'parse_walkforward_blueprint_args'`.
|
||||
|
||||
- [ ] **Step 3: Add `WfBlueprintArgs` + `parse_walkforward_blueprint_args`**
|
||||
|
||||
Insert near `parse_sweep_blueprint_args` (~3987). Mirrors the sweep `--axis`
|
||||
grammar + the `--select` parse from `parse_walkforward_args` (`parse_select`
|
||||
~1614); requires ≥1 `--axis`. This is a content-addressed family verb (always
|
||||
persists, like `aura mc`), so it takes `--name` (default "walkforward") but no
|
||||
`--trace`/persist toggle:
|
||||
|
||||
```rust
|
||||
struct WfBlueprintArgs {
|
||||
axes: Vec<(String, Vec<Scalar>)>,
|
||||
name: String,
|
||||
select: Selection,
|
||||
}
|
||||
|
||||
/// Parse the `aura walkforward <blueprint.json> …` tail: repeatable `--axis
|
||||
/// <name>=<csv>` (the per-window re-fit grid, >= 1 required), `--select
|
||||
/// <argmax|plateau:mean|plateau:worst>` (default argmax), `--name <fam>` (default
|
||||
/// "walkforward"). Reduce-mode + content-addressed, so no `--trace`.
|
||||
fn parse_walkforward_blueprint_args(rest: &[&str]) -> Result<WfBlueprintArgs, String> {
|
||||
let usage = || "walkforward <blueprint.json> --axis <name>=<csv> [--axis …] [--select <argmax|plateau:mean|plateau:worst>] [--name <n>]".to_string();
|
||||
let mut axes: Vec<(String, Vec<Scalar>)> = Vec::new();
|
||||
let mut name = "walkforward".to_string();
|
||||
let mut select = Selection::Argmax;
|
||||
let mut tail = rest;
|
||||
while let Some((flag, t)) = tail.split_first() {
|
||||
let (value, t) = t.split_first().ok_or_else(usage)?;
|
||||
match *flag {
|
||||
"--axis" => {
|
||||
let (n, csv) = value.split_once('=').ok_or_else(usage)?;
|
||||
if n.is_empty() || axes.iter().any(|(a, _)| a == n) { return Err(usage()); }
|
||||
let vals = parse_scalar_csv(csv).ok_or_else(usage)?;
|
||||
axes.push((n.to_string(), vals));
|
||||
}
|
||||
"--select" => select = parse_select(value).ok_or_else(usage)?,
|
||||
"--name" => name = (*value).to_string(),
|
||||
_ => return Err(usage()),
|
||||
}
|
||||
tail = t;
|
||||
}
|
||||
if axes.is_empty() {
|
||||
return Err("walkforward <blueprint.json> requires >= 1 --axis to re-fit per window".to_string());
|
||||
}
|
||||
Ok(WfBlueprintArgs { axes, name, select })
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add `run_blueprint_walkforward` (IO wrapper: persist + print)**
|
||||
|
||||
Insert near `run_blueprint_sweep` (~3340). Mirrors `run_blueprint_sweep`'s persist
|
||||
seam (`put_blueprint` + `append_family`) with the WF family/print:
|
||||
|
||||
```rust
|
||||
/// `aura walkforward <blueprint.json> --axis …`: build the loaded-blueprint
|
||||
/// IS-refit walk-forward, store the canonical blueprint ONCE keyed by the shared
|
||||
/// `topology_hash` (the C18 hook, so `aura reproduce` re-derives it), record it as
|
||||
/// a `FamilyKind::WalkForward` family, and print each OOS member line + the summary.
|
||||
/// Mirrors `run_blueprint_sweep` (content-addressed family verb; no ensure_name_free).
|
||||
fn run_blueprint_walkforward(doc: &str, axes: &[(String, Vec<Scalar>)], name: &str, data: DataSource, select: Selection) {
|
||||
let result = blueprint_walkforward_family(doc, axes, &data, select);
|
||||
let reg = default_registry();
|
||||
let topo = result.windows[0]
|
||||
.run
|
||||
.oos_report
|
||||
.manifest
|
||||
.topology_hash
|
||||
.clone()
|
||||
.expect("a blueprint walk-forward stamps every member's topology_hash");
|
||||
let canonical = blueprint_to_json(
|
||||
&blueprint_from_json(doc, &|t| std_vocabulary(t)).expect("doc parse-validated at the dispatch boundary"),
|
||||
)
|
||||
.expect("a loaded blueprint re-serializes");
|
||||
reg.put_blueprint(&topo, &canonical).unwrap_or_else(|e| { eprintln!("aura: {e}"); std::process::exit(2); });
|
||||
let id = match reg.append_family(name, FamilyKind::WalkForward, &walkforward_member_reports(&result)) {
|
||||
Ok(id) => id,
|
||||
Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
|
||||
};
|
||||
for w in &result.windows {
|
||||
println!("{}", family_member_line(&id, &w.run.oos_report));
|
||||
}
|
||||
println!("{}", walkforward_summary_json(&result));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add the `.json` discriminator to the `["walkforward", ..]` dispatch**
|
||||
|
||||
Modify the dispatch arm (~4175). Add the `.json`-first branch BEFORE the existing
|
||||
`match parse_walkforward_args(rest)`, mirroring the `["sweep", ..]` arm (~4135-4164):
|
||||
|
||||
```rust
|
||||
["walkforward", rest @ ..] => {
|
||||
if let Some(path) =
|
||||
rest.first().filter(|a| a.ends_with(".json") && std::path::Path::new(a).is_file())
|
||||
{
|
||||
let doc = std::fs::read_to_string(path).unwrap_or_else(|e| {
|
||||
eprintln!("aura: {path}: {e}");
|
||||
std::process::exit(2);
|
||||
});
|
||||
if let Err(e) = blueprint_from_json(&doc, &|t| std_vocabulary(t)) {
|
||||
eprintln!("aura: {path}: {e:?}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
let WfBlueprintArgs { axes, name, select } =
|
||||
parse_walkforward_blueprint_args(&rest[1..]).unwrap_or_else(|msg| {
|
||||
eprintln!("aura: {msg}");
|
||||
std::process::exit(2);
|
||||
});
|
||||
run_blueprint_walkforward(&doc, &axes, &name, DataSource::Synthetic, select);
|
||||
return;
|
||||
}
|
||||
match parse_walkforward_args(rest) {
|
||||
Ok((strategy, name, persist, choice, grid, select)) => {
|
||||
run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid, select)
|
||||
}
|
||||
Err(msg) => {
|
||||
eprintln!("aura: {msg}");
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Confirm the existing hard-wired `parse_walkforward_args` arm is preserved verbatim
|
||||
inside the `else`; match its exact current tuple/body when transcribing.)
|
||||
|
||||
- [ ] **Step 6: Verify the unit test passes + the workspace builds**
|
||||
|
||||
Run: `cargo test -p aura-cli --bin aura parse_walkforward_blueprint_args`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo build --workspace`
|
||||
Expected: `Finished` — 0 errors (the verb now wires blueprint_walkforward_family →
|
||||
no more unused-fn warnings for it or the helpers).
|
||||
|
||||
---
|
||||
|
||||
## Task 3: The reproduce WalkForward branch
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` (`reproduce_family_in` match ~2566)
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs`
|
||||
|
||||
- [ ] **Step 1: Write the failing E2E**
|
||||
|
||||
Add to `crates/aura-cli/tests/cli_run.rs` (beside the mc-reproduce template
|
||||
`aura_mc_over_a_blueprint_reproduces_bit_identically` ~3495):
|
||||
|
||||
```rust
|
||||
/// E2E (#173): an IS-refit walk-forward over a loaded blueprint persists a
|
||||
/// content-addressed WalkForward family that `aura reproduce`s bit-identically —
|
||||
/// each OOS member's windowed slice + winner params are recovered from its manifest.
|
||||
#[test]
|
||||
fn aura_walkforward_over_a_blueprint_reproduces_bit_identically() {
|
||||
let cwd = temp_cwd("blueprint-walkforward-reproduce");
|
||||
let bp = format!("{}/tests/fixtures/stage1_signal_open.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let run = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["walkforward", &bp, "--axis", "stage1_signal.fast.length=2,3",
|
||||
"--axis", "stage1_signal.slow.length=4,6", "--name", "wfr"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura walkforward");
|
||||
assert_eq!(run.status.code(), Some(0), "stderr: {}", String::from_utf8_lossy(&run.stderr));
|
||||
let repro = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["reproduce", "wfr-0"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura reproduce");
|
||||
assert_eq!(repro.status.code(), Some(0));
|
||||
let out = String::from_utf8(repro.stdout).expect("utf-8");
|
||||
assert!(out.contains("reproduced 3/3 members bit-identically"), "stdout: {out}");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails (RED — reproduce reconstructs the wrong source)**
|
||||
|
||||
Run: `cargo test -p aura-cli --test cli_run aura_walkforward_over_a_blueprint_reproduces`
|
||||
Expected: FAIL — reproduce falls through the `_` arm (full-window `run_sources`),
|
||||
so the re-run does not match the OOS-windowed original (not bit-identical), or a
|
||||
non-zero exit.
|
||||
|
||||
- [ ] **Step 3: Add the `WalkForward` arm to `reproduce_family_in`**
|
||||
|
||||
In the `match family.kind` block (~2566-2573), add a `FamilyKind::WalkForward` arm
|
||||
before the `_ =>` fallthrough. It supplies only the (sources, window) pair — the
|
||||
winner params are recovered by the SAME shared `point_from_params(&space,
|
||||
&stored.manifest.params)` line the Sweep/MC arms already use:
|
||||
|
||||
```rust
|
||||
FamilyKind::WalkForward => {
|
||||
// each member is one OOS window: rebuild its windowed slice from the
|
||||
// stored window bounds; the winner params come from the shared
|
||||
// manifest->cells recovery below (as a Sweep member's do).
|
||||
let (from, to) = stored.manifest.window;
|
||||
let s = data.windowed_sources(from, to);
|
||||
let w = window_of(&s).expect("non-empty OOS window");
|
||||
(s, w)
|
||||
}
|
||||
```
|
||||
|
||||
(Insert it as a new match arm; keep the existing `FamilyKind::MonteCarlo` arm and
|
||||
the `_ =>` fallthrough. Match the exact binding names the surrounding code uses —
|
||||
transcribe the current arm shape at ~2566 before editing.)
|
||||
|
||||
- [ ] **Step 4: Verify the E2E passes**
|
||||
|
||||
Run: `cargo test -p aura-cli --test cli_run aura_walkforward_over_a_blueprint_reproduces`
|
||||
Expected: PASS — "reproduced 3/3 members bit-identically", exit 0.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: E2E coverage (persist shape, no-axis, malformed) + the full gate
|
||||
|
||||
**Files:**
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs`
|
||||
|
||||
- [ ] **Step 1: Write the three E2Es**
|
||||
|
||||
Add to `crates/aura-cli/tests/cli_run.rs` (beside the sweep-persist template
|
||||
`aura_sweep_loads_a_blueprint_and_persists_a_sweep_family` ~3269):
|
||||
|
||||
```rust
|
||||
/// E2E (#173): the loaded-blueprint walk-forward runs + persists a WalkForward
|
||||
/// family (3 OOS members) with a {"walkforward":…} summary carrying oos_r, and
|
||||
/// stores exactly one content-addressed blueprint.
|
||||
#[test]
|
||||
fn aura_walkforward_over_a_blueprint_persists_a_family() {
|
||||
let cwd = temp_cwd("blueprint-walkforward-persist");
|
||||
let bp = format!("{}/tests/fixtures/stage1_signal_open.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["walkforward", &bp, "--axis", "stage1_signal.fast.length=2,3",
|
||||
"--axis", "stage1_signal.slow.length=4,6", "--name", "wfx"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura walkforward");
|
||||
assert_eq!(out.status.code(), Some(0), "stderr: {}", String::from_utf8_lossy(&out.stderr));
|
||||
let stdout = String::from_utf8(out.stdout).expect("utf-8");
|
||||
assert_eq!(stdout.matches("wfx-").count(), 3, "3 OOS member lines: {stdout}");
|
||||
assert!(stdout.contains("\"walkforward\""), "summary line: {stdout}");
|
||||
assert!(stdout.contains("\"oos_r\""), "R-measured summary: {stdout}");
|
||||
let families = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["runs", "families"]).current_dir(&cwd).output().expect("spawn families");
|
||||
let fo = String::from_utf8(families.stdout).expect("utf-8");
|
||||
assert!(fo.contains("\"kind\":\"WalkForward\""), "families: {fo}");
|
||||
let n = std::fs::read_dir(cwd.join("runs").join("blueprints")).map(|d| d.count()).unwrap_or(0);
|
||||
assert_eq!(n, 1, "exactly one content-addressed blueprint stored");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
}
|
||||
|
||||
/// E2E (#173): an OPEN blueprint with no --axis (nothing to re-fit) is a usage
|
||||
/// error (exit 2), naming that >= 1 --axis is required.
|
||||
#[test]
|
||||
fn aura_walkforward_over_a_blueprint_rejects_no_axis() {
|
||||
let bp = format!("{}/tests/fixtures/stage1_signal_open.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["walkforward", &bp])
|
||||
.output()
|
||||
.expect("spawn aura walkforward (no axis)");
|
||||
assert_eq!(out.status.code(), Some(2));
|
||||
let stderr = String::from_utf8(out.stderr).expect("utf-8");
|
||||
assert!(stderr.contains("requires >= 1 --axis"), "stderr: {stderr}");
|
||||
}
|
||||
|
||||
/// E2E (#173, safety): a malformed blueprint is rejected at the dispatch boundary
|
||||
/// (exit 2, UnknownNodeType) rather than panicking.
|
||||
#[test]
|
||||
fn aura_walkforward_over_a_blueprint_rejects_malformed() {
|
||||
let bp = format!("{}/tests/fixtures/unknown_node.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args(["walkforward", &bp, "--axis", "stage1_signal.fast.length=2,3"])
|
||||
.output()
|
||||
.expect("spawn aura walkforward (malformed)");
|
||||
assert_eq!(out.status.code(), Some(2), "malformed must be rejected, not panic");
|
||||
let stderr = String::from_utf8(out.stderr).expect("utf-8");
|
||||
assert!(stderr.contains("UnknownNodeType"), "stderr: {stderr}");
|
||||
assert!(!stderr.contains("panicked"), "must reject cleanly: {stderr}");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the three new E2Es**
|
||||
|
||||
Run: `cargo test -p aura-cli --test cli_run aura_walkforward_over_a_blueprint`
|
||||
Expected: PASS — 4 tests (the three above + the reproduce test from Task 3, which
|
||||
shares the `aura_walkforward_over_a_blueprint` prefix).
|
||||
|
||||
- [ ] **Step 3: Full-suite + lint gate**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS — no regressions (esp. the hard-wired walkforward tests
|
||||
`walkforward_plateau_select_stamps_plateau_provenance`,
|
||||
`walkforward_select_argmax_is_byte_identical_to_default`, and the sweep/mc/reproduce
|
||||
E2Es — the dispatch + reproduce edits preserved them).
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: `Finished` — 0 warnings (every new fn is now wired; no dead code).
|
||||
|
||||
---
|
||||
|
||||
## Self-review (planner Step 5)
|
||||
|
||||
1. **Spec coverage:** §Concrete shapes (blueprint_sweep_over / run_oos_blueprint /
|
||||
blueprint_walkforward_family / WfBlueprintArgs+parse / dispatch / reproduce) →
|
||||
Tasks 1-3; §Testing (unit family, unit parse, 4 E2Es) → Tasks 1/2/3/4;
|
||||
§Error handling (no-axis, malformed, bad-axis in-closure exit) → Task 2 parse +
|
||||
Task 4 E2Es + Task 1 blueprint_sweep_over Result; §Persist + reproduce → Tasks
|
||||
2+3 (land together, one commit); §Acceptance 1-7 → the E2Es + the Task-4 gate. ✓
|
||||
2. **Placeholder scan:** no TBD/TODO/"similar to"/"add appropriate". The
|
||||
transcribe-the-current-arm notes (dispatch else-branch, reproduce arm) name the
|
||||
exact site + say match it verbatim — not deferred decisions. ✓
|
||||
3. **Type/name consistency:** blueprint_sweep_over / run_oos_blueprint /
|
||||
blueprint_walkforward_family / WfBlueprintArgs / parse_walkforward_blueprint_args /
|
||||
run_blueprint_walkforward / FamilyKind::WalkForward / walkforward_member_reports /
|
||||
Selection / select_winner / walk_forward / WindowRun / WindowBounds consistent
|
||||
across tasks and match the recon signatures. `blueprint_walkforward_family`
|
||||
returns `WalkForwardResult` (not Result) in Task 1, Task 2's wrapper, and the
|
||||
unit test — consistent. ✓
|
||||
4. **Step granularity:** each step is one fn/edit or one command. ✓
|
||||
5. **No commit steps.** ✓
|
||||
6. **Pin/replacement contiguity:** E2E pins — `"reproduced 3/3 members
|
||||
bit-identically"` (the reproduce path's literal, reused from the mc-reproduce
|
||||
template), `"kind":"WalkForward"` (append_family serialization), `"walkforward"`
|
||||
/ `"oos_r"` (walkforward_summary_json keys), `"requires >= 1 --axis"` (the
|
||||
parse Err literal in Task 2 Step 3), `UnknownNodeType` (engine error) — each
|
||||
appears contiguously in the corresponding impl/literal. ✓
|
||||
7. **Compile-gate vs. deferred-caller:** the new `WfBlueprintArgs` struct + its
|
||||
destructure in the dispatch are BOTH in Task 2 (Steps 3+5) before Task 2's
|
||||
build gate (Step 6). No caller deferred. Task 1's fns are unused until Task 2 —
|
||||
handled by using `cargo build` (warning-tolerant) not clippy in Tasks 1-3, with
|
||||
the `-D warnings` clippy gate in Task 4 after wiring. ✓
|
||||
8. **Verification filters resolve:** `blueprint_walkforward_family` (Task 1 new),
|
||||
`parse_walkforward_blueprint_args` (Task 2 new),
|
||||
`aura_walkforward_over_a_blueprint_reproduces` (Task 3 new),
|
||||
`aura_walkforward_over_a_blueprint` (Task 4 — matches the 4 WF E2Es by shared
|
||||
prefix), `walkforward_plateau_select_stamps_plateau_provenance` (existing, Task
|
||||
4 regression) — every filter matches ≥1 named test in the post-task tree. ✓
|
||||
Reference in New Issue
Block a user