plan: 0077 plateau over peak selection
Executable plan for cycle 0077 (#145), decomposed into 5 tasks with the compile-gates sequenced: 1. Reshape FamilySelection/SelectionMode across 3 crates (mechanical, byte-preserving): deflation fields -> Option, add PlateauMean/PlateauWorst + neighbourhood_score/n_neighbours; thread optimize_deflated + all tests + the runs_family display (argmax bytes unchanged). 2. GridSpace::axis_lens() + SweepBinder::sweep_with_lattice (sweep delegates, every existing .sweep() caller byte-unchanged). 3. optimize_plateau + PlateauMode + closed_neighbourhood (mixed-radix grid neighbourhood, argmax of the smoothed surface) + a higher_is_better refactor so direction lives in one place. 4. CLI --select parse + Selection threading + sweep-entry Option<lattice> tuples + select_winner (the RandomSpace-refuse guard) + walkforward_family dispatch; all signature-change callers threaded before the build gate. 5. runs-family plateau E2E golden. Two plan-time forks were derived and recorded on #145 (comment 1974): the lattice surfaces at the engine terminal (only the post-resolve_axes GridSpace holds param_space order), and the RandomSpace-refuse is a structural guard unit-tested at select_winner (walkforward has no --random producer to E2E). refs #145
This commit is contained in:
@@ -0,0 +1,974 @@
|
||||
# Prefer a robust parameter plateau over the in-sample peak — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0077-plateau-over-peak-selection.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to
|
||||
> run this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Add an opt-in `optimize_plateau` selection objective (argmax over the
|
||||
grid-neighbourhood-smoothed metric) reachable via a new `--select` flag, recorded
|
||||
on the `RunManifest.selection` carrier #144 introduced; default `argmax` stays
|
||||
byte-identical (C23).
|
||||
|
||||
**Architecture:** A new family selector lives beside `optimize_deflated` in
|
||||
`aura-registry` (C9), reading each member's metric via the shared `metric_value`
|
||||
and smoothing it over a mixed-radix grid neighbourhood built from `axis_lens`
|
||||
(the `GridSpace` odometer radixes). The lattice is surfaced from the engine via a
|
||||
new `SweepBinder::sweep_with_lattice` terminal (`sweep` delegates to it,
|
||||
byte-unchanged for all existing callers). The shared `FamilySelection` carrier is
|
||||
reshaped — orthogonal rule (`SelectionMode`) × annotation (deflation fields →
|
||||
`Option`, plus plateau fields) — and the CLI threads a `Selection` from
|
||||
`--select` into `walkforward_family`.
|
||||
|
||||
**Tech Stack:** `aura-engine` (`report.rs` carrier, `sweep.rs` accessor,
|
||||
`blueprint.rs` terminal), `aura-registry` (`lib.rs` selector + index math),
|
||||
`aura-cli` (`main.rs` flag/dispatch/display, `tests/cli_run.rs` E2E).
|
||||
|
||||
---
|
||||
|
||||
## Design decisions resolved at plan time (recorded on #145, comment 1974)
|
||||
|
||||
1. **Lattice seam → engine `SweepBinder::sweep_with_lattice`.** Only the engine's
|
||||
post-`resolve_axes` `GridSpace` holds the lattice in `param_space()`/odometer
|
||||
order, which `closed_neighbourhood`'s mixed-radix arithmetic requires; a sibling
|
||||
terminal (`sweep` delegates) contains the blast radius — every existing
|
||||
`.sweep()` caller stays byte-identical.
|
||||
2. **RandomSpace-refuse → structural guard, no new feature.** `walkforward` has no
|
||||
`--random` flag (verified: `--random`/`RandomSpace`/`RandomBinder` appear
|
||||
nowhere in `aura-cli`); wiring a random walk-forward producer is unspecified,
|
||||
out-of-scope. So `sweep_over`/`stage1_r_sweep_over` return
|
||||
`(SweepFamily, Option<Vec<usize>>)` (`Some` today, grid-only), and a
|
||||
`select_winner` helper returns the refuse on `Plateau + None`-lattice; the
|
||||
caller prints the spec's message and exits 2. **Unit-tested** at the helper
|
||||
level (the `Err` path), not via a `walkforward --random` E2E.
|
||||
3. **`FamilySelection.seed` → `Option<u64>`** (spec line 135; the recon carrier's
|
||||
`Option<usize>` was a transcription slip).
|
||||
|
||||
## Files this plan creates or modifies
|
||||
|
||||
- Modify: `crates/aura-engine/src/report.rs:41-66` — reshape `SelectionMode` +
|
||||
`FamilySelection`; `:762-775` round-trip test literal; new plateau-shape test.
|
||||
- Modify: `crates/aura-engine/src/sweep.rs` — new `GridSpace::axis_lens()` + test.
|
||||
- Modify: `crates/aura-engine/src/blueprint.rs:399-411` — new
|
||||
`SweepBinder::sweep_with_lattice`; `sweep` delegates; new test.
|
||||
- Modify: `crates/aura-registry/src/lib.rs:308-312` — `optimize_deflated`
|
||||
construction to `Option` deflation fields; `:163-173` `metric_cmp` refactor to
|
||||
`higher_is_better`; `:744-794` test assertions → `.unwrap()`/`Some`; add
|
||||
`PlateauMode`, `higher_is_better`, `closed_neighbourhood`, `optimize_plateau` +
|
||||
tests.
|
||||
- Modify: `crates/aura-cli/src/main.rs` — import `SelectionMode` (engine),
|
||||
`optimize_plateau`/`PlateauMode` (registry); `Selection` enum; `parse_select` +
|
||||
`--select` in `parse_walkforward_args`; `select_winner`; `sweep_over` /
|
||||
`stage1_r_sweep_over` tuple returns; `walkforward_family` `+select` dispatch;
|
||||
`run_walkforward` `+select`; the dispatch call site; `runs_family` mode-aware
|
||||
display; 4 test callers of `walkforward_family`; unit tests.
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`.
|
||||
- Unchanged (preservation gate): `crates/aura-registry/src/compat.rs:34` embeds
|
||||
`FamilySelection` by value — the reshape flows through with no edit; the
|
||||
legacy-load test `load_lifts_a_selection_block_through_the_compat_mirror`
|
||||
(`lib.rs:520`) and `runs_family_rank_shows_deflated_line` (`cli_run.rs:2358`)
|
||||
must stay green.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Reshape `FamilySelection` / `SelectionMode` (mechanical, byte-preserving across 3 crates)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs:41-66`, `:762-775`
|
||||
- Modify: `crates/aura-registry/src/lib.rs:308-312`, `:744-794`
|
||||
- Modify: `crates/aura-cli/src/main.rs:19-25` (import), `:2215-2223` (display)
|
||||
|
||||
This is the cross-crate compile-gate. The reshape breaks every construction and
|
||||
field-read site; all are threaded here. No new runtime behaviour — the argmax
|
||||
path stays byte-identical; the one new test pins the new fields' serde contract.
|
||||
|
||||
- [ ] **Step 1: Write the failing plateau-shape serde test (RED)**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, in the `#[cfg(test)] mod tests` block,
|
||||
beside `family_selection_round_trips_on_the_manifest` (after line 775), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn family_selection_plateau_shape_round_trips() {
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 16, raw_winner_metric: 1.42,
|
||||
mode: SelectionMode::PlateauMean,
|
||||
deflated_score: None, overfit_probability: None,
|
||||
n_resamples: None, block_len: None, seed: None,
|
||||
neighbourhood_score: Some(1.27), n_neighbours: Some(5),
|
||||
};
|
||||
let json = serde_json::to_string(&sel).unwrap();
|
||||
// plateau annotation present; deflation fields omitted (skip_serializing_if)
|
||||
assert!(json.contains("\"neighbourhood_score\":1.27"), "json: {json}");
|
||||
assert!(json.contains("\"n_neighbours\":5"), "json: {json}");
|
||||
assert!(!json.contains("deflated_score"), "deflation fields omitted under plateau: {json}");
|
||||
assert!(!json.contains("n_resamples"), "json: {json}");
|
||||
let back: FamilySelection = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back, sel);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the new test to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine family_selection_plateau`
|
||||
Expected: FAIL — compile error (`SelectionMode::PlateauMean` / the `neighbourhood_score`/`n_neighbours` fields do not exist yet).
|
||||
|
||||
- [ ] **Step 3: Reshape `SelectionMode` and `FamilySelection`**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, replace the `SelectionMode` enum + doc
|
||||
(lines 41-48) and the `FamilySelection` struct + doc (lines 50-66) with:
|
||||
|
||||
```rust
|
||||
/// Which selection objective produced the record (additive provenance, C23).
|
||||
/// `Argmax` is the bare-best pick (cycle 0076), deflated for the number of trials.
|
||||
/// `PlateauMean` / `PlateauWorst` (cycle 0077) argmax the neighbourhood-smoothed
|
||||
/// surface instead, so peak-vs-plateau runs stay distinguishable on this field.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
PlateauMean,
|
||||
PlateauWorst,
|
||||
}
|
||||
|
||||
/// Selection-provenance for a sweep winner. The selection RULE (`mode`) and its
|
||||
/// ANNOTATION are orthogonal: `Argmax` carries the trials-deflation annotation
|
||||
/// (`deflated_score` / `overfit_probability` / `n_resamples` / `block_len` /
|
||||
/// `seed`); `Plateau*` carries the smoothing annotation (`neighbourhood_score` /
|
||||
/// `n_neighbours`). Each annotation is `Option`, present iff its rule produced the
|
||||
/// record; all are additive — recorded, never re-ranking (C23). A legacy line
|
||||
/// (pre-0077) carries the deflation scalars as bare values that deserialize to
|
||||
/// `Some` (serde default), so it still loads (C14/C18).
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub mode: SelectionMode,
|
||||
// deflation annotation (present iff mode == Argmax with a deflation run)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub deflated_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_resamples: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub block_len: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub seed: Option<u64>,
|
||||
// plateau annotation (present iff mode is Plateau*)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub neighbourhood_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_neighbours: Option<usize>,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update the round-trip test literal**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, in `family_selection_round_trips_on_the_manifest`,
|
||||
replace the `let sel = FamilySelection { … };` literal (lines 764-768) with:
|
||||
|
||||
```rust
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(0.21), overfit_probability: Some(0.06),
|
||||
n_resamples: Some(1000), block_len: Some(5), seed: Some(42),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the engine tests (GREEN, engine internally consistent)**
|
||||
|
||||
Run: `cargo test -p aura-engine family_selection`
|
||||
Expected: PASS — both `family_selection_round_trips_on_the_manifest` and
|
||||
`family_selection_plateau_shape_round_trips` green. (aura-engine does not depend
|
||||
on aura-registry, so its build is consistent even though the registry is not yet
|
||||
updated.)
|
||||
|
||||
- [ ] **Step 6: Update `optimize_deflated`'s construction to the Option fields**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, replace the returned `FamilySelection`
|
||||
literal (lines 308-312) with:
|
||||
|
||||
```rust
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(deflated_score), overfit_probability,
|
||||
n_resamples: Some(n_resamples), block_len: Some(block_len), seed: Some(seed),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
}))
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Update the `optimize_deflated_*` test assertions to `.unwrap()` / `Some`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, apply these exact line edits:
|
||||
|
||||
- Line 749: ` assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw`
|
||||
→ ` assert!(sel.deflated_score.unwrap() <= sel.raw_winner_metric); // dispersion floor ≤ raw`
|
||||
- Line 762: ` assert_eq!(sel.n_resamples, 0);`
|
||||
→ ` assert_eq!(sel.n_resamples, Some(0));`
|
||||
- Line 763: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric, "p95 of an empty null is 0");`
|
||||
→ ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric, "p95 of an empty null is 0");`
|
||||
- Line 782: ` assert!(sel.deflated_score.is_finite(), "deflated_score must be finite, got {}", sel.deflated_score);`
|
||||
→ ` assert!(sel.deflated_score.unwrap().is_finite(), "deflated_score must be finite, got {:?}", sel.deflated_score);`
|
||||
- Line 783: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric);`
|
||||
→ ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric);`
|
||||
- Line 794: ` assert!(sel.deflated_score > 0.0);`
|
||||
→ ` assert!(sel.deflated_score.unwrap() > 0.0);`
|
||||
|
||||
(The compat test `load_lifts_a_selection_block_through_the_compat_mirror` at line
|
||||
520-527 is **unchanged** — its legacy JSON line carries the deflation scalars as
|
||||
bare floats that deserialize to `Some` under the new `Option` + `#[serde(default)]`,
|
||||
and its assertions read only `n_trials`/`mode`. It is the C14/C18 legacy-load gate.)
|
||||
|
||||
- [ ] **Step 8: Run the registry tests (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-registry optimize_deflated`
|
||||
Expected: PASS — all nine `optimize_deflated_*` tests green against the reshaped fields.
|
||||
|
||||
Run: `cargo test -p aura-registry load_lifts`
|
||||
Expected: PASS — the legacy `selection` block still lifts through the compat mirror.
|
||||
|
||||
- [ ] **Step 9: Import `SelectionMode` and rewrite the `runs_family` display (mode-aware, argmax byte-preserved)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, add `SelectionMode` to the `aura_engine` import
|
||||
list (lines 19-24) — e.g. extend the `RunManifest, RunMetrics,` line to
|
||||
`RunManifest, RunMetrics, SelectionMode,` (any slot in the brace list is fine).
|
||||
|
||||
Then replace the selection-display block in `runs_family` (lines 2215-2223):
|
||||
|
||||
```rust
|
||||
for report in &ordered {
|
||||
println!("{}", report.to_json());
|
||||
if let Some(sel) = &report.manifest.selection {
|
||||
match sel.mode {
|
||||
SelectionMode::Argmax => match sel.overfit_probability {
|
||||
Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score.unwrap(), p),
|
||||
None => println!(" deflated={:.4}", sel.deflated_score.unwrap()),
|
||||
},
|
||||
SelectionMode::PlateauMean | SelectionMode::PlateauWorst => {
|
||||
let label = if matches!(sel.mode, SelectionMode::PlateauMean) { "mean" } else { "worst" };
|
||||
if let (Some(score), Some(n)) = (sel.neighbourhood_score, sel.n_neighbours) {
|
||||
println!(" plateau({label})={score:.4} over {n} cells");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Under `Argmax`, `deflated_score` is always `Some` by construction, so `.unwrap()`
|
||||
preserves the exact pre-0077 bytes — the `deflated=…`/`P(overfit)=…` golden stays
|
||||
green. The `Plateau*` branch is exercised by the Task 5 E2E.)
|
||||
|
||||
- [ ] **Step 10: Workspace build + full per-crate regression gate**
|
||||
|
||||
Run: `cargo build --workspace`
|
||||
Expected: clean (0 errors) — all three crates consistent.
|
||||
|
||||
Run: `cargo test -p aura-engine`
|
||||
Expected: PASS (no regressions).
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS (no regressions).
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `GridSpace::axis_lens()` + `SweepBinder::sweep_with_lattice` (engine, additive)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/sweep.rs` (accessor + test)
|
||||
- Modify: `crates/aura-engine/src/blueprint.rs:399-411` (terminal + delegate + test)
|
||||
|
||||
- [ ] **Step 1: Write the failing `axis_lens` golden (RED)**
|
||||
|
||||
In `crates/aura-engine/src/sweep.rs`, in the `#[cfg(test)] mod tests` block (which
|
||||
already imports `aura_core::{Cell, ParamSpec, Scalar, ScalarKind, Timestamp}`),
|
||||
add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn grid_axis_lens_are_the_per_axis_radixes() {
|
||||
let space = vec![
|
||||
ParamSpec { name: "a".into(), kind: ScalarKind::I64 },
|
||||
ParamSpec { name: "b".into(), kind: ScalarKind::I64 },
|
||||
];
|
||||
let grid = GridSpace::new(&space, vec![
|
||||
vec![Scalar::i64(10), Scalar::i64(20)], // axis 0: 2 values
|
||||
vec![Scalar::i64(1), Scalar::i64(2), Scalar::i64(3)], // axis 1: 3 values
|
||||
]).expect("2x3 grid");
|
||||
assert_eq!(grid.axis_lens(), vec![2, 3]);
|
||||
assert_eq!(grid.axis_lens().iter().product::<usize>(), grid.len());
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine grid_axis_lens`
|
||||
Expected: FAIL — `no method named axis_lens found for struct GridSpace`.
|
||||
|
||||
- [ ] **Step 3: Add the `axis_lens` accessor**
|
||||
|
||||
In `crates/aura-engine/src/sweep.rs`, in the `impl GridSpace` block, immediately
|
||||
after `is_empty` (after line 65), add:
|
||||
|
||||
```rust
|
||||
/// Per-axis cardinalities in `param_space()` order (the odometer radixes,
|
||||
/// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau
|
||||
/// neighbourhood walks (cycle 0077).
|
||||
pub fn axis_lens(&self) -> Vec<usize> {
|
||||
self.axes.iter().map(Vec::len).collect()
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the accessor test (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-engine grid_axis_lens`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Write the failing `sweep_with_lattice` test (RED)**
|
||||
|
||||
In `crates/aura-engine/src/blueprint.rs`, in the `#[cfg(test)] mod tests` block
|
||||
(which has `composite_sma_cross_harness()` and `run_point`), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn sweep_with_lattice_surfaces_grid_radixes_in_param_space_order() {
|
||||
let bp = composite_sma_cross_harness().0;
|
||||
let (fam, lattice) = bp
|
||||
.axis("sma_cross.fast.length", vec![Scalar::i64(2), Scalar::i64(3)]) // 2 values
|
||||
.axis("sma_cross.slow.length", vec![Scalar::i64(4), Scalar::i64(5)]) // 2 values
|
||||
.axis("bias.scale", vec![Scalar::f64(0.5)]) // 1 value
|
||||
.sweep_with_lattice(run_point)
|
||||
.expect("named binding resolves and runs");
|
||||
assert_eq!(lattice, vec![2, 2, 1], "radixes in param_space slot order");
|
||||
assert_eq!(lattice.iter().product::<usize>(), fam.points.len()); // 4
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine sweep_with_lattice`
|
||||
Expected: FAIL — `no method named sweep_with_lattice found`.
|
||||
|
||||
- [ ] **Step 7: Add `sweep_with_lattice`; refactor `sweep` to delegate**
|
||||
|
||||
In `crates/aura-engine/src/blueprint.rs`, replace the `SweepBinder::sweep` method
|
||||
(lines 399-411) with:
|
||||
|
||||
```rust
|
||||
/// Resolve the named axes against `param_space()` into a positional grid and
|
||||
/// run the disjoint sweep. `sweep` is [`SweepBinder::sweep_with_lattice`] with
|
||||
/// the lattice dropped, so every existing caller is byte-unchanged.
|
||||
pub fn sweep<F>(self, run_one: F) -> Result<SweepFamily, BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
self.sweep_with_lattice(run_one).map(|(family, _lattice)| family)
|
||||
}
|
||||
|
||||
/// As [`SweepBinder::sweep`], plus the grid's per-axis radixes (`axis_lens`,
|
||||
/// in `param_space()` / odometer order). The lattice is what a plateau
|
||||
/// neighbourhood walk needs (cycle 0077); only the engine's post-`resolve_axes`
|
||||
/// grid holds it in the correct order.
|
||||
pub fn sweep_with_lattice<F>(self, run_one: F) -> Result<(SweepFamily, Vec<usize>), BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
let space = self.bp.param_space();
|
||||
check_param_namespace_injective(&space).map_err(BindError::Compile)?;
|
||||
let ordered = resolve_axes(&space, &self.axes)?;
|
||||
let grid = GridSpace::new(&space, ordered)
|
||||
.expect("named layer pre-validates arity/kind/non-empty");
|
||||
let lattice = grid.axis_lens();
|
||||
Ok((sweep(&grid, run_one), lattice))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Run the new test + the existing sweep tests (GREEN, delegation preserved)**
|
||||
|
||||
Run: `cargo test -p aura-engine sweep_with_lattice`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-engine`
|
||||
Expected: PASS — every existing `.sweep()` test (named-axes parity, grid goldens)
|
||||
stays green; `sweep` delegating to `sweep_with_lattice` is behaviour-preserving.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: `optimize_plateau` + `PlateauMode` + `closed_neighbourhood` (registry, additive)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/lib.rs:163-173` (`metric_cmp` → `higher_is_better`),
|
||||
plus new `PlateauMode`, `higher_is_better`, `closed_neighbourhood`,
|
||||
`optimize_plateau` and their tests.
|
||||
|
||||
- [ ] **Step 1: Write the failing plateau tests (RED)**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, in the `#[cfg(test)] mod tests` block (which
|
||||
has the `member(total_pips, trade_rs)` helper), add:
|
||||
|
||||
```rust
|
||||
/// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle
|
||||
/// plateau in `total_pips`. Bare argmax takes an end spike; the plateau-mean
|
||||
/// argmax takes the interior plateau centre.
|
||||
fn fixture_grid_spike_vs_plateau() -> SweepFamily {
|
||||
let pips = [80.0, 5.0, 48.0, 50.0, 49.0, 5.0, 80.0];
|
||||
SweepFamily { space: vec![], points: pips.iter().map(|&p| member(p, vec![1.0])).collect() }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closed_neighbourhood_2x3_golden() {
|
||||
// 2x3 lattice, odometer (last axis fastest):
|
||||
// (0,0)=0 (0,1)=1 (0,2)=2
|
||||
// (1,0)=3 (1,1)=4 (1,2)=5
|
||||
let sorted = |i: usize| { let mut v = closed_neighbourhood(i, &[2, 3]); v.sort_unstable(); v };
|
||||
assert_eq!(sorted(0), vec![0, 1, 3]); // corner (0,0): self + 2
|
||||
assert_eq!(sorted(4), vec![1, 3, 4, 5]); // interior (1,1): self + 3
|
||||
assert_eq!(sorted(2), vec![1, 2, 5]); // corner (0,2): self + 2
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_picks_the_plateau_not_the_spike() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let spike = optimize(&fam, "total_pips").unwrap();
|
||||
let (centre, sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_ne!(centre.report.metrics.total_pips, spike.report.metrics.total_pips);
|
||||
assert_eq!(centre.report.metrics.total_pips, 50.0, "plateau centre is the middle cell");
|
||||
assert_eq!(sel.mode, SelectionMode::PlateauMean);
|
||||
assert_eq!(sel.raw_winner_metric, 50.0);
|
||||
assert!(sel.neighbourhood_score.unwrap() < spike.report.metrics.total_pips,
|
||||
"smoothed score is below the spike's raw peak");
|
||||
assert_eq!(sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours");
|
||||
assert!(sel.deflated_score.is_none() && sel.n_resamples.is_none(),
|
||||
"deflation fields omitted under plateau");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_worst_is_more_conservative_than_mean() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let (_, mean_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
let (_, worst_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap();
|
||||
// Worst scores each cell by its weakest neighbour, never above the mean.
|
||||
assert!(worst_sel.neighbourhood_score.unwrap() <= mean_sel.neighbourhood_score.unwrap());
|
||||
assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst);
|
||||
assert!(worst_sel.n_neighbours.unwrap() >= 2, "winner is an interior cell");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_single_member_equals_optimize() {
|
||||
let fam = SweepFamily { space: vec![], points: vec![member(42.0, vec![1.0])] };
|
||||
let plain = optimize(&fam, "total_pips").unwrap();
|
||||
let (winner, sel) = optimize_plateau(&fam, &[1], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_eq!(winner.report.metrics.total_pips, plain.report.metrics.total_pips);
|
||||
assert_eq!(sel.n_neighbours, Some(1), "a 1-cell grid's closed neighbourhood is itself");
|
||||
assert_eq!(sel.neighbourhood_score, Some(42.0));
|
||||
assert_eq!(sel.raw_winner_metric, 42.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_plateau_is_deterministic() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let a = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
let b = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
assert_eq!(a, b);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify they fail to compile**
|
||||
|
||||
Run: `cargo test -p aura-registry closed_neighbourhood`
|
||||
Expected: FAIL — `closed_neighbourhood` / `optimize_plateau` / `PlateauMode` do not exist.
|
||||
|
||||
- [ ] **Step 3: Add `higher_is_better` and refactor `metric_cmp` to use it**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, immediately after `is_r_metric` (after line
|
||||
207), add:
|
||||
|
||||
```rust
|
||||
/// The metric's optimisation direction: `true` if a larger value is better
|
||||
/// (`total_pips`, the four R keys), `false` for the lower-is-better keys
|
||||
/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp`
|
||||
/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour)
|
||||
/// both read it, so the per-metric sense lives in exactly one place.
|
||||
fn higher_is_better(m: Metric) -> bool {
|
||||
!matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips)
|
||||
}
|
||||
```
|
||||
|
||||
Then replace the `metric_cmp` body (lines 163-173) with:
|
||||
|
||||
```rust
|
||||
fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Ordering, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let hib = higher_is_better(m);
|
||||
Ok(move |a: &RunReport, b: &RunReport| {
|
||||
let (va, vb) = (metric_value(a, m), metric_value(b, m));
|
||||
// higher-is-better ranks the larger value first; lower-is-better the smaller.
|
||||
if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) }
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
(Behaviour-identical: `rank_by` / `optimize` and their existing tests are the gate.)
|
||||
|
||||
- [ ] **Step 4: Add `closed_neighbourhood`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, after `higher_is_better`, add:
|
||||
|
||||
```rust
|
||||
/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice
|
||||
/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}`
|
||||
/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order
|
||||
/// is `i` first then per-axis neighbours; callers use the result only as a set and
|
||||
/// for its length, so the order is not load-bearing.
|
||||
fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> {
|
||||
// decompose i to per-axis coords (last axis fastest)
|
||||
let mut coords = vec![0usize; axis_lens.len()];
|
||||
let mut rem = i;
|
||||
for k in (0..axis_lens.len()).rev() {
|
||||
coords[k] = rem % axis_lens[k];
|
||||
rem /= axis_lens[k];
|
||||
}
|
||||
// recompose a coord vector to its flat index (Horner, last axis fastest)
|
||||
let recompose = |c: &[usize]| c.iter().zip(axis_lens).fold(0usize, |flat, (&ck, &len)| flat * len + ck);
|
||||
let mut out = vec![i];
|
||||
for k in 0..axis_lens.len() {
|
||||
for delta in [-1isize, 1] {
|
||||
let ck = coords[k] as isize + delta;
|
||||
if ck < 0 || ck as usize >= axis_lens[k] { continue; }
|
||||
let mut nb = coords.clone();
|
||||
nb[k] = ck as usize;
|
||||
out.push(recompose(&nb));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add `PlateauMode` and `optimize_plateau`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, immediately before `optimize_deflated` (before
|
||||
line 258, before its doc comment), add:
|
||||
|
||||
```rust
|
||||
/// Plateau-selection aggregate: how a member's grid-neighbourhood metric is
|
||||
/// reduced to one smoothed score before the argmax. `Mean` averages the closed
|
||||
/// neighbourhood; `Worst` takes the most-pessimistic neighbour by the metric's
|
||||
/// direction (biasing toward interior cells, which keep more neighbours).
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum PlateauMode {
|
||||
Mean,
|
||||
Worst,
|
||||
}
|
||||
|
||||
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau
|
||||
/// provenance. Each member scores as the mean (or worst-case) of its closed grid
|
||||
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the
|
||||
/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are
|
||||
/// the grid radixes (`param_space()` order, last axis fastest). Pure,
|
||||
/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL;
|
||||
/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the
|
||||
/// smoothed value the argmax maximised.
|
||||
pub fn optimize_plateau(
|
||||
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let n = family.points.len();
|
||||
debug_assert_eq!(
|
||||
axis_lens.iter().product::<usize>(), n,
|
||||
"axis_lens product must equal the family size (a valid grid lattice)",
|
||||
);
|
||||
let hib = higher_is_better(m);
|
||||
// smoothed score + closed-neighbourhood size per member, in odometer order
|
||||
let scored: Vec<(usize, f64, usize)> = (0..n)
|
||||
.map(|i| {
|
||||
let nbrs = closed_neighbourhood(i, axis_lens);
|
||||
let vals: Vec<f64> = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect();
|
||||
let score = match mode {
|
||||
PlateauMode::Mean => vals.iter().sum::<f64>() / vals.len() as f64,
|
||||
PlateauMode::Worst => {
|
||||
if hib {
|
||||
vals.iter().copied().fold(f64::INFINITY, f64::min)
|
||||
} else {
|
||||
vals.iter().copied().fold(f64::NEG_INFINITY, f64::max)
|
||||
}
|
||||
}
|
||||
};
|
||||
(i, score, nbrs.len())
|
||||
})
|
||||
.collect();
|
||||
// argmax the smoothed surface by direction; earliest-odometer tie (only a
|
||||
// strictly-better later cell displaces the incumbent, as `optimize`).
|
||||
let &(wi, wscore, wn) = scored
|
||||
.iter()
|
||||
.reduce(|best, cur| {
|
||||
let better = if hib { cur.1 > best.1 } else { cur.1 < best.1 };
|
||||
if better { cur } else { best }
|
||||
})
|
||||
.expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)");
|
||||
let winner = family.points[wi].clone();
|
||||
let raw = metric_value(&winner.report, m);
|
||||
Ok((
|
||||
winner,
|
||||
FamilySelection {
|
||||
selection_metric: metric.to_string(),
|
||||
n_trials: n,
|
||||
raw_winner_metric: raw,
|
||||
mode: match mode {
|
||||
PlateauMode::Mean => SelectionMode::PlateauMean,
|
||||
PlateauMode::Worst => SelectionMode::PlateauWorst,
|
||||
},
|
||||
deflated_score: None,
|
||||
overfit_probability: None,
|
||||
n_resamples: None,
|
||||
block_len: None,
|
||||
seed: None,
|
||||
neighbourhood_score: Some(wscore),
|
||||
n_neighbours: Some(wn),
|
||||
},
|
||||
))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the plateau tests + the existing rank/optimize gate (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-registry plateau`
|
||||
Expected: PASS — `plateau_picks_the_plateau_not_the_spike`,
|
||||
`plateau_worst_is_more_conservative_than_mean`, `plateau_single_member_equals_optimize`,
|
||||
`optimize_plateau_is_deterministic` all green.
|
||||
|
||||
Run: `cargo test -p aura-registry closed_neighbourhood`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS — `rank_by` / `optimize` / `optimize_deflated` stay green after the
|
||||
`metric_cmp` → `higher_is_better` refactor.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: CLI `--select` parse + threading + sweep-entry tuples + `select_winner` (compile-gate)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` — `aura_registry` import; `Selection` enum;
|
||||
`parse_select`; `parse_walkforward_args` (`+--select`, 6-tuple); the dispatch
|
||||
call site (`:2935`); `run_walkforward` (`+select`); `walkforward_family`
|
||||
(`+select` dispatch via `select_winner`); `sweep_over` / `stage1_r_sweep_over`
|
||||
(tuple returns); `select_winner`; the 4 test callers of `walkforward_family`;
|
||||
unit tests.
|
||||
|
||||
This is the CLI compile-gate: the signature changes break their callers, all
|
||||
threaded here. Default `Argmax` keeps the existing walk-forward output
|
||||
byte-identical (C23).
|
||||
|
||||
- [ ] **Step 1: Write the failing `select_winner` refuse + parse tests (RED)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, in the `#[cfg(test)] mod tests` block at line
|
||||
2965, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn select_winner_refuses_plateau_without_a_lattice() {
|
||||
// A plateau request with no lattice (a random sweep would yield None) is
|
||||
// refused, never silently argmaxed. The refuse short-circuits before the
|
||||
// family is read, so an empty family is fine here.
|
||||
let fam = SweepFamily { space: vec![], points: vec![] };
|
||||
let err = select_winner(&fam, "total_pips", Selection::Plateau(PlateauMode::Mean), None)
|
||||
.unwrap_err();
|
||||
assert!(err.contains("requires a grid sweep"), "refuse message: {err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_walkforward_select_flag() {
|
||||
let argmax = parse_walkforward_args(&["--strategy", "stage1-r"]).unwrap();
|
||||
assert!(matches!(argmax.5, Selection::Argmax), "default is argmax");
|
||||
let mean = parse_walkforward_args(&["--select", "plateau:mean"]).unwrap();
|
||||
assert!(matches!(mean.5, Selection::Plateau(PlateauMode::Mean)));
|
||||
let worst = parse_walkforward_args(&["--select", "plateau:worst"]).unwrap();
|
||||
assert!(matches!(worst.5, Selection::Plateau(PlateauMode::Worst)));
|
||||
assert!(parse_walkforward_args(&["--select", "bogus"]).is_err(),
|
||||
"unknown --select token is a usage error");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify they fail to compile**
|
||||
|
||||
Run: `cargo test -p aura-cli select_winner_refuses`
|
||||
Expected: FAIL — `Selection` / `PlateauMode` / `select_winner` not in scope; the
|
||||
`parse_walkforward_args` tuple has no `.5`.
|
||||
|
||||
- [ ] **Step 3: Imports + add the `Selection` enum and `parse_select`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, extend the `aura_registry` import (lines 26-30) —
|
||||
add `optimize_plateau, PlateauMode,` to the brace list (alongside `optimize_deflated`).
|
||||
Also add `FamilySelection` to the `aura_engine` import list (lines 19-24; it is not
|
||||
currently imported, and `select_winner` in Step 4 names it). (`SelectionMode` was
|
||||
already added to that list in Task 1 Step 9.)
|
||||
|
||||
Then, immediately after the `Strategy` enum (after line 1551's enum block; place it
|
||||
near the other CLI enums), add:
|
||||
|
||||
```rust
|
||||
/// In-sample winner-selection objective for walk-forward (cycle 0077). `Argmax` is
|
||||
/// the bare-best pick deflated for trials (#144, the default); `Plateau` argmaxes
|
||||
/// the neighbourhood-smoothed surface instead (opt-in via `--select`).
|
||||
#[derive(Clone, Copy)]
|
||||
enum Selection {
|
||||
Argmax,
|
||||
Plateau(PlateauMode),
|
||||
}
|
||||
|
||||
/// Parse a `--select` token: `argmax` | `plateau:mean` | `plateau:worst`. Unknown
|
||||
/// tokens are a usage error (the caller maps `Err(())` to exit 2).
|
||||
fn parse_select(s: &str) -> Result<Selection, ()> {
|
||||
match s {
|
||||
"argmax" => Ok(Selection::Argmax),
|
||||
"plateau:mean" => Ok(Selection::Plateau(PlateauMode::Mean)),
|
||||
"plateau:worst" => Ok(Selection::Plateau(PlateauMode::Worst)),
|
||||
_ => Err(()),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add the `select_winner` dispatch helper**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, immediately before `walkforward_family` (before
|
||||
line 1786), add:
|
||||
|
||||
```rust
|
||||
/// Resolve the in-sample winner under the chosen selection objective. `Argmax`
|
||||
/// defers to the trials-deflation pick (#144). `Plateau` argmaxes the smoothed grid
|
||||
/// surface — it needs the grid lattice, so a sweep with no lattice (a future random
|
||||
/// walk-forward producer) is refused rather than silently argmaxed. The metric is
|
||||
/// always known at the call sites, so a metric error is unreachable (`expect`); the
|
||||
/// only fallible outcome is the plateau-without-lattice refusal, returned as
|
||||
/// `Err(message)` for the caller to print and exit 2.
|
||||
fn select_winner(
|
||||
family: &SweepFamily, metric: &str, select: Selection, lattice: Option<&[usize]>,
|
||||
) -> Result<(SweepPoint, FamilySelection), String> {
|
||||
match select {
|
||||
Selection::Argmax => Ok(optimize_deflated(
|
||||
family, metric, DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED,
|
||||
).expect("walk-forward metrics are known")),
|
||||
Selection::Plateau(mode) => match lattice {
|
||||
Some(lens) => Ok(optimize_plateau(family, lens, metric, mode)
|
||||
.expect("walk-forward metrics are known")),
|
||||
None => Err(
|
||||
"--select plateau requires a grid sweep; a random sweep has no parameter lattice"
|
||||
.to_string(),
|
||||
),
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`select_winner` names `FamilySelection`, which is **not** currently imported in
|
||||
`main.rs` — Step 3 above adds it to the `aura_engine` import list.)
|
||||
|
||||
- [ ] **Step 5: Switch `sweep_over` / `stage1_r_sweep_over` to return `(SweepFamily, Option<Vec<usize>>)`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`:
|
||||
|
||||
`stage1_r_sweep_over` (line 1301): change the signature return type
|
||||
`-> SweepFamily` to `-> (SweepFamily, Option<Vec<usize>>)`; change its terminal
|
||||
`.sweep(|point| {` (line 1318) to `.sweep_with_lattice(|point| {`, and insert a
|
||||
`.map(...)` line before the existing trailing `.expect(...)` (line 1342, currently
|
||||
` .expect("the stage1-r named grid matches the stage1-r param-space")`). The
|
||||
closure's closing ` })` and the existing `.expect(...)` line both stay; the
|
||||
new `.map` goes between them, so the function tail reads:
|
||||
|
||||
```rust
|
||||
})
|
||||
.map(|(fam, lat)| (fam, Some(lat)))
|
||||
.expect("the stage1-r named grid matches the stage1-r param-space")
|
||||
```
|
||||
|
||||
`sweep_over` (line 1841): change the signature return type `-> SweepFamily` to
|
||||
`-> (SweepFamily, Option<Vec<usize>>)`; change its terminal `.sweep(|point| {`
|
||||
(line 1854) to `.sweep_with_lattice(|point| {`, and insert the `.map(...)` line
|
||||
before the existing trailing
|
||||
` .expect("the built-in named grid matches the sample param-space")` (line
|
||||
1869), so the function tail reads:
|
||||
|
||||
```rust
|
||||
})
|
||||
.map(|(fam, lat)| (fam, Some(lat)))
|
||||
.expect("the built-in named grid matches the sample param-space")
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Thread `Selection` through `walkforward_family` (signature + both arms)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, change `walkforward_family`'s signature (line
|
||||
1786-1788) to add the `select` parameter:
|
||||
|
||||
```rust
|
||||
fn walkforward_family(
|
||||
strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid,
|
||||
select: Selection,
|
||||
) -> WalkForwardResult {
|
||||
```
|
||||
|
||||
In the `Strategy::SmaCross` arm, replace the sweep + optimize lines (1802-1807):
|
||||
|
||||
```rust
|
||||
let (is_family, lattice) = sweep_over(w.is.0, w.is.1, data);
|
||||
let (best, selection) = match select_winner(&is_family, "total_pips", select, lattice.as_deref()) {
|
||||
Ok(v) => v,
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
};
|
||||
let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
```
|
||||
|
||||
In the `Strategy::Stage1R` arm, replace the sweep + optimize lines (1820-1825):
|
||||
|
||||
```rust
|
||||
let (is_family, lattice) = stage1_r_sweep_over(w.is.0, w.is.1, data, grid);
|
||||
let (best, selection) = match select_winner(&is_family, "sqn_normalized", select, lattice.as_deref()) {
|
||||
Ok(v) => v,
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
};
|
||||
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Thread the new `select` argument through every `walkforward_family` caller**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, update the production caller and the four test
|
||||
callers to pass a `Selection`:
|
||||
|
||||
- `run_walkforward` (line 1755): add `select: Selection` to its signature:
|
||||
`fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid, select: Selection) {`
|
||||
and update its `walkforward_family(...)` call (line 1763) to
|
||||
`let result = walkforward_family(strategy, persist.then_some(name), &data, grid, select);`
|
||||
- Line 1963: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
- Line 2131: `walkforward_family(Strategy::Stage1R, None, data, grid, Selection::Argmax)`
|
||||
- Line 3456: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
- Line 4131: `walkforward_family(Strategy::Stage1R, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
|
||||
- [ ] **Step 8: Add `--select` to `parse_walkforward_args` and thread it to the dispatch**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, in `parse_walkforward_args` (line 1648):
|
||||
|
||||
Change the return type to add `Selection`:
|
||||
`) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid, Selection), String> {`
|
||||
|
||||
Update the usage string to mention `--select` (append `[--select <argmax|plateau:mean|plateau:worst>]`
|
||||
to both the `///` doc usage line and the `usage` closure string).
|
||||
|
||||
Add `let mut select = Selection::Argmax;` beside the other `let mut` locals.
|
||||
|
||||
Add a match arm for the flag (beside `--stop-k`):
|
||||
`"--select" => select = parse_select(value).map_err(|()| usage())?,`
|
||||
|
||||
Change the final return to:
|
||||
`Ok((strategy, name, persist, real.finish(&usage)?, grid, select))`
|
||||
|
||||
In the command dispatch (line 2935-2939), update the destructure + call:
|
||||
|
||||
```rust
|
||||
["walkforward", rest @ ..] => match parse_walkforward_args(rest) {
|
||||
Ok((strategy, name, persist, choice, grid, select)) => {
|
||||
run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid, select)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 9: Compile-gate — workspace + tests build clean**
|
||||
|
||||
Run: `cargo build -p aura-cli --tests`
|
||||
Expected: clean (0 errors) — every caller of the changed signatures threaded.
|
||||
|
||||
- [ ] **Step 10: Run the new unit tests (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-cli select_winner_refuses`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-cli parse_walkforward_select`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 11: C23 — existing walk-forward goldens stay byte-identical**
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_strategy_stage1_r`
|
||||
Expected: PASS — argmax path unchanged.
|
||||
|
||||
Run: `cargo test -p aura-cli runs_family_rank_shows_deflated`
|
||||
Expected: PASS — the deflated display line is byte-identical.
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_bare_sma`
|
||||
Expected: PASS — the SMA summary golden is unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: `runs family rank` plateau E2E golden
|
||||
|
||||
**Files:**
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`
|
||||
|
||||
Tasks 1-4 deliver the behaviour; this pins it end-to-end (C18: the plateau
|
||||
provenance reaches disk and the display renders it).
|
||||
|
||||
- [ ] **Step 1: Write the plateau E2E (RED until Tasks 1-4 land; GREEN after)**
|
||||
|
||||
In `crates/aura-cli/tests/cli_run.rs`, beside `runs_family_rank_shows_deflated_line`
|
||||
(after line 2388), add:
|
||||
|
||||
```rust
|
||||
/// Property: a stage1-r walk-forward run with `--select plateau:mean` selects the
|
||||
/// neighbourhood-smoothed winner and stamps each OOS manifest with the plateau
|
||||
/// provenance (`mode = PlateauMean`, `neighbourhood_score`, `n_neighbours`); `runs
|
||||
/// family … rank` renders the `plateau(mean)=… over N cells` line, and the
|
||||
/// deflation fields are omitted (orthogonal annotation). End-to-end witness that
|
||||
/// the opt-in selection rule reaches disk (C18) and the display path renders it.
|
||||
#[test]
|
||||
fn walkforward_plateau_select_stamps_plateau_provenance() {
|
||||
let cwd = temp_cwd("runs-plateau");
|
||||
let wf = Command::new(BIN)
|
||||
.args([
|
||||
"walkforward", "--strategy", "stage1-r", "--select", "plateau:mean",
|
||||
"--fast", "50,100", "--slow", "200,400",
|
||||
"--stop-length", "14,21", "--stop-k", "2.0,3.0",
|
||||
"--name", "wf-plat",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn walkforward --select plateau:mean");
|
||||
assert!(
|
||||
wf.status.success(),
|
||||
"walkforward exit: {:?}; stderr: {}",
|
||||
wf.status,
|
||||
String::from_utf8_lossy(&wf.stderr)
|
||||
);
|
||||
|
||||
let rank = Command::new(BIN)
|
||||
.args(["runs", "family", "wf-plat-0", "rank", "sqn_normalized"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn runs family wf-plat-0 rank sqn_normalized");
|
||||
assert!(rank.status.success(), "rank exit: {:?}", rank.status);
|
||||
let rank_out = String::from_utf8(rank.stdout).expect("utf-8");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
|
||||
assert!(rank_out.contains("plateau(mean)="), "rank output missing plateau(mean)=: {rank_out:?}");
|
||||
assert!(rank_out.contains(" cells"), "rank output missing 'over N cells': {rank_out:?}");
|
||||
assert!(rank_out.contains("\"mode\":\"PlateauMean\""), "manifest carries PlateauMean: {rank_out:?}");
|
||||
assert!(!rank_out.contains("\"deflated_score\""), "plateau run omits deflated_score: {rank_out:?}");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the plateau E2E (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_plateau`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 3: Full CLI suite + workspace + clippy (no regressions)**
|
||||
|
||||
Run: `cargo test -p aura-cli`
|
||||
Expected: PASS — every existing golden green; the plateau E2E green.
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: clean (no warnings).
|
||||
Reference in New Issue
Block a user