From fd221c81ecfbc1d7f79e3f673195a3914be857f6 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 26 Jun 2026 15:54:57 +0200 Subject: [PATCH] plan: 0077 plateau over peak selection Executable plan for cycle 0077 (#145), decomposed into 5 tasks with the compile-gates sequenced: 1. Reshape FamilySelection/SelectionMode across 3 crates (mechanical, byte-preserving): deflation fields -> Option, add PlateauMean/PlateauWorst + neighbourhood_score/n_neighbours; thread optimize_deflated + all tests + the runs_family display (argmax bytes unchanged). 2. GridSpace::axis_lens() + SweepBinder::sweep_with_lattice (sweep delegates, every existing .sweep() caller byte-unchanged). 3. optimize_plateau + PlateauMode + closed_neighbourhood (mixed-radix grid neighbourhood, argmax of the smoothed surface) + a higher_is_better refactor so direction lives in one place. 4. CLI --select parse + Selection threading + sweep-entry Option tuples + select_winner (the RandomSpace-refuse guard) + walkforward_family dispatch; all signature-change callers threaded before the build gate. 5. runs-family plateau E2E golden. Two plan-time forks were derived and recorded on #145 (comment 1974): the lattice surfaces at the engine terminal (only the post-resolve_axes GridSpace holds param_space order), and the RandomSpace-refuse is a structural guard unit-tested at select_winner (walkforward has no --random producer to E2E). refs #145 --- .../plans/0077-plateau-over-peak-selection.md | 974 ++++++++++++++++++ 1 file changed, 974 insertions(+) create mode 100644 docs/plans/0077-plateau-over-peak-selection.md diff --git a/docs/plans/0077-plateau-over-peak-selection.md b/docs/plans/0077-plateau-over-peak-selection.md new file mode 100644 index 0000000..284e2c3 --- /dev/null +++ b/docs/plans/0077-plateau-over-peak-selection.md @@ -0,0 +1,974 @@ +# Prefer a robust parameter plateau over the in-sample peak — Implementation Plan + +> **Parent spec:** `docs/specs/0077-plateau-over-peak-selection.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to +> run this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Add an opt-in `optimize_plateau` selection objective (argmax over the +grid-neighbourhood-smoothed metric) reachable via a new `--select` flag, recorded +on the `RunManifest.selection` carrier #144 introduced; default `argmax` stays +byte-identical (C23). + +**Architecture:** A new family selector lives beside `optimize_deflated` in +`aura-registry` (C9), reading each member's metric via the shared `metric_value` +and smoothing it over a mixed-radix grid neighbourhood built from `axis_lens` +(the `GridSpace` odometer radixes). The lattice is surfaced from the engine via a +new `SweepBinder::sweep_with_lattice` terminal (`sweep` delegates to it, +byte-unchanged for all existing callers). The shared `FamilySelection` carrier is +reshaped — orthogonal rule (`SelectionMode`) × annotation (deflation fields → +`Option`, plus plateau fields) — and the CLI threads a `Selection` from +`--select` into `walkforward_family`. + +**Tech Stack:** `aura-engine` (`report.rs` carrier, `sweep.rs` accessor, +`blueprint.rs` terminal), `aura-registry` (`lib.rs` selector + index math), +`aura-cli` (`main.rs` flag/dispatch/display, `tests/cli_run.rs` E2E). + +--- + +## Design decisions resolved at plan time (recorded on #145, comment 1974) + +1. **Lattice seam → engine `SweepBinder::sweep_with_lattice`.** Only the engine's + post-`resolve_axes` `GridSpace` holds the lattice in `param_space()`/odometer + order, which `closed_neighbourhood`'s mixed-radix arithmetic requires; a sibling + terminal (`sweep` delegates) contains the blast radius — every existing + `.sweep()` caller stays byte-identical. +2. **RandomSpace-refuse → structural guard, no new feature.** `walkforward` has no + `--random` flag (verified: `--random`/`RandomSpace`/`RandomBinder` appear + nowhere in `aura-cli`); wiring a random walk-forward producer is unspecified, + out-of-scope. So `sweep_over`/`stage1_r_sweep_over` return + `(SweepFamily, Option>)` (`Some` today, grid-only), and a + `select_winner` helper returns the refuse on `Plateau + None`-lattice; the + caller prints the spec's message and exits 2. **Unit-tested** at the helper + level (the `Err` path), not via a `walkforward --random` E2E. +3. **`FamilySelection.seed` → `Option`** (spec line 135; the recon carrier's + `Option` was a transcription slip). + +## Files this plan creates or modifies + +- Modify: `crates/aura-engine/src/report.rs:41-66` — reshape `SelectionMode` + + `FamilySelection`; `:762-775` round-trip test literal; new plateau-shape test. +- Modify: `crates/aura-engine/src/sweep.rs` — new `GridSpace::axis_lens()` + test. +- Modify: `crates/aura-engine/src/blueprint.rs:399-411` — new + `SweepBinder::sweep_with_lattice`; `sweep` delegates; new test. +- Modify: `crates/aura-registry/src/lib.rs:308-312` — `optimize_deflated` + construction to `Option` deflation fields; `:163-173` `metric_cmp` refactor to + `higher_is_better`; `:744-794` test assertions → `.unwrap()`/`Some`; add + `PlateauMode`, `higher_is_better`, `closed_neighbourhood`, `optimize_plateau` + + tests. +- Modify: `crates/aura-cli/src/main.rs` — import `SelectionMode` (engine), + `optimize_plateau`/`PlateauMode` (registry); `Selection` enum; `parse_select` + + `--select` in `parse_walkforward_args`; `select_winner`; `sweep_over` / + `stage1_r_sweep_over` tuple returns; `walkforward_family` `+select` dispatch; + `run_walkforward` `+select`; the dispatch call site; `runs_family` mode-aware + display; 4 test callers of `walkforward_family`; unit tests. +- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`. +- Unchanged (preservation gate): `crates/aura-registry/src/compat.rs:34` embeds + `FamilySelection` by value — the reshape flows through with no edit; the + legacy-load test `load_lifts_a_selection_block_through_the_compat_mirror` + (`lib.rs:520`) and `runs_family_rank_shows_deflated_line` (`cli_run.rs:2358`) + must stay green. + +--- + +## Task 1: Reshape `FamilySelection` / `SelectionMode` (mechanical, byte-preserving across 3 crates) + +**Files:** +- Modify: `crates/aura-engine/src/report.rs:41-66`, `:762-775` +- Modify: `crates/aura-registry/src/lib.rs:308-312`, `:744-794` +- Modify: `crates/aura-cli/src/main.rs:19-25` (import), `:2215-2223` (display) + +This is the cross-crate compile-gate. The reshape breaks every construction and +field-read site; all are threaded here. No new runtime behaviour — the argmax +path stays byte-identical; the one new test pins the new fields' serde contract. + +- [ ] **Step 1: Write the failing plateau-shape serde test (RED)** + +In `crates/aura-engine/src/report.rs`, in the `#[cfg(test)] mod tests` block, +beside `family_selection_round_trips_on_the_manifest` (after line 775), add: + +```rust + #[test] + fn family_selection_plateau_shape_round_trips() { + let sel = FamilySelection { + selection_metric: "sqn_normalized".into(), n_trials: 16, raw_winner_metric: 1.42, + mode: SelectionMode::PlateauMean, + deflated_score: None, overfit_probability: None, + n_resamples: None, block_len: None, seed: None, + neighbourhood_score: Some(1.27), n_neighbours: Some(5), + }; + let json = serde_json::to_string(&sel).unwrap(); + // plateau annotation present; deflation fields omitted (skip_serializing_if) + assert!(json.contains("\"neighbourhood_score\":1.27"), "json: {json}"); + assert!(json.contains("\"n_neighbours\":5"), "json: {json}"); + assert!(!json.contains("deflated_score"), "deflation fields omitted under plateau: {json}"); + assert!(!json.contains("n_resamples"), "json: {json}"); + let back: FamilySelection = serde_json::from_str(&json).unwrap(); + assert_eq!(back, sel); + } +``` + +- [ ] **Step 2: Run the new test to verify it fails to compile** + +Run: `cargo test -p aura-engine family_selection_plateau` +Expected: FAIL — compile error (`SelectionMode::PlateauMean` / the `neighbourhood_score`/`n_neighbours` fields do not exist yet). + +- [ ] **Step 3: Reshape `SelectionMode` and `FamilySelection`** + +In `crates/aura-engine/src/report.rs`, replace the `SelectionMode` enum + doc +(lines 41-48) and the `FamilySelection` struct + doc (lines 50-66) with: + +```rust +/// Which selection objective produced the record (additive provenance, C23). +/// `Argmax` is the bare-best pick (cycle 0076), deflated for the number of trials. +/// `PlateauMean` / `PlateauWorst` (cycle 0077) argmax the neighbourhood-smoothed +/// surface instead, so peak-vs-plateau runs stay distinguishable on this field. +#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub enum SelectionMode { + Argmax, + PlateauMean, + PlateauWorst, +} + +/// Selection-provenance for a sweep winner. The selection RULE (`mode`) and its +/// ANNOTATION are orthogonal: `Argmax` carries the trials-deflation annotation +/// (`deflated_score` / `overfit_probability` / `n_resamples` / `block_len` / +/// `seed`); `Plateau*` carries the smoothing annotation (`neighbourhood_score` / +/// `n_neighbours`). Each annotation is `Option`, present iff its rule produced the +/// record; all are additive — recorded, never re-ranking (C23). A legacy line +/// (pre-0077) carries the deflation scalars as bare values that deserialize to +/// `Some` (serde default), so it still loads (C14/C18). +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct FamilySelection { + pub selection_metric: String, + pub n_trials: usize, + pub raw_winner_metric: f64, + pub mode: SelectionMode, + // deflation annotation (present iff mode == Argmax with a deflation run) + #[serde(default, skip_serializing_if = "Option::is_none")] + pub deflated_score: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub overfit_probability: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n_resamples: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub block_len: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub seed: Option, + // plateau annotation (present iff mode is Plateau*) + #[serde(default, skip_serializing_if = "Option::is_none")] + pub neighbourhood_score: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n_neighbours: Option, +} +``` + +- [ ] **Step 4: Update the round-trip test literal** + +In `crates/aura-engine/src/report.rs`, in `family_selection_round_trips_on_the_manifest`, +replace the `let sel = FamilySelection { … };` literal (lines 764-768) with: + +```rust + let sel = FamilySelection { + selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83, + mode: SelectionMode::Argmax, + deflated_score: Some(0.21), overfit_probability: Some(0.06), + n_resamples: Some(1000), block_len: Some(5), seed: Some(42), + neighbourhood_score: None, n_neighbours: None, + }; +``` + +- [ ] **Step 5: Run the engine tests (GREEN, engine internally consistent)** + +Run: `cargo test -p aura-engine family_selection` +Expected: PASS — both `family_selection_round_trips_on_the_manifest` and +`family_selection_plateau_shape_round_trips` green. (aura-engine does not depend +on aura-registry, so its build is consistent even though the registry is not yet +updated.) + +- [ ] **Step 6: Update `optimize_deflated`'s construction to the Option fields** + +In `crates/aura-registry/src/lib.rs`, replace the returned `FamilySelection` +literal (lines 308-312) with: + +```rust + Ok((winner.clone(), FamilySelection { + selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, + mode: SelectionMode::Argmax, + deflated_score: Some(deflated_score), overfit_probability, + n_resamples: Some(n_resamples), block_len: Some(block_len), seed: Some(seed), + neighbourhood_score: None, n_neighbours: None, + })) +``` + +- [ ] **Step 7: Update the `optimize_deflated_*` test assertions to `.unwrap()` / `Some`** + +In `crates/aura-registry/src/lib.rs`, apply these exact line edits: + +- Line 749: ` assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw` + → ` assert!(sel.deflated_score.unwrap() <= sel.raw_winner_metric); // dispersion floor ≤ raw` +- Line 762: ` assert_eq!(sel.n_resamples, 0);` + → ` assert_eq!(sel.n_resamples, Some(0));` +- Line 763: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric, "p95 of an empty null is 0");` + → ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric, "p95 of an empty null is 0");` +- Line 782: ` assert!(sel.deflated_score.is_finite(), "deflated_score must be finite, got {}", sel.deflated_score);` + → ` assert!(sel.deflated_score.unwrap().is_finite(), "deflated_score must be finite, got {:?}", sel.deflated_score);` +- Line 783: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric);` + → ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric);` +- Line 794: ` assert!(sel.deflated_score > 0.0);` + → ` assert!(sel.deflated_score.unwrap() > 0.0);` + +(The compat test `load_lifts_a_selection_block_through_the_compat_mirror` at line +520-527 is **unchanged** — its legacy JSON line carries the deflation scalars as +bare floats that deserialize to `Some` under the new `Option` + `#[serde(default)]`, +and its assertions read only `n_trials`/`mode`. It is the C14/C18 legacy-load gate.) + +- [ ] **Step 8: Run the registry tests (GREEN)** + +Run: `cargo test -p aura-registry optimize_deflated` +Expected: PASS — all nine `optimize_deflated_*` tests green against the reshaped fields. + +Run: `cargo test -p aura-registry load_lifts` +Expected: PASS — the legacy `selection` block still lifts through the compat mirror. + +- [ ] **Step 9: Import `SelectionMode` and rewrite the `runs_family` display (mode-aware, argmax byte-preserved)** + +In `crates/aura-cli/src/main.rs`, add `SelectionMode` to the `aura_engine` import +list (lines 19-24) — e.g. extend the `RunManifest, RunMetrics,` line to +`RunManifest, RunMetrics, SelectionMode,` (any slot in the brace list is fine). + +Then replace the selection-display block in `runs_family` (lines 2215-2223): + +```rust + for report in &ordered { + println!("{}", report.to_json()); + if let Some(sel) = &report.manifest.selection { + match sel.mode { + SelectionMode::Argmax => match sel.overfit_probability { + Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score.unwrap(), p), + None => println!(" deflated={:.4}", sel.deflated_score.unwrap()), + }, + SelectionMode::PlateauMean | SelectionMode::PlateauWorst => { + let label = if matches!(sel.mode, SelectionMode::PlateauMean) { "mean" } else { "worst" }; + if let (Some(score), Some(n)) = (sel.neighbourhood_score, sel.n_neighbours) { + println!(" plateau({label})={score:.4} over {n} cells"); + } + } + } + } + } +``` + +(Under `Argmax`, `deflated_score` is always `Some` by construction, so `.unwrap()` +preserves the exact pre-0077 bytes — the `deflated=…`/`P(overfit)=…` golden stays +green. The `Plateau*` branch is exercised by the Task 5 E2E.) + +- [ ] **Step 10: Workspace build + full per-crate regression gate** + +Run: `cargo build --workspace` +Expected: clean (0 errors) — all three crates consistent. + +Run: `cargo test -p aura-engine` +Expected: PASS (no regressions). + +Run: `cargo test -p aura-registry` +Expected: PASS (no regressions). + +--- + +## Task 2: `GridSpace::axis_lens()` + `SweepBinder::sweep_with_lattice` (engine, additive) + +**Files:** +- Modify: `crates/aura-engine/src/sweep.rs` (accessor + test) +- Modify: `crates/aura-engine/src/blueprint.rs:399-411` (terminal + delegate + test) + +- [ ] **Step 1: Write the failing `axis_lens` golden (RED)** + +In `crates/aura-engine/src/sweep.rs`, in the `#[cfg(test)] mod tests` block (which +already imports `aura_core::{Cell, ParamSpec, Scalar, ScalarKind, Timestamp}`), +add: + +```rust + #[test] + fn grid_axis_lens_are_the_per_axis_radixes() { + let space = vec![ + ParamSpec { name: "a".into(), kind: ScalarKind::I64 }, + ParamSpec { name: "b".into(), kind: ScalarKind::I64 }, + ]; + let grid = GridSpace::new(&space, vec![ + vec![Scalar::i64(10), Scalar::i64(20)], // axis 0: 2 values + vec![Scalar::i64(1), Scalar::i64(2), Scalar::i64(3)], // axis 1: 3 values + ]).expect("2x3 grid"); + assert_eq!(grid.axis_lens(), vec![2, 3]); + assert_eq!(grid.axis_lens().iter().product::(), grid.len()); + } +``` + +- [ ] **Step 2: Run to verify it fails to compile** + +Run: `cargo test -p aura-engine grid_axis_lens` +Expected: FAIL — `no method named axis_lens found for struct GridSpace`. + +- [ ] **Step 3: Add the `axis_lens` accessor** + +In `crates/aura-engine/src/sweep.rs`, in the `impl GridSpace` block, immediately +after `is_empty` (after line 65), add: + +```rust + /// Per-axis cardinalities in `param_space()` order (the odometer radixes, + /// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau + /// neighbourhood walks (cycle 0077). + pub fn axis_lens(&self) -> Vec { + self.axes.iter().map(Vec::len).collect() + } +``` + +- [ ] **Step 4: Run the accessor test (GREEN)** + +Run: `cargo test -p aura-engine grid_axis_lens` +Expected: PASS. + +- [ ] **Step 5: Write the failing `sweep_with_lattice` test (RED)** + +In `crates/aura-engine/src/blueprint.rs`, in the `#[cfg(test)] mod tests` block +(which has `composite_sma_cross_harness()` and `run_point`), add: + +```rust + #[test] + fn sweep_with_lattice_surfaces_grid_radixes_in_param_space_order() { + let bp = composite_sma_cross_harness().0; + let (fam, lattice) = bp + .axis("sma_cross.fast.length", vec![Scalar::i64(2), Scalar::i64(3)]) // 2 values + .axis("sma_cross.slow.length", vec![Scalar::i64(4), Scalar::i64(5)]) // 2 values + .axis("bias.scale", vec![Scalar::f64(0.5)]) // 1 value + .sweep_with_lattice(run_point) + .expect("named binding resolves and runs"); + assert_eq!(lattice, vec![2, 2, 1], "radixes in param_space slot order"); + assert_eq!(lattice.iter().product::(), fam.points.len()); // 4 + } +``` + +- [ ] **Step 6: Run to verify it fails to compile** + +Run: `cargo test -p aura-engine sweep_with_lattice` +Expected: FAIL — `no method named sweep_with_lattice found`. + +- [ ] **Step 7: Add `sweep_with_lattice`; refactor `sweep` to delegate** + +In `crates/aura-engine/src/blueprint.rs`, replace the `SweepBinder::sweep` method +(lines 399-411) with: + +```rust + /// Resolve the named axes against `param_space()` into a positional grid and + /// run the disjoint sweep. `sweep` is [`SweepBinder::sweep_with_lattice`] with + /// the lattice dropped, so every existing caller is byte-unchanged. + pub fn sweep(self, run_one: F) -> Result + where + F: Fn(&[Cell]) -> RunReport + Sync, + { + self.sweep_with_lattice(run_one).map(|(family, _lattice)| family) + } + + /// As [`SweepBinder::sweep`], plus the grid's per-axis radixes (`axis_lens`, + /// in `param_space()` / odometer order). The lattice is what a plateau + /// neighbourhood walk needs (cycle 0077); only the engine's post-`resolve_axes` + /// grid holds it in the correct order. + pub fn sweep_with_lattice(self, run_one: F) -> Result<(SweepFamily, Vec), BindError> + where + F: Fn(&[Cell]) -> RunReport + Sync, + { + let space = self.bp.param_space(); + check_param_namespace_injective(&space).map_err(BindError::Compile)?; + let ordered = resolve_axes(&space, &self.axes)?; + let grid = GridSpace::new(&space, ordered) + .expect("named layer pre-validates arity/kind/non-empty"); + let lattice = grid.axis_lens(); + Ok((sweep(&grid, run_one), lattice)) + } +``` + +- [ ] **Step 8: Run the new test + the existing sweep tests (GREEN, delegation preserved)** + +Run: `cargo test -p aura-engine sweep_with_lattice` +Expected: PASS. + +Run: `cargo test -p aura-engine` +Expected: PASS — every existing `.sweep()` test (named-axes parity, grid goldens) +stays green; `sweep` delegating to `sweep_with_lattice` is behaviour-preserving. + +--- + +## Task 3: `optimize_plateau` + `PlateauMode` + `closed_neighbourhood` (registry, additive) + +**Files:** +- Modify: `crates/aura-registry/src/lib.rs:163-173` (`metric_cmp` → `higher_is_better`), + plus new `PlateauMode`, `higher_is_better`, `closed_neighbourhood`, + `optimize_plateau` and their tests. + +- [ ] **Step 1: Write the failing plateau tests (RED)** + +In `crates/aura-registry/src/lib.rs`, in the `#[cfg(test)] mod tests` block (which +has the `member(total_pips, trade_rs)` helper), add: + +```rust + /// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle + /// plateau in `total_pips`. Bare argmax takes an end spike; the plateau-mean + /// argmax takes the interior plateau centre. + fn fixture_grid_spike_vs_plateau() -> SweepFamily { + let pips = [80.0, 5.0, 48.0, 50.0, 49.0, 5.0, 80.0]; + SweepFamily { space: vec![], points: pips.iter().map(|&p| member(p, vec![1.0])).collect() } + } + + #[test] + fn closed_neighbourhood_2x3_golden() { + // 2x3 lattice, odometer (last axis fastest): + // (0,0)=0 (0,1)=1 (0,2)=2 + // (1,0)=3 (1,1)=4 (1,2)=5 + let sorted = |i: usize| { let mut v = closed_neighbourhood(i, &[2, 3]); v.sort_unstable(); v }; + assert_eq!(sorted(0), vec![0, 1, 3]); // corner (0,0): self + 2 + assert_eq!(sorted(4), vec![1, 3, 4, 5]); // interior (1,1): self + 3 + assert_eq!(sorted(2), vec![1, 2, 5]); // corner (0,2): self + 2 + } + + #[test] + fn plateau_picks_the_plateau_not_the_spike() { + let fam = fixture_grid_spike_vs_plateau(); + let spike = optimize(&fam, "total_pips").unwrap(); + let (centre, sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap(); + assert_ne!(centre.report.metrics.total_pips, spike.report.metrics.total_pips); + assert_eq!(centre.report.metrics.total_pips, 50.0, "plateau centre is the middle cell"); + assert_eq!(sel.mode, SelectionMode::PlateauMean); + assert_eq!(sel.raw_winner_metric, 50.0); + assert!(sel.neighbourhood_score.unwrap() < spike.report.metrics.total_pips, + "smoothed score is below the spike's raw peak"); + assert_eq!(sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours"); + assert!(sel.deflated_score.is_none() && sel.n_resamples.is_none(), + "deflation fields omitted under plateau"); + } + + #[test] + fn plateau_worst_is_more_conservative_than_mean() { + let fam = fixture_grid_spike_vs_plateau(); + let (_, mean_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap(); + let (_, worst_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap(); + // Worst scores each cell by its weakest neighbour, never above the mean. + assert!(worst_sel.neighbourhood_score.unwrap() <= mean_sel.neighbourhood_score.unwrap()); + assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst); + assert!(worst_sel.n_neighbours.unwrap() >= 2, "winner is an interior cell"); + } + + #[test] + fn plateau_single_member_equals_optimize() { + let fam = SweepFamily { space: vec![], points: vec![member(42.0, vec![1.0])] }; + let plain = optimize(&fam, "total_pips").unwrap(); + let (winner, sel) = optimize_plateau(&fam, &[1], "total_pips", PlateauMode::Mean).unwrap(); + assert_eq!(winner.report.metrics.total_pips, plain.report.metrics.total_pips); + assert_eq!(sel.n_neighbours, Some(1), "a 1-cell grid's closed neighbourhood is itself"); + assert_eq!(sel.neighbourhood_score, Some(42.0)); + assert_eq!(sel.raw_winner_metric, 42.0); + } + + #[test] + fn optimize_plateau_is_deterministic() { + let fam = fixture_grid_spike_vs_plateau(); + let a = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1; + let b = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1; + assert_eq!(a, b); + } +``` + +- [ ] **Step 2: Run to verify they fail to compile** + +Run: `cargo test -p aura-registry closed_neighbourhood` +Expected: FAIL — `closed_neighbourhood` / `optimize_plateau` / `PlateauMode` do not exist. + +- [ ] **Step 3: Add `higher_is_better` and refactor `metric_cmp` to use it** + +In `crates/aura-registry/src/lib.rs`, immediately after `is_r_metric` (after line +207), add: + +```rust +/// The metric's optimisation direction: `true` if a larger value is better +/// (`total_pips`, the four R keys), `false` for the lower-is-better keys +/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp` +/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour) +/// both read it, so the per-metric sense lives in exactly one place. +fn higher_is_better(m: Metric) -> bool { + !matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips) +} +``` + +Then replace the `metric_cmp` body (lines 163-173) with: + +```rust +fn metric_cmp(metric: &str) -> Result Ordering, RegistryError> { + let m = resolve_metric(metric)?; + let hib = higher_is_better(m); + Ok(move |a: &RunReport, b: &RunReport| { + let (va, vb) = (metric_value(a, m), metric_value(b, m)); + // higher-is-better ranks the larger value first; lower-is-better the smaller. + if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) } + }) +} +``` + +(Behaviour-identical: `rank_by` / `optimize` and their existing tests are the gate.) + +- [ ] **Step 4: Add `closed_neighbourhood`** + +In `crates/aura-registry/src/lib.rs`, after `higher_is_better`, add: + +```rust +/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice +/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}` +/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order +/// is `i` first then per-axis neighbours; callers use the result only as a set and +/// for its length, so the order is not load-bearing. +fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec { + // decompose i to per-axis coords (last axis fastest) + let mut coords = vec![0usize; axis_lens.len()]; + let mut rem = i; + for k in (0..axis_lens.len()).rev() { + coords[k] = rem % axis_lens[k]; + rem /= axis_lens[k]; + } + // recompose a coord vector to its flat index (Horner, last axis fastest) + let recompose = |c: &[usize]| c.iter().zip(axis_lens).fold(0usize, |flat, (&ck, &len)| flat * len + ck); + let mut out = vec![i]; + for k in 0..axis_lens.len() { + for delta in [-1isize, 1] { + let ck = coords[k] as isize + delta; + if ck < 0 || ck as usize >= axis_lens[k] { continue; } + let mut nb = coords.clone(); + nb[k] = ck as usize; + out.push(recompose(&nb)); + } + } + out +} +``` + +- [ ] **Step 5: Add `PlateauMode` and `optimize_plateau`** + +In `crates/aura-registry/src/lib.rs`, immediately before `optimize_deflated` (before +line 258, before its doc comment), add: + +```rust +/// Plateau-selection aggregate: how a member's grid-neighbourhood metric is +/// reduced to one smoothed score before the argmax. `Mean` averages the closed +/// neighbourhood; `Worst` takes the most-pessimistic neighbour by the metric's +/// direction (biasing toward interior cells, which keep more neighbours). +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum PlateauMode { + Mean, + Worst, +} + +/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau +/// provenance. Each member scores as the mean (or worst-case) of its closed grid +/// neighbourhood's `metric_value`; the winner is the best smoothed score by the +/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are +/// the grid radixes (`param_space()` order, last axis fastest). Pure, +/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL; +/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the +/// smoothed value the argmax maximised. +pub fn optimize_plateau( + family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode, +) -> Result<(SweepPoint, FamilySelection), RegistryError> { + let m = resolve_metric(metric)?; + let n = family.points.len(); + debug_assert_eq!( + axis_lens.iter().product::(), n, + "axis_lens product must equal the family size (a valid grid lattice)", + ); + let hib = higher_is_better(m); + // smoothed score + closed-neighbourhood size per member, in odometer order + let scored: Vec<(usize, f64, usize)> = (0..n) + .map(|i| { + let nbrs = closed_neighbourhood(i, axis_lens); + let vals: Vec = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect(); + let score = match mode { + PlateauMode::Mean => vals.iter().sum::() / vals.len() as f64, + PlateauMode::Worst => { + if hib { + vals.iter().copied().fold(f64::INFINITY, f64::min) + } else { + vals.iter().copied().fold(f64::NEG_INFINITY, f64::max) + } + } + }; + (i, score, nbrs.len()) + }) + .collect(); + // argmax the smoothed surface by direction; earliest-odometer tie (only a + // strictly-better later cell displaces the incumbent, as `optimize`). + let &(wi, wscore, wn) = scored + .iter() + .reduce(|best, cur| { + let better = if hib { cur.1 > best.1 } else { cur.1 < best.1 }; + if better { cur } else { best } + }) + .expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)"); + let winner = family.points[wi].clone(); + let raw = metric_value(&winner.report, m); + Ok(( + winner, + FamilySelection { + selection_metric: metric.to_string(), + n_trials: n, + raw_winner_metric: raw, + mode: match mode { + PlateauMode::Mean => SelectionMode::PlateauMean, + PlateauMode::Worst => SelectionMode::PlateauWorst, + }, + deflated_score: None, + overfit_probability: None, + n_resamples: None, + block_len: None, + seed: None, + neighbourhood_score: Some(wscore), + n_neighbours: Some(wn), + }, + )) +} +``` + +- [ ] **Step 6: Run the plateau tests + the existing rank/optimize gate (GREEN)** + +Run: `cargo test -p aura-registry plateau` +Expected: PASS — `plateau_picks_the_plateau_not_the_spike`, +`plateau_worst_is_more_conservative_than_mean`, `plateau_single_member_equals_optimize`, +`optimize_plateau_is_deterministic` all green. + +Run: `cargo test -p aura-registry closed_neighbourhood` +Expected: PASS. + +Run: `cargo test -p aura-registry` +Expected: PASS — `rank_by` / `optimize` / `optimize_deflated` stay green after the +`metric_cmp` → `higher_is_better` refactor. + +--- + +## Task 4: CLI `--select` parse + threading + sweep-entry tuples + `select_winner` (compile-gate) + +**Files:** +- Modify: `crates/aura-cli/src/main.rs` — `aura_registry` import; `Selection` enum; + `parse_select`; `parse_walkforward_args` (`+--select`, 6-tuple); the dispatch + call site (`:2935`); `run_walkforward` (`+select`); `walkforward_family` + (`+select` dispatch via `select_winner`); `sweep_over` / `stage1_r_sweep_over` + (tuple returns); `select_winner`; the 4 test callers of `walkforward_family`; + unit tests. + +This is the CLI compile-gate: the signature changes break their callers, all +threaded here. Default `Argmax` keeps the existing walk-forward output +byte-identical (C23). + +- [ ] **Step 1: Write the failing `select_winner` refuse + parse tests (RED)** + +In `crates/aura-cli/src/main.rs`, in the `#[cfg(test)] mod tests` block at line +2965, add: + +```rust + #[test] + fn select_winner_refuses_plateau_without_a_lattice() { + // A plateau request with no lattice (a random sweep would yield None) is + // refused, never silently argmaxed. The refuse short-circuits before the + // family is read, so an empty family is fine here. + let fam = SweepFamily { space: vec![], points: vec![] }; + let err = select_winner(&fam, "total_pips", Selection::Plateau(PlateauMode::Mean), None) + .unwrap_err(); + assert!(err.contains("requires a grid sweep"), "refuse message: {err}"); + } + + #[test] + fn parse_walkforward_select_flag() { + let argmax = parse_walkforward_args(&["--strategy", "stage1-r"]).unwrap(); + assert!(matches!(argmax.5, Selection::Argmax), "default is argmax"); + let mean = parse_walkforward_args(&["--select", "plateau:mean"]).unwrap(); + assert!(matches!(mean.5, Selection::Plateau(PlateauMode::Mean))); + let worst = parse_walkforward_args(&["--select", "plateau:worst"]).unwrap(); + assert!(matches!(worst.5, Selection::Plateau(PlateauMode::Worst))); + assert!(parse_walkforward_args(&["--select", "bogus"]).is_err(), + "unknown --select token is a usage error"); + } +``` + +- [ ] **Step 2: Run to verify they fail to compile** + +Run: `cargo test -p aura-cli select_winner_refuses` +Expected: FAIL — `Selection` / `PlateauMode` / `select_winner` not in scope; the +`parse_walkforward_args` tuple has no `.5`. + +- [ ] **Step 3: Imports + add the `Selection` enum and `parse_select`** + +In `crates/aura-cli/src/main.rs`, extend the `aura_registry` import (lines 26-30) — +add `optimize_plateau, PlateauMode,` to the brace list (alongside `optimize_deflated`). +Also add `FamilySelection` to the `aura_engine` import list (lines 19-24; it is not +currently imported, and `select_winner` in Step 4 names it). (`SelectionMode` was +already added to that list in Task 1 Step 9.) + +Then, immediately after the `Strategy` enum (after line 1551's enum block; place it +near the other CLI enums), add: + +```rust +/// In-sample winner-selection objective for walk-forward (cycle 0077). `Argmax` is +/// the bare-best pick deflated for trials (#144, the default); `Plateau` argmaxes +/// the neighbourhood-smoothed surface instead (opt-in via `--select`). +#[derive(Clone, Copy)] +enum Selection { + Argmax, + Plateau(PlateauMode), +} + +/// Parse a `--select` token: `argmax` | `plateau:mean` | `plateau:worst`. Unknown +/// tokens are a usage error (the caller maps `Err(())` to exit 2). +fn parse_select(s: &str) -> Result { + match s { + "argmax" => Ok(Selection::Argmax), + "plateau:mean" => Ok(Selection::Plateau(PlateauMode::Mean)), + "plateau:worst" => Ok(Selection::Plateau(PlateauMode::Worst)), + _ => Err(()), + } +} +``` + +- [ ] **Step 4: Add the `select_winner` dispatch helper** + +In `crates/aura-cli/src/main.rs`, immediately before `walkforward_family` (before +line 1786), add: + +```rust +/// Resolve the in-sample winner under the chosen selection objective. `Argmax` +/// defers to the trials-deflation pick (#144). `Plateau` argmaxes the smoothed grid +/// surface — it needs the grid lattice, so a sweep with no lattice (a future random +/// walk-forward producer) is refused rather than silently argmaxed. The metric is +/// always known at the call sites, so a metric error is unreachable (`expect`); the +/// only fallible outcome is the plateau-without-lattice refusal, returned as +/// `Err(message)` for the caller to print and exit 2. +fn select_winner( + family: &SweepFamily, metric: &str, select: Selection, lattice: Option<&[usize]>, +) -> Result<(SweepPoint, FamilySelection), String> { + match select { + Selection::Argmax => Ok(optimize_deflated( + family, metric, DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, + ).expect("walk-forward metrics are known")), + Selection::Plateau(mode) => match lattice { + Some(lens) => Ok(optimize_plateau(family, lens, metric, mode) + .expect("walk-forward metrics are known")), + None => Err( + "--select plateau requires a grid sweep; a random sweep has no parameter lattice" + .to_string(), + ), + }, + } +} +``` + +(`select_winner` names `FamilySelection`, which is **not** currently imported in +`main.rs` — Step 3 above adds it to the `aura_engine` import list.) + +- [ ] **Step 5: Switch `sweep_over` / `stage1_r_sweep_over` to return `(SweepFamily, Option>)`** + +In `crates/aura-cli/src/main.rs`: + +`stage1_r_sweep_over` (line 1301): change the signature return type +`-> SweepFamily` to `-> (SweepFamily, Option>)`; change its terminal +`.sweep(|point| {` (line 1318) to `.sweep_with_lattice(|point| {`, and insert a +`.map(...)` line before the existing trailing `.expect(...)` (line 1342, currently +` .expect("the stage1-r named grid matches the stage1-r param-space")`). The +closure's closing ` })` and the existing `.expect(...)` line both stay; the +new `.map` goes between them, so the function tail reads: + +```rust + }) + .map(|(fam, lat)| (fam, Some(lat))) + .expect("the stage1-r named grid matches the stage1-r param-space") +``` + +`sweep_over` (line 1841): change the signature return type `-> SweepFamily` to +`-> (SweepFamily, Option>)`; change its terminal `.sweep(|point| {` +(line 1854) to `.sweep_with_lattice(|point| {`, and insert the `.map(...)` line +before the existing trailing +` .expect("the built-in named grid matches the sample param-space")` (line +1869), so the function tail reads: + +```rust + }) + .map(|(fam, lat)| (fam, Some(lat))) + .expect("the built-in named grid matches the sample param-space") +``` + +- [ ] **Step 6: Thread `Selection` through `walkforward_family` (signature + both arms)** + +In `crates/aura-cli/src/main.rs`, change `walkforward_family`'s signature (line +1786-1788) to add the `select` parameter: + +```rust +fn walkforward_family( + strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid, + select: Selection, +) -> WalkForwardResult { +``` + +In the `Strategy::SmaCross` arm, replace the sweep + optimize lines (1802-1807): + +```rust + let (is_family, lattice) = sweep_over(w.is.0, w.is.1, data); + let (best, selection) = match select_winner(&is_family, "total_pips", select, lattice.as_deref()) { + Ok(v) => v, + Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } + }; + let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); + oos_report.manifest.selection = Some(selection); +``` + +In the `Strategy::Stage1R` arm, replace the sweep + optimize lines (1820-1825): + +```rust + let (is_family, lattice) = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); + let (best, selection) = match select_winner(&is_family, "sqn_normalized", select, lattice.as_deref()) { + Ok(v) => v, + Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } + }; + let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); + oos_report.manifest.selection = Some(selection); +``` + +- [ ] **Step 7: Thread the new `select` argument through every `walkforward_family` caller** + +In `crates/aura-cli/src/main.rs`, update the production caller and the four test +callers to pass a `Selection`: + +- `run_walkforward` (line 1755): add `select: Selection` to its signature: + `fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid, select: Selection) {` + and update its `walkforward_family(...)` call (line 1763) to + `let result = walkforward_family(strategy, persist.then_some(name), &data, grid, select);` +- Line 1963: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` +- Line 2131: `walkforward_family(Strategy::Stage1R, None, data, grid, Selection::Argmax)` +- Line 3456: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` +- Line 4131: `walkforward_family(Strategy::Stage1R, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` + +- [ ] **Step 8: Add `--select` to `parse_walkforward_args` and thread it to the dispatch** + +In `crates/aura-cli/src/main.rs`, in `parse_walkforward_args` (line 1648): + +Change the return type to add `Selection`: +`) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid, Selection), String> {` + +Update the usage string to mention `--select` (append `[--select ]` +to both the `///` doc usage line and the `usage` closure string). + +Add `let mut select = Selection::Argmax;` beside the other `let mut` locals. + +Add a match arm for the flag (beside `--stop-k`): +`"--select" => select = parse_select(value).map_err(|()| usage())?,` + +Change the final return to: +`Ok((strategy, name, persist, real.finish(&usage)?, grid, select))` + +In the command dispatch (line 2935-2939), update the destructure + call: + +```rust + ["walkforward", rest @ ..] => match parse_walkforward_args(rest) { + Ok((strategy, name, persist, choice, grid, select)) => { + run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid, select) + } +``` + +- [ ] **Step 9: Compile-gate — workspace + tests build clean** + +Run: `cargo build -p aura-cli --tests` +Expected: clean (0 errors) — every caller of the changed signatures threaded. + +- [ ] **Step 10: Run the new unit tests (GREEN)** + +Run: `cargo test -p aura-cli select_winner_refuses` +Expected: PASS. + +Run: `cargo test -p aura-cli parse_walkforward_select` +Expected: PASS. + +- [ ] **Step 11: C23 — existing walk-forward goldens stay byte-identical** + +Run: `cargo test -p aura-cli walkforward_strategy_stage1_r` +Expected: PASS — argmax path unchanged. + +Run: `cargo test -p aura-cli runs_family_rank_shows_deflated` +Expected: PASS — the deflated display line is byte-identical. + +Run: `cargo test -p aura-cli walkforward_bare_sma` +Expected: PASS — the SMA summary golden is unchanged. + +--- + +## Task 5: `runs family rank` plateau E2E golden + +**Files:** +- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance` + +Tasks 1-4 deliver the behaviour; this pins it end-to-end (C18: the plateau +provenance reaches disk and the display renders it). + +- [ ] **Step 1: Write the plateau E2E (RED until Tasks 1-4 land; GREEN after)** + +In `crates/aura-cli/tests/cli_run.rs`, beside `runs_family_rank_shows_deflated_line` +(after line 2388), add: + +```rust +/// Property: a stage1-r walk-forward run with `--select plateau:mean` selects the +/// neighbourhood-smoothed winner and stamps each OOS manifest with the plateau +/// provenance (`mode = PlateauMean`, `neighbourhood_score`, `n_neighbours`); `runs +/// family … rank` renders the `plateau(mean)=… over N cells` line, and the +/// deflation fields are omitted (orthogonal annotation). End-to-end witness that +/// the opt-in selection rule reaches disk (C18) and the display path renders it. +#[test] +fn walkforward_plateau_select_stamps_plateau_provenance() { + let cwd = temp_cwd("runs-plateau"); + let wf = Command::new(BIN) + .args([ + "walkforward", "--strategy", "stage1-r", "--select", "plateau:mean", + "--fast", "50,100", "--slow", "200,400", + "--stop-length", "14,21", "--stop-k", "2.0,3.0", + "--name", "wf-plat", + ]) + .current_dir(&cwd) + .output() + .expect("spawn walkforward --select plateau:mean"); + assert!( + wf.status.success(), + "walkforward exit: {:?}; stderr: {}", + wf.status, + String::from_utf8_lossy(&wf.stderr) + ); + + let rank = Command::new(BIN) + .args(["runs", "family", "wf-plat-0", "rank", "sqn_normalized"]) + .current_dir(&cwd) + .output() + .expect("spawn runs family wf-plat-0 rank sqn_normalized"); + assert!(rank.status.success(), "rank exit: {:?}", rank.status); + let rank_out = String::from_utf8(rank.stdout).expect("utf-8"); + let _ = std::fs::remove_dir_all(&cwd); + + assert!(rank_out.contains("plateau(mean)="), "rank output missing plateau(mean)=: {rank_out:?}"); + assert!(rank_out.contains(" cells"), "rank output missing 'over N cells': {rank_out:?}"); + assert!(rank_out.contains("\"mode\":\"PlateauMean\""), "manifest carries PlateauMean: {rank_out:?}"); + assert!(!rank_out.contains("\"deflated_score\""), "plateau run omits deflated_score: {rank_out:?}"); +} +``` + +- [ ] **Step 2: Run the plateau E2E (GREEN)** + +Run: `cargo test -p aura-cli walkforward_plateau` +Expected: PASS. + +- [ ] **Step 3: Full CLI suite + workspace + clippy (no regressions)** + +Run: `cargo test -p aura-cli` +Expected: PASS — every existing golden green; the plateau E2E green. + +Run: `cargo test --workspace` +Expected: PASS. + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: clean (no warnings).