From d50a4828ad26c420b01fcdd18300277264303ba9 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 26 Jun 2026 16:42:49 +0200 Subject: [PATCH] =?UTF-8?q?audit(0077):=20cycle=20close=20=E2=80=94=20drif?= =?UTF-8?q?t=20addressed;=20legacy-load=20fixture=20+=20ledger=20sync;=20d?= =?UTF-8?q?rop=20ephemera?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architect drift review over 9c6b9c7..cb32658 returned drift_found (3 items); all resolved here. What holds (architect): C23 default-argmax byte-identical, C9 separation (selector in aura-registry, walk_forward selection-agnostic), C1/C2 (plateau pure/deterministic/in-sample), one higher_is_better direction source. Resolutions: - [medium] Backfilled the pre-0077 FamilySelection legacy-load fixture (report.rs family_selection_loads_a_pre_0077_legacy_line). The feat commit and the struct doc-comment claimed "a legacy line still loads (C14/C18)" as landed, but no test pinned the field-level mapping: bare deflation scalars -> Some(...), absent plateau keys -> None. families.jsonl/runs.jsonl are append-only, so a winner stamped pre-0077 must read back; now pinned. - [low] Synced the ledger's inferential-validation open thread (docs/design/INDEX.md): #145 plateau-over-peak landed (cycle 0077), selection is now a pluggable objective (bare argmax -> trials-deflated -> plateau) on the manifest carrier; only #146 (cross-instrument generalization) remains of the milestone's three pieces. - [medium-high] Removed the cycle ephemera (git rm docs/specs/0077-*, docs/plans/0077-*) per the cycle-close convention. Regression: full workspace suite green and clippy --all-targets -D warnings clean (verified at feat-review time). No baseline scripts configured; the test + clippy gates are the regression gate. Cycle 0077 tidy (clean) — drift-clean, not a milestone close (the milestone fieldtest is a separate deliberate gate). --- crates/aura-engine/src/report.rs | 20 + docs/design/INDEX.md | 18 +- .../plans/0077-plateau-over-peak-selection.md | 974 ------------------ .../specs/0077-plateau-over-peak-selection.md | 301 ------ 4 files changed, 35 insertions(+), 1278 deletions(-) delete mode 100644 docs/plans/0077-plateau-over-peak-selection.md delete mode 100644 docs/specs/0077-plateau-over-peak-selection.md diff --git a/crates/aura-engine/src/report.rs b/crates/aura-engine/src/report.rs index 9abb318..bbd0fbe 100644 --- a/crates/aura-engine/src/report.rs +++ b/crates/aura-engine/src/report.rs @@ -811,6 +811,26 @@ mod tests { assert_eq!(back, sel); } + #[test] + fn family_selection_loads_a_pre_0077_legacy_line() { + // The pre-0077 on-disk shape (#144): the deflation fields as BARE scalars, + // no plateau keys. `families.jsonl` / `runs.jsonl` are append-only, so a + // winner stamped last cycle must still read back — the bare scalars land in + // the `Some(...)` deflation Options, the absent plateau keys default to + // `None` (the C14/C18 back-compat the struct's serde attrs claim). + let legacy = r#"{"selection_metric":"sqn_normalized","n_trials":4,"raw_winner_metric":1.83,"deflated_score":0.21,"overfit_probability":0.06,"mode":"Argmax","n_resamples":1000,"block_len":5,"seed":42}"#; + let sel: FamilySelection = + serde_json::from_str(legacy).expect("a pre-0077 selection line still loads"); + assert_eq!(sel.mode, SelectionMode::Argmax); + assert_eq!(sel.deflated_score, Some(0.21)); + assert_eq!(sel.overfit_probability, Some(0.06)); + assert_eq!(sel.n_resamples, Some(1000)); + assert_eq!(sel.block_len, Some(5)); + assert_eq!(sel.seed, Some(42)); + assert_eq!(sel.neighbourhood_score, None); + assert_eq!(sel.n_neighbours, None); + } + #[test] fn position_action_round_trips_through_i64() { for a in [PositionAction::Buy, PositionAction::Sell, PositionAction::Close] { diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index 8d3d3db..201f5ab 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -1422,7 +1422,8 @@ load-bearing in the flat graph. significance read (#139, landed cycle 0075). Distinct from the two threads above — neither orchestration *composability* (#109) nor a search *policy* (Bayesian/ genetic), but the statistical *validity* of the selection itself. Not a - C-invariant until built; recorded as the World's third (now partly built) half. + C-invariant until built; recorded as the World's third (now two of three pieces + built) half. **Status (cycle 0076):** the trials-deflation piece (#144) landed — `optimize_deflated` (aura-registry) wraps `optimize` and records, additively on the winning member's manifest (C18, the new `RunManifest.selection`), a deflated @@ -1430,8 +1431,19 @@ load-bearing in the flat graph. member wins** (C23; `optimize` / `rank_by` stay a bare argmax — the deflation is recorded provenance, not a re-ranking). The R arm is a centred moving-block reality-check (reusing the `r_bootstrap` kernel); the `total_pips` arm a - closed-form expected-max-of-K dispersion floor. #145 (plateau-over-peak) and #146 - (cross-instrument generalization) remain open. + closed-form expected-max-of-K dispersion floor. **Status (cycle 0077):** the + plateau-over-peak piece (#145) landed — `optimize_plateau` (aura-registry) + argmaxes the **neighbourhood-smoothed** metric surface (mean or worst-case over + each member's closed mixed-radix grid neighbourhood) instead of the bare peak, + recorded on the same `RunManifest.selection` carrier, now an orthogonal *rule × + annotation*: `SelectionMode::{Argmax, PlateauMean, PlateauWorst}` with either the + deflation annotation (#144) or the plateau annotation (`neighbourhood_score` / + `n_neighbours`). It is opt-in via a `--select` flag — default argmax stays + byte-identical (C23). The grid lattice surfaces from the engine + (`SweepBinder::sweep_with_lattice`); the policy stays in aura-registry with + `walk_forward` selection-agnostic (C9). Selection is now a *pluggable objective* + (bare argmax → trials-deflated → plateau), not a single hard-wired argmax. #146 + (cross-instrument generalization) remains the last open piece of the milestone. - **`aura-std` contents** — the crate exists (doc-only); which universal blocks land first follows the walking-skeleton's needs. - **`strategies/` split** — a later split, *inside a project*, of top-level diff --git a/docs/plans/0077-plateau-over-peak-selection.md b/docs/plans/0077-plateau-over-peak-selection.md deleted file mode 100644 index 284e2c3..0000000 --- a/docs/plans/0077-plateau-over-peak-selection.md +++ /dev/null @@ -1,974 +0,0 @@ -# Prefer a robust parameter plateau over the in-sample peak — Implementation Plan - -> **Parent spec:** `docs/specs/0077-plateau-over-peak-selection.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to -> run this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Add an opt-in `optimize_plateau` selection objective (argmax over the -grid-neighbourhood-smoothed metric) reachable via a new `--select` flag, recorded -on the `RunManifest.selection` carrier #144 introduced; default `argmax` stays -byte-identical (C23). - -**Architecture:** A new family selector lives beside `optimize_deflated` in -`aura-registry` (C9), reading each member's metric via the shared `metric_value` -and smoothing it over a mixed-radix grid neighbourhood built from `axis_lens` -(the `GridSpace` odometer radixes). The lattice is surfaced from the engine via a -new `SweepBinder::sweep_with_lattice` terminal (`sweep` delegates to it, -byte-unchanged for all existing callers). The shared `FamilySelection` carrier is -reshaped — orthogonal rule (`SelectionMode`) × annotation (deflation fields → -`Option`, plus plateau fields) — and the CLI threads a `Selection` from -`--select` into `walkforward_family`. - -**Tech Stack:** `aura-engine` (`report.rs` carrier, `sweep.rs` accessor, -`blueprint.rs` terminal), `aura-registry` (`lib.rs` selector + index math), -`aura-cli` (`main.rs` flag/dispatch/display, `tests/cli_run.rs` E2E). - ---- - -## Design decisions resolved at plan time (recorded on #145, comment 1974) - -1. **Lattice seam → engine `SweepBinder::sweep_with_lattice`.** Only the engine's - post-`resolve_axes` `GridSpace` holds the lattice in `param_space()`/odometer - order, which `closed_neighbourhood`'s mixed-radix arithmetic requires; a sibling - terminal (`sweep` delegates) contains the blast radius — every existing - `.sweep()` caller stays byte-identical. -2. **RandomSpace-refuse → structural guard, no new feature.** `walkforward` has no - `--random` flag (verified: `--random`/`RandomSpace`/`RandomBinder` appear - nowhere in `aura-cli`); wiring a random walk-forward producer is unspecified, - out-of-scope. So `sweep_over`/`stage1_r_sweep_over` return - `(SweepFamily, Option>)` (`Some` today, grid-only), and a - `select_winner` helper returns the refuse on `Plateau + None`-lattice; the - caller prints the spec's message and exits 2. **Unit-tested** at the helper - level (the `Err` path), not via a `walkforward --random` E2E. -3. **`FamilySelection.seed` → `Option`** (spec line 135; the recon carrier's - `Option` was a transcription slip). - -## Files this plan creates or modifies - -- Modify: `crates/aura-engine/src/report.rs:41-66` — reshape `SelectionMode` + - `FamilySelection`; `:762-775` round-trip test literal; new plateau-shape test. -- Modify: `crates/aura-engine/src/sweep.rs` — new `GridSpace::axis_lens()` + test. -- Modify: `crates/aura-engine/src/blueprint.rs:399-411` — new - `SweepBinder::sweep_with_lattice`; `sweep` delegates; new test. -- Modify: `crates/aura-registry/src/lib.rs:308-312` — `optimize_deflated` - construction to `Option` deflation fields; `:163-173` `metric_cmp` refactor to - `higher_is_better`; `:744-794` test assertions → `.unwrap()`/`Some`; add - `PlateauMode`, `higher_is_better`, `closed_neighbourhood`, `optimize_plateau` + - tests. -- Modify: `crates/aura-cli/src/main.rs` — import `SelectionMode` (engine), - `optimize_plateau`/`PlateauMode` (registry); `Selection` enum; `parse_select` + - `--select` in `parse_walkforward_args`; `select_winner`; `sweep_over` / - `stage1_r_sweep_over` tuple returns; `walkforward_family` `+select` dispatch; - `run_walkforward` `+select`; the dispatch call site; `runs_family` mode-aware - display; 4 test callers of `walkforward_family`; unit tests. -- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`. -- Unchanged (preservation gate): `crates/aura-registry/src/compat.rs:34` embeds - `FamilySelection` by value — the reshape flows through with no edit; the - legacy-load test `load_lifts_a_selection_block_through_the_compat_mirror` - (`lib.rs:520`) and `runs_family_rank_shows_deflated_line` (`cli_run.rs:2358`) - must stay green. - ---- - -## Task 1: Reshape `FamilySelection` / `SelectionMode` (mechanical, byte-preserving across 3 crates) - -**Files:** -- Modify: `crates/aura-engine/src/report.rs:41-66`, `:762-775` -- Modify: `crates/aura-registry/src/lib.rs:308-312`, `:744-794` -- Modify: `crates/aura-cli/src/main.rs:19-25` (import), `:2215-2223` (display) - -This is the cross-crate compile-gate. The reshape breaks every construction and -field-read site; all are threaded here. No new runtime behaviour — the argmax -path stays byte-identical; the one new test pins the new fields' serde contract. - -- [ ] **Step 1: Write the failing plateau-shape serde test (RED)** - -In `crates/aura-engine/src/report.rs`, in the `#[cfg(test)] mod tests` block, -beside `family_selection_round_trips_on_the_manifest` (after line 775), add: - -```rust - #[test] - fn family_selection_plateau_shape_round_trips() { - let sel = FamilySelection { - selection_metric: "sqn_normalized".into(), n_trials: 16, raw_winner_metric: 1.42, - mode: SelectionMode::PlateauMean, - deflated_score: None, overfit_probability: None, - n_resamples: None, block_len: None, seed: None, - neighbourhood_score: Some(1.27), n_neighbours: Some(5), - }; - let json = serde_json::to_string(&sel).unwrap(); - // plateau annotation present; deflation fields omitted (skip_serializing_if) - assert!(json.contains("\"neighbourhood_score\":1.27"), "json: {json}"); - assert!(json.contains("\"n_neighbours\":5"), "json: {json}"); - assert!(!json.contains("deflated_score"), "deflation fields omitted under plateau: {json}"); - assert!(!json.contains("n_resamples"), "json: {json}"); - let back: FamilySelection = serde_json::from_str(&json).unwrap(); - assert_eq!(back, sel); - } -``` - -- [ ] **Step 2: Run the new test to verify it fails to compile** - -Run: `cargo test -p aura-engine family_selection_plateau` -Expected: FAIL — compile error (`SelectionMode::PlateauMean` / the `neighbourhood_score`/`n_neighbours` fields do not exist yet). - -- [ ] **Step 3: Reshape `SelectionMode` and `FamilySelection`** - -In `crates/aura-engine/src/report.rs`, replace the `SelectionMode` enum + doc -(lines 41-48) and the `FamilySelection` struct + doc (lines 50-66) with: - -```rust -/// Which selection objective produced the record (additive provenance, C23). -/// `Argmax` is the bare-best pick (cycle 0076), deflated for the number of trials. -/// `PlateauMean` / `PlateauWorst` (cycle 0077) argmax the neighbourhood-smoothed -/// surface instead, so peak-vs-plateau runs stay distinguishable on this field. -#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub enum SelectionMode { - Argmax, - PlateauMean, - PlateauWorst, -} - -/// Selection-provenance for a sweep winner. The selection RULE (`mode`) and its -/// ANNOTATION are orthogonal: `Argmax` carries the trials-deflation annotation -/// (`deflated_score` / `overfit_probability` / `n_resamples` / `block_len` / -/// `seed`); `Plateau*` carries the smoothing annotation (`neighbourhood_score` / -/// `n_neighbours`). Each annotation is `Option`, present iff its rule produced the -/// record; all are additive — recorded, never re-ranking (C23). A legacy line -/// (pre-0077) carries the deflation scalars as bare values that deserialize to -/// `Some` (serde default), so it still loads (C14/C18). -#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub struct FamilySelection { - pub selection_metric: String, - pub n_trials: usize, - pub raw_winner_metric: f64, - pub mode: SelectionMode, - // deflation annotation (present iff mode == Argmax with a deflation run) - #[serde(default, skip_serializing_if = "Option::is_none")] - pub deflated_score: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub overfit_probability: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub n_resamples: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub block_len: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub seed: Option, - // plateau annotation (present iff mode is Plateau*) - #[serde(default, skip_serializing_if = "Option::is_none")] - pub neighbourhood_score: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub n_neighbours: Option, -} -``` - -- [ ] **Step 4: Update the round-trip test literal** - -In `crates/aura-engine/src/report.rs`, in `family_selection_round_trips_on_the_manifest`, -replace the `let sel = FamilySelection { … };` literal (lines 764-768) with: - -```rust - let sel = FamilySelection { - selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83, - mode: SelectionMode::Argmax, - deflated_score: Some(0.21), overfit_probability: Some(0.06), - n_resamples: Some(1000), block_len: Some(5), seed: Some(42), - neighbourhood_score: None, n_neighbours: None, - }; -``` - -- [ ] **Step 5: Run the engine tests (GREEN, engine internally consistent)** - -Run: `cargo test -p aura-engine family_selection` -Expected: PASS — both `family_selection_round_trips_on_the_manifest` and -`family_selection_plateau_shape_round_trips` green. (aura-engine does not depend -on aura-registry, so its build is consistent even though the registry is not yet -updated.) - -- [ ] **Step 6: Update `optimize_deflated`'s construction to the Option fields** - -In `crates/aura-registry/src/lib.rs`, replace the returned `FamilySelection` -literal (lines 308-312) with: - -```rust - Ok((winner.clone(), FamilySelection { - selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, - mode: SelectionMode::Argmax, - deflated_score: Some(deflated_score), overfit_probability, - n_resamples: Some(n_resamples), block_len: Some(block_len), seed: Some(seed), - neighbourhood_score: None, n_neighbours: None, - })) -``` - -- [ ] **Step 7: Update the `optimize_deflated_*` test assertions to `.unwrap()` / `Some`** - -In `crates/aura-registry/src/lib.rs`, apply these exact line edits: - -- Line 749: ` assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw` - → ` assert!(sel.deflated_score.unwrap() <= sel.raw_winner_metric); // dispersion floor ≤ raw` -- Line 762: ` assert_eq!(sel.n_resamples, 0);` - → ` assert_eq!(sel.n_resamples, Some(0));` -- Line 763: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric, "p95 of an empty null is 0");` - → ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric, "p95 of an empty null is 0");` -- Line 782: ` assert!(sel.deflated_score.is_finite(), "deflated_score must be finite, got {}", sel.deflated_score);` - → ` assert!(sel.deflated_score.unwrap().is_finite(), "deflated_score must be finite, got {:?}", sel.deflated_score);` -- Line 783: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric);` - → ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric);` -- Line 794: ` assert!(sel.deflated_score > 0.0);` - → ` assert!(sel.deflated_score.unwrap() > 0.0);` - -(The compat test `load_lifts_a_selection_block_through_the_compat_mirror` at line -520-527 is **unchanged** — its legacy JSON line carries the deflation scalars as -bare floats that deserialize to `Some` under the new `Option` + `#[serde(default)]`, -and its assertions read only `n_trials`/`mode`. It is the C14/C18 legacy-load gate.) - -- [ ] **Step 8: Run the registry tests (GREEN)** - -Run: `cargo test -p aura-registry optimize_deflated` -Expected: PASS — all nine `optimize_deflated_*` tests green against the reshaped fields. - -Run: `cargo test -p aura-registry load_lifts` -Expected: PASS — the legacy `selection` block still lifts through the compat mirror. - -- [ ] **Step 9: Import `SelectionMode` and rewrite the `runs_family` display (mode-aware, argmax byte-preserved)** - -In `crates/aura-cli/src/main.rs`, add `SelectionMode` to the `aura_engine` import -list (lines 19-24) — e.g. extend the `RunManifest, RunMetrics,` line to -`RunManifest, RunMetrics, SelectionMode,` (any slot in the brace list is fine). - -Then replace the selection-display block in `runs_family` (lines 2215-2223): - -```rust - for report in &ordered { - println!("{}", report.to_json()); - if let Some(sel) = &report.manifest.selection { - match sel.mode { - SelectionMode::Argmax => match sel.overfit_probability { - Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score.unwrap(), p), - None => println!(" deflated={:.4}", sel.deflated_score.unwrap()), - }, - SelectionMode::PlateauMean | SelectionMode::PlateauWorst => { - let label = if matches!(sel.mode, SelectionMode::PlateauMean) { "mean" } else { "worst" }; - if let (Some(score), Some(n)) = (sel.neighbourhood_score, sel.n_neighbours) { - println!(" plateau({label})={score:.4} over {n} cells"); - } - } - } - } - } -``` - -(Under `Argmax`, `deflated_score` is always `Some` by construction, so `.unwrap()` -preserves the exact pre-0077 bytes — the `deflated=…`/`P(overfit)=…` golden stays -green. The `Plateau*` branch is exercised by the Task 5 E2E.) - -- [ ] **Step 10: Workspace build + full per-crate regression gate** - -Run: `cargo build --workspace` -Expected: clean (0 errors) — all three crates consistent. - -Run: `cargo test -p aura-engine` -Expected: PASS (no regressions). - -Run: `cargo test -p aura-registry` -Expected: PASS (no regressions). - ---- - -## Task 2: `GridSpace::axis_lens()` + `SweepBinder::sweep_with_lattice` (engine, additive) - -**Files:** -- Modify: `crates/aura-engine/src/sweep.rs` (accessor + test) -- Modify: `crates/aura-engine/src/blueprint.rs:399-411` (terminal + delegate + test) - -- [ ] **Step 1: Write the failing `axis_lens` golden (RED)** - -In `crates/aura-engine/src/sweep.rs`, in the `#[cfg(test)] mod tests` block (which -already imports `aura_core::{Cell, ParamSpec, Scalar, ScalarKind, Timestamp}`), -add: - -```rust - #[test] - fn grid_axis_lens_are_the_per_axis_radixes() { - let space = vec![ - ParamSpec { name: "a".into(), kind: ScalarKind::I64 }, - ParamSpec { name: "b".into(), kind: ScalarKind::I64 }, - ]; - let grid = GridSpace::new(&space, vec![ - vec![Scalar::i64(10), Scalar::i64(20)], // axis 0: 2 values - vec![Scalar::i64(1), Scalar::i64(2), Scalar::i64(3)], // axis 1: 3 values - ]).expect("2x3 grid"); - assert_eq!(grid.axis_lens(), vec![2, 3]); - assert_eq!(grid.axis_lens().iter().product::(), grid.len()); - } -``` - -- [ ] **Step 2: Run to verify it fails to compile** - -Run: `cargo test -p aura-engine grid_axis_lens` -Expected: FAIL — `no method named axis_lens found for struct GridSpace`. - -- [ ] **Step 3: Add the `axis_lens` accessor** - -In `crates/aura-engine/src/sweep.rs`, in the `impl GridSpace` block, immediately -after `is_empty` (after line 65), add: - -```rust - /// Per-axis cardinalities in `param_space()` order (the odometer radixes, - /// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau - /// neighbourhood walks (cycle 0077). - pub fn axis_lens(&self) -> Vec { - self.axes.iter().map(Vec::len).collect() - } -``` - -- [ ] **Step 4: Run the accessor test (GREEN)** - -Run: `cargo test -p aura-engine grid_axis_lens` -Expected: PASS. - -- [ ] **Step 5: Write the failing `sweep_with_lattice` test (RED)** - -In `crates/aura-engine/src/blueprint.rs`, in the `#[cfg(test)] mod tests` block -(which has `composite_sma_cross_harness()` and `run_point`), add: - -```rust - #[test] - fn sweep_with_lattice_surfaces_grid_radixes_in_param_space_order() { - let bp = composite_sma_cross_harness().0; - let (fam, lattice) = bp - .axis("sma_cross.fast.length", vec![Scalar::i64(2), Scalar::i64(3)]) // 2 values - .axis("sma_cross.slow.length", vec![Scalar::i64(4), Scalar::i64(5)]) // 2 values - .axis("bias.scale", vec![Scalar::f64(0.5)]) // 1 value - .sweep_with_lattice(run_point) - .expect("named binding resolves and runs"); - assert_eq!(lattice, vec![2, 2, 1], "radixes in param_space slot order"); - assert_eq!(lattice.iter().product::(), fam.points.len()); // 4 - } -``` - -- [ ] **Step 6: Run to verify it fails to compile** - -Run: `cargo test -p aura-engine sweep_with_lattice` -Expected: FAIL — `no method named sweep_with_lattice found`. - -- [ ] **Step 7: Add `sweep_with_lattice`; refactor `sweep` to delegate** - -In `crates/aura-engine/src/blueprint.rs`, replace the `SweepBinder::sweep` method -(lines 399-411) with: - -```rust - /// Resolve the named axes against `param_space()` into a positional grid and - /// run the disjoint sweep. `sweep` is [`SweepBinder::sweep_with_lattice`] with - /// the lattice dropped, so every existing caller is byte-unchanged. - pub fn sweep(self, run_one: F) -> Result - where - F: Fn(&[Cell]) -> RunReport + Sync, - { - self.sweep_with_lattice(run_one).map(|(family, _lattice)| family) - } - - /// As [`SweepBinder::sweep`], plus the grid's per-axis radixes (`axis_lens`, - /// in `param_space()` / odometer order). The lattice is what a plateau - /// neighbourhood walk needs (cycle 0077); only the engine's post-`resolve_axes` - /// grid holds it in the correct order. - pub fn sweep_with_lattice(self, run_one: F) -> Result<(SweepFamily, Vec), BindError> - where - F: Fn(&[Cell]) -> RunReport + Sync, - { - let space = self.bp.param_space(); - check_param_namespace_injective(&space).map_err(BindError::Compile)?; - let ordered = resolve_axes(&space, &self.axes)?; - let grid = GridSpace::new(&space, ordered) - .expect("named layer pre-validates arity/kind/non-empty"); - let lattice = grid.axis_lens(); - Ok((sweep(&grid, run_one), lattice)) - } -``` - -- [ ] **Step 8: Run the new test + the existing sweep tests (GREEN, delegation preserved)** - -Run: `cargo test -p aura-engine sweep_with_lattice` -Expected: PASS. - -Run: `cargo test -p aura-engine` -Expected: PASS — every existing `.sweep()` test (named-axes parity, grid goldens) -stays green; `sweep` delegating to `sweep_with_lattice` is behaviour-preserving. - ---- - -## Task 3: `optimize_plateau` + `PlateauMode` + `closed_neighbourhood` (registry, additive) - -**Files:** -- Modify: `crates/aura-registry/src/lib.rs:163-173` (`metric_cmp` → `higher_is_better`), - plus new `PlateauMode`, `higher_is_better`, `closed_neighbourhood`, - `optimize_plateau` and their tests. - -- [ ] **Step 1: Write the failing plateau tests (RED)** - -In `crates/aura-registry/src/lib.rs`, in the `#[cfg(test)] mod tests` block (which -has the `member(total_pips, trade_rs)` helper), add: - -```rust - /// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle - /// plateau in `total_pips`. Bare argmax takes an end spike; the plateau-mean - /// argmax takes the interior plateau centre. - fn fixture_grid_spike_vs_plateau() -> SweepFamily { - let pips = [80.0, 5.0, 48.0, 50.0, 49.0, 5.0, 80.0]; - SweepFamily { space: vec![], points: pips.iter().map(|&p| member(p, vec![1.0])).collect() } - } - - #[test] - fn closed_neighbourhood_2x3_golden() { - // 2x3 lattice, odometer (last axis fastest): - // (0,0)=0 (0,1)=1 (0,2)=2 - // (1,0)=3 (1,1)=4 (1,2)=5 - let sorted = |i: usize| { let mut v = closed_neighbourhood(i, &[2, 3]); v.sort_unstable(); v }; - assert_eq!(sorted(0), vec![0, 1, 3]); // corner (0,0): self + 2 - assert_eq!(sorted(4), vec![1, 3, 4, 5]); // interior (1,1): self + 3 - assert_eq!(sorted(2), vec![1, 2, 5]); // corner (0,2): self + 2 - } - - #[test] - fn plateau_picks_the_plateau_not_the_spike() { - let fam = fixture_grid_spike_vs_plateau(); - let spike = optimize(&fam, "total_pips").unwrap(); - let (centre, sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap(); - assert_ne!(centre.report.metrics.total_pips, spike.report.metrics.total_pips); - assert_eq!(centre.report.metrics.total_pips, 50.0, "plateau centre is the middle cell"); - assert_eq!(sel.mode, SelectionMode::PlateauMean); - assert_eq!(sel.raw_winner_metric, 50.0); - assert!(sel.neighbourhood_score.unwrap() < spike.report.metrics.total_pips, - "smoothed score is below the spike's raw peak"); - assert_eq!(sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours"); - assert!(sel.deflated_score.is_none() && sel.n_resamples.is_none(), - "deflation fields omitted under plateau"); - } - - #[test] - fn plateau_worst_is_more_conservative_than_mean() { - let fam = fixture_grid_spike_vs_plateau(); - let (_, mean_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap(); - let (_, worst_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap(); - // Worst scores each cell by its weakest neighbour, never above the mean. - assert!(worst_sel.neighbourhood_score.unwrap() <= mean_sel.neighbourhood_score.unwrap()); - assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst); - assert!(worst_sel.n_neighbours.unwrap() >= 2, "winner is an interior cell"); - } - - #[test] - fn plateau_single_member_equals_optimize() { - let fam = SweepFamily { space: vec![], points: vec![member(42.0, vec![1.0])] }; - let plain = optimize(&fam, "total_pips").unwrap(); - let (winner, sel) = optimize_plateau(&fam, &[1], "total_pips", PlateauMode::Mean).unwrap(); - assert_eq!(winner.report.metrics.total_pips, plain.report.metrics.total_pips); - assert_eq!(sel.n_neighbours, Some(1), "a 1-cell grid's closed neighbourhood is itself"); - assert_eq!(sel.neighbourhood_score, Some(42.0)); - assert_eq!(sel.raw_winner_metric, 42.0); - } - - #[test] - fn optimize_plateau_is_deterministic() { - let fam = fixture_grid_spike_vs_plateau(); - let a = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1; - let b = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1; - assert_eq!(a, b); - } -``` - -- [ ] **Step 2: Run to verify they fail to compile** - -Run: `cargo test -p aura-registry closed_neighbourhood` -Expected: FAIL — `closed_neighbourhood` / `optimize_plateau` / `PlateauMode` do not exist. - -- [ ] **Step 3: Add `higher_is_better` and refactor `metric_cmp` to use it** - -In `crates/aura-registry/src/lib.rs`, immediately after `is_r_metric` (after line -207), add: - -```rust -/// The metric's optimisation direction: `true` if a larger value is better -/// (`total_pips`, the four R keys), `false` for the lower-is-better keys -/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp` -/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour) -/// both read it, so the per-metric sense lives in exactly one place. -fn higher_is_better(m: Metric) -> bool { - !matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips) -} -``` - -Then replace the `metric_cmp` body (lines 163-173) with: - -```rust -fn metric_cmp(metric: &str) -> Result Ordering, RegistryError> { - let m = resolve_metric(metric)?; - let hib = higher_is_better(m); - Ok(move |a: &RunReport, b: &RunReport| { - let (va, vb) = (metric_value(a, m), metric_value(b, m)); - // higher-is-better ranks the larger value first; lower-is-better the smaller. - if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) } - }) -} -``` - -(Behaviour-identical: `rank_by` / `optimize` and their existing tests are the gate.) - -- [ ] **Step 4: Add `closed_neighbourhood`** - -In `crates/aura-registry/src/lib.rs`, after `higher_is_better`, add: - -```rust -/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice -/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}` -/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order -/// is `i` first then per-axis neighbours; callers use the result only as a set and -/// for its length, so the order is not load-bearing. -fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec { - // decompose i to per-axis coords (last axis fastest) - let mut coords = vec![0usize; axis_lens.len()]; - let mut rem = i; - for k in (0..axis_lens.len()).rev() { - coords[k] = rem % axis_lens[k]; - rem /= axis_lens[k]; - } - // recompose a coord vector to its flat index (Horner, last axis fastest) - let recompose = |c: &[usize]| c.iter().zip(axis_lens).fold(0usize, |flat, (&ck, &len)| flat * len + ck); - let mut out = vec![i]; - for k in 0..axis_lens.len() { - for delta in [-1isize, 1] { - let ck = coords[k] as isize + delta; - if ck < 0 || ck as usize >= axis_lens[k] { continue; } - let mut nb = coords.clone(); - nb[k] = ck as usize; - out.push(recompose(&nb)); - } - } - out -} -``` - -- [ ] **Step 5: Add `PlateauMode` and `optimize_plateau`** - -In `crates/aura-registry/src/lib.rs`, immediately before `optimize_deflated` (before -line 258, before its doc comment), add: - -```rust -/// Plateau-selection aggregate: how a member's grid-neighbourhood metric is -/// reduced to one smoothed score before the argmax. `Mean` averages the closed -/// neighbourhood; `Worst` takes the most-pessimistic neighbour by the metric's -/// direction (biasing toward interior cells, which keep more neighbours). -#[derive(Clone, Copy, Debug, PartialEq)] -pub enum PlateauMode { - Mean, - Worst, -} - -/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau -/// provenance. Each member scores as the mean (or worst-case) of its closed grid -/// neighbourhood's `metric_value`; the winner is the best smoothed score by the -/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are -/// the grid radixes (`param_space()` order, last axis fastest). Pure, -/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL; -/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the -/// smoothed value the argmax maximised. -pub fn optimize_plateau( - family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode, -) -> Result<(SweepPoint, FamilySelection), RegistryError> { - let m = resolve_metric(metric)?; - let n = family.points.len(); - debug_assert_eq!( - axis_lens.iter().product::(), n, - "axis_lens product must equal the family size (a valid grid lattice)", - ); - let hib = higher_is_better(m); - // smoothed score + closed-neighbourhood size per member, in odometer order - let scored: Vec<(usize, f64, usize)> = (0..n) - .map(|i| { - let nbrs = closed_neighbourhood(i, axis_lens); - let vals: Vec = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect(); - let score = match mode { - PlateauMode::Mean => vals.iter().sum::() / vals.len() as f64, - PlateauMode::Worst => { - if hib { - vals.iter().copied().fold(f64::INFINITY, f64::min) - } else { - vals.iter().copied().fold(f64::NEG_INFINITY, f64::max) - } - } - }; - (i, score, nbrs.len()) - }) - .collect(); - // argmax the smoothed surface by direction; earliest-odometer tie (only a - // strictly-better later cell displaces the incumbent, as `optimize`). - let &(wi, wscore, wn) = scored - .iter() - .reduce(|best, cur| { - let better = if hib { cur.1 > best.1 } else { cur.1 < best.1 }; - if better { cur } else { best } - }) - .expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)"); - let winner = family.points[wi].clone(); - let raw = metric_value(&winner.report, m); - Ok(( - winner, - FamilySelection { - selection_metric: metric.to_string(), - n_trials: n, - raw_winner_metric: raw, - mode: match mode { - PlateauMode::Mean => SelectionMode::PlateauMean, - PlateauMode::Worst => SelectionMode::PlateauWorst, - }, - deflated_score: None, - overfit_probability: None, - n_resamples: None, - block_len: None, - seed: None, - neighbourhood_score: Some(wscore), - n_neighbours: Some(wn), - }, - )) -} -``` - -- [ ] **Step 6: Run the plateau tests + the existing rank/optimize gate (GREEN)** - -Run: `cargo test -p aura-registry plateau` -Expected: PASS — `plateau_picks_the_plateau_not_the_spike`, -`plateau_worst_is_more_conservative_than_mean`, `plateau_single_member_equals_optimize`, -`optimize_plateau_is_deterministic` all green. - -Run: `cargo test -p aura-registry closed_neighbourhood` -Expected: PASS. - -Run: `cargo test -p aura-registry` -Expected: PASS — `rank_by` / `optimize` / `optimize_deflated` stay green after the -`metric_cmp` → `higher_is_better` refactor. - ---- - -## Task 4: CLI `--select` parse + threading + sweep-entry tuples + `select_winner` (compile-gate) - -**Files:** -- Modify: `crates/aura-cli/src/main.rs` — `aura_registry` import; `Selection` enum; - `parse_select`; `parse_walkforward_args` (`+--select`, 6-tuple); the dispatch - call site (`:2935`); `run_walkforward` (`+select`); `walkforward_family` - (`+select` dispatch via `select_winner`); `sweep_over` / `stage1_r_sweep_over` - (tuple returns); `select_winner`; the 4 test callers of `walkforward_family`; - unit tests. - -This is the CLI compile-gate: the signature changes break their callers, all -threaded here. Default `Argmax` keeps the existing walk-forward output -byte-identical (C23). - -- [ ] **Step 1: Write the failing `select_winner` refuse + parse tests (RED)** - -In `crates/aura-cli/src/main.rs`, in the `#[cfg(test)] mod tests` block at line -2965, add: - -```rust - #[test] - fn select_winner_refuses_plateau_without_a_lattice() { - // A plateau request with no lattice (a random sweep would yield None) is - // refused, never silently argmaxed. The refuse short-circuits before the - // family is read, so an empty family is fine here. - let fam = SweepFamily { space: vec![], points: vec![] }; - let err = select_winner(&fam, "total_pips", Selection::Plateau(PlateauMode::Mean), None) - .unwrap_err(); - assert!(err.contains("requires a grid sweep"), "refuse message: {err}"); - } - - #[test] - fn parse_walkforward_select_flag() { - let argmax = parse_walkforward_args(&["--strategy", "stage1-r"]).unwrap(); - assert!(matches!(argmax.5, Selection::Argmax), "default is argmax"); - let mean = parse_walkforward_args(&["--select", "plateau:mean"]).unwrap(); - assert!(matches!(mean.5, Selection::Plateau(PlateauMode::Mean))); - let worst = parse_walkforward_args(&["--select", "plateau:worst"]).unwrap(); - assert!(matches!(worst.5, Selection::Plateau(PlateauMode::Worst))); - assert!(parse_walkforward_args(&["--select", "bogus"]).is_err(), - "unknown --select token is a usage error"); - } -``` - -- [ ] **Step 2: Run to verify they fail to compile** - -Run: `cargo test -p aura-cli select_winner_refuses` -Expected: FAIL — `Selection` / `PlateauMode` / `select_winner` not in scope; the -`parse_walkforward_args` tuple has no `.5`. - -- [ ] **Step 3: Imports + add the `Selection` enum and `parse_select`** - -In `crates/aura-cli/src/main.rs`, extend the `aura_registry` import (lines 26-30) — -add `optimize_plateau, PlateauMode,` to the brace list (alongside `optimize_deflated`). -Also add `FamilySelection` to the `aura_engine` import list (lines 19-24; it is not -currently imported, and `select_winner` in Step 4 names it). (`SelectionMode` was -already added to that list in Task 1 Step 9.) - -Then, immediately after the `Strategy` enum (after line 1551's enum block; place it -near the other CLI enums), add: - -```rust -/// In-sample winner-selection objective for walk-forward (cycle 0077). `Argmax` is -/// the bare-best pick deflated for trials (#144, the default); `Plateau` argmaxes -/// the neighbourhood-smoothed surface instead (opt-in via `--select`). -#[derive(Clone, Copy)] -enum Selection { - Argmax, - Plateau(PlateauMode), -} - -/// Parse a `--select` token: `argmax` | `plateau:mean` | `plateau:worst`. Unknown -/// tokens are a usage error (the caller maps `Err(())` to exit 2). -fn parse_select(s: &str) -> Result { - match s { - "argmax" => Ok(Selection::Argmax), - "plateau:mean" => Ok(Selection::Plateau(PlateauMode::Mean)), - "plateau:worst" => Ok(Selection::Plateau(PlateauMode::Worst)), - _ => Err(()), - } -} -``` - -- [ ] **Step 4: Add the `select_winner` dispatch helper** - -In `crates/aura-cli/src/main.rs`, immediately before `walkforward_family` (before -line 1786), add: - -```rust -/// Resolve the in-sample winner under the chosen selection objective. `Argmax` -/// defers to the trials-deflation pick (#144). `Plateau` argmaxes the smoothed grid -/// surface — it needs the grid lattice, so a sweep with no lattice (a future random -/// walk-forward producer) is refused rather than silently argmaxed. The metric is -/// always known at the call sites, so a metric error is unreachable (`expect`); the -/// only fallible outcome is the plateau-without-lattice refusal, returned as -/// `Err(message)` for the caller to print and exit 2. -fn select_winner( - family: &SweepFamily, metric: &str, select: Selection, lattice: Option<&[usize]>, -) -> Result<(SweepPoint, FamilySelection), String> { - match select { - Selection::Argmax => Ok(optimize_deflated( - family, metric, DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, - ).expect("walk-forward metrics are known")), - Selection::Plateau(mode) => match lattice { - Some(lens) => Ok(optimize_plateau(family, lens, metric, mode) - .expect("walk-forward metrics are known")), - None => Err( - "--select plateau requires a grid sweep; a random sweep has no parameter lattice" - .to_string(), - ), - }, - } -} -``` - -(`select_winner` names `FamilySelection`, which is **not** currently imported in -`main.rs` — Step 3 above adds it to the `aura_engine` import list.) - -- [ ] **Step 5: Switch `sweep_over` / `stage1_r_sweep_over` to return `(SweepFamily, Option>)`** - -In `crates/aura-cli/src/main.rs`: - -`stage1_r_sweep_over` (line 1301): change the signature return type -`-> SweepFamily` to `-> (SweepFamily, Option>)`; change its terminal -`.sweep(|point| {` (line 1318) to `.sweep_with_lattice(|point| {`, and insert a -`.map(...)` line before the existing trailing `.expect(...)` (line 1342, currently -` .expect("the stage1-r named grid matches the stage1-r param-space")`). The -closure's closing ` })` and the existing `.expect(...)` line both stay; the -new `.map` goes between them, so the function tail reads: - -```rust - }) - .map(|(fam, lat)| (fam, Some(lat))) - .expect("the stage1-r named grid matches the stage1-r param-space") -``` - -`sweep_over` (line 1841): change the signature return type `-> SweepFamily` to -`-> (SweepFamily, Option>)`; change its terminal `.sweep(|point| {` -(line 1854) to `.sweep_with_lattice(|point| {`, and insert the `.map(...)` line -before the existing trailing -` .expect("the built-in named grid matches the sample param-space")` (line -1869), so the function tail reads: - -```rust - }) - .map(|(fam, lat)| (fam, Some(lat))) - .expect("the built-in named grid matches the sample param-space") -``` - -- [ ] **Step 6: Thread `Selection` through `walkforward_family` (signature + both arms)** - -In `crates/aura-cli/src/main.rs`, change `walkforward_family`'s signature (line -1786-1788) to add the `select` parameter: - -```rust -fn walkforward_family( - strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid, - select: Selection, -) -> WalkForwardResult { -``` - -In the `Strategy::SmaCross` arm, replace the sweep + optimize lines (1802-1807): - -```rust - let (is_family, lattice) = sweep_over(w.is.0, w.is.1, data); - let (best, selection) = match select_winner(&is_family, "total_pips", select, lattice.as_deref()) { - Ok(v) => v, - Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } - }; - let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); - oos_report.manifest.selection = Some(selection); -``` - -In the `Strategy::Stage1R` arm, replace the sweep + optimize lines (1820-1825): - -```rust - let (is_family, lattice) = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); - let (best, selection) = match select_winner(&is_family, "sqn_normalized", select, lattice.as_deref()) { - Ok(v) => v, - Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } - }; - let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); - oos_report.manifest.selection = Some(selection); -``` - -- [ ] **Step 7: Thread the new `select` argument through every `walkforward_family` caller** - -In `crates/aura-cli/src/main.rs`, update the production caller and the four test -callers to pass a `Selection`: - -- `run_walkforward` (line 1755): add `select: Selection` to its signature: - `fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid, select: Selection) {` - and update its `walkforward_family(...)` call (line 1763) to - `let result = walkforward_family(strategy, persist.then_some(name), &data, grid, select);` -- Line 1963: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` -- Line 2131: `walkforward_family(Strategy::Stage1R, None, data, grid, Selection::Argmax)` -- Line 3456: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` -- Line 4131: `walkforward_family(Strategy::Stage1R, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)` - -- [ ] **Step 8: Add `--select` to `parse_walkforward_args` and thread it to the dispatch** - -In `crates/aura-cli/src/main.rs`, in `parse_walkforward_args` (line 1648): - -Change the return type to add `Selection`: -`) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid, Selection), String> {` - -Update the usage string to mention `--select` (append `[--select ]` -to both the `///` doc usage line and the `usage` closure string). - -Add `let mut select = Selection::Argmax;` beside the other `let mut` locals. - -Add a match arm for the flag (beside `--stop-k`): -`"--select" => select = parse_select(value).map_err(|()| usage())?,` - -Change the final return to: -`Ok((strategy, name, persist, real.finish(&usage)?, grid, select))` - -In the command dispatch (line 2935-2939), update the destructure + call: - -```rust - ["walkforward", rest @ ..] => match parse_walkforward_args(rest) { - Ok((strategy, name, persist, choice, grid, select)) => { - run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid, select) - } -``` - -- [ ] **Step 9: Compile-gate — workspace + tests build clean** - -Run: `cargo build -p aura-cli --tests` -Expected: clean (0 errors) — every caller of the changed signatures threaded. - -- [ ] **Step 10: Run the new unit tests (GREEN)** - -Run: `cargo test -p aura-cli select_winner_refuses` -Expected: PASS. - -Run: `cargo test -p aura-cli parse_walkforward_select` -Expected: PASS. - -- [ ] **Step 11: C23 — existing walk-forward goldens stay byte-identical** - -Run: `cargo test -p aura-cli walkforward_strategy_stage1_r` -Expected: PASS — argmax path unchanged. - -Run: `cargo test -p aura-cli runs_family_rank_shows_deflated` -Expected: PASS — the deflated display line is byte-identical. - -Run: `cargo test -p aura-cli walkforward_bare_sma` -Expected: PASS — the SMA summary golden is unchanged. - ---- - -## Task 5: `runs family rank` plateau E2E golden - -**Files:** -- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance` - -Tasks 1-4 deliver the behaviour; this pins it end-to-end (C18: the plateau -provenance reaches disk and the display renders it). - -- [ ] **Step 1: Write the plateau E2E (RED until Tasks 1-4 land; GREEN after)** - -In `crates/aura-cli/tests/cli_run.rs`, beside `runs_family_rank_shows_deflated_line` -(after line 2388), add: - -```rust -/// Property: a stage1-r walk-forward run with `--select plateau:mean` selects the -/// neighbourhood-smoothed winner and stamps each OOS manifest with the plateau -/// provenance (`mode = PlateauMean`, `neighbourhood_score`, `n_neighbours`); `runs -/// family … rank` renders the `plateau(mean)=… over N cells` line, and the -/// deflation fields are omitted (orthogonal annotation). End-to-end witness that -/// the opt-in selection rule reaches disk (C18) and the display path renders it. -#[test] -fn walkforward_plateau_select_stamps_plateau_provenance() { - let cwd = temp_cwd("runs-plateau"); - let wf = Command::new(BIN) - .args([ - "walkforward", "--strategy", "stage1-r", "--select", "plateau:mean", - "--fast", "50,100", "--slow", "200,400", - "--stop-length", "14,21", "--stop-k", "2.0,3.0", - "--name", "wf-plat", - ]) - .current_dir(&cwd) - .output() - .expect("spawn walkforward --select plateau:mean"); - assert!( - wf.status.success(), - "walkforward exit: {:?}; stderr: {}", - wf.status, - String::from_utf8_lossy(&wf.stderr) - ); - - let rank = Command::new(BIN) - .args(["runs", "family", "wf-plat-0", "rank", "sqn_normalized"]) - .current_dir(&cwd) - .output() - .expect("spawn runs family wf-plat-0 rank sqn_normalized"); - assert!(rank.status.success(), "rank exit: {:?}", rank.status); - let rank_out = String::from_utf8(rank.stdout).expect("utf-8"); - let _ = std::fs::remove_dir_all(&cwd); - - assert!(rank_out.contains("plateau(mean)="), "rank output missing plateau(mean)=: {rank_out:?}"); - assert!(rank_out.contains(" cells"), "rank output missing 'over N cells': {rank_out:?}"); - assert!(rank_out.contains("\"mode\":\"PlateauMean\""), "manifest carries PlateauMean: {rank_out:?}"); - assert!(!rank_out.contains("\"deflated_score\""), "plateau run omits deflated_score: {rank_out:?}"); -} -``` - -- [ ] **Step 2: Run the plateau E2E (GREEN)** - -Run: `cargo test -p aura-cli walkforward_plateau` -Expected: PASS. - -- [ ] **Step 3: Full CLI suite + workspace + clippy (no regressions)** - -Run: `cargo test -p aura-cli` -Expected: PASS — every existing golden green; the plateau E2E green. - -Run: `cargo test --workspace` -Expected: PASS. - -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: clean (no warnings). diff --git a/docs/specs/0077-plateau-over-peak-selection.md b/docs/specs/0077-plateau-over-peak-selection.md deleted file mode 100644 index b8312a5..0000000 --- a/docs/specs/0077-plateau-over-peak-selection.md +++ /dev/null @@ -1,301 +0,0 @@ -# Prefer a robust parameter plateau over the in-sample peak — Design Spec - -**Date:** 2026-06-26 -**Status:** Draft — awaiting user spec review -**Authors:** orchestrator + Claude - -Cycle 0077 — second cycle of the milestone *Inferential validation*. Closes #145 -(refs the milestone). Builds on #144's selection-provenance carrier. - -## Goal - -`optimize` (and the walk-forward in-sample step that calls it) picks the single -sharpest in-sample peak by a bare argmax — the configuration most likely fit to -noise, since a small shift in the data or the parameters collapses it. There is no -way to ask instead for a *robust region*: a broad neighbourhood of the parameter -grid that is uniformly good. - -This cycle adds an **opt-in plateau selection objective**: a new `optimize_plateau` -that scores each grid member by the aggregate (mean or worst-case) of its grid -neighbours' metric and argmaxes that *smoothed* surface, so the chosen member sits -at the centre of a broad plateau rather than on a lucky spike. It is **strictly -opt-in** via a new `--select` flag (default `argmax` — every existing run stays -byte-identical, C23), and records its provenance on the same `RunManifest.selection` -carrier #144 introduced (extended for the two selection rules). - -## Architecture - -A new family-level selector sits beside `optimize` / `optimize_deflated` in -`aura-registry`: - -`optimize_plateau(family, axis_lens, metric, mode)` reads each member's metric via -the shared `metric_value` (#144's refactor — one direction source), computes the -neighbourhood-aggregated score over the grid lattice, and argmaxes the smoothed -surface by the metric's own direction (earliest-odometer tie, as `optimize`). The -**grid neighbours** come from mixed-radix arithmetic over `axis_lens` (the per-axis -cardinalities, in `param_space()` order, last-axis-fastest — the `GridSpace` -odometer convention): a member's flat index decomposes to per-axis coordinates; its -±1-per-axis cells are its neighbours; the closed neighbourhood is `{self} ∪ -{in-range ±1 neighbours}`. Pure, no RNG → deterministic (C1). - -`axis_lens` is **an argument, not a `SweepFamily` field** (adding a field breaks -`SweepFamily`'s derived `PartialEq` + its struct-literal sites). The `GridSpace` -already owns the cardinalities at the live walk-forward call site; a new -`GridSpace::axis_lens()` surfaces them, and the sweep builder returns them alongside -the family so the call site can pass them in. The lattice is needed only transiently -at selection time. - -The selection RULE (argmax vs plateau) and the deflation ANNOTATION (#144) are -**orthogonal**, so the shared `FamilySelection` carrier is extended — not duplicated -— to record either: `SelectionMode` gains `PlateauMean` / `PlateauWorst` (the slot -#144 reserved), `deflated_score` becomes `Option` (None under plateau), and two -plateau fields are added (None under argmax). `optimize_deflated` fills the -deflation fields under `mode = Argmax`; `optimize_plateau` fills the plateau fields -under `mode = Plateau*`. - -The policy lives in `aura-registry` (C9); `walk_forward` stays selection-agnostic. -A `--select` flag is parsed in the CLI and threaded to `walkforward_family`, which -dispatches `optimize_deflated` (argmax) or `optimize_plateau` (plateau). A -**RandomSpace** sweep has no lattice, so `--select plateau` on one is **refused** -(exit 2) — plateau adjacency falls out of a grid only. - -## Concrete code shapes - -### User-facing program (the acceptance evidence) - -```console -$ aura walkforward --strategy stage1-r --select plateau:mean \ - --fast 50,100 --slow 200,400 --stop-length 14,21 --stop-k 2.0,3.0 -``` - -Default (no `--select`) is `argmax` — byte-identical to today. With -`--select plateau:mean`, each window's in-sample winner is the centre of the -broadest 4-D grid plateau, and its OOS manifest records the plateau provenance: - -```jsonc -"selection": { - "selection_metric": "sqn_normalized", - "n_trials": 16, - "raw_winner_metric": 1.42, // the winner's own metric (still surfaced) - "mode": "PlateauMean", - "neighbourhood_score": 1.27, // the smoothed score the argmax maximised - "n_neighbours": 5 // closed-neighbourhood size at this cell (≤ 1 + 2·dim) - // deflated_score / overfit_probability / n_resamples … omitted under plateau -} -``` - -`aura runs family rank sqn_normalized` shows a human-readable line: - -```console -# … each member, best-first: sqn_normalized=1.42 plateau(mean)=1.27 over 5 cells -``` - -A plateau request on a random sweep is refused: - -```console -$ aura walkforward --strategy stage1-r --real EURUSD --random 64 --select plateau:mean -aura: --select plateau requires a grid sweep; a random sweep has no parameter lattice (exit 2) -``` - -### `FamilySelection` / `SelectionMode` — before → after (`aura-engine/src/report.rs`) - -```rust -// before (#144) -pub enum SelectionMode { Argmax } -pub struct FamilySelection { - pub selection_metric: String, - pub n_trials: usize, - pub raw_winner_metric: f64, - pub deflated_score: f64, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub overfit_probability: Option, - pub mode: SelectionMode, - pub n_resamples: usize, - pub block_len: usize, - pub seed: u64, -} - -// after (#145) — orthogonal rule × annotation; legacy lines still load (serde default) -pub enum SelectionMode { Argmax, PlateauMean, PlateauWorst } -pub struct FamilySelection { - pub selection_metric: String, - pub n_trials: usize, - pub raw_winner_metric: f64, - pub mode: SelectionMode, - // deflation annotation (present iff mode == Argmax with a deflation run) - #[serde(default, skip_serializing_if = "Option::is_none")] - pub deflated_score: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub overfit_probability: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub n_resamples: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub block_len: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub seed: Option, - // plateau annotation (present iff mode is Plateau*) - #[serde(default, skip_serializing_if = "Option::is_none")] - pub neighbourhood_score: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub n_neighbours: Option, -} -``` - -(`#144`'s `optimize_deflated` and its tests update to the `Option` deflation fields -— `deflated_score: Some(raw - p95)`, etc.; the existing assertions become -`sel.deflated_score.unwrap()`. This reshape is a small wire change to a -one-cycle-old, test-only-data type; the #144 serde-back-compat tests are the green -gate it must preserve.) - -### `GridSpace::axis_lens` + sweep-builder surfacing (`aura-engine/src/sweep.rs`) - -```rust -impl GridSpace { - /// Per-axis cardinalities in `param_space()` order (the odometer radixes, - /// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau - /// neighbourhood walks. - pub fn axis_lens(&self) -> Vec { self.axes.iter().map(Vec::len).collect() } -} -``` - -The sweep entry the CLI builder calls returns the family **and** its `axis_lens` -(e.g. a `sweep_with_lattice` variant, or the builder exposes the built `GridSpace`), -so `sweep_over` / `stage1_r_sweep_over` hand `(SweepFamily, Vec)` back to -`walkforward_family`. A `RandomSpace` yields `None` for the lattice (no axes). - -### The selector (`aura-registry/src/lib.rs`, beside `optimize_deflated`) - -```rust -#[derive(Clone, Copy)] -pub enum PlateauMode { Mean, Worst } - -/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface plus its plateau -/// provenance. Each member scores as the mean (or worst-case) of its closed grid -/// neighbourhood's `metric_value`; the winner is the best smoothed score by the -/// metric's own direction (earliest-odometer tie). `axis_lens` are the grid -/// radixes (last-axis-fastest). Pure, deterministic (C1). -pub fn optimize_plateau( - family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode, -) -> Result<(SweepPoint, FamilySelection), RegistryError> { - let m = resolve_metric(metric)?; - // smoothed score per member i: - // nbrs = closed_neighbourhood(i, axis_lens) // {i} ∪ in-range ±1-per-axis - // vals = nbrs.map(|j| metric_value(&family.points[j].report, m)) - // score[i] = match mode { Mean => mean(vals), Worst => worst(vals, dir) } - // winner = argmax_by_direction(score, m) // earliest-odometer tie, as optimize - // ... returns (winner.clone(), FamilySelection { mode, raw_winner_metric, - // neighbourhood_score: Some(score[winner]), n_neighbours: Some(nbrs.len()), … }) -} - -// closed_neighbourhood(i, axis_lens): decompose i to mixed-radix coords (last -// radix fastest), emit i plus each coord±1 that stays in [0, axis_lens[k]). -fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec { /* pure index math */ } -``` - -`Worst` is the most-conservative neighbour by the metric's direction (the `min` for -a higher-is-better metric); it biases toward interior cells (a boundary cell has -fewer neighbours), which the testing strategy pins. - -### CLI `--select` (`aura-cli/src/main.rs`) - -```rust -enum Selection { Argmax, Plateau(PlateauMode) } -// parse_walkforward_args: "--select argmax" | "plateau:mean" | "plateau:worst"; -// default Argmax; an unknown token is a usage error (exit 2). -// walkforward_family, per arm: -// let (is_family, axis_lens) = sweep_over(...); // now returns the lattice -// let (best, selection) = match select { -// Selection::Argmax => optimize_deflated(&is_family, metric, …), -// Selection::Plateau(pm) => match axis_lens { -// Some(lens) => optimize_plateau(&is_family, &lens, metric, pm), -// None => { eprintln!("aura: --select plateau requires a grid sweep …"); exit(2) } -// }, -// }; -// oos_report.manifest.selection = Some(selection); // the #144 stamping seam -``` - -`runs_family` display gains a `plateau()= over cells` line when -`mode` is `Plateau*` (beside the existing deflated line for argmax). - -## Components - -| Component | Crate / file | Change | -|---|---|---| -| `SelectionMode` + `FamilySelection` reshape | `aura-engine/report.rs` | 2 enum variants; `deflated_score`→`Option`; deflation-trio→`Option`; add `neighbourhood_score`/`n_neighbours` | -| `GridSpace::axis_lens` + sweep-builder lattice return | `aura-engine/sweep.rs` | new accessor; sweep returns `(SweepFamily, axis_lens)` | -| `optimize_plateau`, `PlateauMode`, `closed_neighbourhood` | `aura-registry/lib.rs` | new selector + pure index math; `optimize_deflated` updated to `Option` deflation fields | -| compat mirror | `aura-registry/compat.rs` | already carries `selection` (#144) — no new field, but confirm the reshaped `FamilySelection` round-trips | -| `--select` parse + dispatch + display | `aura-cli/main.rs` | flag, `walkforward_family` dispatch, lattice threading, `runs_family` line | - -## Data flow - -1. `parse_walkforward_args` reads `--select` → `Selection` (default `Argmax`). -2. Per window, the arm's sweep returns `(is_family, Option)`. -3. `Argmax` → `optimize_deflated` (unchanged path, #144). `Plateau(mode)` → with - `Some(lens)` → `optimize_plateau`; with `None` (RandomSpace) → refuse, exit 2. -4. `optimize_plateau` smooths each member's metric over its closed grid - neighbourhood, argmaxes by direction, returns the winner + a `FamilySelection` - with `mode = Plateau*`, `neighbourhood_score`, `n_neighbours`. -5. The record stamps `oos_report.manifest.selection`; flows through `append_family` - unchanged; `runs_family` renders it. - -## Error handling - -- `--select plateau` on a random sweep (no lattice): `exit 2` with a clear message - (refuse-don't-guess, C18 discipline) — never a silent argmax fallback. -- An unknown `--select` token: usage error, `exit 2`. -- A degenerate single-member family: its closed neighbourhood is itself; the plateau - score equals the raw — `optimize_plateau`'s winner equals `optimize`'s (no spurious - reshaping of a 1-point grid). -- `axis_lens` whose product ≠ `family.points.len()` (a malformed lattice) is an - internal invariant violation, asserted (`debug_assert`); the shipped call sites - pass the grid's own lengths, so it is unreachable. - -## Testing strategy - -- **C23 default-argmax byte-identical:** `--select argmax` (and no `--select`) - produces a walk-forward run byte-for-byte identical to the pre-0077 output (the - existing walk-forward goldens stay green unchanged); plateau is strictly opt-in. -- **#144 reshape preserved:** `optimize_deflated`'s tests + the serde back-compat - tests stay green after `deflated_score` → `Option` (a legacy `selection`-less line - loads as `None`; a stamped argmax line round-trips with the deflation fields). -- **Plateau picks the plateau, not the spike:** a fabricated grid `SweepFamily` with - a sharp isolated peak and a separate broad plateau — `optimize` picks the spike, - `optimize_plateau(Mean)` picks the plateau centre (a different `SweepPoint`); the - winner is the smoothed argmax, recorded `neighbourhood_score < raw_winner_metric` - of the spike. -- **mean vs worst:** on the same fixture, `Worst` is more conservative and biases - toward interior cells (a boundary cell loses neighbours) — pinned so the - edge-truncation asymmetry is documented, not silent. -- **Mixed-radix neighbours:** `closed_neighbourhood(i, axis_lens)` golden on a known - small lattice (e.g. 2×3) — exact neighbour index sets, last-axis-fastest. -- **Small-grid degeneracy:** a 2-value axis → the neighbourhood is most of the grid; - a documented caveat test (plateau ≈ global mean), not a failure. -- **RandomSpace refuse:** `--select plateau` on a `--random` sweep exits 2 with the - message; never silently argmaxes. -- **C1 determinism:** identical family + `axis_lens` + mode → identical winner + - `FamilySelection` (pure, no RNG). - -## Acceptance criteria - -- `aura walkforward … --select plateau:mean|plateau:worst` selects the - neighbourhood-smoothed winner and stamps `mode`/`neighbourhood_score`/`n_neighbours` - on each OOS winner's manifest; `runs family … rank` surfaces the plateau line. -- Default `argmax` keeps every existing walk-forward run byte-identical (C23) and the - #144 deflation path intact. -- `--select plateau` on a random sweep is refused (exit 2). -- The selection is deterministic from the family + axis_lens + mode (C1), reads only - the in-sample family (C2), and lives in `aura-registry` with `walk_forward` - selection-agnostic (C9). Legacy registry lines still load (C14/C18). - -### Out of scope (deferred) - -- RandomSpace plateau via a kNN distance metric (an invented adjacency — its own - cycle if ever wanted). -- Offline re-plateau of a reloaded family (the lattice is transient at live - selection; a reloaded family carries no `axis_lens`). -- Per-arm selection (one global `--select` suffices; the metric already differs per - arm and the objective is metric-agnostic). -- Composing plateau selection with trials-deflation on the same winner (the two - annotations are orthogonal but their composition is a later concern; this cycle - records one rule per run).