From 45b8e2b28b836de0bbd07bf9774392d1b09d0117 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 26 Jun 2026 15:14:48 +0200 Subject: [PATCH] spec: 0077 plateau over peak selection (boss-signed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second cycle of the inferential-validation milestone. Adds an opt-in plateau selection objective: optimize_plateau beside optimize/optimize_deflated scores each grid member by the mean/worst of its grid-neighbourhood metric and argmaxes the smoothed surface (mixed-radix neighbours over GridSpace axis_lens, passed as an argument not a SweepFamily field). Strictly opt-in via --select , default argmax (C23 byte-preserving). Reconciles the shared RunManifest.selection carrier with #144's landed reality: the selection RULE (argmax vs plateau) and the deflation ANNOTATION are orthogonal, so FamilySelection is extended — deflation fields become Option, SelectionMode gains PlateauMean/PlateauWorst (the reserved slot), and neighbourhood_score/n_neighbours are added. A small wire change to the one-cycle-old type; legacy lines still load. RandomSpace plateau refused (exit 2 — no lattice); kNN deferred. Grounding-check PASS (12 assumptions ratified). refs #145 --- .../specs/0077-plateau-over-peak-selection.md | 301 ++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 docs/specs/0077-plateau-over-peak-selection.md diff --git a/docs/specs/0077-plateau-over-peak-selection.md b/docs/specs/0077-plateau-over-peak-selection.md new file mode 100644 index 0000000..b8312a5 --- /dev/null +++ b/docs/specs/0077-plateau-over-peak-selection.md @@ -0,0 +1,301 @@ +# Prefer a robust parameter plateau over the in-sample peak — Design Spec + +**Date:** 2026-06-26 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +Cycle 0077 — second cycle of the milestone *Inferential validation*. Closes #145 +(refs the milestone). Builds on #144's selection-provenance carrier. + +## Goal + +`optimize` (and the walk-forward in-sample step that calls it) picks the single +sharpest in-sample peak by a bare argmax — the configuration most likely fit to +noise, since a small shift in the data or the parameters collapses it. There is no +way to ask instead for a *robust region*: a broad neighbourhood of the parameter +grid that is uniformly good. + +This cycle adds an **opt-in plateau selection objective**: a new `optimize_plateau` +that scores each grid member by the aggregate (mean or worst-case) of its grid +neighbours' metric and argmaxes that *smoothed* surface, so the chosen member sits +at the centre of a broad plateau rather than on a lucky spike. It is **strictly +opt-in** via a new `--select` flag (default `argmax` — every existing run stays +byte-identical, C23), and records its provenance on the same `RunManifest.selection` +carrier #144 introduced (extended for the two selection rules). + +## Architecture + +A new family-level selector sits beside `optimize` / `optimize_deflated` in +`aura-registry`: + +`optimize_plateau(family, axis_lens, metric, mode)` reads each member's metric via +the shared `metric_value` (#144's refactor — one direction source), computes the +neighbourhood-aggregated score over the grid lattice, and argmaxes the smoothed +surface by the metric's own direction (earliest-odometer tie, as `optimize`). The +**grid neighbours** come from mixed-radix arithmetic over `axis_lens` (the per-axis +cardinalities, in `param_space()` order, last-axis-fastest — the `GridSpace` +odometer convention): a member's flat index decomposes to per-axis coordinates; its +±1-per-axis cells are its neighbours; the closed neighbourhood is `{self} ∪ +{in-range ±1 neighbours}`. Pure, no RNG → deterministic (C1). + +`axis_lens` is **an argument, not a `SweepFamily` field** (adding a field breaks +`SweepFamily`'s derived `PartialEq` + its struct-literal sites). The `GridSpace` +already owns the cardinalities at the live walk-forward call site; a new +`GridSpace::axis_lens()` surfaces them, and the sweep builder returns them alongside +the family so the call site can pass them in. The lattice is needed only transiently +at selection time. + +The selection RULE (argmax vs plateau) and the deflation ANNOTATION (#144) are +**orthogonal**, so the shared `FamilySelection` carrier is extended — not duplicated +— to record either: `SelectionMode` gains `PlateauMean` / `PlateauWorst` (the slot +#144 reserved), `deflated_score` becomes `Option` (None under plateau), and two +plateau fields are added (None under argmax). `optimize_deflated` fills the +deflation fields under `mode = Argmax`; `optimize_plateau` fills the plateau fields +under `mode = Plateau*`. + +The policy lives in `aura-registry` (C9); `walk_forward` stays selection-agnostic. +A `--select` flag is parsed in the CLI and threaded to `walkforward_family`, which +dispatches `optimize_deflated` (argmax) or `optimize_plateau` (plateau). A +**RandomSpace** sweep has no lattice, so `--select plateau` on one is **refused** +(exit 2) — plateau adjacency falls out of a grid only. + +## Concrete code shapes + +### User-facing program (the acceptance evidence) + +```console +$ aura walkforward --strategy stage1-r --select plateau:mean \ + --fast 50,100 --slow 200,400 --stop-length 14,21 --stop-k 2.0,3.0 +``` + +Default (no `--select`) is `argmax` — byte-identical to today. With +`--select plateau:mean`, each window's in-sample winner is the centre of the +broadest 4-D grid plateau, and its OOS manifest records the plateau provenance: + +```jsonc +"selection": { + "selection_metric": "sqn_normalized", + "n_trials": 16, + "raw_winner_metric": 1.42, // the winner's own metric (still surfaced) + "mode": "PlateauMean", + "neighbourhood_score": 1.27, // the smoothed score the argmax maximised + "n_neighbours": 5 // closed-neighbourhood size at this cell (≤ 1 + 2·dim) + // deflated_score / overfit_probability / n_resamples … omitted under plateau +} +``` + +`aura runs family rank sqn_normalized` shows a human-readable line: + +```console +# … each member, best-first: sqn_normalized=1.42 plateau(mean)=1.27 over 5 cells +``` + +A plateau request on a random sweep is refused: + +```console +$ aura walkforward --strategy stage1-r --real EURUSD --random 64 --select plateau:mean +aura: --select plateau requires a grid sweep; a random sweep has no parameter lattice (exit 2) +``` + +### `FamilySelection` / `SelectionMode` — before → after (`aura-engine/src/report.rs`) + +```rust +// before (#144) +pub enum SelectionMode { Argmax } +pub struct FamilySelection { + pub selection_metric: String, + pub n_trials: usize, + pub raw_winner_metric: f64, + pub deflated_score: f64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub overfit_probability: Option, + pub mode: SelectionMode, + pub n_resamples: usize, + pub block_len: usize, + pub seed: u64, +} + +// after (#145) — orthogonal rule × annotation; legacy lines still load (serde default) +pub enum SelectionMode { Argmax, PlateauMean, PlateauWorst } +pub struct FamilySelection { + pub selection_metric: String, + pub n_trials: usize, + pub raw_winner_metric: f64, + pub mode: SelectionMode, + // deflation annotation (present iff mode == Argmax with a deflation run) + #[serde(default, skip_serializing_if = "Option::is_none")] + pub deflated_score: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub overfit_probability: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n_resamples: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub block_len: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub seed: Option, + // plateau annotation (present iff mode is Plateau*) + #[serde(default, skip_serializing_if = "Option::is_none")] + pub neighbourhood_score: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub n_neighbours: Option, +} +``` + +(`#144`'s `optimize_deflated` and its tests update to the `Option` deflation fields +— `deflated_score: Some(raw - p95)`, etc.; the existing assertions become +`sel.deflated_score.unwrap()`. This reshape is a small wire change to a +one-cycle-old, test-only-data type; the #144 serde-back-compat tests are the green +gate it must preserve.) + +### `GridSpace::axis_lens` + sweep-builder surfacing (`aura-engine/src/sweep.rs`) + +```rust +impl GridSpace { + /// Per-axis cardinalities in `param_space()` order (the odometer radixes, + /// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau + /// neighbourhood walks. + pub fn axis_lens(&self) -> Vec { self.axes.iter().map(Vec::len).collect() } +} +``` + +The sweep entry the CLI builder calls returns the family **and** its `axis_lens` +(e.g. a `sweep_with_lattice` variant, or the builder exposes the built `GridSpace`), +so `sweep_over` / `stage1_r_sweep_over` hand `(SweepFamily, Vec)` back to +`walkforward_family`. A `RandomSpace` yields `None` for the lattice (no axes). + +### The selector (`aura-registry/src/lib.rs`, beside `optimize_deflated`) + +```rust +#[derive(Clone, Copy)] +pub enum PlateauMode { Mean, Worst } + +/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface plus its plateau +/// provenance. Each member scores as the mean (or worst-case) of its closed grid +/// neighbourhood's `metric_value`; the winner is the best smoothed score by the +/// metric's own direction (earliest-odometer tie). `axis_lens` are the grid +/// radixes (last-axis-fastest). Pure, deterministic (C1). +pub fn optimize_plateau( + family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode, +) -> Result<(SweepPoint, FamilySelection), RegistryError> { + let m = resolve_metric(metric)?; + // smoothed score per member i: + // nbrs = closed_neighbourhood(i, axis_lens) // {i} ∪ in-range ±1-per-axis + // vals = nbrs.map(|j| metric_value(&family.points[j].report, m)) + // score[i] = match mode { Mean => mean(vals), Worst => worst(vals, dir) } + // winner = argmax_by_direction(score, m) // earliest-odometer tie, as optimize + // ... returns (winner.clone(), FamilySelection { mode, raw_winner_metric, + // neighbourhood_score: Some(score[winner]), n_neighbours: Some(nbrs.len()), … }) +} + +// closed_neighbourhood(i, axis_lens): decompose i to mixed-radix coords (last +// radix fastest), emit i plus each coord±1 that stays in [0, axis_lens[k]). +fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec { /* pure index math */ } +``` + +`Worst` is the most-conservative neighbour by the metric's direction (the `min` for +a higher-is-better metric); it biases toward interior cells (a boundary cell has +fewer neighbours), which the testing strategy pins. + +### CLI `--select` (`aura-cli/src/main.rs`) + +```rust +enum Selection { Argmax, Plateau(PlateauMode) } +// parse_walkforward_args: "--select argmax" | "plateau:mean" | "plateau:worst"; +// default Argmax; an unknown token is a usage error (exit 2). +// walkforward_family, per arm: +// let (is_family, axis_lens) = sweep_over(...); // now returns the lattice +// let (best, selection) = match select { +// Selection::Argmax => optimize_deflated(&is_family, metric, …), +// Selection::Plateau(pm) => match axis_lens { +// Some(lens) => optimize_plateau(&is_family, &lens, metric, pm), +// None => { eprintln!("aura: --select plateau requires a grid sweep …"); exit(2) } +// }, +// }; +// oos_report.manifest.selection = Some(selection); // the #144 stamping seam +``` + +`runs_family` display gains a `plateau()= over cells` line when +`mode` is `Plateau*` (beside the existing deflated line for argmax). + +## Components + +| Component | Crate / file | Change | +|---|---|---| +| `SelectionMode` + `FamilySelection` reshape | `aura-engine/report.rs` | 2 enum variants; `deflated_score`→`Option`; deflation-trio→`Option`; add `neighbourhood_score`/`n_neighbours` | +| `GridSpace::axis_lens` + sweep-builder lattice return | `aura-engine/sweep.rs` | new accessor; sweep returns `(SweepFamily, axis_lens)` | +| `optimize_plateau`, `PlateauMode`, `closed_neighbourhood` | `aura-registry/lib.rs` | new selector + pure index math; `optimize_deflated` updated to `Option` deflation fields | +| compat mirror | `aura-registry/compat.rs` | already carries `selection` (#144) — no new field, but confirm the reshaped `FamilySelection` round-trips | +| `--select` parse + dispatch + display | `aura-cli/main.rs` | flag, `walkforward_family` dispatch, lattice threading, `runs_family` line | + +## Data flow + +1. `parse_walkforward_args` reads `--select` → `Selection` (default `Argmax`). +2. Per window, the arm's sweep returns `(is_family, Option)`. +3. `Argmax` → `optimize_deflated` (unchanged path, #144). `Plateau(mode)` → with + `Some(lens)` → `optimize_plateau`; with `None` (RandomSpace) → refuse, exit 2. +4. `optimize_plateau` smooths each member's metric over its closed grid + neighbourhood, argmaxes by direction, returns the winner + a `FamilySelection` + with `mode = Plateau*`, `neighbourhood_score`, `n_neighbours`. +5. The record stamps `oos_report.manifest.selection`; flows through `append_family` + unchanged; `runs_family` renders it. + +## Error handling + +- `--select plateau` on a random sweep (no lattice): `exit 2` with a clear message + (refuse-don't-guess, C18 discipline) — never a silent argmax fallback. +- An unknown `--select` token: usage error, `exit 2`. +- A degenerate single-member family: its closed neighbourhood is itself; the plateau + score equals the raw — `optimize_plateau`'s winner equals `optimize`'s (no spurious + reshaping of a 1-point grid). +- `axis_lens` whose product ≠ `family.points.len()` (a malformed lattice) is an + internal invariant violation, asserted (`debug_assert`); the shipped call sites + pass the grid's own lengths, so it is unreachable. + +## Testing strategy + +- **C23 default-argmax byte-identical:** `--select argmax` (and no `--select`) + produces a walk-forward run byte-for-byte identical to the pre-0077 output (the + existing walk-forward goldens stay green unchanged); plateau is strictly opt-in. +- **#144 reshape preserved:** `optimize_deflated`'s tests + the serde back-compat + tests stay green after `deflated_score` → `Option` (a legacy `selection`-less line + loads as `None`; a stamped argmax line round-trips with the deflation fields). +- **Plateau picks the plateau, not the spike:** a fabricated grid `SweepFamily` with + a sharp isolated peak and a separate broad plateau — `optimize` picks the spike, + `optimize_plateau(Mean)` picks the plateau centre (a different `SweepPoint`); the + winner is the smoothed argmax, recorded `neighbourhood_score < raw_winner_metric` + of the spike. +- **mean vs worst:** on the same fixture, `Worst` is more conservative and biases + toward interior cells (a boundary cell loses neighbours) — pinned so the + edge-truncation asymmetry is documented, not silent. +- **Mixed-radix neighbours:** `closed_neighbourhood(i, axis_lens)` golden on a known + small lattice (e.g. 2×3) — exact neighbour index sets, last-axis-fastest. +- **Small-grid degeneracy:** a 2-value axis → the neighbourhood is most of the grid; + a documented caveat test (plateau ≈ global mean), not a failure. +- **RandomSpace refuse:** `--select plateau` on a `--random` sweep exits 2 with the + message; never silently argmaxes. +- **C1 determinism:** identical family + `axis_lens` + mode → identical winner + + `FamilySelection` (pure, no RNG). + +## Acceptance criteria + +- `aura walkforward … --select plateau:mean|plateau:worst` selects the + neighbourhood-smoothed winner and stamps `mode`/`neighbourhood_score`/`n_neighbours` + on each OOS winner's manifest; `runs family … rank` surfaces the plateau line. +- Default `argmax` keeps every existing walk-forward run byte-identical (C23) and the + #144 deflation path intact. +- `--select plateau` on a random sweep is refused (exit 2). +- The selection is deterministic from the family + axis_lens + mode (C1), reads only + the in-sample family (C2), and lives in `aura-registry` with `walk_forward` + selection-agnostic (C9). Legacy registry lines still load (C14/C18). + +### Out of scope (deferred) + +- RandomSpace plateau via a kNN distance metric (an invented adjacency — its own + cycle if ever wanted). +- Offline re-plateau of a reloaded family (the lattice is transient at live + selection; a reloaded family carries no `axis_lens`). +- Per-arm selection (one global `--select` suffices; the metric already differs per + arm and the objective is metric-agnostic). +- Composing plateau selection with trials-deflation on the same winner (the two + annotations are orthogonal but their composition is a later concern; this cycle + records one rule per run).