Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

spec: 0077 plateau over peak selection (boss-signed)

Browse Source 
Second cycle of the inferential-validation milestone. Adds an opt-in plateau
selection objective: optimize_plateau beside optimize/optimize_deflated scores
each grid member by the mean/worst of its grid-neighbourhood metric and argmaxes
the smoothed surface (mixed-radix neighbours over GridSpace axis_lens, passed as
an argument not a SweepFamily field). Strictly opt-in via --select
<argmax|plateau:mean|plateau:worst>, default argmax (C23 byte-preserving).

Reconciles the shared RunManifest.selection carrier with #144's landed reality:
the selection RULE (argmax vs plateau) and the deflation ANNOTATION are orthogonal,
so FamilySelection is extended — deflation fields become Option, SelectionMode gains
PlateauMean/PlateauWorst (the reserved slot), and neighbourhood_score/n_neighbours
are added. A small wire change to the one-cycle-old type; legacy lines still load.
RandomSpace plateau refused (exit 2 — no lattice); kNN deferred.

Grounding-check PASS (12 assumptions ratified). refs #145
This commit is contained in:
Brummel
2026-06-26 15:14:48 +02:00
parent 9c6b9c7ea9
commit 45b8e2b28b
1 changed files with 301 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/0077-plateau-over-peak-selection.md
+301
View File
@@ -0,0 +1,301 @@
# Prefer a robust parameter plateau over the in-sample peak — Design Spec
**Date:** 2026-06-26
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude
Cycle 0077 — second cycle of the milestone *Inferential validation*. Closes #145
(refs the milestone). Builds on #144's selection-provenance carrier.
## Goal
`optimize` (and the walk-forward in-sample step that calls it) picks the single
sharpest in-sample peak by a bare argmax — the configuration most likely fit to
noise, since a small shift in the data or the parameters collapses it. There is no
way to ask instead for a *robust region*: a broad neighbourhood of the parameter
grid that is uniformly good.
This cycle adds an **opt-in plateau selection objective**: a new `optimize_plateau`
that scores each grid member by the aggregate (mean or worst-case) of its grid
neighbours' metric and argmaxes that *smoothed* surface, so the chosen member sits
at the centre of a broad plateau rather than on a lucky spike. It is **strictly
opt-in** via a new `--select` flag (default `argmax` — every existing run stays
byte-identical, C23), and records its provenance on the same `RunManifest.selection`
carrier #144 introduced (extended for the two selection rules).
## Architecture
A new family-level selector sits beside `optimize` / `optimize_deflated` in
`aura-registry`:
`optimize_plateau(family, axis_lens, metric, mode)` reads each member's metric via
the shared `metric_value` (#144's refactor — one direction source), computes the
neighbourhood-aggregated score over the grid lattice, and argmaxes the smoothed
surface by the metric's own direction (earliest-odometer tie, as `optimize`). The
**grid neighbours** come from mixed-radix arithmetic over `axis_lens` (the per-axis
cardinalities, in `param_space()` order, last-axis-fastest — the `GridSpace`
odometer convention): a member's flat index decomposes to per-axis coordinates; its
±1-per-axis cells are its neighbours; the closed neighbourhood is `{self} ∪
{in-range ±1 neighbours}`. Pure, no RNG → deterministic (C1).
`axis_lens` is **an argument, not a `SweepFamily` field** (adding a field breaks
`SweepFamily`'s derived `PartialEq` + its struct-literal sites). The `GridSpace`
already owns the cardinalities at the live walk-forward call site; a new
`GridSpace::axis_lens()` surfaces them, and the sweep builder returns them alongside
the family so the call site can pass them in. The lattice is needed only transiently
at selection time.
The selection RULE (argmax vs plateau) and the deflation ANNOTATION (#144) are
**orthogonal**, so the shared `FamilySelection` carrier is extended — not duplicated
— to record either: `SelectionMode` gains `PlateauMean` / `PlateauWorst` (the slot
#144 reserved), `deflated_score` becomes `Option` (None under plateau), and two
plateau fields are added (None under argmax). `optimize_deflated` fills the
deflation fields under `mode = Argmax`; `optimize_plateau` fills the plateau fields
under `mode = Plateau*`.
The policy lives in `aura-registry` (C9); `walk_forward` stays selection-agnostic.
A `--select` flag is parsed in the CLI and threaded to `walkforward_family`, which
dispatches `optimize_deflated` (argmax) or `optimize_plateau` (plateau). A
**RandomSpace** sweep has no lattice, so `--select plateau` on one is **refused**
(exit 2) — plateau adjacency falls out of a grid only.
## Concrete code shapes
### User-facing program (the acceptance evidence)
```console
$ aura walkforward --strategy stage1-r --select plateau:mean \
--fast 50,100 --slow 200,400 --stop-length 14,21 --stop-k 2.0,3.0
```
Default (no `--select`) is `argmax` — byte-identical to today. With
`--select plateau:mean`, each window's in-sample winner is the centre of the
broadest 4-D grid plateau, and its OOS manifest records the plateau provenance:
```jsonc
"selection": {
"selection_metric": "sqn_normalized",
"n_trials": 16,
"raw_winner_metric": 1.42, // the winner's own metric (still surfaced)
"mode": "PlateauMean",
"neighbourhood_score": 1.27, // the smoothed score the argmax maximised
"n_neighbours": 5 // closed-neighbourhood size at this cell (≤ 1 + 2·dim)
// deflated_score / overfit_probability / n_resamples … omitted under plateau
}
```
`aura runs family <id> rank sqn_normalized` shows a human-readable line:
```console
# … each member, best-first: sqn_normalized=1.42 plateau(mean)=1.27 over 5 cells
```
A plateau request on a random sweep is refused:
```console
$ aura walkforward --strategy stage1-r --real EURUSD --random 64 --select plateau:mean
aura: --select plateau requires a grid sweep; a random sweep has no parameter lattice (exit 2)
```
### `FamilySelection` / `SelectionMode` — before → after (`aura-engine/src/report.rs`)
```rust
// before (#144)
pub enum SelectionMode { Argmax }
pub struct FamilySelection {
pub selection_metric: String,
pub n_trials: usize,
pub raw_winner_metric: f64,
pub deflated_score: f64,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub overfit_probability: Option<f64>,
pub mode: SelectionMode,
pub n_resamples: usize,
pub block_len: usize,
pub seed: u64,
}
// after (#145) — orthogonal rule × annotation; legacy lines still load (serde default)
pub enum SelectionMode { Argmax, PlateauMean, PlateauWorst }
pub struct FamilySelection {
pub selection_metric: String,
pub n_trials: usize,
pub raw_winner_metric: f64,
pub mode: SelectionMode,
// deflation annotation (present iff mode == Argmax with a deflation run)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub deflated_score: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub overfit_probability: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub n_resamples: Option<usize>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub block_len: Option<usize>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub seed: Option<u64>,
// plateau annotation (present iff mode is Plateau*)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub neighbourhood_score: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub n_neighbours: Option<usize>,
}
```
(`#144`'s `optimize_deflated` and its tests update to the `Option` deflation fields
— `deflated_score: Some(raw - p95)`, etc.; the existing assertions become
`sel.deflated_score.unwrap()`. This reshape is a small wire change to a
one-cycle-old, test-only-data type; the #144 serde-back-compat tests are the green
gate it must preserve.)
### `GridSpace::axis_lens` + sweep-builder surfacing (`aura-engine/src/sweep.rs`)
```rust
impl GridSpace {
/// Per-axis cardinalities in `param_space()` order (the odometer radixes,
/// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau
/// neighbourhood walks.
pub fn axis_lens(&self) -> Vec<usize> { self.axes.iter().map(Vec::len).collect() }
}
```
The sweep entry the CLI builder calls returns the family **and** its `axis_lens`
(e.g. a `sweep_with_lattice` variant, or the builder exposes the built `GridSpace`),
so `sweep_over` / `stage1_r_sweep_over` hand `(SweepFamily, Vec<usize>)` back to
`walkforward_family`. A `RandomSpace` yields `None` for the lattice (no axes).
### The selector (`aura-registry/src/lib.rs`, beside `optimize_deflated`)
```rust
#[derive(Clone, Copy)]
pub enum PlateauMode { Mean, Worst }
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface plus its plateau
/// provenance. Each member scores as the mean (or worst-case) of its closed grid
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the
/// metric's own direction (earliest-odometer tie). `axis_lens` are the grid
/// radixes (last-axis-fastest). Pure, deterministic (C1).
pub fn optimize_plateau(
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode,
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
let m = resolve_metric(metric)?;
// smoothed score per member i:
// nbrs = closed_neighbourhood(i, axis_lens) // {i} ∪ in-range ±1-per-axis
// vals = nbrs.map(|j| metric_value(&family.points[j].report, m))
// score[i] = match mode { Mean => mean(vals), Worst => worst(vals, dir) }
// winner = argmax_by_direction(score, m) // earliest-odometer tie, as optimize
// ... returns (winner.clone(), FamilySelection { mode, raw_winner_metric,
// neighbourhood_score: Some(score[winner]), n_neighbours: Some(nbrs.len()), … })
}
// closed_neighbourhood(i, axis_lens): decompose i to mixed-radix coords (last
// radix fastest), emit i plus each coord±1 that stays in [0, axis_lens[k]).
fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> { /* pure index math */ }
```
`Worst` is the most-conservative neighbour by the metric's direction (the `min` for
a higher-is-better metric); it biases toward interior cells (a boundary cell has
fewer neighbours), which the testing strategy pins.
### CLI `--select` (`aura-cli/src/main.rs`)
```rust
enum Selection { Argmax, Plateau(PlateauMode) }
// parse_walkforward_args: "--select argmax" | "plateau:mean" | "plateau:worst";
// default Argmax; an unknown token is a usage error (exit 2).
// walkforward_family, per arm:
// let (is_family, axis_lens) = sweep_over(...); // now returns the lattice
// let (best, selection) = match select {
// Selection::Argmax => optimize_deflated(&is_family, metric, …),
// Selection::Plateau(pm) => match axis_lens {
// Some(lens) => optimize_plateau(&is_family, &lens, metric, pm),
// None => { eprintln!("aura: --select plateau requires a grid sweep …"); exit(2) }
// },
// };
// oos_report.manifest.selection = Some(selection); // the #144 stamping seam
```
`runs_family` display gains a `plateau(<mode>)=<score> over <n> cells` line when
`mode` is `Plateau*` (beside the existing deflated line for argmax).
## Components
| Component | Crate / file | Change |
|---|---|---|
| `SelectionMode` + `FamilySelection` reshape | `aura-engine/report.rs` | 2 enum variants; `deflated_score`→`Option`; deflation-trio→`Option`; add `neighbourhood_score`/`n_neighbours` |
| `GridSpace::axis_lens` + sweep-builder lattice return | `aura-engine/sweep.rs` | new accessor; sweep returns `(SweepFamily, axis_lens)` |
| `optimize_plateau`, `PlateauMode`, `closed_neighbourhood` | `aura-registry/lib.rs` | new selector + pure index math; `optimize_deflated` updated to `Option` deflation fields |
| compat mirror | `aura-registry/compat.rs` | already carries `selection` (#144) — no new field, but confirm the reshaped `FamilySelection` round-trips |
| `--select` parse + dispatch + display | `aura-cli/main.rs` | flag, `walkforward_family` dispatch, lattice threading, `runs_family` line |
## Data flow
1. `parse_walkforward_args` reads `--select` → `Selection` (default `Argmax`).
2. Per window, the arm's sweep returns `(is_family, Option<axis_lens>)`.
3. `Argmax` → `optimize_deflated` (unchanged path, #144). `Plateau(mode)` → with
`Some(lens)` → `optimize_plateau`; with `None` (RandomSpace) → refuse, exit 2.
4. `optimize_plateau` smooths each member's metric over its closed grid
neighbourhood, argmaxes by direction, returns the winner + a `FamilySelection`
with `mode = Plateau*`, `neighbourhood_score`, `n_neighbours`.
5. The record stamps `oos_report.manifest.selection`; flows through `append_family`
unchanged; `runs_family` renders it.
## Error handling
- `--select plateau` on a random sweep (no lattice): `exit 2` with a clear message
(refuse-don't-guess, C18 discipline) — never a silent argmax fallback.
- An unknown `--select` token: usage error, `exit 2`.
- A degenerate single-member family: its closed neighbourhood is itself; the plateau
score equals the raw — `optimize_plateau`'s winner equals `optimize`'s (no spurious
reshaping of a 1-point grid).
- `axis_lens` whose product ≠ `family.points.len()` (a malformed lattice) is an
internal invariant violation, asserted (`debug_assert`); the shipped call sites
pass the grid's own lengths, so it is unreachable.
## Testing strategy
- **C23 default-argmax byte-identical:** `--select argmax` (and no `--select`)
produces a walk-forward run byte-for-byte identical to the pre-0077 output (the
existing walk-forward goldens stay green unchanged); plateau is strictly opt-in.
- **#144 reshape preserved:** `optimize_deflated`'s tests + the serde back-compat
tests stay green after `deflated_score` → `Option` (a legacy `selection`-less line
loads as `None`; a stamped argmax line round-trips with the deflation fields).
- **Plateau picks the plateau, not the spike:** a fabricated grid `SweepFamily` with
a sharp isolated peak and a separate broad plateau — `optimize` picks the spike,
`optimize_plateau(Mean)` picks the plateau centre (a different `SweepPoint`); the
winner is the smoothed argmax, recorded `neighbourhood_score < raw_winner_metric`
of the spike.
- **mean vs worst:** on the same fixture, `Worst` is more conservative and biases
toward interior cells (a boundary cell loses neighbours) — pinned so the
edge-truncation asymmetry is documented, not silent.
- **Mixed-radix neighbours:** `closed_neighbourhood(i, axis_lens)` golden on a known
small lattice (e.g. 2×3) — exact neighbour index sets, last-axis-fastest.
- **Small-grid degeneracy:** a 2-value axis → the neighbourhood is most of the grid;
a documented caveat test (plateau ≈ global mean), not a failure.
- **RandomSpace refuse:** `--select plateau` on a `--random` sweep exits 2 with the
message; never silently argmaxes.
- **C1 determinism:** identical family + `axis_lens` + mode → identical winner +
`FamilySelection` (pure, no RNG).
## Acceptance criteria
- `aura walkforward … --select plateau:mean|plateau:worst` selects the
neighbourhood-smoothed winner and stamps `mode`/`neighbourhood_score`/`n_neighbours`
on each OOS winner's manifest; `runs family … rank` surfaces the plateau line.
- Default `argmax` keeps every existing walk-forward run byte-identical (C23) and the
#144 deflation path intact.
- `--select plateau` on a random sweep is refused (exit 2).
- The selection is deterministic from the family + axis_lens + mode (C1), reads only
the in-sample family (C2), and lives in `aura-registry` with `walk_forward`
selection-agnostic (C9). Legacy registry lines still load (C14/C18).
### Out of scope (deferred)
- RandomSpace plateau via a kNN distance metric (an invented adjacency — its own
cycle if ever wanted).
- Offline re-plateau of a reloaded family (the lattice is transient at live
selection; a reloaded family carries no `axis_lens`).
- Per-arm selection (one global `--select` suffices; the metric already differs per
arm and the objective is metric-agnostic).
- Composing plateau selection with trials-deflation on the same winner (the two
annotations are orthogonal but their composition is a later concern; this cycle
records one rule per run).