Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Prefer a robust parameter plateau over the in-sample peak in optimize() #145

New Issue
Closed
opened 2026-06-26 10:30:55 +02:00 by Brummel · 2 comments
Brummel commented 2026-06-26 10:30:55 +02:00
Owner
Copy Link
Copy Source

Concern

optimize(family, metric) returns the single argmax member — crates/aura-registry/src/lib.rs:186. By construction it selects the sharpest in-sample peak, with no preference for a parameter region that is stable in its neighbourhood. A peak that sits on a narrow spike of the metric surface is the configuration most likely to be fit to noise: a small shift in the data, or in the parameters themselves, collapses it.

The one adjacent primitive, param_stability (crates/aura-engine/src/walkforward.rs:254), reports the dispersion of the chosen parameters across walk-forward windows, but it is an on-demand post-hoc diagnostic emitted as a CLI field — it never feeds back into selection.

What to build

A selection objective that prefers a broad, stable region of the parameter space over an isolated optimum:

  • A neighbourhood-aware score (a metric averaged or worst-cased over a member's grid neighbours, or a flatness penalty on the local metric gradient) selectable as an alternative to bare argmax in optimize.
  • param_stability, or an equivalent across-window stability measure, usable as a selection criterion and not only as a diagnostic.
  • The chosen selection mode recorded on the manifest (C18) so two runs that selected by peak vs plateau are distinguishable and reproducible.

Relationship to #139 and the trials-deflation issue

Complementary. #139 tests a chosen candidate out-of-sample; the trials-deflation issue corrects the winner's score for family size. This issue changes which member is chosen in the first place — a robust plateau rather than the noise-fit peak — and applies equally to #139's per-window in-sample sweep.

Notes

  • GridSpace already enumerates a cartesian lattice (crates/aura-engine/src/sweep.rs:19), so grid-neighbour lookup is well-defined; RandomSpace neighbourhoods need a distance metric, flagged as a design point.
## Concern `optimize(family, metric)` returns the single argmax member — `crates/aura-registry/src/lib.rs:186`. By construction it selects the sharpest in-sample peak, with no preference for a parameter region that is stable in its neighbourhood. A peak that sits on a narrow spike of the metric surface is the configuration most likely to be fit to noise: a small shift in the data, or in the parameters themselves, collapses it. The one adjacent primitive, `param_stability` (`crates/aura-engine/src/walkforward.rs:254`), reports the dispersion of the chosen parameters across walk-forward windows, but it is an on-demand post-hoc diagnostic emitted as a CLI field — it never feeds back into selection. ## What to build A selection objective that prefers a broad, stable region of the parameter space over an isolated optimum: - [ ] A neighbourhood-aware score (a metric averaged or worst-cased over a member's grid neighbours, or a flatness penalty on the local metric gradient) selectable as an alternative to bare argmax in `optimize`. - [ ] `param_stability`, or an equivalent across-window stability measure, usable as a selection criterion and not only as a diagnostic. - [ ] The chosen selection mode recorded on the manifest (C18) so two runs that selected by peak vs plateau are distinguishable and reproducible. ## Relationship to #139 and the trials-deflation issue Complementary. #139 tests a chosen candidate out-of-sample; the trials-deflation issue corrects the winner's score for family size. This issue changes which member is chosen in the first place — a robust plateau rather than the noise-fit peak — and applies equally to #139's per-window in-sample sweep. ## Notes - `GridSpace` already enumerates a cartesian lattice (`crates/aura-engine/src/sweep.rs:19`), so grid-neighbour lookup is well-defined; `RandomSpace` neighbourhoods need a distance metric, flagged as a design point.
Brummel added this to the Inferential validation (defend against false discovery at sweep scale) milestone 2026-06-26 10:30:55 +02:00
Brummel added the feature label 2026-06-26 10:30:55 +02:00
Brummel commented 2026-06-26 15:09:11 +02:00
Author
Owner
Copy Link
Copy Source

Design reconciliation (specify, /boss) — spec 0077

Entry path: specify (design settled by the milestone design-recon + judge verdict;
the forks below are derived, not user-preference). Trunk: a new
optimize_plateau(family, axis_lens, metric, mode) beside optimize/optimize_deflated
that argmaxes a neighbourhood-aggregated (mean/worst over grid neighbours) score —
a parameter-free robustness statement, NOT a gradient-penalty (which needs a
metric-scale-sensitive free knob with no principled default).

Derived decisions (rationale, recorded for after-the-fact audit):

  • axis_lens carry → passed as an ARGUMENT to optimize_plateau, never a
    SweepFamily field. Basis: derived — a field breaks SweepFamily's derived
    PartialEq + four SweepFamily { space, points } struct-literal sites; the
    GridSpace owns the per-axis cardinalities at the live walk-forward call site, and
    the lattice is only needed transiently at selection. Needs a new
    GridSpace::axis_lens() (per-axis lengths in param_space() order) surfaced out
    of the sweep builder.
  • RandomSpace plateau → REFUSE --select plateau on a random sweep (exit 2).
    Basis: derived — plateau adjacency falls out of a GridSpace lattice; a kNN
    metric+radius is an invented design choice, and the CLI walk-forward path is
    all-GridSpace, so RandomSpace plateau is unexercised. kNN deferred to its own cycle.
  • Aggregate → expose both --select plateau:mean|plateau:worst, default mean.
    Basis: derived — mean is the less-surprising statement; worst has an
    edge-neighbour-truncation bias toward interior points (worth a test before a
    default); the chosen mode is recorded, so runs stay reproducible.
  • Flag scope → one global --select <argmax|plateau:mean|plateau:worst>,
    default argmax. Basis: derived — C23 byte-preserving (every existing run stays
    identical, plateau strictly opt-in); per-arm selection is unmotivated.
  • FamilySelection carrier evolution (the load-bearing reconciliation) → the
    recon predated #144, which landed a deflation-shaped FamilySelection +
    SelectionMode{Argmax}. The selection RULE (argmax vs plateau — which member
    wins) and the deflation ANNOTATION (how inflated the winner is) are ORTHOGONAL.
    Resolution: extend the shared carrier minimally — add SelectionMode::PlateauMean
    / PlateauWorst (the slot #144 reserved), make deflated_score: f64 → Option<f64>
    (None under plateau, the one genuinely deflation-specific scalar), and add
    neighbourhood_score: Option<f64> + n_neighbours: Option<usize> (None under
    argmax). optimize_deflated populates the deflation fields under mode=Argmax;
    optimize_plateau populates the plateau fields under mode=Plateau*. Basis:
    derived — one composable carrier across both milestone pieces beats a second
    parallel type; the reshape is a small wire change to #144's one-cycle-old,
    test-only-data type (legacy lines still load via serde(default)); #144's existing
    deflation + serde-back-compat tests are the green gate the reshape must preserve.

Out of scope (deferred): RandomSpace kNN plateau; offline re-plateau of a reloaded
family; per-arm selection. C23 default-argmax keeps every existing walk-forward run
byte-identical; plateau is strictly opt-in.

## Design reconciliation (specify, /boss) — spec 0077 Entry path: specify (design settled by the milestone design-recon + judge verdict; the forks below are derived, not user-preference). Trunk: a new `optimize_plateau(family, axis_lens, metric, mode)` beside `optimize`/`optimize_deflated` that argmaxes a neighbourhood-aggregated (mean/worst over grid neighbours) score — a parameter-free robustness statement, NOT a gradient-penalty (which needs a metric-scale-sensitive free knob with no principled default). **Derived decisions (rationale, recorded for after-the-fact audit):** - **axis_lens carry** → passed as an ARGUMENT to `optimize_plateau`, never a `SweepFamily` field. Basis: derived — a field breaks `SweepFamily`'s derived PartialEq + four `SweepFamily { space, points }` struct-literal sites; the GridSpace owns the per-axis cardinalities at the live walk-forward call site, and the lattice is only needed transiently at selection. Needs a new `GridSpace::axis_lens()` (per-axis lengths in `param_space()` order) surfaced out of the sweep builder. - **RandomSpace plateau** → REFUSE `--select plateau` on a random sweep (exit 2). Basis: derived — plateau adjacency falls out of a GridSpace lattice; a kNN metric+radius is an invented design choice, and the CLI walk-forward path is all-GridSpace, so RandomSpace plateau is unexercised. kNN deferred to its own cycle. - **Aggregate** → expose both `--select plateau:mean|plateau:worst`, default `mean`. Basis: derived — mean is the less-surprising statement; `worst` has an edge-neighbour-truncation bias toward interior points (worth a test before a default); the chosen mode is recorded, so runs stay reproducible. - **Flag scope** → one global `--select <argmax|plateau:mean|plateau:worst>`, default `argmax`. Basis: derived — C23 byte-preserving (every existing run stays identical, plateau strictly opt-in); per-arm selection is unmotivated. - **FamilySelection carrier evolution (the load-bearing reconciliation)** → the recon predated #144, which landed a deflation-shaped `FamilySelection` + `SelectionMode{Argmax}`. The selection RULE (argmax vs plateau — which member wins) and the deflation ANNOTATION (how inflated the winner is) are ORTHOGONAL. Resolution: extend the shared carrier minimally — add `SelectionMode::PlateauMean` / `PlateauWorst` (the slot #144 reserved), make `deflated_score: f64 → Option<f64>` (None under plateau, the one genuinely deflation-specific scalar), and add `neighbourhood_score: Option<f64>` + `n_neighbours: Option<usize>` (None under argmax). `optimize_deflated` populates the deflation fields under `mode=Argmax`; `optimize_plateau` populates the plateau fields under `mode=Plateau*`. Basis: derived — one composable carrier across both milestone pieces beats a second parallel type; the reshape is a small wire change to #144's one-cycle-old, test-only-data type (legacy lines still load via serde(default)); #144's existing deflation + serde-back-compat tests are the green gate the reshape must preserve. Out of scope (deferred): RandomSpace kNN plateau; offline re-plateau of a reloaded family; per-arm selection. C23 default-argmax keeps every existing walk-forward run byte-identical; plateau is strictly opt-in.
Brummel commented 2026-06-26 15:38:45 +02:00
Author
Owner
Copy Link
Copy Source

Planner-stage fork decisions (/boss) — plan 0077

Two load-bearing forks surfaced while mapping spec 0077 onto the current tree (plan-recon DONE_WITH_CONCERNS). Both derived and recorded here for after-the-fact audit/veto.

Fork: where axis_lens surfaces (engine terminal vs CLI-local recompute).
→ Engine. A new SweepBinder::sweep_with_lattice terminal returns (SweepFamily, Vec<usize>) from the GridSpace it already builds (aura-engine/src/blueprint.rs:401); sweep() is refactored to delegate to it (byte-unchanged for every existing .sweep() caller, C23). GridSpace::axis_lens() is the accessor.
Basis: derived — only the engine's post-resolve_axes GridSpace holds the lattice in param_space()/odometer order, which closed_neighbourhood's mixed-radix arithmetic requires; recomputing it CLI-side from .axis(name, vals) call-order would duplicate resolve_axes and risk a silent order mismatch (wrong neighbours). The spec itself leans this way ("a sweep_with_lattice variant, or the builder exposes the built GridSpace", lines 161-164). A sibling terminal, not a signature change to sweep, contains the blast radius.

Fork: the RandomSpace-refuse — spec assumes a --random walk-forward producer that does not exist.
→ Structural guard, no new feature. walkforward has no --random flag (verified: --random/RandomSpace/RandomBinder appear nowhere in aura-cli), and the spec specifies no random-walk-forward producer (no component, no data-flow). So the refuse is implemented as the type-level guard the spec wants — sweep_over/stage1_r_sweep_over return (SweepFamily, Option<Vec<usize>>) (Some today, grid-only), and a select_winner helper returns the refuse on Plateau + None-lattice; the caller prints the spec's message and exits 2. It is unit-tested at the helper level (the Err path), not via a walkforward --random E2E — the spec's --random 64 example (line 96) uses a flag this cycle does not build.
Basis: derived — the spec's authoritative scope (Components table + "Out of scope", which defers RandomSpace-plateau support entirely) grounds the minimal reading: refuse is forward-looking structure, wiring a random walk-forward sweep is unspecified out-of-scope work. The guard becomes CLI-reachable the moment a random walk-forward producer is ever added.

Minor (spec already settles it): FamilySelection.seed reshapes to Option<u64> (spec line 135); the recon carrier's Option<usize> was a transcription slip.

## Planner-stage fork decisions (/boss) — plan 0077 Two load-bearing forks surfaced while mapping spec 0077 onto the current tree (plan-recon `DONE_WITH_CONCERNS`). Both derived and recorded here for after-the-fact audit/veto. **Fork: where `axis_lens` surfaces (engine terminal vs CLI-local recompute).** → **Engine.** A new `SweepBinder::sweep_with_lattice` terminal returns `(SweepFamily, Vec<usize>)` from the `GridSpace` it already builds (`aura-engine/src/blueprint.rs:401`); `sweep()` is refactored to delegate to it (byte-unchanged for every existing `.sweep()` caller, C23). `GridSpace::axis_lens()` is the accessor. Basis: derived — only the engine's post-`resolve_axes` `GridSpace` holds the lattice in `param_space()`/odometer order, which `closed_neighbourhood`'s mixed-radix arithmetic requires; recomputing it CLI-side from `.axis(name, vals)` call-order would duplicate `resolve_axes` and risk a silent order mismatch (wrong neighbours). The spec itself leans this way ("a `sweep_with_lattice` variant, or the builder exposes the built `GridSpace`", lines 161-164). A sibling terminal, not a signature change to `sweep`, contains the blast radius. **Fork: the RandomSpace-refuse — spec assumes a `--random` walk-forward producer that does not exist.** → **Structural guard, no new feature.** `walkforward` has **no** `--random` flag (verified: `--random`/`RandomSpace`/`RandomBinder` appear nowhere in `aura-cli`), and the spec specifies no random-walk-forward producer (no component, no data-flow). So the refuse is implemented as the type-level guard the spec wants — `sweep_over`/`stage1_r_sweep_over` return `(SweepFamily, Option<Vec<usize>>)` (`Some` today, grid-only), and a `select_winner` helper returns the refuse on `Plateau + None-lattice`; the caller prints the spec's message and exits 2. It is **unit-tested** at the helper level (the `Err` path), not via a `walkforward --random` E2E — the spec's `--random 64` example (line 96) uses a flag this cycle does not build. Basis: derived — the spec's authoritative scope (Components table + "Out of scope", which defers RandomSpace-plateau support entirely) grounds the minimal reading: refuse is forward-looking structure, wiring a random walk-forward sweep is unspecified out-of-scope work. The guard becomes CLI-reachable the moment a random walk-forward producer is ever added. **Minor (spec already settles it):** `FamilySelection.seed` reshapes to `Option<u64>` (spec line 135); the recon carrier's `Option<usize>` was a transcription slip.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone Inferential validation (defend against false discovery at sweep scale)
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#145
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?