audit(0077): cycle close — drift addressed; legacy-load fixture + ledger sync; drop ephemera
Architect drift review over 9c6b9c7..cb32658 returned drift_found (3 items); all resolved here. What holds (architect): C23 default-argmax byte-identical, C9 separation (selector in aura-registry, walk_forward selection-agnostic), C1/C2 (plateau pure/deterministic/in-sample), one higher_is_better direction source. Resolutions: - [medium] Backfilled the pre-0077 FamilySelection legacy-load fixture (report.rs family_selection_loads_a_pre_0077_legacy_line). The feat commit and the struct doc-comment claimed "a legacy line still loads (C14/C18)" as landed, but no test pinned the field-level mapping: bare deflation scalars -> Some(...), absent plateau keys -> None. families.jsonl/runs.jsonl are append-only, so a winner stamped pre-0077 must read back; now pinned. - [low] Synced the ledger's inferential-validation open thread (docs/design/INDEX.md): #145 plateau-over-peak landed (cycle 0077), selection is now a pluggable objective (bare argmax -> trials-deflated -> plateau) on the manifest carrier; only #146 (cross-instrument generalization) remains of the milestone's three pieces. - [medium-high] Removed the cycle ephemera (git rm docs/specs/0077-*, docs/plans/0077-*) per the cycle-close convention. Regression: full workspace suite green and clippy --all-targets -D warnings clean (verified at feat-review time). No baseline scripts configured; the test + clippy gates are the regression gate. Cycle 0077 tidy (clean) — drift-clean, not a milestone close (the milestone fieldtest is a separate deliberate gate).
This commit is contained in:
@@ -811,6 +811,26 @@ mod tests {
|
||||
assert_eq!(back, sel);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn family_selection_loads_a_pre_0077_legacy_line() {
|
||||
// The pre-0077 on-disk shape (#144): the deflation fields as BARE scalars,
|
||||
// no plateau keys. `families.jsonl` / `runs.jsonl` are append-only, so a
|
||||
// winner stamped last cycle must still read back — the bare scalars land in
|
||||
// the `Some(...)` deflation Options, the absent plateau keys default to
|
||||
// `None` (the C14/C18 back-compat the struct's serde attrs claim).
|
||||
let legacy = r#"{"selection_metric":"sqn_normalized","n_trials":4,"raw_winner_metric":1.83,"deflated_score":0.21,"overfit_probability":0.06,"mode":"Argmax","n_resamples":1000,"block_len":5,"seed":42}"#;
|
||||
let sel: FamilySelection =
|
||||
serde_json::from_str(legacy).expect("a pre-0077 selection line still loads");
|
||||
assert_eq!(sel.mode, SelectionMode::Argmax);
|
||||
assert_eq!(sel.deflated_score, Some(0.21));
|
||||
assert_eq!(sel.overfit_probability, Some(0.06));
|
||||
assert_eq!(sel.n_resamples, Some(1000));
|
||||
assert_eq!(sel.block_len, Some(5));
|
||||
assert_eq!(sel.seed, Some(42));
|
||||
assert_eq!(sel.neighbourhood_score, None);
|
||||
assert_eq!(sel.n_neighbours, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn position_action_round_trips_through_i64() {
|
||||
for a in [PositionAction::Buy, PositionAction::Sell, PositionAction::Close] {
|
||||
|
||||
+15
-3
@@ -1422,7 +1422,8 @@ load-bearing in the flat graph.
|
||||
significance read (#139, landed cycle 0075). Distinct from the two threads above —
|
||||
neither orchestration *composability* (#109) nor a search *policy* (Bayesian/
|
||||
genetic), but the statistical *validity* of the selection itself. Not a
|
||||
C-invariant until built; recorded as the World's third (now partly built) half.
|
||||
C-invariant until built; recorded as the World's third (now two of three pieces
|
||||
built) half.
|
||||
**Status (cycle 0076):** the trials-deflation piece (#144) landed —
|
||||
`optimize_deflated` (aura-registry) wraps `optimize` and records, additively on
|
||||
the winning member's manifest (C18, the new `RunManifest.selection`), a deflated
|
||||
@@ -1430,8 +1431,19 @@ load-bearing in the flat graph.
|
||||
member wins** (C23; `optimize` / `rank_by` stay a bare argmax — the deflation is
|
||||
recorded provenance, not a re-ranking). The R arm is a centred moving-block
|
||||
reality-check (reusing the `r_bootstrap` kernel); the `total_pips` arm a
|
||||
closed-form expected-max-of-K dispersion floor. #145 (plateau-over-peak) and #146
|
||||
(cross-instrument generalization) remain open.
|
||||
closed-form expected-max-of-K dispersion floor. **Status (cycle 0077):** the
|
||||
plateau-over-peak piece (#145) landed — `optimize_plateau` (aura-registry)
|
||||
argmaxes the **neighbourhood-smoothed** metric surface (mean or worst-case over
|
||||
each member's closed mixed-radix grid neighbourhood) instead of the bare peak,
|
||||
recorded on the same `RunManifest.selection` carrier, now an orthogonal *rule ×
|
||||
annotation*: `SelectionMode::{Argmax, PlateauMean, PlateauWorst}` with either the
|
||||
deflation annotation (#144) or the plateau annotation (`neighbourhood_score` /
|
||||
`n_neighbours`). It is opt-in via a `--select` flag — default argmax stays
|
||||
byte-identical (C23). The grid lattice surfaces from the engine
|
||||
(`SweepBinder::sweep_with_lattice`); the policy stays in aura-registry with
|
||||
`walk_forward` selection-agnostic (C9). Selection is now a *pluggable objective*
|
||||
(bare argmax → trials-deflated → plateau), not a single hard-wired argmax. #146
|
||||
(cross-instrument generalization) remains the last open piece of the milestone.
|
||||
- **`aura-std` contents** — the crate exists (doc-only); which universal blocks
|
||||
land first follows the walking-skeleton's needs.
|
||||
- **`strategies/` split** — a later split, *inside a project*, of top-level
|
||||
|
||||
@@ -1,974 +0,0 @@
|
||||
# Prefer a robust parameter plateau over the in-sample peak — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0077-plateau-over-peak-selection.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to
|
||||
> run this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Add an opt-in `optimize_plateau` selection objective (argmax over the
|
||||
grid-neighbourhood-smoothed metric) reachable via a new `--select` flag, recorded
|
||||
on the `RunManifest.selection` carrier #144 introduced; default `argmax` stays
|
||||
byte-identical (C23).
|
||||
|
||||
**Architecture:** A new family selector lives beside `optimize_deflated` in
|
||||
`aura-registry` (C9), reading each member's metric via the shared `metric_value`
|
||||
and smoothing it over a mixed-radix grid neighbourhood built from `axis_lens`
|
||||
(the `GridSpace` odometer radixes). The lattice is surfaced from the engine via a
|
||||
new `SweepBinder::sweep_with_lattice` terminal (`sweep` delegates to it,
|
||||
byte-unchanged for all existing callers). The shared `FamilySelection` carrier is
|
||||
reshaped — orthogonal rule (`SelectionMode`) × annotation (deflation fields →
|
||||
`Option`, plus plateau fields) — and the CLI threads a `Selection` from
|
||||
`--select` into `walkforward_family`.
|
||||
|
||||
**Tech Stack:** `aura-engine` (`report.rs` carrier, `sweep.rs` accessor,
|
||||
`blueprint.rs` terminal), `aura-registry` (`lib.rs` selector + index math),
|
||||
`aura-cli` (`main.rs` flag/dispatch/display, `tests/cli_run.rs` E2E).
|
||||
|
||||
---
|
||||
|
||||
## Design decisions resolved at plan time (recorded on #145, comment 1974)
|
||||
|
||||
1. **Lattice seam → engine `SweepBinder::sweep_with_lattice`.** Only the engine's
|
||||
post-`resolve_axes` `GridSpace` holds the lattice in `param_space()`/odometer
|
||||
order, which `closed_neighbourhood`'s mixed-radix arithmetic requires; a sibling
|
||||
terminal (`sweep` delegates) contains the blast radius — every existing
|
||||
`.sweep()` caller stays byte-identical.
|
||||
2. **RandomSpace-refuse → structural guard, no new feature.** `walkforward` has no
|
||||
`--random` flag (verified: `--random`/`RandomSpace`/`RandomBinder` appear
|
||||
nowhere in `aura-cli`); wiring a random walk-forward producer is unspecified,
|
||||
out-of-scope. So `sweep_over`/`stage1_r_sweep_over` return
|
||||
`(SweepFamily, Option<Vec<usize>>)` (`Some` today, grid-only), and a
|
||||
`select_winner` helper returns the refuse on `Plateau + None`-lattice; the
|
||||
caller prints the spec's message and exits 2. **Unit-tested** at the helper
|
||||
level (the `Err` path), not via a `walkforward --random` E2E.
|
||||
3. **`FamilySelection.seed` → `Option<u64>`** (spec line 135; the recon carrier's
|
||||
`Option<usize>` was a transcription slip).
|
||||
|
||||
## Files this plan creates or modifies
|
||||
|
||||
- Modify: `crates/aura-engine/src/report.rs:41-66` — reshape `SelectionMode` +
|
||||
`FamilySelection`; `:762-775` round-trip test literal; new plateau-shape test.
|
||||
- Modify: `crates/aura-engine/src/sweep.rs` — new `GridSpace::axis_lens()` + test.
|
||||
- Modify: `crates/aura-engine/src/blueprint.rs:399-411` — new
|
||||
`SweepBinder::sweep_with_lattice`; `sweep` delegates; new test.
|
||||
- Modify: `crates/aura-registry/src/lib.rs:308-312` — `optimize_deflated`
|
||||
construction to `Option` deflation fields; `:163-173` `metric_cmp` refactor to
|
||||
`higher_is_better`; `:744-794` test assertions → `.unwrap()`/`Some`; add
|
||||
`PlateauMode`, `higher_is_better`, `closed_neighbourhood`, `optimize_plateau` +
|
||||
tests.
|
||||
- Modify: `crates/aura-cli/src/main.rs` — import `SelectionMode` (engine),
|
||||
`optimize_plateau`/`PlateauMode` (registry); `Selection` enum; `parse_select` +
|
||||
`--select` in `parse_walkforward_args`; `select_winner`; `sweep_over` /
|
||||
`stage1_r_sweep_over` tuple returns; `walkforward_family` `+select` dispatch;
|
||||
`run_walkforward` `+select`; the dispatch call site; `runs_family` mode-aware
|
||||
display; 4 test callers of `walkforward_family`; unit tests.
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`.
|
||||
- Unchanged (preservation gate): `crates/aura-registry/src/compat.rs:34` embeds
|
||||
`FamilySelection` by value — the reshape flows through with no edit; the
|
||||
legacy-load test `load_lifts_a_selection_block_through_the_compat_mirror`
|
||||
(`lib.rs:520`) and `runs_family_rank_shows_deflated_line` (`cli_run.rs:2358`)
|
||||
must stay green.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Reshape `FamilySelection` / `SelectionMode` (mechanical, byte-preserving across 3 crates)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs:41-66`, `:762-775`
|
||||
- Modify: `crates/aura-registry/src/lib.rs:308-312`, `:744-794`
|
||||
- Modify: `crates/aura-cli/src/main.rs:19-25` (import), `:2215-2223` (display)
|
||||
|
||||
This is the cross-crate compile-gate. The reshape breaks every construction and
|
||||
field-read site; all are threaded here. No new runtime behaviour — the argmax
|
||||
path stays byte-identical; the one new test pins the new fields' serde contract.
|
||||
|
||||
- [ ] **Step 1: Write the failing plateau-shape serde test (RED)**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, in the `#[cfg(test)] mod tests` block,
|
||||
beside `family_selection_round_trips_on_the_manifest` (after line 775), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn family_selection_plateau_shape_round_trips() {
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 16, raw_winner_metric: 1.42,
|
||||
mode: SelectionMode::PlateauMean,
|
||||
deflated_score: None, overfit_probability: None,
|
||||
n_resamples: None, block_len: None, seed: None,
|
||||
neighbourhood_score: Some(1.27), n_neighbours: Some(5),
|
||||
};
|
||||
let json = serde_json::to_string(&sel).unwrap();
|
||||
// plateau annotation present; deflation fields omitted (skip_serializing_if)
|
||||
assert!(json.contains("\"neighbourhood_score\":1.27"), "json: {json}");
|
||||
assert!(json.contains("\"n_neighbours\":5"), "json: {json}");
|
||||
assert!(!json.contains("deflated_score"), "deflation fields omitted under plateau: {json}");
|
||||
assert!(!json.contains("n_resamples"), "json: {json}");
|
||||
let back: FamilySelection = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back, sel);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the new test to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine family_selection_plateau`
|
||||
Expected: FAIL — compile error (`SelectionMode::PlateauMean` / the `neighbourhood_score`/`n_neighbours` fields do not exist yet).
|
||||
|
||||
- [ ] **Step 3: Reshape `SelectionMode` and `FamilySelection`**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, replace the `SelectionMode` enum + doc
|
||||
(lines 41-48) and the `FamilySelection` struct + doc (lines 50-66) with:
|
||||
|
||||
```rust
|
||||
/// Which selection objective produced the record (additive provenance, C23).
|
||||
/// `Argmax` is the bare-best pick (cycle 0076), deflated for the number of trials.
|
||||
/// `PlateauMean` / `PlateauWorst` (cycle 0077) argmax the neighbourhood-smoothed
|
||||
/// surface instead, so peak-vs-plateau runs stay distinguishable on this field.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
PlateauMean,
|
||||
PlateauWorst,
|
||||
}
|
||||
|
||||
/// Selection-provenance for a sweep winner. The selection RULE (`mode`) and its
|
||||
/// ANNOTATION are orthogonal: `Argmax` carries the trials-deflation annotation
|
||||
/// (`deflated_score` / `overfit_probability` / `n_resamples` / `block_len` /
|
||||
/// `seed`); `Plateau*` carries the smoothing annotation (`neighbourhood_score` /
|
||||
/// `n_neighbours`). Each annotation is `Option`, present iff its rule produced the
|
||||
/// record; all are additive — recorded, never re-ranking (C23). A legacy line
|
||||
/// (pre-0077) carries the deflation scalars as bare values that deserialize to
|
||||
/// `Some` (serde default), so it still loads (C14/C18).
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub mode: SelectionMode,
|
||||
// deflation annotation (present iff mode == Argmax with a deflation run)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub deflated_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_resamples: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub block_len: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub seed: Option<u64>,
|
||||
// plateau annotation (present iff mode is Plateau*)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub neighbourhood_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_neighbours: Option<usize>,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update the round-trip test literal**
|
||||
|
||||
In `crates/aura-engine/src/report.rs`, in `family_selection_round_trips_on_the_manifest`,
|
||||
replace the `let sel = FamilySelection { … };` literal (lines 764-768) with:
|
||||
|
||||
```rust
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(0.21), overfit_probability: Some(0.06),
|
||||
n_resamples: Some(1000), block_len: Some(5), seed: Some(42),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the engine tests (GREEN, engine internally consistent)**
|
||||
|
||||
Run: `cargo test -p aura-engine family_selection`
|
||||
Expected: PASS — both `family_selection_round_trips_on_the_manifest` and
|
||||
`family_selection_plateau_shape_round_trips` green. (aura-engine does not depend
|
||||
on aura-registry, so its build is consistent even though the registry is not yet
|
||||
updated.)
|
||||
|
||||
- [ ] **Step 6: Update `optimize_deflated`'s construction to the Option fields**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, replace the returned `FamilySelection`
|
||||
literal (lines 308-312) with:
|
||||
|
||||
```rust
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(deflated_score), overfit_probability,
|
||||
n_resamples: Some(n_resamples), block_len: Some(block_len), seed: Some(seed),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
}))
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Update the `optimize_deflated_*` test assertions to `.unwrap()` / `Some`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, apply these exact line edits:
|
||||
|
||||
- Line 749: ` assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw`
|
||||
→ ` assert!(sel.deflated_score.unwrap() <= sel.raw_winner_metric); // dispersion floor ≤ raw`
|
||||
- Line 762: ` assert_eq!(sel.n_resamples, 0);`
|
||||
→ ` assert_eq!(sel.n_resamples, Some(0));`
|
||||
- Line 763: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric, "p95 of an empty null is 0");`
|
||||
→ ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric, "p95 of an empty null is 0");`
|
||||
- Line 782: ` assert!(sel.deflated_score.is_finite(), "deflated_score must be finite, got {}", sel.deflated_score);`
|
||||
→ ` assert!(sel.deflated_score.unwrap().is_finite(), "deflated_score must be finite, got {:?}", sel.deflated_score);`
|
||||
- Line 783: ` assert_eq!(sel.deflated_score, sel.raw_winner_metric);`
|
||||
→ ` assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric);`
|
||||
- Line 794: ` assert!(sel.deflated_score > 0.0);`
|
||||
→ ` assert!(sel.deflated_score.unwrap() > 0.0);`
|
||||
|
||||
(The compat test `load_lifts_a_selection_block_through_the_compat_mirror` at line
|
||||
520-527 is **unchanged** — its legacy JSON line carries the deflation scalars as
|
||||
bare floats that deserialize to `Some` under the new `Option` + `#[serde(default)]`,
|
||||
and its assertions read only `n_trials`/`mode`. It is the C14/C18 legacy-load gate.)
|
||||
|
||||
- [ ] **Step 8: Run the registry tests (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-registry optimize_deflated`
|
||||
Expected: PASS — all nine `optimize_deflated_*` tests green against the reshaped fields.
|
||||
|
||||
Run: `cargo test -p aura-registry load_lifts`
|
||||
Expected: PASS — the legacy `selection` block still lifts through the compat mirror.
|
||||
|
||||
- [ ] **Step 9: Import `SelectionMode` and rewrite the `runs_family` display (mode-aware, argmax byte-preserved)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, add `SelectionMode` to the `aura_engine` import
|
||||
list (lines 19-24) — e.g. extend the `RunManifest, RunMetrics,` line to
|
||||
`RunManifest, RunMetrics, SelectionMode,` (any slot in the brace list is fine).
|
||||
|
||||
Then replace the selection-display block in `runs_family` (lines 2215-2223):
|
||||
|
||||
```rust
|
||||
for report in &ordered {
|
||||
println!("{}", report.to_json());
|
||||
if let Some(sel) = &report.manifest.selection {
|
||||
match sel.mode {
|
||||
SelectionMode::Argmax => match sel.overfit_probability {
|
||||
Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score.unwrap(), p),
|
||||
None => println!(" deflated={:.4}", sel.deflated_score.unwrap()),
|
||||
},
|
||||
SelectionMode::PlateauMean | SelectionMode::PlateauWorst => {
|
||||
let label = if matches!(sel.mode, SelectionMode::PlateauMean) { "mean" } else { "worst" };
|
||||
if let (Some(score), Some(n)) = (sel.neighbourhood_score, sel.n_neighbours) {
|
||||
println!(" plateau({label})={score:.4} over {n} cells");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Under `Argmax`, `deflated_score` is always `Some` by construction, so `.unwrap()`
|
||||
preserves the exact pre-0077 bytes — the `deflated=…`/`P(overfit)=…` golden stays
|
||||
green. The `Plateau*` branch is exercised by the Task 5 E2E.)
|
||||
|
||||
- [ ] **Step 10: Workspace build + full per-crate regression gate**
|
||||
|
||||
Run: `cargo build --workspace`
|
||||
Expected: clean (0 errors) — all three crates consistent.
|
||||
|
||||
Run: `cargo test -p aura-engine`
|
||||
Expected: PASS (no regressions).
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS (no regressions).
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `GridSpace::axis_lens()` + `SweepBinder::sweep_with_lattice` (engine, additive)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/sweep.rs` (accessor + test)
|
||||
- Modify: `crates/aura-engine/src/blueprint.rs:399-411` (terminal + delegate + test)
|
||||
|
||||
- [ ] **Step 1: Write the failing `axis_lens` golden (RED)**
|
||||
|
||||
In `crates/aura-engine/src/sweep.rs`, in the `#[cfg(test)] mod tests` block (which
|
||||
already imports `aura_core::{Cell, ParamSpec, Scalar, ScalarKind, Timestamp}`),
|
||||
add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn grid_axis_lens_are_the_per_axis_radixes() {
|
||||
let space = vec![
|
||||
ParamSpec { name: "a".into(), kind: ScalarKind::I64 },
|
||||
ParamSpec { name: "b".into(), kind: ScalarKind::I64 },
|
||||
];
|
||||
let grid = GridSpace::new(&space, vec![
|
||||
vec![Scalar::i64(10), Scalar::i64(20)], // axis 0: 2 values
|
||||
vec![Scalar::i64(1), Scalar::i64(2), Scalar::i64(3)], // axis 1: 3 values
|
||||
]).expect("2x3 grid");
|
||||
assert_eq!(grid.axis_lens(), vec![2, 3]);
|
||||
assert_eq!(grid.axis_lens().iter().product::<usize>(), grid.len());
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine grid_axis_lens`
|
||||
Expected: FAIL — `no method named axis_lens found for struct GridSpace`.
|
||||
|
||||
- [ ] **Step 3: Add the `axis_lens` accessor**
|
||||
|
||||
In `crates/aura-engine/src/sweep.rs`, in the `impl GridSpace` block, immediately
|
||||
after `is_empty` (after line 65), add:
|
||||
|
||||
```rust
|
||||
/// Per-axis cardinalities in `param_space()` order (the odometer radixes,
|
||||
/// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau
|
||||
/// neighbourhood walks (cycle 0077).
|
||||
pub fn axis_lens(&self) -> Vec<usize> {
|
||||
self.axes.iter().map(Vec::len).collect()
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the accessor test (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-engine grid_axis_lens`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Write the failing `sweep_with_lattice` test (RED)**
|
||||
|
||||
In `crates/aura-engine/src/blueprint.rs`, in the `#[cfg(test)] mod tests` block
|
||||
(which has `composite_sma_cross_harness()` and `run_point`), add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn sweep_with_lattice_surfaces_grid_radixes_in_param_space_order() {
|
||||
let bp = composite_sma_cross_harness().0;
|
||||
let (fam, lattice) = bp
|
||||
.axis("sma_cross.fast.length", vec![Scalar::i64(2), Scalar::i64(3)]) // 2 values
|
||||
.axis("sma_cross.slow.length", vec![Scalar::i64(4), Scalar::i64(5)]) // 2 values
|
||||
.axis("bias.scale", vec![Scalar::f64(0.5)]) // 1 value
|
||||
.sweep_with_lattice(run_point)
|
||||
.expect("named binding resolves and runs");
|
||||
assert_eq!(lattice, vec![2, 2, 1], "radixes in param_space slot order");
|
||||
assert_eq!(lattice.iter().product::<usize>(), fam.points.len()); // 4
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run to verify it fails to compile**
|
||||
|
||||
Run: `cargo test -p aura-engine sweep_with_lattice`
|
||||
Expected: FAIL — `no method named sweep_with_lattice found`.
|
||||
|
||||
- [ ] **Step 7: Add `sweep_with_lattice`; refactor `sweep` to delegate**
|
||||
|
||||
In `crates/aura-engine/src/blueprint.rs`, replace the `SweepBinder::sweep` method
|
||||
(lines 399-411) with:
|
||||
|
||||
```rust
|
||||
/// Resolve the named axes against `param_space()` into a positional grid and
|
||||
/// run the disjoint sweep. `sweep` is [`SweepBinder::sweep_with_lattice`] with
|
||||
/// the lattice dropped, so every existing caller is byte-unchanged.
|
||||
pub fn sweep<F>(self, run_one: F) -> Result<SweepFamily, BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
self.sweep_with_lattice(run_one).map(|(family, _lattice)| family)
|
||||
}
|
||||
|
||||
/// As [`SweepBinder::sweep`], plus the grid's per-axis radixes (`axis_lens`,
|
||||
/// in `param_space()` / odometer order). The lattice is what a plateau
|
||||
/// neighbourhood walk needs (cycle 0077); only the engine's post-`resolve_axes`
|
||||
/// grid holds it in the correct order.
|
||||
pub fn sweep_with_lattice<F>(self, run_one: F) -> Result<(SweepFamily, Vec<usize>), BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
let space = self.bp.param_space();
|
||||
check_param_namespace_injective(&space).map_err(BindError::Compile)?;
|
||||
let ordered = resolve_axes(&space, &self.axes)?;
|
||||
let grid = GridSpace::new(&space, ordered)
|
||||
.expect("named layer pre-validates arity/kind/non-empty");
|
||||
let lattice = grid.axis_lens();
|
||||
Ok((sweep(&grid, run_one), lattice))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Run the new test + the existing sweep tests (GREEN, delegation preserved)**
|
||||
|
||||
Run: `cargo test -p aura-engine sweep_with_lattice`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-engine`
|
||||
Expected: PASS — every existing `.sweep()` test (named-axes parity, grid goldens)
|
||||
stays green; `sweep` delegating to `sweep_with_lattice` is behaviour-preserving.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: `optimize_plateau` + `PlateauMode` + `closed_neighbourhood` (registry, additive)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/lib.rs:163-173` (`metric_cmp` → `higher_is_better`),
|
||||
plus new `PlateauMode`, `higher_is_better`, `closed_neighbourhood`,
|
||||
`optimize_plateau` and their tests.
|
||||
|
||||
- [ ] **Step 1: Write the failing plateau tests (RED)**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, in the `#[cfg(test)] mod tests` block (which
|
||||
has the `member(total_pips, trade_rs)` helper), add:
|
||||
|
||||
```rust
|
||||
/// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle
|
||||
/// plateau in `total_pips`. Bare argmax takes an end spike; the plateau-mean
|
||||
/// argmax takes the interior plateau centre.
|
||||
fn fixture_grid_spike_vs_plateau() -> SweepFamily {
|
||||
let pips = [80.0, 5.0, 48.0, 50.0, 49.0, 5.0, 80.0];
|
||||
SweepFamily { space: vec![], points: pips.iter().map(|&p| member(p, vec![1.0])).collect() }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closed_neighbourhood_2x3_golden() {
|
||||
// 2x3 lattice, odometer (last axis fastest):
|
||||
// (0,0)=0 (0,1)=1 (0,2)=2
|
||||
// (1,0)=3 (1,1)=4 (1,2)=5
|
||||
let sorted = |i: usize| { let mut v = closed_neighbourhood(i, &[2, 3]); v.sort_unstable(); v };
|
||||
assert_eq!(sorted(0), vec![0, 1, 3]); // corner (0,0): self + 2
|
||||
assert_eq!(sorted(4), vec![1, 3, 4, 5]); // interior (1,1): self + 3
|
||||
assert_eq!(sorted(2), vec![1, 2, 5]); // corner (0,2): self + 2
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_picks_the_plateau_not_the_spike() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let spike = optimize(&fam, "total_pips").unwrap();
|
||||
let (centre, sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_ne!(centre.report.metrics.total_pips, spike.report.metrics.total_pips);
|
||||
assert_eq!(centre.report.metrics.total_pips, 50.0, "plateau centre is the middle cell");
|
||||
assert_eq!(sel.mode, SelectionMode::PlateauMean);
|
||||
assert_eq!(sel.raw_winner_metric, 50.0);
|
||||
assert!(sel.neighbourhood_score.unwrap() < spike.report.metrics.total_pips,
|
||||
"smoothed score is below the spike's raw peak");
|
||||
assert_eq!(sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours");
|
||||
assert!(sel.deflated_score.is_none() && sel.n_resamples.is_none(),
|
||||
"deflation fields omitted under plateau");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_worst_is_more_conservative_than_mean() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let (_, mean_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
let (_, worst_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap();
|
||||
// Worst scores each cell by its weakest neighbour, never above the mean.
|
||||
assert!(worst_sel.neighbourhood_score.unwrap() <= mean_sel.neighbourhood_score.unwrap());
|
||||
assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst);
|
||||
assert!(worst_sel.n_neighbours.unwrap() >= 2, "winner is an interior cell");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_single_member_equals_optimize() {
|
||||
let fam = SweepFamily { space: vec![], points: vec![member(42.0, vec![1.0])] };
|
||||
let plain = optimize(&fam, "total_pips").unwrap();
|
||||
let (winner, sel) = optimize_plateau(&fam, &[1], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_eq!(winner.report.metrics.total_pips, plain.report.metrics.total_pips);
|
||||
assert_eq!(sel.n_neighbours, Some(1), "a 1-cell grid's closed neighbourhood is itself");
|
||||
assert_eq!(sel.neighbourhood_score, Some(42.0));
|
||||
assert_eq!(sel.raw_winner_metric, 42.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_plateau_is_deterministic() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let a = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
let b = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
assert_eq!(a, b);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify they fail to compile**
|
||||
|
||||
Run: `cargo test -p aura-registry closed_neighbourhood`
|
||||
Expected: FAIL — `closed_neighbourhood` / `optimize_plateau` / `PlateauMode` do not exist.
|
||||
|
||||
- [ ] **Step 3: Add `higher_is_better` and refactor `metric_cmp` to use it**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, immediately after `is_r_metric` (after line
|
||||
207), add:
|
||||
|
||||
```rust
|
||||
/// The metric's optimisation direction: `true` if a larger value is better
|
||||
/// (`total_pips`, the four R keys), `false` for the lower-is-better keys
|
||||
/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp`
|
||||
/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour)
|
||||
/// both read it, so the per-metric sense lives in exactly one place.
|
||||
fn higher_is_better(m: Metric) -> bool {
|
||||
!matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips)
|
||||
}
|
||||
```
|
||||
|
||||
Then replace the `metric_cmp` body (lines 163-173) with:
|
||||
|
||||
```rust
|
||||
fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Ordering, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let hib = higher_is_better(m);
|
||||
Ok(move |a: &RunReport, b: &RunReport| {
|
||||
let (va, vb) = (metric_value(a, m), metric_value(b, m));
|
||||
// higher-is-better ranks the larger value first; lower-is-better the smaller.
|
||||
if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) }
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
(Behaviour-identical: `rank_by` / `optimize` and their existing tests are the gate.)
|
||||
|
||||
- [ ] **Step 4: Add `closed_neighbourhood`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, after `higher_is_better`, add:
|
||||
|
||||
```rust
|
||||
/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice
|
||||
/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}`
|
||||
/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order
|
||||
/// is `i` first then per-axis neighbours; callers use the result only as a set and
|
||||
/// for its length, so the order is not load-bearing.
|
||||
fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> {
|
||||
// decompose i to per-axis coords (last axis fastest)
|
||||
let mut coords = vec![0usize; axis_lens.len()];
|
||||
let mut rem = i;
|
||||
for k in (0..axis_lens.len()).rev() {
|
||||
coords[k] = rem % axis_lens[k];
|
||||
rem /= axis_lens[k];
|
||||
}
|
||||
// recompose a coord vector to its flat index (Horner, last axis fastest)
|
||||
let recompose = |c: &[usize]| c.iter().zip(axis_lens).fold(0usize, |flat, (&ck, &len)| flat * len + ck);
|
||||
let mut out = vec![i];
|
||||
for k in 0..axis_lens.len() {
|
||||
for delta in [-1isize, 1] {
|
||||
let ck = coords[k] as isize + delta;
|
||||
if ck < 0 || ck as usize >= axis_lens[k] { continue; }
|
||||
let mut nb = coords.clone();
|
||||
nb[k] = ck as usize;
|
||||
out.push(recompose(&nb));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add `PlateauMode` and `optimize_plateau`**
|
||||
|
||||
In `crates/aura-registry/src/lib.rs`, immediately before `optimize_deflated` (before
|
||||
line 258, before its doc comment), add:
|
||||
|
||||
```rust
|
||||
/// Plateau-selection aggregate: how a member's grid-neighbourhood metric is
|
||||
/// reduced to one smoothed score before the argmax. `Mean` averages the closed
|
||||
/// neighbourhood; `Worst` takes the most-pessimistic neighbour by the metric's
|
||||
/// direction (biasing toward interior cells, which keep more neighbours).
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum PlateauMode {
|
||||
Mean,
|
||||
Worst,
|
||||
}
|
||||
|
||||
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau
|
||||
/// provenance. Each member scores as the mean (or worst-case) of its closed grid
|
||||
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the
|
||||
/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are
|
||||
/// the grid radixes (`param_space()` order, last axis fastest). Pure,
|
||||
/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL;
|
||||
/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the
|
||||
/// smoothed value the argmax maximised.
|
||||
pub fn optimize_plateau(
|
||||
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let n = family.points.len();
|
||||
debug_assert_eq!(
|
||||
axis_lens.iter().product::<usize>(), n,
|
||||
"axis_lens product must equal the family size (a valid grid lattice)",
|
||||
);
|
||||
let hib = higher_is_better(m);
|
||||
// smoothed score + closed-neighbourhood size per member, in odometer order
|
||||
let scored: Vec<(usize, f64, usize)> = (0..n)
|
||||
.map(|i| {
|
||||
let nbrs = closed_neighbourhood(i, axis_lens);
|
||||
let vals: Vec<f64> = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect();
|
||||
let score = match mode {
|
||||
PlateauMode::Mean => vals.iter().sum::<f64>() / vals.len() as f64,
|
||||
PlateauMode::Worst => {
|
||||
if hib {
|
||||
vals.iter().copied().fold(f64::INFINITY, f64::min)
|
||||
} else {
|
||||
vals.iter().copied().fold(f64::NEG_INFINITY, f64::max)
|
||||
}
|
||||
}
|
||||
};
|
||||
(i, score, nbrs.len())
|
||||
})
|
||||
.collect();
|
||||
// argmax the smoothed surface by direction; earliest-odometer tie (only a
|
||||
// strictly-better later cell displaces the incumbent, as `optimize`).
|
||||
let &(wi, wscore, wn) = scored
|
||||
.iter()
|
||||
.reduce(|best, cur| {
|
||||
let better = if hib { cur.1 > best.1 } else { cur.1 < best.1 };
|
||||
if better { cur } else { best }
|
||||
})
|
||||
.expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)");
|
||||
let winner = family.points[wi].clone();
|
||||
let raw = metric_value(&winner.report, m);
|
||||
Ok((
|
||||
winner,
|
||||
FamilySelection {
|
||||
selection_metric: metric.to_string(),
|
||||
n_trials: n,
|
||||
raw_winner_metric: raw,
|
||||
mode: match mode {
|
||||
PlateauMode::Mean => SelectionMode::PlateauMean,
|
||||
PlateauMode::Worst => SelectionMode::PlateauWorst,
|
||||
},
|
||||
deflated_score: None,
|
||||
overfit_probability: None,
|
||||
n_resamples: None,
|
||||
block_len: None,
|
||||
seed: None,
|
||||
neighbourhood_score: Some(wscore),
|
||||
n_neighbours: Some(wn),
|
||||
},
|
||||
))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the plateau tests + the existing rank/optimize gate (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-registry plateau`
|
||||
Expected: PASS — `plateau_picks_the_plateau_not_the_spike`,
|
||||
`plateau_worst_is_more_conservative_than_mean`, `plateau_single_member_equals_optimize`,
|
||||
`optimize_plateau_is_deterministic` all green.
|
||||
|
||||
Run: `cargo test -p aura-registry closed_neighbourhood`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS — `rank_by` / `optimize` / `optimize_deflated` stay green after the
|
||||
`metric_cmp` → `higher_is_better` refactor.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: CLI `--select` parse + threading + sweep-entry tuples + `select_winner` (compile-gate)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs` — `aura_registry` import; `Selection` enum;
|
||||
`parse_select`; `parse_walkforward_args` (`+--select`, 6-tuple); the dispatch
|
||||
call site (`:2935`); `run_walkforward` (`+select`); `walkforward_family`
|
||||
(`+select` dispatch via `select_winner`); `sweep_over` / `stage1_r_sweep_over`
|
||||
(tuple returns); `select_winner`; the 4 test callers of `walkforward_family`;
|
||||
unit tests.
|
||||
|
||||
This is the CLI compile-gate: the signature changes break their callers, all
|
||||
threaded here. Default `Argmax` keeps the existing walk-forward output
|
||||
byte-identical (C23).
|
||||
|
||||
- [ ] **Step 1: Write the failing `select_winner` refuse + parse tests (RED)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, in the `#[cfg(test)] mod tests` block at line
|
||||
2965, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn select_winner_refuses_plateau_without_a_lattice() {
|
||||
// A plateau request with no lattice (a random sweep would yield None) is
|
||||
// refused, never silently argmaxed. The refuse short-circuits before the
|
||||
// family is read, so an empty family is fine here.
|
||||
let fam = SweepFamily { space: vec![], points: vec![] };
|
||||
let err = select_winner(&fam, "total_pips", Selection::Plateau(PlateauMode::Mean), None)
|
||||
.unwrap_err();
|
||||
assert!(err.contains("requires a grid sweep"), "refuse message: {err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_walkforward_select_flag() {
|
||||
let argmax = parse_walkforward_args(&["--strategy", "stage1-r"]).unwrap();
|
||||
assert!(matches!(argmax.5, Selection::Argmax), "default is argmax");
|
||||
let mean = parse_walkforward_args(&["--select", "plateau:mean"]).unwrap();
|
||||
assert!(matches!(mean.5, Selection::Plateau(PlateauMode::Mean)));
|
||||
let worst = parse_walkforward_args(&["--select", "plateau:worst"]).unwrap();
|
||||
assert!(matches!(worst.5, Selection::Plateau(PlateauMode::Worst)));
|
||||
assert!(parse_walkforward_args(&["--select", "bogus"]).is_err(),
|
||||
"unknown --select token is a usage error");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify they fail to compile**
|
||||
|
||||
Run: `cargo test -p aura-cli select_winner_refuses`
|
||||
Expected: FAIL — `Selection` / `PlateauMode` / `select_winner` not in scope; the
|
||||
`parse_walkforward_args` tuple has no `.5`.
|
||||
|
||||
- [ ] **Step 3: Imports + add the `Selection` enum and `parse_select`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, extend the `aura_registry` import (lines 26-30) —
|
||||
add `optimize_plateau, PlateauMode,` to the brace list (alongside `optimize_deflated`).
|
||||
Also add `FamilySelection` to the `aura_engine` import list (lines 19-24; it is not
|
||||
currently imported, and `select_winner` in Step 4 names it). (`SelectionMode` was
|
||||
already added to that list in Task 1 Step 9.)
|
||||
|
||||
Then, immediately after the `Strategy` enum (after line 1551's enum block; place it
|
||||
near the other CLI enums), add:
|
||||
|
||||
```rust
|
||||
/// In-sample winner-selection objective for walk-forward (cycle 0077). `Argmax` is
|
||||
/// the bare-best pick deflated for trials (#144, the default); `Plateau` argmaxes
|
||||
/// the neighbourhood-smoothed surface instead (opt-in via `--select`).
|
||||
#[derive(Clone, Copy)]
|
||||
enum Selection {
|
||||
Argmax,
|
||||
Plateau(PlateauMode),
|
||||
}
|
||||
|
||||
/// Parse a `--select` token: `argmax` | `plateau:mean` | `plateau:worst`. Unknown
|
||||
/// tokens are a usage error (the caller maps `Err(())` to exit 2).
|
||||
fn parse_select(s: &str) -> Result<Selection, ()> {
|
||||
match s {
|
||||
"argmax" => Ok(Selection::Argmax),
|
||||
"plateau:mean" => Ok(Selection::Plateau(PlateauMode::Mean)),
|
||||
"plateau:worst" => Ok(Selection::Plateau(PlateauMode::Worst)),
|
||||
_ => Err(()),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add the `select_winner` dispatch helper**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, immediately before `walkforward_family` (before
|
||||
line 1786), add:
|
||||
|
||||
```rust
|
||||
/// Resolve the in-sample winner under the chosen selection objective. `Argmax`
|
||||
/// defers to the trials-deflation pick (#144). `Plateau` argmaxes the smoothed grid
|
||||
/// surface — it needs the grid lattice, so a sweep with no lattice (a future random
|
||||
/// walk-forward producer) is refused rather than silently argmaxed. The metric is
|
||||
/// always known at the call sites, so a metric error is unreachable (`expect`); the
|
||||
/// only fallible outcome is the plateau-without-lattice refusal, returned as
|
||||
/// `Err(message)` for the caller to print and exit 2.
|
||||
fn select_winner(
|
||||
family: &SweepFamily, metric: &str, select: Selection, lattice: Option<&[usize]>,
|
||||
) -> Result<(SweepPoint, FamilySelection), String> {
|
||||
match select {
|
||||
Selection::Argmax => Ok(optimize_deflated(
|
||||
family, metric, DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED,
|
||||
).expect("walk-forward metrics are known")),
|
||||
Selection::Plateau(mode) => match lattice {
|
||||
Some(lens) => Ok(optimize_plateau(family, lens, metric, mode)
|
||||
.expect("walk-forward metrics are known")),
|
||||
None => Err(
|
||||
"--select plateau requires a grid sweep; a random sweep has no parameter lattice"
|
||||
.to_string(),
|
||||
),
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`select_winner` names `FamilySelection`, which is **not** currently imported in
|
||||
`main.rs` — Step 3 above adds it to the `aura_engine` import list.)
|
||||
|
||||
- [ ] **Step 5: Switch `sweep_over` / `stage1_r_sweep_over` to return `(SweepFamily, Option<Vec<usize>>)`**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`:
|
||||
|
||||
`stage1_r_sweep_over` (line 1301): change the signature return type
|
||||
`-> SweepFamily` to `-> (SweepFamily, Option<Vec<usize>>)`; change its terminal
|
||||
`.sweep(|point| {` (line 1318) to `.sweep_with_lattice(|point| {`, and insert a
|
||||
`.map(...)` line before the existing trailing `.expect(...)` (line 1342, currently
|
||||
` .expect("the stage1-r named grid matches the stage1-r param-space")`). The
|
||||
closure's closing ` })` and the existing `.expect(...)` line both stay; the
|
||||
new `.map` goes between them, so the function tail reads:
|
||||
|
||||
```rust
|
||||
})
|
||||
.map(|(fam, lat)| (fam, Some(lat)))
|
||||
.expect("the stage1-r named grid matches the stage1-r param-space")
|
||||
```
|
||||
|
||||
`sweep_over` (line 1841): change the signature return type `-> SweepFamily` to
|
||||
`-> (SweepFamily, Option<Vec<usize>>)`; change its terminal `.sweep(|point| {`
|
||||
(line 1854) to `.sweep_with_lattice(|point| {`, and insert the `.map(...)` line
|
||||
before the existing trailing
|
||||
` .expect("the built-in named grid matches the sample param-space")` (line
|
||||
1869), so the function tail reads:
|
||||
|
||||
```rust
|
||||
})
|
||||
.map(|(fam, lat)| (fam, Some(lat)))
|
||||
.expect("the built-in named grid matches the sample param-space")
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Thread `Selection` through `walkforward_family` (signature + both arms)**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, change `walkforward_family`'s signature (line
|
||||
1786-1788) to add the `select` parameter:
|
||||
|
||||
```rust
|
||||
fn walkforward_family(
|
||||
strategy: Strategy, trace: Option<&str>, data: &DataSource, grid: &Stage1RGrid,
|
||||
select: Selection,
|
||||
) -> WalkForwardResult {
|
||||
```
|
||||
|
||||
In the `Strategy::SmaCross` arm, replace the sweep + optimize lines (1802-1807):
|
||||
|
||||
```rust
|
||||
let (is_family, lattice) = sweep_over(w.is.0, w.is.1, data);
|
||||
let (best, selection) = match select_winner(&is_family, "total_pips", select, lattice.as_deref()) {
|
||||
Ok(v) => v,
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
};
|
||||
let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
```
|
||||
|
||||
In the `Strategy::Stage1R` arm, replace the sweep + optimize lines (1820-1825):
|
||||
|
||||
```rust
|
||||
let (is_family, lattice) = stage1_r_sweep_over(w.is.0, w.is.1, data, grid);
|
||||
let (best, selection) = match select_winner(&is_family, "sqn_normalized", select, lattice.as_deref()) {
|
||||
Ok(v) => v,
|
||||
Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
|
||||
};
|
||||
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Thread the new `select` argument through every `walkforward_family` caller**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, update the production caller and the four test
|
||||
callers to pass a `Selection`:
|
||||
|
||||
- `run_walkforward` (line 1755): add `select: Selection` to its signature:
|
||||
`fn run_walkforward(strategy: Strategy, name: &str, persist: bool, data: DataSource, grid: &Stage1RGrid, select: Selection) {`
|
||||
and update its `walkforward_family(...)` call (line 1763) to
|
||||
`let result = walkforward_family(strategy, persist.then_some(name), &data, grid, select);`
|
||||
- Line 1963: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
- Line 2131: `walkforward_family(Strategy::Stage1R, None, data, grid, Selection::Argmax)`
|
||||
- Line 3456: `walkforward_family(Strategy::SmaCross, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
- Line 4131: `walkforward_family(Strategy::Stage1R, None, &DataSource::Synthetic, &Stage1RGrid::default(), Selection::Argmax)`
|
||||
|
||||
- [ ] **Step 8: Add `--select` to `parse_walkforward_args` and thread it to the dispatch**
|
||||
|
||||
In `crates/aura-cli/src/main.rs`, in `parse_walkforward_args` (line 1648):
|
||||
|
||||
Change the return type to add `Selection`:
|
||||
`) -> Result<(Strategy, String, bool, DataChoice, Stage1RGrid, Selection), String> {`
|
||||
|
||||
Update the usage string to mention `--select` (append `[--select <argmax|plateau:mean|plateau:worst>]`
|
||||
to both the `///` doc usage line and the `usage` closure string).
|
||||
|
||||
Add `let mut select = Selection::Argmax;` beside the other `let mut` locals.
|
||||
|
||||
Add a match arm for the flag (beside `--stop-k`):
|
||||
`"--select" => select = parse_select(value).map_err(|()| usage())?,`
|
||||
|
||||
Change the final return to:
|
||||
`Ok((strategy, name, persist, real.finish(&usage)?, grid, select))`
|
||||
|
||||
In the command dispatch (line 2935-2939), update the destructure + call:
|
||||
|
||||
```rust
|
||||
["walkforward", rest @ ..] => match parse_walkforward_args(rest) {
|
||||
Ok((strategy, name, persist, choice, grid, select)) => {
|
||||
run_walkforward(strategy, &name, persist, DataSource::from_choice(choice), &grid, select)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 9: Compile-gate — workspace + tests build clean**
|
||||
|
||||
Run: `cargo build -p aura-cli --tests`
|
||||
Expected: clean (0 errors) — every caller of the changed signatures threaded.
|
||||
|
||||
- [ ] **Step 10: Run the new unit tests (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-cli select_winner_refuses`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo test -p aura-cli parse_walkforward_select`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 11: C23 — existing walk-forward goldens stay byte-identical**
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_strategy_stage1_r`
|
||||
Expected: PASS — argmax path unchanged.
|
||||
|
||||
Run: `cargo test -p aura-cli runs_family_rank_shows_deflated`
|
||||
Expected: PASS — the deflated display line is byte-identical.
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_bare_sma`
|
||||
Expected: PASS — the SMA summary golden is unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: `runs family rank` plateau E2E golden
|
||||
|
||||
**Files:**
|
||||
- Test: `crates/aura-cli/tests/cli_run.rs` — `walkforward_plateau_select_stamps_plateau_provenance`
|
||||
|
||||
Tasks 1-4 deliver the behaviour; this pins it end-to-end (C18: the plateau
|
||||
provenance reaches disk and the display renders it).
|
||||
|
||||
- [ ] **Step 1: Write the plateau E2E (RED until Tasks 1-4 land; GREEN after)**
|
||||
|
||||
In `crates/aura-cli/tests/cli_run.rs`, beside `runs_family_rank_shows_deflated_line`
|
||||
(after line 2388), add:
|
||||
|
||||
```rust
|
||||
/// Property: a stage1-r walk-forward run with `--select plateau:mean` selects the
|
||||
/// neighbourhood-smoothed winner and stamps each OOS manifest with the plateau
|
||||
/// provenance (`mode = PlateauMean`, `neighbourhood_score`, `n_neighbours`); `runs
|
||||
/// family … rank` renders the `plateau(mean)=… over N cells` line, and the
|
||||
/// deflation fields are omitted (orthogonal annotation). End-to-end witness that
|
||||
/// the opt-in selection rule reaches disk (C18) and the display path renders it.
|
||||
#[test]
|
||||
fn walkforward_plateau_select_stamps_plateau_provenance() {
|
||||
let cwd = temp_cwd("runs-plateau");
|
||||
let wf = Command::new(BIN)
|
||||
.args([
|
||||
"walkforward", "--strategy", "stage1-r", "--select", "plateau:mean",
|
||||
"--fast", "50,100", "--slow", "200,400",
|
||||
"--stop-length", "14,21", "--stop-k", "2.0,3.0",
|
||||
"--name", "wf-plat",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn walkforward --select plateau:mean");
|
||||
assert!(
|
||||
wf.status.success(),
|
||||
"walkforward exit: {:?}; stderr: {}",
|
||||
wf.status,
|
||||
String::from_utf8_lossy(&wf.stderr)
|
||||
);
|
||||
|
||||
let rank = Command::new(BIN)
|
||||
.args(["runs", "family", "wf-plat-0", "rank", "sqn_normalized"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn runs family wf-plat-0 rank sqn_normalized");
|
||||
assert!(rank.status.success(), "rank exit: {:?}", rank.status);
|
||||
let rank_out = String::from_utf8(rank.stdout).expect("utf-8");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
|
||||
assert!(rank_out.contains("plateau(mean)="), "rank output missing plateau(mean)=: {rank_out:?}");
|
||||
assert!(rank_out.contains(" cells"), "rank output missing 'over N cells': {rank_out:?}");
|
||||
assert!(rank_out.contains("\"mode\":\"PlateauMean\""), "manifest carries PlateauMean: {rank_out:?}");
|
||||
assert!(!rank_out.contains("\"deflated_score\""), "plateau run omits deflated_score: {rank_out:?}");
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the plateau E2E (GREEN)**
|
||||
|
||||
Run: `cargo test -p aura-cli walkforward_plateau`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 3: Full CLI suite + workspace + clippy (no regressions)**
|
||||
|
||||
Run: `cargo test -p aura-cli`
|
||||
Expected: PASS — every existing golden green; the plateau E2E green.
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS.
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: clean (no warnings).
|
||||
@@ -1,301 +0,0 @@
|
||||
# Prefer a robust parameter plateau over the in-sample peak — Design Spec
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
Cycle 0077 — second cycle of the milestone *Inferential validation*. Closes #145
|
||||
(refs the milestone). Builds on #144's selection-provenance carrier.
|
||||
|
||||
## Goal
|
||||
|
||||
`optimize` (and the walk-forward in-sample step that calls it) picks the single
|
||||
sharpest in-sample peak by a bare argmax — the configuration most likely fit to
|
||||
noise, since a small shift in the data or the parameters collapses it. There is no
|
||||
way to ask instead for a *robust region*: a broad neighbourhood of the parameter
|
||||
grid that is uniformly good.
|
||||
|
||||
This cycle adds an **opt-in plateau selection objective**: a new `optimize_plateau`
|
||||
that scores each grid member by the aggregate (mean or worst-case) of its grid
|
||||
neighbours' metric and argmaxes that *smoothed* surface, so the chosen member sits
|
||||
at the centre of a broad plateau rather than on a lucky spike. It is **strictly
|
||||
opt-in** via a new `--select` flag (default `argmax` — every existing run stays
|
||||
byte-identical, C23), and records its provenance on the same `RunManifest.selection`
|
||||
carrier #144 introduced (extended for the two selection rules).
|
||||
|
||||
## Architecture
|
||||
|
||||
A new family-level selector sits beside `optimize` / `optimize_deflated` in
|
||||
`aura-registry`:
|
||||
|
||||
`optimize_plateau(family, axis_lens, metric, mode)` reads each member's metric via
|
||||
the shared `metric_value` (#144's refactor — one direction source), computes the
|
||||
neighbourhood-aggregated score over the grid lattice, and argmaxes the smoothed
|
||||
surface by the metric's own direction (earliest-odometer tie, as `optimize`). The
|
||||
**grid neighbours** come from mixed-radix arithmetic over `axis_lens` (the per-axis
|
||||
cardinalities, in `param_space()` order, last-axis-fastest — the `GridSpace`
|
||||
odometer convention): a member's flat index decomposes to per-axis coordinates; its
|
||||
±1-per-axis cells are its neighbours; the closed neighbourhood is `{self} ∪
|
||||
{in-range ±1 neighbours}`. Pure, no RNG → deterministic (C1).
|
||||
|
||||
`axis_lens` is **an argument, not a `SweepFamily` field** (adding a field breaks
|
||||
`SweepFamily`'s derived `PartialEq` + its struct-literal sites). The `GridSpace`
|
||||
already owns the cardinalities at the live walk-forward call site; a new
|
||||
`GridSpace::axis_lens()` surfaces them, and the sweep builder returns them alongside
|
||||
the family so the call site can pass them in. The lattice is needed only transiently
|
||||
at selection time.
|
||||
|
||||
The selection RULE (argmax vs plateau) and the deflation ANNOTATION (#144) are
|
||||
**orthogonal**, so the shared `FamilySelection` carrier is extended — not duplicated
|
||||
— to record either: `SelectionMode` gains `PlateauMean` / `PlateauWorst` (the slot
|
||||
#144 reserved), `deflated_score` becomes `Option` (None under plateau), and two
|
||||
plateau fields are added (None under argmax). `optimize_deflated` fills the
|
||||
deflation fields under `mode = Argmax`; `optimize_plateau` fills the plateau fields
|
||||
under `mode = Plateau*`.
|
||||
|
||||
The policy lives in `aura-registry` (C9); `walk_forward` stays selection-agnostic.
|
||||
A `--select` flag is parsed in the CLI and threaded to `walkforward_family`, which
|
||||
dispatches `optimize_deflated` (argmax) or `optimize_plateau` (plateau). A
|
||||
**RandomSpace** sweep has no lattice, so `--select plateau` on one is **refused**
|
||||
(exit 2) — plateau adjacency falls out of a grid only.
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### User-facing program (the acceptance evidence)
|
||||
|
||||
```console
|
||||
$ aura walkforward --strategy stage1-r --select plateau:mean \
|
||||
--fast 50,100 --slow 200,400 --stop-length 14,21 --stop-k 2.0,3.0
|
||||
```
|
||||
|
||||
Default (no `--select`) is `argmax` — byte-identical to today. With
|
||||
`--select plateau:mean`, each window's in-sample winner is the centre of the
|
||||
broadest 4-D grid plateau, and its OOS manifest records the plateau provenance:
|
||||
|
||||
```jsonc
|
||||
"selection": {
|
||||
"selection_metric": "sqn_normalized",
|
||||
"n_trials": 16,
|
||||
"raw_winner_metric": 1.42, // the winner's own metric (still surfaced)
|
||||
"mode": "PlateauMean",
|
||||
"neighbourhood_score": 1.27, // the smoothed score the argmax maximised
|
||||
"n_neighbours": 5 // closed-neighbourhood size at this cell (≤ 1 + 2·dim)
|
||||
// deflated_score / overfit_probability / n_resamples … omitted under plateau
|
||||
}
|
||||
```
|
||||
|
||||
`aura runs family <id> rank sqn_normalized` shows a human-readable line:
|
||||
|
||||
```console
|
||||
# … each member, best-first: sqn_normalized=1.42 plateau(mean)=1.27 over 5 cells
|
||||
```
|
||||
|
||||
A plateau request on a random sweep is refused:
|
||||
|
||||
```console
|
||||
$ aura walkforward --strategy stage1-r --real EURUSD --random 64 --select plateau:mean
|
||||
aura: --select plateau requires a grid sweep; a random sweep has no parameter lattice (exit 2)
|
||||
```
|
||||
|
||||
### `FamilySelection` / `SelectionMode` — before → after (`aura-engine/src/report.rs`)
|
||||
|
||||
```rust
|
||||
// before (#144)
|
||||
pub enum SelectionMode { Argmax }
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub deflated_score: f64,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
pub mode: SelectionMode,
|
||||
pub n_resamples: usize,
|
||||
pub block_len: usize,
|
||||
pub seed: u64,
|
||||
}
|
||||
|
||||
// after (#145) — orthogonal rule × annotation; legacy lines still load (serde default)
|
||||
pub enum SelectionMode { Argmax, PlateauMean, PlateauWorst }
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub mode: SelectionMode,
|
||||
// deflation annotation (present iff mode == Argmax with a deflation run)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub deflated_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_resamples: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub block_len: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub seed: Option<u64>,
|
||||
// plateau annotation (present iff mode is Plateau*)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub neighbourhood_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_neighbours: Option<usize>,
|
||||
}
|
||||
```
|
||||
|
||||
(`#144`'s `optimize_deflated` and its tests update to the `Option` deflation fields
|
||||
— `deflated_score: Some(raw - p95)`, etc.; the existing assertions become
|
||||
`sel.deflated_score.unwrap()`. This reshape is a small wire change to a
|
||||
one-cycle-old, test-only-data type; the #144 serde-back-compat tests are the green
|
||||
gate it must preserve.)
|
||||
|
||||
### `GridSpace::axis_lens` + sweep-builder surfacing (`aura-engine/src/sweep.rs`)
|
||||
|
||||
```rust
|
||||
impl GridSpace {
|
||||
/// Per-axis cardinalities in `param_space()` order (the odometer radixes,
|
||||
/// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau
|
||||
/// neighbourhood walks.
|
||||
pub fn axis_lens(&self) -> Vec<usize> { self.axes.iter().map(Vec::len).collect() }
|
||||
}
|
||||
```
|
||||
|
||||
The sweep entry the CLI builder calls returns the family **and** its `axis_lens`
|
||||
(e.g. a `sweep_with_lattice` variant, or the builder exposes the built `GridSpace`),
|
||||
so `sweep_over` / `stage1_r_sweep_over` hand `(SweepFamily, Vec<usize>)` back to
|
||||
`walkforward_family`. A `RandomSpace` yields `None` for the lattice (no axes).
|
||||
|
||||
### The selector (`aura-registry/src/lib.rs`, beside `optimize_deflated`)
|
||||
|
||||
```rust
|
||||
#[derive(Clone, Copy)]
|
||||
pub enum PlateauMode { Mean, Worst }
|
||||
|
||||
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface plus its plateau
|
||||
/// provenance. Each member scores as the mean (or worst-case) of its closed grid
|
||||
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the
|
||||
/// metric's own direction (earliest-odometer tie). `axis_lens` are the grid
|
||||
/// radixes (last-axis-fastest). Pure, deterministic (C1).
|
||||
pub fn optimize_plateau(
|
||||
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
// smoothed score per member i:
|
||||
// nbrs = closed_neighbourhood(i, axis_lens) // {i} ∪ in-range ±1-per-axis
|
||||
// vals = nbrs.map(|j| metric_value(&family.points[j].report, m))
|
||||
// score[i] = match mode { Mean => mean(vals), Worst => worst(vals, dir) }
|
||||
// winner = argmax_by_direction(score, m) // earliest-odometer tie, as optimize
|
||||
// ... returns (winner.clone(), FamilySelection { mode, raw_winner_metric,
|
||||
// neighbourhood_score: Some(score[winner]), n_neighbours: Some(nbrs.len()), … })
|
||||
}
|
||||
|
||||
// closed_neighbourhood(i, axis_lens): decompose i to mixed-radix coords (last
|
||||
// radix fastest), emit i plus each coord±1 that stays in [0, axis_lens[k]).
|
||||
fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> { /* pure index math */ }
|
||||
```
|
||||
|
||||
`Worst` is the most-conservative neighbour by the metric's direction (the `min` for
|
||||
a higher-is-better metric); it biases toward interior cells (a boundary cell has
|
||||
fewer neighbours), which the testing strategy pins.
|
||||
|
||||
### CLI `--select` (`aura-cli/src/main.rs`)
|
||||
|
||||
```rust
|
||||
enum Selection { Argmax, Plateau(PlateauMode) }
|
||||
// parse_walkforward_args: "--select argmax" | "plateau:mean" | "plateau:worst";
|
||||
// default Argmax; an unknown token is a usage error (exit 2).
|
||||
// walkforward_family, per arm:
|
||||
// let (is_family, axis_lens) = sweep_over(...); // now returns the lattice
|
||||
// let (best, selection) = match select {
|
||||
// Selection::Argmax => optimize_deflated(&is_family, metric, …),
|
||||
// Selection::Plateau(pm) => match axis_lens {
|
||||
// Some(lens) => optimize_plateau(&is_family, &lens, metric, pm),
|
||||
// None => { eprintln!("aura: --select plateau requires a grid sweep …"); exit(2) }
|
||||
// },
|
||||
// };
|
||||
// oos_report.manifest.selection = Some(selection); // the #144 stamping seam
|
||||
```
|
||||
|
||||
`runs_family` display gains a `plateau(<mode>)=<score> over <n> cells` line when
|
||||
`mode` is `Plateau*` (beside the existing deflated line for argmax).
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Crate / file | Change |
|
||||
|---|---|---|
|
||||
| `SelectionMode` + `FamilySelection` reshape | `aura-engine/report.rs` | 2 enum variants; `deflated_score`→`Option`; deflation-trio→`Option`; add `neighbourhood_score`/`n_neighbours` |
|
||||
| `GridSpace::axis_lens` + sweep-builder lattice return | `aura-engine/sweep.rs` | new accessor; sweep returns `(SweepFamily, axis_lens)` |
|
||||
| `optimize_plateau`, `PlateauMode`, `closed_neighbourhood` | `aura-registry/lib.rs` | new selector + pure index math; `optimize_deflated` updated to `Option` deflation fields |
|
||||
| compat mirror | `aura-registry/compat.rs` | already carries `selection` (#144) — no new field, but confirm the reshaped `FamilySelection` round-trips |
|
||||
| `--select` parse + dispatch + display | `aura-cli/main.rs` | flag, `walkforward_family` dispatch, lattice threading, `runs_family` line |
|
||||
|
||||
## Data flow
|
||||
|
||||
1. `parse_walkforward_args` reads `--select` → `Selection` (default `Argmax`).
|
||||
2. Per window, the arm's sweep returns `(is_family, Option<axis_lens>)`.
|
||||
3. `Argmax` → `optimize_deflated` (unchanged path, #144). `Plateau(mode)` → with
|
||||
`Some(lens)` → `optimize_plateau`; with `None` (RandomSpace) → refuse, exit 2.
|
||||
4. `optimize_plateau` smooths each member's metric over its closed grid
|
||||
neighbourhood, argmaxes by direction, returns the winner + a `FamilySelection`
|
||||
with `mode = Plateau*`, `neighbourhood_score`, `n_neighbours`.
|
||||
5. The record stamps `oos_report.manifest.selection`; flows through `append_family`
|
||||
unchanged; `runs_family` renders it.
|
||||
|
||||
## Error handling
|
||||
|
||||
- `--select plateau` on a random sweep (no lattice): `exit 2` with a clear message
|
||||
(refuse-don't-guess, C18 discipline) — never a silent argmax fallback.
|
||||
- An unknown `--select` token: usage error, `exit 2`.
|
||||
- A degenerate single-member family: its closed neighbourhood is itself; the plateau
|
||||
score equals the raw — `optimize_plateau`'s winner equals `optimize`'s (no spurious
|
||||
reshaping of a 1-point grid).
|
||||
- `axis_lens` whose product ≠ `family.points.len()` (a malformed lattice) is an
|
||||
internal invariant violation, asserted (`debug_assert`); the shipped call sites
|
||||
pass the grid's own lengths, so it is unreachable.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **C23 default-argmax byte-identical:** `--select argmax` (and no `--select`)
|
||||
produces a walk-forward run byte-for-byte identical to the pre-0077 output (the
|
||||
existing walk-forward goldens stay green unchanged); plateau is strictly opt-in.
|
||||
- **#144 reshape preserved:** `optimize_deflated`'s tests + the serde back-compat
|
||||
tests stay green after `deflated_score` → `Option` (a legacy `selection`-less line
|
||||
loads as `None`; a stamped argmax line round-trips with the deflation fields).
|
||||
- **Plateau picks the plateau, not the spike:** a fabricated grid `SweepFamily` with
|
||||
a sharp isolated peak and a separate broad plateau — `optimize` picks the spike,
|
||||
`optimize_plateau(Mean)` picks the plateau centre (a different `SweepPoint`); the
|
||||
winner is the smoothed argmax, recorded `neighbourhood_score < raw_winner_metric`
|
||||
of the spike.
|
||||
- **mean vs worst:** on the same fixture, `Worst` is more conservative and biases
|
||||
toward interior cells (a boundary cell loses neighbours) — pinned so the
|
||||
edge-truncation asymmetry is documented, not silent.
|
||||
- **Mixed-radix neighbours:** `closed_neighbourhood(i, axis_lens)` golden on a known
|
||||
small lattice (e.g. 2×3) — exact neighbour index sets, last-axis-fastest.
|
||||
- **Small-grid degeneracy:** a 2-value axis → the neighbourhood is most of the grid;
|
||||
a documented caveat test (plateau ≈ global mean), not a failure.
|
||||
- **RandomSpace refuse:** `--select plateau` on a `--random` sweep exits 2 with the
|
||||
message; never silently argmaxes.
|
||||
- **C1 determinism:** identical family + `axis_lens` + mode → identical winner +
|
||||
`FamilySelection` (pure, no RNG).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `aura walkforward … --select plateau:mean|plateau:worst` selects the
|
||||
neighbourhood-smoothed winner and stamps `mode`/`neighbourhood_score`/`n_neighbours`
|
||||
on each OOS winner's manifest; `runs family … rank` surfaces the plateau line.
|
||||
- Default `argmax` keeps every existing walk-forward run byte-identical (C23) and the
|
||||
#144 deflation path intact.
|
||||
- `--select plateau` on a random sweep is refused (exit 2).
|
||||
- The selection is deterministic from the family + axis_lens + mode (C1), reads only
|
||||
the in-sample family (C2), and lives in `aura-registry` with `walk_forward`
|
||||
selection-agnostic (C9). Legacy registry lines still load (C14/C18).
|
||||
|
||||
### Out of scope (deferred)
|
||||
|
||||
- RandomSpace plateau via a kNN distance metric (an invented adjacency — its own
|
||||
cycle if ever wanted).
|
||||
- Offline re-plateau of a reloaded family (the lattice is transient at live
|
||||
selection; a reloaded family carries no `axis_lens`).
|
||||
- Per-arm selection (one global `--select` suffices; the metric already differs per
|
||||
arm and the objective is metric-agnostic).
|
||||
- Composing plateau selection with trials-deflation on the same winner (the two
|
||||
annotations are orthogonal but their composition is a later concern; this cycle
|
||||
records one rule per run).
|
||||
Reference in New Issue
Block a user