feat(0077): prefer a robust parameter plateau over the in-sample peak
Add an opt-in plateau selection objective for walk-forward, recorded on the same RunManifest.selection carrier #144 introduced. Default argmax is byte-identical (C23); plateau is reached only via a new --select flag. What landed: - FamilySelection / SelectionMode reshape (report.rs): the selection RULE (mode) and its ANNOTATION are orthogonal — deflation fields become Option (present iff Argmax), plus PlateauMean/PlateauWorst variants and neighbourhood_score/n_neighbours (present iff Plateau*). Legacy lines still load: the deflation scalars deserialize from bare values via serde default (C14/C18). compat.rs embeds FamilySelection by value — reshape flows through untouched. - GridSpace::axis_lens() + SweepBinder::sweep_with_lattice (engine): the grid radixes in param_space()/odometer order. sweep() delegates to sweep_with_lattice with the lattice dropped, so every existing .sweep() caller is byte-unchanged. - optimize_plateau + PlateauMode + closed_neighbourhood (aura-registry, C9): each member scores as the mean/worst of its closed mixed-radix grid neighbourhood's metric_value; the winner is the smoothed argmax by the metric's direction (earliest-odometer tie, as optimize). Pure, no RNG (C1); in-sample only (C2). A higher_is_better helper now sources the per-metric direction once, shared by metric_cmp and optimize_plateau. - CLI --select <argmax|plateau:mean|plateau:worst> (default argmax; unknown token exits 2). walkforward_family dispatches via a select_winner helper; sweep_over/stage1_r_sweep_over carry the lattice as Option<Vec<usize>>. runs-family display gains a plateau(<mode>)=<score> over <n> cells line. RandomSpace-refuse: walkforward has no --random producer, so the plateau-without-lattice guard is structural — select_winner returns the refuse on a None lattice (the caller prints the message and exits 2), unit-tested at the helper, not via a --random E2E (decided + recorded on #145 comment 1974, with the lattice-seam fork). Tests: plateau-picks-plateau-not-spike, mean-vs-worst, lower-is-better direction-flip, closed-neighbourhood golden, single-member degeneracy, C1 determinism; CLI select-parse, select_winner refuse; E2E argmax-byte-identical (C23), plateau:mean and plateau:worst provenance. Full workspace suite green; clippy --all-targets -D warnings clean. closes #145
This commit is contained in:
@@ -2387,6 +2387,130 @@ fn runs_family_rank_shows_deflated_line() {
|
||||
);
|
||||
}
|
||||
|
||||
/// Property: a stage1-r walk-forward run with `--select plateau:mean` selects the
|
||||
/// neighbourhood-smoothed winner and stamps each OOS manifest with the plateau
|
||||
/// provenance (`mode = PlateauMean`, `neighbourhood_score`, `n_neighbours`); `runs
|
||||
/// family … rank` renders the `plateau(mean)=… over N cells` line, and the
|
||||
/// deflation fields are omitted (orthogonal annotation). End-to-end witness that
|
||||
/// the opt-in selection rule reaches disk (C18) and the display path renders it.
|
||||
#[test]
|
||||
fn walkforward_plateau_select_stamps_plateau_provenance() {
|
||||
let cwd = temp_cwd("runs-plateau");
|
||||
let wf = Command::new(BIN)
|
||||
.args([
|
||||
"walkforward", "--strategy", "stage1-r", "--select", "plateau:mean",
|
||||
"--fast", "50,100", "--slow", "200,400",
|
||||
"--stop-length", "14,21", "--stop-k", "2.0,3.0",
|
||||
"--name", "wf-plat",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn walkforward --select plateau:mean");
|
||||
assert!(
|
||||
wf.status.success(),
|
||||
"walkforward exit: {:?}; stderr: {}",
|
||||
wf.status,
|
||||
String::from_utf8_lossy(&wf.stderr)
|
||||
);
|
||||
|
||||
let rank = Command::new(BIN)
|
||||
.args(["runs", "family", "wf-plat-0", "rank", "sqn_normalized"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn runs family wf-plat-0 rank sqn_normalized");
|
||||
assert!(rank.status.success(), "rank exit: {:?}", rank.status);
|
||||
let rank_out = String::from_utf8(rank.stdout).expect("utf-8");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
|
||||
assert!(rank_out.contains("plateau(mean)="), "rank output missing plateau(mean)=: {rank_out:?}");
|
||||
assert!(rank_out.contains(" cells"), "rank output missing 'over N cells': {rank_out:?}");
|
||||
assert!(rank_out.contains("\"mode\":\"PlateauMean\""), "manifest carries PlateauMean: {rank_out:?}");
|
||||
assert!(!rank_out.contains("\"deflated_score\""), "plateau run omits deflated_score: {rank_out:?}");
|
||||
}
|
||||
|
||||
/// Property (C23, the opt-in feature's headline guarantee): `--select argmax` is a
|
||||
/// no-op against the default. A stage1-r walk-forward run with an EXPLICIT
|
||||
/// `--select argmax` produces stdout byte-for-byte identical to the same run with
|
||||
/// no `--select` flag at all, AND keeps the trials-deflation provenance
|
||||
/// (`mode == Argmax`, the deflated annotation) on each OOS manifest. Pins both
|
||||
/// halves of "plateau is strictly opt-in": the flag's default IS argmax (no
|
||||
/// divergence), and threading the new `Selection` did not silently demote argmax to
|
||||
/// a bare pick (the #144 deflation path survives). A regression where `--select`
|
||||
/// perturbed the default path, or where the default switched away from argmax,
|
||||
/// passes every other test but fails this byte-equality.
|
||||
#[test]
|
||||
fn walkforward_select_argmax_is_byte_identical_to_default() {
|
||||
let run = |args: &[&str]| {
|
||||
let cwd = temp_cwd("wf-argmax-noop");
|
||||
let out = Command::new(BIN)
|
||||
.arg("walkforward")
|
||||
.args(args)
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn aura walkforward");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"walkforward exit: {:?}; stderr: {}",
|
||||
out.status,
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
String::from_utf8(out.stdout).expect("utf-8")
|
||||
};
|
||||
let default = run(&["--strategy", "stage1-r"]);
|
||||
let explicit = run(&["--strategy", "stage1-r", "--select", "argmax"]);
|
||||
assert_eq!(default, explicit, "explicit --select argmax must be a no-op vs the default");
|
||||
// argmax stayed the deflation path (not a bare argmax): the per-window member
|
||||
// JSON carries the Argmax mode and its deflation annotation.
|
||||
assert!(default.contains("\"mode\":\"Argmax\""), "argmax manifest mode: {default:?}");
|
||||
assert!(default.contains("\"deflated_score\""), "argmax keeps the deflation annotation: {default:?}");
|
||||
assert!(!default.contains("\"neighbourhood_score\""), "argmax omits the plateau annotation: {default:?}");
|
||||
}
|
||||
|
||||
/// Property: `--select plateau:worst` reaches disk and the display path through the
|
||||
/// DISTINCT worst-case branch — a separate `SelectionMode::PlateauWorst` enum
|
||||
/// variant on the wire and a separate `plateau(worst)=` display label. The landed
|
||||
/// happy-path E2E exercises only `plateau:mean`; the worst arm's serialization
|
||||
/// variant and its display label are otherwise untouched end-to-end, so a
|
||||
/// regression that mislabelled worst as mean (or failed to round-trip the variant)
|
||||
/// would pass the mean test. Same fixture grid as the mean E2E, only the rule
|
||||
/// differs.
|
||||
#[test]
|
||||
fn walkforward_plateau_worst_stamps_worst_variant_and_label() {
|
||||
let cwd = temp_cwd("runs-plateau-worst");
|
||||
let wf = Command::new(BIN)
|
||||
.args([
|
||||
"walkforward", "--strategy", "stage1-r", "--select", "plateau:worst",
|
||||
"--fast", "50,100", "--slow", "200,400",
|
||||
"--stop-length", "14,21", "--stop-k", "2.0,3.0",
|
||||
"--name", "wf-worst",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn walkforward --select plateau:worst");
|
||||
assert!(
|
||||
wf.status.success(),
|
||||
"walkforward exit: {:?}; stderr: {}",
|
||||
wf.status,
|
||||
String::from_utf8_lossy(&wf.stderr)
|
||||
);
|
||||
|
||||
let rank = Command::new(BIN)
|
||||
.args(["runs", "family", "wf-worst-0", "rank", "sqn_normalized"])
|
||||
.current_dir(&cwd)
|
||||
.output()
|
||||
.expect("spawn runs family wf-worst-0 rank sqn_normalized");
|
||||
assert!(rank.status.success(), "rank exit: {:?}", rank.status);
|
||||
let rank_out = String::from_utf8(rank.stdout).expect("utf-8");
|
||||
let _ = std::fs::remove_dir_all(&cwd);
|
||||
|
||||
assert!(rank_out.contains("plateau(worst)="), "rank output missing plateau(worst)=: {rank_out:?}");
|
||||
assert!(rank_out.contains("\"mode\":\"PlateauWorst\""), "manifest carries PlateauWorst: {rank_out:?}");
|
||||
// the worst label must not be rendered or stamped as mean
|
||||
assert!(!rank_out.contains("plateau(mean)="), "worst run must not render the mean label: {rank_out:?}");
|
||||
assert!(!rank_out.contains("\"mode\":\"PlateauMean\""), "worst run must not stamp PlateauMean: {rank_out:?}");
|
||||
}
|
||||
|
||||
/// Property: the bare `aura walkforward` (default SMA-cross) summary is preserved
|
||||
/// byte-for-byte by the new `--strategy`/grid plumbing — it still carries exactly
|
||||
/// `windows`, `stitched_total_pips`, `param_stability` and NOT the stage1-r-only
|
||||
|
||||
Reference in New Issue
Block a user