feat(0077): prefer a robust parameter plateau over the in-sample peak
Add an opt-in plateau selection objective for walk-forward, recorded on the same RunManifest.selection carrier #144 introduced. Default argmax is byte-identical (C23); plateau is reached only via a new --select flag. What landed: - FamilySelection / SelectionMode reshape (report.rs): the selection RULE (mode) and its ANNOTATION are orthogonal — deflation fields become Option (present iff Argmax), plus PlateauMean/PlateauWorst variants and neighbourhood_score/n_neighbours (present iff Plateau*). Legacy lines still load: the deflation scalars deserialize from bare values via serde default (C14/C18). compat.rs embeds FamilySelection by value — reshape flows through untouched. - GridSpace::axis_lens() + SweepBinder::sweep_with_lattice (engine): the grid radixes in param_space()/odometer order. sweep() delegates to sweep_with_lattice with the lattice dropped, so every existing .sweep() caller is byte-unchanged. - optimize_plateau + PlateauMode + closed_neighbourhood (aura-registry, C9): each member scores as the mean/worst of its closed mixed-radix grid neighbourhood's metric_value; the winner is the smoothed argmax by the metric's direction (earliest-odometer tie, as optimize). Pure, no RNG (C1); in-sample only (C2). A higher_is_better helper now sources the per-metric direction once, shared by metric_cmp and optimize_plateau. - CLI --select <argmax|plateau:mean|plateau:worst> (default argmax; unknown token exits 2). walkforward_family dispatches via a select_winner helper; sweep_over/stage1_r_sweep_over carry the lattice as Option<Vec<usize>>. runs-family display gains a plateau(<mode>)=<score> over <n> cells line. RandomSpace-refuse: walkforward has no --random producer, so the plateau-without-lattice guard is structural — select_winner returns the refuse on a None lattice (the caller prints the message and exits 2), unit-tested at the helper, not via a --random E2E (decided + recorded on #145 comment 1974, with the lattice-seam fork). Tests: plateau-picks-plateau-not-spike, mean-vs-worst, lower-is-better direction-flip, closed-neighbourhood golden, single-member degeneracy, C1 determinism; CLI select-parse, select_winner refuse; E2E argmax-byte-identical (C23), plateau:mean and plateau:worst provenance. Full workspace suite green; clippy --all-targets -D warnings clean. closes #145
This commit is contained in:
+258
-19
@@ -155,21 +155,20 @@ fn metric_value(rep: &RunReport, m: Metric) -> f64 {
|
||||
/// deterministically without panicking. An unknown metric name is a
|
||||
/// `RegistryError::UnknownMetric`.
|
||||
///
|
||||
/// The single source of truth for "best" — both [`rank_by`] (sort by it) and
|
||||
/// [`optimize`] (argmax by it) call this, so the per-metric direction lives in
|
||||
/// exactly one place. The metric name is resolved once via `resolve_metric`; the
|
||||
/// returned closure carries the resolved `Metric`, so per-comparison work is just
|
||||
/// reading each value (`metric_value`) and the key compare.
|
||||
/// The single source of best-first *ordering* — both [`rank_by`] (sort by it) and
|
||||
/// [`optimize`] (argmax by it) call this. The per-metric direction itself lives in
|
||||
/// `higher_is_better` (which this and `optimize_plateau` both read); `metric_cmp`
|
||||
/// turns that direction into a `total_cmp` over the read values. The metric name is
|
||||
/// resolved once via `resolve_metric`; the returned closure carries the resolved
|
||||
/// `Metric`, so per-comparison work is just reading each value (`metric_value`) and
|
||||
/// the key compare.
|
||||
fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Ordering, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let hib = higher_is_better(m);
|
||||
Ok(move |a: &RunReport, b: &RunReport| {
|
||||
let (va, vb) = (metric_value(a, m), metric_value(b, m));
|
||||
match m {
|
||||
// lower-is-better
|
||||
Metric::MaxDrawdown | Metric::BiasSignFlips => va.total_cmp(&vb),
|
||||
// higher-is-better
|
||||
_ => vb.total_cmp(&va),
|
||||
}
|
||||
// higher-is-better ranks the larger value first; lower-is-better the smaller.
|
||||
if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) }
|
||||
})
|
||||
}
|
||||
|
||||
@@ -206,6 +205,43 @@ fn is_r_metric(m: Metric) -> bool {
|
||||
matches!(m, Metric::Sqn | Metric::SqnNormalized | Metric::ExpectancyR | Metric::NetExpectancyR)
|
||||
}
|
||||
|
||||
/// The metric's optimisation direction: `true` if a larger value is better
|
||||
/// (`total_pips`, the four R keys), `false` for the lower-is-better keys
|
||||
/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp`
|
||||
/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour)
|
||||
/// both read it, so the per-metric sense lives in exactly one place.
|
||||
fn higher_is_better(m: Metric) -> bool {
|
||||
!matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips)
|
||||
}
|
||||
|
||||
/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice
|
||||
/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}`
|
||||
/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order
|
||||
/// is `i` first then per-axis neighbours; callers use the result only as a set and
|
||||
/// for its length, so the order is not load-bearing.
|
||||
fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> {
|
||||
// decompose i to per-axis coords (last axis fastest)
|
||||
let mut coords = vec![0usize; axis_lens.len()];
|
||||
let mut rem = i;
|
||||
for k in (0..axis_lens.len()).rev() {
|
||||
coords[k] = rem % axis_lens[k];
|
||||
rem /= axis_lens[k];
|
||||
}
|
||||
// recompose a coord vector to its flat index (Horner, last axis fastest)
|
||||
let recompose = |c: &[usize]| c.iter().zip(axis_lens).fold(0usize, |flat, (&ck, &len)| flat * len + ck);
|
||||
let mut out = vec![i];
|
||||
for k in 0..axis_lens.len() {
|
||||
for delta in [-1isize, 1] {
|
||||
let ck = coords[k] as isize + delta;
|
||||
if ck < 0 || ck as usize >= axis_lens[k] { continue; }
|
||||
let mut nb = coords.clone();
|
||||
nb[k] = ck as usize;
|
||||
out.push(recompose(&nb));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn member_trade_rs(rep: &RunReport) -> &[f64] {
|
||||
rep.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[])
|
||||
}
|
||||
@@ -255,6 +291,84 @@ fn member_sd(family: &SweepFamily, m: Metric) -> f64 {
|
||||
(vals.iter().map(|v| (v - mean).powi(2)).sum::<f64>() / (n - 1) as f64).sqrt()
|
||||
}
|
||||
|
||||
/// Plateau-selection aggregate: how a member's grid-neighbourhood metric is
|
||||
/// reduced to one smoothed score before the argmax. `Mean` averages the closed
|
||||
/// neighbourhood; `Worst` takes the most-pessimistic neighbour by the metric's
|
||||
/// direction (biasing toward interior cells, which keep more neighbours).
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum PlateauMode {
|
||||
Mean,
|
||||
Worst,
|
||||
}
|
||||
|
||||
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau
|
||||
/// provenance. Each member scores as the mean (or worst-case) of its closed grid
|
||||
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the
|
||||
/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are
|
||||
/// the grid radixes (`param_space()` order, last axis fastest). Pure,
|
||||
/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL;
|
||||
/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the
|
||||
/// smoothed value the argmax maximised.
|
||||
pub fn optimize_plateau(
|
||||
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
let n = family.points.len();
|
||||
debug_assert_eq!(
|
||||
axis_lens.iter().product::<usize>(), n,
|
||||
"axis_lens product must equal the family size (a valid grid lattice)",
|
||||
);
|
||||
let hib = higher_is_better(m);
|
||||
// smoothed score + closed-neighbourhood size per member, in odometer order
|
||||
let scored: Vec<(usize, f64, usize)> = (0..n)
|
||||
.map(|i| {
|
||||
let nbrs = closed_neighbourhood(i, axis_lens);
|
||||
let vals: Vec<f64> = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect();
|
||||
let score = match mode {
|
||||
PlateauMode::Mean => vals.iter().sum::<f64>() / vals.len() as f64,
|
||||
PlateauMode::Worst => {
|
||||
if hib {
|
||||
vals.iter().copied().fold(f64::INFINITY, f64::min)
|
||||
} else {
|
||||
vals.iter().copied().fold(f64::NEG_INFINITY, f64::max)
|
||||
}
|
||||
}
|
||||
};
|
||||
(i, score, nbrs.len())
|
||||
})
|
||||
.collect();
|
||||
// argmax the smoothed surface by direction; earliest-odometer tie (only a
|
||||
// strictly-better later cell displaces the incumbent, as `optimize`).
|
||||
let &(wi, wscore, wn) = scored
|
||||
.iter()
|
||||
.reduce(|best, cur| {
|
||||
let better = if hib { cur.1 > best.1 } else { cur.1 < best.1 };
|
||||
if better { cur } else { best }
|
||||
})
|
||||
.expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)");
|
||||
let winner = family.points[wi].clone();
|
||||
let raw = metric_value(&winner.report, m);
|
||||
Ok((
|
||||
winner,
|
||||
FamilySelection {
|
||||
selection_metric: metric.to_string(),
|
||||
n_trials: n,
|
||||
raw_winner_metric: raw,
|
||||
mode: match mode {
|
||||
PlateauMode::Mean => SelectionMode::PlateauMean,
|
||||
PlateauMode::Worst => SelectionMode::PlateauWorst,
|
||||
},
|
||||
deflated_score: None,
|
||||
overfit_probability: None,
|
||||
n_resamples: None,
|
||||
block_len: None,
|
||||
seed: None,
|
||||
neighbourhood_score: Some(wscore),
|
||||
n_neighbours: Some(wn),
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
|
||||
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23).
|
||||
/// R arm: a centred moving-block reality-check (`overfit_probability` =
|
||||
@@ -307,8 +421,10 @@ pub fn optimize_deflated(
|
||||
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
deflated_score, overfit_probability, mode: SelectionMode::Argmax,
|
||||
n_resamples, block_len, seed,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(deflated_score), overfit_probability,
|
||||
n_resamples: Some(n_resamples), block_len: Some(block_len), seed: Some(seed),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -746,7 +862,7 @@ mod tests {
|
||||
let sel = optimize_deflated(&fam, "total_pips", 100, 3, 1).unwrap().1;
|
||||
assert!(sel.overfit_probability.is_none());
|
||||
assert_eq!(sel.selection_metric, "total_pips");
|
||||
assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw
|
||||
assert!(sel.deflated_score.unwrap() <= sel.raw_winner_metric); // dispersion floor ≤ raw
|
||||
}
|
||||
|
||||
/// Sibling-contract parity with `r_bootstrap`, which defines `n_resamples ==
|
||||
@@ -759,8 +875,8 @@ mod tests {
|
||||
fn optimize_deflated_zero_resamples_floors_like_r_bootstrap() {
|
||||
let fam = fixture_family_with_r();
|
||||
let sel = optimize_deflated(&fam, "expectancy_r", 0, 3, 1).unwrap().1;
|
||||
assert_eq!(sel.n_resamples, 0);
|
||||
assert_eq!(sel.deflated_score, sel.raw_winner_metric, "p95 of an empty null is 0");
|
||||
assert_eq!(sel.n_resamples, Some(0));
|
||||
assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric, "p95 of an empty null is 0");
|
||||
assert_eq!(sel.overfit_probability, Some(1.0), "(0+1)/(0+1) Laplace floor");
|
||||
}
|
||||
|
||||
@@ -779,8 +895,8 @@ mod tests {
|
||||
}
|
||||
}
|
||||
let sel = optimize_deflated(&fam, "expectancy_r", 500, 3, 1).unwrap().1;
|
||||
assert!(sel.deflated_score.is_finite(), "deflated_score must be finite, got {}", sel.deflated_score);
|
||||
assert_eq!(sel.deflated_score, sel.raw_winner_metric);
|
||||
assert!(sel.deflated_score.unwrap().is_finite(), "deflated_score must be finite, got {:?}", sel.deflated_score);
|
||||
assert_eq!(sel.deflated_score.unwrap(), sel.raw_winner_metric);
|
||||
assert_eq!(sel.overfit_probability, Some(1.0));
|
||||
}
|
||||
|
||||
@@ -791,7 +907,7 @@ mod tests {
|
||||
let fam = fixture_family_one_strong_edge();
|
||||
let sel = optimize_deflated(&fam, "expectancy_r", 1000, 1, 9).unwrap().1;
|
||||
assert!(sel.overfit_probability.unwrap() < 0.2, "p={:?}", sel.overfit_probability);
|
||||
assert!(sel.deflated_score > 0.0);
|
||||
assert!(sel.deflated_score.unwrap() > 0.0);
|
||||
}
|
||||
|
||||
/// Every member shares the SAME strong +R edge. An *uncentred* null (each
|
||||
@@ -856,4 +972,127 @@ mod tests {
|
||||
let m = resolve_metric("sqn_normalized").unwrap();
|
||||
assert_eq!(sel.raw_winner_metric, metric_value(&winner.report, m));
|
||||
}
|
||||
|
||||
/// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle
|
||||
/// plateau in `total_pips`. Bare argmax takes an end spike; the plateau-mean
|
||||
/// argmax takes the interior plateau centre.
|
||||
fn fixture_grid_spike_vs_plateau() -> SweepFamily {
|
||||
let pips = [80.0, 5.0, 48.0, 50.0, 49.0, 5.0, 80.0];
|
||||
SweepFamily { space: vec![], points: pips.iter().map(|&p| member(p, vec![1.0])).collect() }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn closed_neighbourhood_2x3_golden() {
|
||||
// 2x3 lattice, odometer (last axis fastest):
|
||||
// (0,0)=0 (0,1)=1 (0,2)=2
|
||||
// (1,0)=3 (1,1)=4 (1,2)=5
|
||||
let sorted = |i: usize| { let mut v = closed_neighbourhood(i, &[2, 3]); v.sort_unstable(); v };
|
||||
assert_eq!(sorted(0), vec![0, 1, 3]); // corner (0,0): self + 2
|
||||
assert_eq!(sorted(4), vec![1, 3, 4, 5]); // interior (1,1): self + 3
|
||||
assert_eq!(sorted(2), vec![1, 2, 5]); // corner (0,2): self + 2
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_picks_the_plateau_not_the_spike() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let spike = optimize(&fam, "total_pips").unwrap();
|
||||
let (centre, sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_ne!(centre.report.metrics.total_pips, spike.report.metrics.total_pips);
|
||||
assert_eq!(centre.report.metrics.total_pips, 50.0, "plateau centre is the middle cell");
|
||||
assert_eq!(sel.mode, SelectionMode::PlateauMean);
|
||||
assert_eq!(sel.raw_winner_metric, 50.0);
|
||||
assert!(sel.neighbourhood_score.unwrap() < spike.report.metrics.total_pips,
|
||||
"smoothed score is below the spike's raw peak");
|
||||
assert_eq!(sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours");
|
||||
assert!(sel.deflated_score.is_none() && sel.n_resamples.is_none(),
|
||||
"deflation fields omitted under plateau");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_worst_is_more_conservative_than_mean() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let (_, mean_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Mean).unwrap();
|
||||
let (_, worst_sel) = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap();
|
||||
// Worst scores each cell by its weakest neighbour, never above the mean.
|
||||
assert!(worst_sel.neighbourhood_score.unwrap() <= mean_sel.neighbourhood_score.unwrap());
|
||||
assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst);
|
||||
assert!(worst_sel.n_neighbours.unwrap() >= 2, "winner is an interior cell");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plateau_single_member_equals_optimize() {
|
||||
let fam = SweepFamily { space: vec![], points: vec![member(42.0, vec![1.0])] };
|
||||
let plain = optimize(&fam, "total_pips").unwrap();
|
||||
let (winner, sel) = optimize_plateau(&fam, &[1], "total_pips", PlateauMode::Mean).unwrap();
|
||||
assert_eq!(winner.report.metrics.total_pips, plain.report.metrics.total_pips);
|
||||
assert_eq!(sel.n_neighbours, Some(1), "a 1-cell grid's closed neighbourhood is itself");
|
||||
assert_eq!(sel.neighbourhood_score, Some(42.0));
|
||||
assert_eq!(sel.raw_winner_metric, 42.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_plateau_is_deterministic() {
|
||||
let fam = fixture_grid_spike_vs_plateau();
|
||||
let a = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
let b = optimize_plateau(&fam, &[7], "total_pips", PlateauMode::Worst).unwrap().1;
|
||||
assert_eq!(a, b);
|
||||
}
|
||||
|
||||
/// A 1-D grid (7 cells, odometer order) in `max_drawdown` — the canonical
|
||||
/// lower-is-better key. Two end *dips* (drawdown 1.0, the lure for a bare
|
||||
/// lower-is-better argmax) flank a broad middle plateau of moderately-low
|
||||
/// drawdown (≈30), with worse shoulders. A member carries only `max_drawdown`;
|
||||
/// `total_pips`/`r` are inert here.
|
||||
fn fixture_grid_dip_vs_plateau_lower_is_better() -> SweepFamily {
|
||||
let dds = [1.0, 60.0, 30.0, 30.0, 31.0, 60.0, 1.0];
|
||||
let point = |dd: f64| SweepPoint {
|
||||
params: vec![],
|
||||
report: report_with(0.0, dd, 0),
|
||||
};
|
||||
SweepFamily { space: vec![], points: dds.iter().map(|&dd| point(dd)).collect() }
|
||||
}
|
||||
|
||||
/// Direction-flip coverage: on a lower-is-better metric (`max_drawdown`),
|
||||
/// plateau selection exercises the `else` (smaller-is-better) argmax branch AND
|
||||
/// the `Worst` arm's `NEG_INFINITY`/`max` fold. Bare argmax takes an end dip
|
||||
/// (the global minimum drawdown); the plateau-mean argmax takes the interior
|
||||
/// plateau centre, whose neighbourhood mean is the lowest. This is the
|
||||
/// lower-is-better companion the higher-is-better plateau tests do not reach.
|
||||
#[test]
|
||||
fn plateau_lower_is_better_picks_the_plateau_not_the_dip() {
|
||||
let fam = fixture_grid_dip_vs_plateau_lower_is_better();
|
||||
let dip = optimize(&fam, "max_drawdown").unwrap();
|
||||
assert_eq!(dip.report.metrics.max_drawdown, 1.0, "bare argmax takes the end dip");
|
||||
|
||||
let (centre, mean_sel) =
|
||||
optimize_plateau(&fam, &[7], "max_drawdown", PlateauMode::Mean).unwrap();
|
||||
// the plateau centre is an interior cell, not the end dip
|
||||
assert_ne!(centre.report.metrics.max_drawdown, 1.0);
|
||||
assert_eq!(mean_sel.mode, SelectionMode::PlateauMean);
|
||||
assert_eq!(mean_sel.n_neighbours, Some(3), "interior cell has self + 2 neighbours");
|
||||
// the winning cell's neighbourhood mean beats the end dip's (whose 60.0
|
||||
// shoulder drags its mean up), so the smaller-is-better argmax did not pick
|
||||
// the dip.
|
||||
assert!(
|
||||
mean_sel.neighbourhood_score.unwrap() < (1.0 + 60.0) / 2.0,
|
||||
"plateau mean is below the end dip's neighbourhood mean; score={:?}",
|
||||
mean_sel.neighbourhood_score,
|
||||
);
|
||||
|
||||
// Worst arm: each cell scores by its highest (worst) drawdown neighbour
|
||||
// (the NEG_INFINITY/max fold). It is never below the mean for a
|
||||
// lower-is-better metric, and still avoids the dip whose worst neighbour is
|
||||
// 60.0.
|
||||
let (_, worst_sel) =
|
||||
optimize_plateau(&fam, &[7], "max_drawdown", PlateauMode::Worst).unwrap();
|
||||
assert_eq!(worst_sel.mode, SelectionMode::PlateauWorst);
|
||||
assert!(
|
||||
worst_sel.neighbourhood_score.unwrap() >= mean_sel.neighbourhood_score.unwrap(),
|
||||
"worst (max drawdown) ≥ mean for a lower-is-better metric",
|
||||
);
|
||||
assert!(
|
||||
worst_sel.neighbourhood_score.unwrap() < 60.0,
|
||||
"the plateau interior's worst neighbour is below the dip's 60.0 shoulder",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user