feat(0077): prefer a robust parameter plateau over the in-sample peak
Add an opt-in plateau selection objective for walk-forward, recorded on the same RunManifest.selection carrier #144 introduced. Default argmax is byte-identical (C23); plateau is reached only via a new --select flag. What landed: - FamilySelection / SelectionMode reshape (report.rs): the selection RULE (mode) and its ANNOTATION are orthogonal — deflation fields become Option (present iff Argmax), plus PlateauMean/PlateauWorst variants and neighbourhood_score/n_neighbours (present iff Plateau*). Legacy lines still load: the deflation scalars deserialize from bare values via serde default (C14/C18). compat.rs embeds FamilySelection by value — reshape flows through untouched. - GridSpace::axis_lens() + SweepBinder::sweep_with_lattice (engine): the grid radixes in param_space()/odometer order. sweep() delegates to sweep_with_lattice with the lattice dropped, so every existing .sweep() caller is byte-unchanged. - optimize_plateau + PlateauMode + closed_neighbourhood (aura-registry, C9): each member scores as the mean/worst of its closed mixed-radix grid neighbourhood's metric_value; the winner is the smoothed argmax by the metric's direction (earliest-odometer tie, as optimize). Pure, no RNG (C1); in-sample only (C2). A higher_is_better helper now sources the per-metric direction once, shared by metric_cmp and optimize_plateau. - CLI --select <argmax|plateau:mean|plateau:worst> (default argmax; unknown token exits 2). walkforward_family dispatches via a select_winner helper; sweep_over/stage1_r_sweep_over carry the lattice as Option<Vec<usize>>. runs-family display gains a plateau(<mode>)=<score> over <n> cells line. RandomSpace-refuse: walkforward has no --random producer, so the plateau-without-lattice guard is structural — select_winner returns the refuse on a None lattice (the caller prints the message and exits 2), unit-tested at the helper, not via a --random E2E (decided + recorded on #145 comment 1974, with the lattice-seam fork). Tests: plateau-picks-plateau-not-spike, mean-vs-worst, lower-is-better direction-flip, closed-neighbourhood golden, single-member degeneracy, C1 determinism; CLI select-parse, select_winner refuse; E2E argmax-byte-identical (C23), plateau:mean and plateau:worst provenance. Full workspace suite green; clippy --all-targets -D warnings clean. closes #145
This commit is contained in:
@@ -397,8 +397,20 @@ impl SweepBinder {
|
||||
}
|
||||
|
||||
/// Resolve the named axes against `param_space()` into a positional grid and
|
||||
/// run the disjoint sweep.
|
||||
/// run the disjoint sweep. `sweep` is [`SweepBinder::sweep_with_lattice`] with
|
||||
/// the lattice dropped, so every existing caller is byte-unchanged.
|
||||
pub fn sweep<F>(self, run_one: F) -> Result<SweepFamily, BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
self.sweep_with_lattice(run_one).map(|(family, _lattice)| family)
|
||||
}
|
||||
|
||||
/// As [`SweepBinder::sweep`], plus the grid's per-axis radixes (`axis_lens`,
|
||||
/// in `param_space()` / odometer order). The lattice is what a plateau
|
||||
/// neighbourhood walk needs (cycle 0077); only the engine's post-`resolve_axes`
|
||||
/// grid holds it in the correct order.
|
||||
pub fn sweep_with_lattice<F>(self, run_one: F) -> Result<(SweepFamily, Vec<usize>), BindError>
|
||||
where
|
||||
F: Fn(&[Cell]) -> RunReport + Sync,
|
||||
{
|
||||
@@ -407,7 +419,8 @@ impl SweepBinder {
|
||||
let ordered = resolve_axes(&space, &self.axes)?;
|
||||
let grid = GridSpace::new(&space, ordered)
|
||||
.expect("named layer pre-validates arity/kind/non-empty");
|
||||
Ok(sweep(&grid, run_one))
|
||||
let lattice = grid.axis_lens();
|
||||
Ok((sweep(&grid, run_one), lattice))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -938,6 +951,19 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sweep_with_lattice_surfaces_grid_radixes_in_param_space_order() {
|
||||
let bp = composite_sma_cross_harness().0;
|
||||
let (fam, lattice) = bp
|
||||
.axis("sma_cross.fast.length", vec![Scalar::i64(2), Scalar::i64(3)]) // 2 values
|
||||
.axis("sma_cross.slow.length", vec![Scalar::i64(4), Scalar::i64(5)]) // 2 values
|
||||
.axis("bias.scale", vec![Scalar::f64(0.5)]) // 1 value
|
||||
.sweep_with_lattice(run_point)
|
||||
.expect("named binding resolves and runs");
|
||||
assert_eq!(lattice, vec![2, 2, 1], "radixes in param_space slot order");
|
||||
assert_eq!(lattice.iter().product::<usize>(), fam.points.len()); // 4
|
||||
}
|
||||
|
||||
/// Property (the reason `RandomBinder` exists): building a random sweep **by
|
||||
/// name** is order-independent and binds the same knob to the same range
|
||||
/// regardless of `.range(...)` call order — exactly the safety the grid's
|
||||
|
||||
@@ -39,30 +39,46 @@ pub struct RunMetrics {
|
||||
}
|
||||
|
||||
/// Which selection objective produced the record (additive provenance, C23).
|
||||
/// `Argmax` is the bare-best pick (cycle 0076). The enum is deliberately left
|
||||
/// open: cycle 0145 adds a `Plateau*` variant on this same field, so peak-vs-
|
||||
/// plateau runs stay distinguishable.
|
||||
/// `Argmax` is the bare-best pick (cycle 0076), deflated for the number of trials.
|
||||
/// `PlateauMean` / `PlateauWorst` (cycle 0077) argmax the neighbourhood-smoothed
|
||||
/// surface instead, so peak-vs-plateau runs stay distinguishable on this field.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
PlateauMean,
|
||||
PlateauWorst,
|
||||
}
|
||||
|
||||
/// Selection-provenance for a sweep winner: how its metric was deflated for the
|
||||
/// number of configurations it beat. Additive — recorded, never re-ranking (C23).
|
||||
/// `overfit_probability` is the empirical data-snooping p-value (R arm only);
|
||||
/// `None` on the `total_pips` dispersion-floor arm.
|
||||
/// Selection-provenance for a sweep winner. The selection RULE (`mode`) and its
|
||||
/// ANNOTATION are orthogonal: `Argmax` carries the trials-deflation annotation
|
||||
/// (`deflated_score` / `overfit_probability` / `n_resamples` / `block_len` /
|
||||
/// `seed`); `Plateau*` carries the smoothing annotation (`neighbourhood_score` /
|
||||
/// `n_neighbours`). Each annotation is `Option`, present iff its rule produced the
|
||||
/// record; all are additive — recorded, never re-ranking (C23). A legacy line
|
||||
/// (pre-0077) carries the deflation scalars as bare values that deserialize to
|
||||
/// `Some` (serde default), so it still loads (C14/C18).
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub deflated_score: f64,
|
||||
pub mode: SelectionMode,
|
||||
// deflation annotation (present iff mode == Argmax with a deflation run)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub deflated_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
pub mode: SelectionMode,
|
||||
pub n_resamples: usize,
|
||||
pub block_len: usize,
|
||||
pub seed: u64,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_resamples: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub block_len: Option<usize>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub seed: Option<u64>,
|
||||
// plateau annotation (present iff mode is Plateau*)
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub neighbourhood_score: Option<f64>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub n_neighbours: Option<usize>,
|
||||
}
|
||||
|
||||
/// R-based signal-quality metrics (Stage-1), reduced from a `PositionManagement` dense
|
||||
@@ -763,8 +779,10 @@ mod tests {
|
||||
fn family_selection_round_trips_on_the_manifest() {
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83,
|
||||
deflated_score: 0.21, overfit_probability: Some(0.06), mode: SelectionMode::Argmax,
|
||||
n_resamples: 1000, block_len: 5, seed: 42,
|
||||
mode: SelectionMode::Argmax,
|
||||
deflated_score: Some(0.21), overfit_probability: Some(0.06),
|
||||
n_resamples: Some(1000), block_len: Some(5), seed: Some(42),
|
||||
neighbourhood_score: None, n_neighbours: None,
|
||||
};
|
||||
let m = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
@@ -774,6 +792,25 @@ mod tests {
|
||||
assert_eq!(back.selection, Some(sel));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn family_selection_plateau_shape_round_trips() {
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 16, raw_winner_metric: 1.42,
|
||||
mode: SelectionMode::PlateauMean,
|
||||
deflated_score: None, overfit_probability: None,
|
||||
n_resamples: None, block_len: None, seed: None,
|
||||
neighbourhood_score: Some(1.27), n_neighbours: Some(5),
|
||||
};
|
||||
let json = serde_json::to_string(&sel).unwrap();
|
||||
// plateau annotation present; deflation fields omitted (skip_serializing_if)
|
||||
assert!(json.contains("\"neighbourhood_score\":1.27"), "json: {json}");
|
||||
assert!(json.contains("\"n_neighbours\":5"), "json: {json}");
|
||||
assert!(!json.contains("deflated_score"), "deflation fields omitted under plateau: {json}");
|
||||
assert!(!json.contains("n_resamples"), "json: {json}");
|
||||
let back: FamilySelection = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back, sel);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn position_action_round_trips_through_i64() {
|
||||
for a in [PositionAction::Buy, PositionAction::Sell, PositionAction::Close] {
|
||||
|
||||
@@ -64,6 +64,13 @@ impl GridSpace {
|
||||
false
|
||||
}
|
||||
|
||||
/// Per-axis cardinalities in `param_space()` order (the odometer radixes,
|
||||
/// last-axis-fastest). `∏ axis_lens() == len()`. The lattice shape a plateau
|
||||
/// neighbourhood walks (cycle 0077).
|
||||
pub fn axis_lens(&self) -> Vec<usize> {
|
||||
self.axes.iter().map(Vec::len).collect()
|
||||
}
|
||||
|
||||
/// The cartesian product, in odometer order: the **last** axis varies
|
||||
/// fastest. Deterministic — the same grid yields the same point sequence.
|
||||
pub fn points(&self) -> Vec<Vec<Cell>> {
|
||||
@@ -597,6 +604,20 @@ mod tests {
|
||||
assert!(!grid.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn grid_axis_lens_are_the_per_axis_radixes() {
|
||||
let space = vec![
|
||||
ParamSpec { name: "a".into(), kind: ScalarKind::I64 },
|
||||
ParamSpec { name: "b".into(), kind: ScalarKind::I64 },
|
||||
];
|
||||
let grid = GridSpace::new(&space, vec![
|
||||
vec![Scalar::i64(10), Scalar::i64(20)], // axis 0: 2 values
|
||||
vec![Scalar::i64(1), Scalar::i64(2), Scalar::i64(3)], // axis 1: 3 values
|
||||
]).expect("2x3 grid");
|
||||
assert_eq!(grid.axis_lens(), vec![2, 3]);
|
||||
assert_eq!(grid.axis_lens().iter().product::<usize>(), grid.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn arity_mismatch_is_an_error() {
|
||||
let space = i64_space(2);
|
||||
|
||||
Reference in New Issue
Block a user