diff --git a/docs/plans/0076-deflate-sweep-winner-trials.md b/docs/plans/0076-deflate-sweep-winner-trials.md new file mode 100644 index 0000000..269b849 --- /dev/null +++ b/docs/plans/0076-deflate-sweep-winner-trials.md @@ -0,0 +1,674 @@ +# Deflate the sweep winner for trials — Implementation Plan + +> **Parent spec:** `docs/specs/0076-deflate-sweep-winner-trials.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Stamp an additive trials-deflation record (`deflated_score` + +`overfit_probability`) on each walk-forward OOS winner's manifest, without changing +which member wins. + +**Architecture:** A new `optimize_deflated` sits beside `optimize` in +`aura-registry`; it returns `optimize`'s byte-identical argmax winner plus a +`FamilySelection` record (an `aura-engine` type on `RunManifest`). The R arm runs a +centred moving-block reality-check reusing the `r_bootstrap` kernel; the `total_pips` +arm uses a closed-form expected-max-of-K dispersion floor. The two `walkforward_family` +seams call it and stamp the record on the chosen OOS report. + +**Tech Stack:** `aura-engine` (report.rs types + mc.rs kernel + stats helpers), +`aura-registry` (selector + compat mirror), `aura-cli` (wiring + display). + +--- + +## Files this plan creates or modifies + +- Modify: `crates/aura-engine/src/report.rs` — `FamilySelection`/`SelectionMode` types, `RunManifest.selection` field, `inv_norm_cdf`/`expected_max_of_normals`, thread engine-internal fixtures +- Modify: `crates/aura-engine/src/lib.rs:64-68` — export the new public items +- Modify: `crates/aura-engine/src/mc.rs` — extract `resample_block` from `r_bootstrap` +- Modify: `crates/aura-engine/src/harness.rs:143` — make `SplitMix64` `pub` +- Modify: `crates/aura-engine/src/{blueprint.rs,mc.rs,sweep.rs,walkforward.rs}` + test fixtures — thread `selection: None` +- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112` — thread `selection: None` +- Modify: `crates/aura-registry/src/compat.rs:27,65,67` — mirror + destructure + build +- Modify: `crates/aura-registry/src/lib.rs` — `resolve_metric`/`metric_value` refactor, `optimize_deflated` + helpers; thread fixture `:239` +- Modify: `crates/aura-registry/src/trace_store.rs:250` — thread `selection: None` +- Modify: `crates/aura-cli/src/main.rs:26,163,1795,1810,2178` — import, `make_manifest`, wiring, display +- Modify: `crates/aura-ingest/{examples,tests}/*` — thread `selection: None` (6 sites) +- Test: new serde / extraction / golden / statistic tests in the files above + +--- + +### Task 1: aura-engine — record types, manifest field, engine-internal threading + +**Files:** +- Modify: `crates/aura-engine/src/report.rs` (types beside `RMetrics` ~:44-86; `RunManifest` :419-434; fixtures :779,:1154,:1182,:1202) +- Modify: `crates/aura-engine/src/lib.rs:64-68` +- Modify: `crates/aura-engine/src/{blueprint.rs:929, mc.rs:224, mc.rs:247, sweep.rs:643, walkforward.rs:288}` +- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112` + +- [ ] **Step 1: Add the record types** in `report.rs`, immediately before `pub struct RMetrics` (~:44) + +```rust +/// Which selection objective produced the record (additive provenance, C23). +/// `Argmax` is the bare-best pick (cycle 0076). The enum is deliberately left +/// open: cycle 0145 adds a `Plateau*` variant on this same field, so peak-vs- +/// plateau runs stay distinguishable. +#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub enum SelectionMode { + Argmax, +} + +/// Selection-provenance for a sweep winner: how its metric was deflated for the +/// number of configurations it beat. Additive — recorded, never re-ranking (C23). +/// `overfit_probability` is the empirical data-snooping p-value (R arm only); +/// `None` on the `total_pips` dispersion-floor arm. +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct FamilySelection { + pub selection_metric: String, + pub n_trials: usize, + pub raw_winner_metric: f64, + pub deflated_score: f64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub overfit_probability: Option, + pub mode: SelectionMode, + pub n_resamples: usize, + pub block_len: usize, + pub seed: u64, +} +``` + +- [ ] **Step 2: Add the `selection` field** to `RunManifest` (after `pub broker: String,`, ~:433) + +```rust + /// Selection provenance, present only on a sweep/walk-forward winner; a + /// standalone run and every pre-0076 line read back as `None`. One-directional + /// serde widening (C14/C18), identical idiom to `RunMetrics.r`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selection: Option, +``` + +- [ ] **Step 3: Thread `selection: None` into every engine-internal `RunManifest { … }` literal.** Add `selection: None,` to the struct literal at each of: `blueprint.rs:929`, `mc.rs:224`, `mc.rs:247`, `report.rs:779`, `report.rs:1154`, `report.rs:1182`, `report.rs:1202`, `sweep.rs:643`, `walkforward.rs:288`, `tests/random_sweep_e2e.rs:112`. (Each is a `RunManifest { commit: …, …, broker: … }` body; append `selection: None,`.) + +- [ ] **Step 4: Export the new types** — extend the `pub use report::{…}` list at `lib.rs:64-68` with `FamilySelection, SelectionMode` (keep alphabetical/existing order). + +- [ ] **Step 5: Compile-gate the engine** + +Run: `cargo test -p aura-engine --no-run` +Expected: builds with 0 errors (all engine-internal literals threaded). + +- [ ] **Step 6: Add serde tests** in `report.rs`'s `#[cfg(test)] mod tests` (Timestamp is already in scope there) + +```rust + #[test] + fn runmanifest_without_selection_field_deserialises_to_none() { + let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#; + let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads"); + assert!(m.selection.is_none()); + } + + #[test] + fn runmanifest_without_selection_serialises_without_the_key() { + let m = RunManifest { + commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), + seed: 0, broker: "b".into(), selection: None, + }; + assert!(!serde_json::to_string(&m).unwrap().contains("selection")); + } + + #[test] + fn family_selection_round_trips_on_the_manifest() { + let sel = FamilySelection { + selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83, + deflated_score: 0.21, overfit_probability: Some(0.06), mode: SelectionMode::Argmax, + n_resamples: 1000, block_len: 5, seed: 42, + }; + let m = RunManifest { + commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), + seed: 0, broker: "b".into(), selection: Some(sel.clone()), + }; + let back: RunManifest = serde_json::from_str(&serde_json::to_string(&m).unwrap()).unwrap(); + assert_eq!(back.selection, Some(sel)); + } +``` + +- [ ] **Step 7: Run the engine tests (incl. the existing serde/round-trip goldens)** + +Run: `cargo test -p aura-engine` +Expected: PASS — the three new tests pass; the existing `to_json_equals_serde_disk_shape`, `runreport_serde_round_trips`, `to_json_renders_the_canonical_form` stay green (unstamped manifests emit no `selection` key, byte-identical). + +--- + +### Task 2: Downstream compile completion — compat mirror + remaining literals + +**Files:** +- Modify: `crates/aura-registry/src/compat.rs:13,27,65,67` +- Modify: `crates/aura-registry/src/lib.rs:239` (fixture); `crates/aura-registry/src/trace_store.rs:250` +- Modify: `crates/aura-cli/src/main.rs:163` (`make_manifest`) +- Modify: `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`, `examples/ger40_breakout_walkforward.rs:73`, `examples/shared/breakout_real.rs:390`, `tests/ger40_breakout_world.rs:65`, `tests/real_bars.rs:82`, `tests/streaming_seam.rs:82` + +- [ ] **Step 1: Extend the compat import** — `compat.rs:13`, add `FamilySelection`: + +```rust +use aura_engine::{FamilySelection, RunManifest, RunMetrics, RunReport, Scalar, Timestamp}; +``` + +- [ ] **Step 2: Mirror the field** in `RunManifestRead` (`compat.rs:27-33`), after `broker: String,`: + +```rust + #[serde(default)] + selection: Option, +``` + +- [ ] **Step 3: Fix the destructure + build** (`compat.rs:65-72`) — this is the load-bearing compile tripwire: + +```rust + let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest; + RunReport { + manifest: RunManifest { + commit, + params: params.into_iter().map(|(name, v)| (name, v.0)).collect(), + window, + seed, + broker, + selection, + }, + metrics: r.metrics, + } +``` + +- [ ] **Step 4: Thread `selection: None`** into the remaining literals: `aura-registry/src/lib.rs:239`, `aura-registry/src/trace_store.rs:250`, `aura-cli/src/main.rs:163`, and the six `aura-ingest` sites listed in **Files**. + +- [ ] **Step 5: Workspace compile-gate** + +Run: `cargo check --workspace --all-targets` +Expected: 0 errors (all 21 literals + the compat destructure threaded). + +- [ ] **Step 6: Add a compat lift test** in `compat.rs`'s test module (or `lib.rs` tests) — a legacy line *with* a selection block lifts it: + +```rust + #[test] + fn load_lifts_a_selection_block_through_the_compat_mirror() { + let line = r#"{"manifest":{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b","selection":{"selection_metric":"sqn_normalized","n_trials":4,"raw_winner_metric":1.8,"deflated_score":0.2,"overfit_probability":0.06,"mode":"Argmax","n_resamples":1000,"block_len":5,"seed":42}},"metrics":{"total_pips":0.0,"max_drawdown":0.0,"bias_sign_flips":0}}"#; + let rep: RunReport = serde_json::from_str::(line).unwrap().into(); + let sel = rep.manifest.selection.expect("selection lifted"); + assert_eq!(sel.n_trials, 4); + assert_eq!(sel.mode, aura_engine::SelectionMode::Argmax); + } +``` + +(If `RunReportRead` is not reachable from the chosen test module, place this test inside `compat.rs`'s own `#[cfg(test)] mod tests`. The existing `append_then_load_round_trips_in_order` and `load_reads_a_legacy_bare_float_params_line` must stay green.) + +- [ ] **Step 7: Run registry tests** + +Run: `cargo test -p aura-registry` +Expected: PASS — the new lift test passes; `append_then_load_round_trips_in_order`, `load_reads_a_legacy_bare_float_params_line`, `lineage_round_trips_one_family` stay green. + +--- + +### Task 3: aura-engine — extract `resample_block`, make `SplitMix64` public + +**Files:** +- Modify: `crates/aura-engine/src/mc.rs:173-199` (extract), `:14` (import unchanged) +- Modify: `crates/aura-engine/src/harness.rs:143` (visibility) +- Modify: `crates/aura-engine/src/lib.rs` (re-export `SplitMix64`, `resample_block`) + +- [ ] **Step 1: Make `SplitMix64` public** — at `harness.rs:143`, change `pub(crate) struct SplitMix64` to `pub struct SplitMix64`, and ensure its `new` and `next_u64` methods are `pub` (widen any `pub(crate)` on them). + +- [ ] **Step 2: Re-export** from `lib.rs` — add `pub use harness::SplitMix64;` (beside the existing `harness` re-exports) and `pub use mc::{… resample_block …}` once Step 3 lands. + +- [ ] **Step 3: Extract the kernel** — insert `resample_block` above `r_bootstrap` (~:169) and call it from the loop: + +```rust +/// One moving-block resample of `rs` to length `rs.len()` (non-circular; the final +/// block is truncated so the resample has exactly `n` values). `block_len` must be +/// pre-clamped to `[1, rs.len()]` by the caller. Pure given the `rng` state — the +/// shared kernel behind `r_bootstrap` and the trials-deflation reality-check (C1). +pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec { + let n = rs.len(); + let mut sample: Vec = Vec::with_capacity(n); + while sample.len() < n { + let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize; + let take = block_len.min(n - sample.len()); + sample.extend_from_slice(&rs[start..start + take]); + } + sample +} +``` + +Then replace the loop body at `:187-194` so `r_bootstrap` calls it: + +```rust + for _ in 0..n_resamples { + let sample = resample_block(rs, block_len, &mut rng); + means.push(sample.iter().sum::() / n as f64); + } +``` + +- [ ] **Step 4: Verify the kernel is byte-identical** + +Run: `cargo test -p aura-engine r_bootstrap` +Expected: PASS — `r_bootstrap_is_deterministic_given_seed`, `r_bootstrap_single_block_equals_full_series_mean`, `r_bootstrap_block_len_is_clamped_to_series_len`, `r_bootstrap_empty_series_is_all_zero` all green (proves extraction, not change). + +--- + +### Task 4: aura-engine — `inv_norm_cdf` + `expected_max_of_normals` + +**Files:** +- Modify: `crates/aura-engine/src/report.rs` (module-level fns beside `summarize`/`r_metrics_from_rs`) +- Modify: `crates/aura-engine/src/lib.rs:64-68` (export) + +- [ ] **Step 1: Write the failing test** in `report.rs` tests + +```rust + #[test] + fn inv_norm_cdf_matches_known_quantiles() { + assert!((inv_norm_cdf(0.975) - 1.959964).abs() < 1e-3); + assert!(inv_norm_cdf(0.5).abs() < 1e-9); + assert!((inv_norm_cdf(0.1) + inv_norm_cdf(0.9)).abs() < 1e-9); // symmetry + } + + #[test] + fn expected_max_of_normals_is_zero_at_one_and_monotone() { + assert_eq!(expected_max_of_normals(1), 0.0); + assert_eq!(expected_max_of_normals(0), 0.0); + let (e2, e4, e10, e100) = ( + expected_max_of_normals(2), expected_max_of_normals(4), + expected_max_of_normals(10), expected_max_of_normals(100), + ); + assert!(e2 < e4 && e4 < e10 && e10 < e100); // strictly increasing + assert!((1.0..1.1).contains(&e4)); // ~1.05 (not the √(2 ln 4)=1.66 asymptote) + assert!((2.4..2.7).contains(&e100)); // ~2.53 + } +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cargo test -p aura-engine expected_max_of_normals` +Expected: FAIL to compile — `cannot find function 'expected_max_of_normals'`. + +- [ ] **Step 3: Write the implementation** (module-level in `report.rs`) + +```rust +/// Inverse standard-normal CDF (quantile function), Acklam's rational +/// approximation — absolute error < ~1.2e-9 over `p ∈ (0,1)`. Pure (C1). +/// Callers pass strictly-interior `p` (`k >= 2` keeps the argument off the +/// boundaries); the boundary branches return ±inf as a defined limit. +pub fn inv_norm_cdf(p: f64) -> f64 { + const A: [f64; 6] = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02, + 1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00]; + const B: [f64; 5] = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02, + 6.680131188771972e+01, -1.328068155288572e+01]; + const C: [f64; 6] = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00, + -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00]; + const D: [f64; 4] = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00, + 3.754408661907416e+00]; + const P_LOW: f64 = 0.02425; + if p <= 0.0 { return f64::NEG_INFINITY; } + if p >= 1.0 { return f64::INFINITY; } + if p < P_LOW { + let q = (-2.0 * p.ln()).sqrt(); + (((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5]) + / ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0) + } else if p <= 1.0 - P_LOW { + let q = p - 0.5; + let r = q * q; + (((((A[0]*r + A[1])*r + A[2])*r + A[3])*r + A[4])*r + A[5]) * q + / (((((B[0]*r + B[1])*r + B[2])*r + B[3])*r + B[4])*r + 1.0) + } else { + let q = (-2.0 * (1.0 - p).ln()).sqrt(); + -(((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5]) + / ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0) + } +} + +/// Expected maximum of `k` i.i.d. standard normals — the hurdle a search of `k` +/// configurations clears by chance alone. `k <= 1` -> `0.0` (a single trial has no +/// multiple-comparison inflation, and guards the `Φ⁻¹(1 - 1/k) -> Φ⁻¹(0)` +/// divergence). For `k >= 2`: +/// `(1 - γ)·Φ⁻¹(1 - 1/k) + γ·Φ⁻¹(1 - 1/(k·e))`, `γ = 0.5772156649…`. Pure (C1). +pub fn expected_max_of_normals(k: usize) -> f64 { + if k <= 1 { + return 0.0; + } + const GAMMA: f64 = 0.577_215_664_901_532_9; + let kf = k as f64; + (1.0 - GAMMA) * inv_norm_cdf(1.0 - 1.0 / kf) + + GAMMA * inv_norm_cdf(1.0 - 1.0 / (kf * std::f64::consts::E)) +} +``` + +- [ ] **Step 4: Export** — add `expected_max_of_normals, inv_norm_cdf` to the `pub use report::{…}` list at `lib.rs:64-68`. + +- [ ] **Step 5: Run to verify it passes** + +Run: `cargo test -p aura-engine expected_max_of_normals` +Expected: PASS. Then `cargo test -p aura-engine inv_norm_cdf` — PASS. + +--- + +### Task 5: aura-registry — `optimize_deflated` + the value-read refactor + +**Files:** +- Modify: `crates/aura-registry/src/lib.rs:20` (imports), `:125-165` (refactor), beside `:186` (`optimize_deflated` + helpers) + +- [ ] **Step 1: Extend imports** (`lib.rs:20`) + +```rust +use aura_engine::{ + expected_max_of_normals, r_metrics_from_rs, resample_block, FamilySelection, MetricStats, + RunReport, SelectionMode, SplitMix64, SweepFamily, SweepPoint, +}; +``` + +- [ ] **Step 2: Add fixture helpers** to `lib.rs`'s `#[cfg(test)] mod tests` (import what is missing — `use aura_engine::{RunManifest, RunMetrics, RunReport, Timestamp};`). Members are built from a trade-R slice via `r_metrics_from_rs`, so every `RMetrics` field is populated consistently: + +```rust + fn member(total_pips: f64, trade_rs: Vec) -> SweepPoint { + SweepPoint { + params: vec![], + report: RunReport { + manifest: RunManifest { + commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), + seed: 0, broker: "b".into(), selection: None, + }, + metrics: RunMetrics { + total_pips, max_drawdown: 0.0, bias_sign_flips: 0, + r: Some(aura_engine::r_metrics_from_rs(&trade_rs)), + }, + }, + } + } + + fn fixture_family_with_r() -> SweepFamily { + SweepFamily { space: vec![], points: vec![ + member(10.0, vec![1.0, -0.5, 0.8, -0.2, 0.6]), + member(30.0, vec![2.0, 1.5, -0.3, 1.0, 0.9]), // best by every metric here + member(5.0, vec![-0.5, 0.2, -0.8, 0.1, -0.3]), + ]} + } + + fn fixture_family_one_strong_edge() -> SweepFamily { + SweepFamily { space: vec![], points: vec![ + member(1.0, vec![2.0, 2.5, 1.8, 2.2, 2.1, 1.9]), // strong, consistent +R edge + member(1.0, vec![0.05, -0.1, 0.0, 0.1, -0.05, 0.02]), // ~zero-edge noise + member(1.0, vec![-0.1, 0.1, 0.0, -0.05, 0.05, 0.0]), + ]} + } +``` + +- [ ] **Step 2a: Write the failing tests** (same module) — the C23 tripwire + determinism + arm behaviour: + +```rust + #[test] + fn optimize_deflated_winner_is_byte_identical_to_optimize() { + let fam = fixture_family_with_r(); // helper: ≥3 members, varied sqn_normalized + trade_rs + for metric in ["total_pips", "sqn_normalized", "expectancy_r"] { + let plain = optimize(&fam, metric).unwrap(); + let (defl, _) = optimize_deflated(&fam, metric, 200, 3, 7).unwrap(); + assert_eq!(defl, plain, "deflation must not change the winner ({metric})"); + } + } + + #[test] + fn optimize_deflated_is_reproducible_from_its_seed() { + let fam = fixture_family_with_r(); + let a = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1; + let b = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1; + assert_eq!(a, b); + } + + #[test] + fn optimize_deflated_total_pips_arm_floors_without_a_probability() { + let fam = fixture_family_with_r(); + let sel = optimize_deflated(&fam, "total_pips", 100, 3, 1).unwrap().1; + assert!(sel.overfit_probability.is_none()); + assert_eq!(sel.selection_metric, "total_pips"); + assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw + } + + #[test] + fn optimize_deflated_overfit_probability_is_low_for_a_clear_edge() { + // A family whose winner has a strong real R-edge over near-zero-edge peers: + // the centred best-of-K null rarely reaches the winner's raw metric. + let fam = fixture_family_one_strong_edge(); + let sel = optimize_deflated(&fam, "expectancy_r", 1000, 1, 9).unwrap().1; + assert!(sel.overfit_probability.unwrap() < 0.2, "p={:?}", sel.overfit_probability); + assert!(sel.deflated_score > 0.0); + } +``` + +- [ ] **Step 2b: Run to verify it fails** + +Run: `cargo test -p aura-registry optimize_deflated` +Expected: FAIL to compile — `cannot find function 'optimize_deflated'`. + +- [ ] **Step 3: Refactor `metric_cmp`'s value-read into shared helpers** — extract the name→enum match (`:128-138`) and the value read (`:141-164`): + +```rust +/// Resolve a metric NAME to the closed `Metric` kind (the single name→kind map). +fn resolve_metric(name: &str) -> Result { + Ok(match name { + "total_pips" => Metric::TotalPips, + "max_drawdown" => Metric::MaxDrawdown, + "bias_sign_flips" | "exposure_sign_flips" => Metric::BiasSignFlips, + "sqn" => Metric::Sqn, + "sqn_normalized" => Metric::SqnNormalized, + "expectancy_r" => Metric::ExpectancyR, + "net_expectancy_r" => Metric::NetExpectancyR, + other => return Err(RegistryError::UnknownMetric(other.to_string())), + }) +} + +/// The metric's scalar value for one report. R metrics live in the optional `r` +/// block; a missing block reads `NEG_INFINITY` (the worst rank for the +/// higher-is-better R keys), preserving `metric_cmp`'s documented contract. +fn metric_value(rep: &RunReport, m: Metric) -> f64 { + fn r_get(rep: &RunReport, f: impl Fn(&aura_engine::RMetrics) -> f64) -> f64 { + rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY) + } + match m { + Metric::TotalPips => rep.metrics.total_pips, + Metric::MaxDrawdown => rep.metrics.max_drawdown, + Metric::BiasSignFlips => rep.metrics.bias_sign_flips as f64, + Metric::Sqn => r_get(rep, |r| r.sqn), + Metric::SqnNormalized => r_get(rep, |r| r.sqn_normalized), + Metric::ExpectancyR => r_get(rep, |r| r.expectancy_r), + Metric::NetExpectancyR => r_get(rep, |r| r.net_expectancy_r), + } +} +``` + +Rewrite `metric_cmp` to resolve via `resolve_metric` and compare via `metric_value`, keeping the per-metric direction: + +```rust +fn metric_cmp(metric: &str) -> Result Ordering, RegistryError> { + let m = resolve_metric(metric)?; + Ok(move |a: &RunReport, b: &RunReport| { + let (va, vb) = (metric_value(a, m), metric_value(b, m)); + match m { + // lower-is-better + Metric::MaxDrawdown | Metric::BiasSignFlips => va.total_cmp(&vb), + // higher-is-better + _ => vb.total_cmp(&va), + } + }) +} +``` + +- [ ] **Step 4: Add `optimize_deflated` + helpers** beside `optimize` (~:194) + +```rust +fn is_r_metric(m: Metric) -> bool { + matches!(m, Metric::Sqn | Metric::SqnNormalized | Metric::ExpectancyR | Metric::NetExpectancyR) +} + +fn member_trade_rs(rep: &RunReport) -> &[f64] { + rep.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[]) +} + +/// Recompute the selected R metric from a trade-R slice (reuses the engine's +/// `r_metrics_from_rs`, the same arithmetic `summarize_r` uses). +fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64 { + let rm = r_metrics_from_rs(rs); + match m { + Metric::Sqn => rm.sqn, + Metric::SqnNormalized => rm.sqn_normalized, + Metric::ExpectancyR => rm.expectancy_r, + Metric::NetExpectancyR => rm.net_expectancy_r, + _ => unreachable!("member_metric_from_rs is R-only"), + } +} + +/// The centred best-of-K null-max distribution: each member's `trade_rs` is +/// mean-subtracted (the no-edge null), then for each resample every member is +/// moving-block-resampled (in odometer order, from a per-iteration seed) and the +/// max recomputed metric across members is taken. Deterministic given `seed` (C1). +fn null_best_of_k(family: &SweepFamily, m: Metric, n_resamples: usize, block_len: usize, seed: u64) -> Vec { + let centred: Vec> = family.points.iter().map(|p| { + let rs = member_trade_rs(&p.report); + if rs.is_empty() { return Vec::new(); } + let mean = rs.iter().sum::() / rs.len() as f64; + rs.iter().map(|x| x - mean).collect() + }).collect(); + (0..n_resamples).map(|i| { + let mut rng = SplitMix64::new(seed ^ i as u64); + centred.iter().fold(f64::NEG_INFINITY, |best, rs| { + if rs.is_empty() { return best; } + let bl = block_len.clamp(1, rs.len()); + let v = member_metric_from_rs(&resample_block(rs, bl, &mut rng), m); + best.max(v) + }) + }).collect() +} + +/// Sample standard deviation of the K members' metric values (the `total_pips` +/// dispersion-floor arm). `< 2` members -> `0.0`. +fn member_sd(family: &SweepFamily, m: Metric) -> f64 { + let vals: Vec = family.points.iter().map(|p| metric_value(&p.report, m)).collect(); + let n = vals.len(); + if n < 2 { return 0.0; } + let mean = vals.iter().sum::() / n as f64; + (vals.iter().map(|v| (v - mean).powi(2)).sum::() / (n - 1) as f64).sqrt() +} + +/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned +/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23). +/// R arm: a centred moving-block reality-check (`overfit_probability` = +/// `(count(null ≥ raw) + 1)/(n + 1)`, `deflated_score` = `raw − p95(null)`). +/// `total_pips` arm: a closed-form expected-max-of-K dispersion floor, no +/// probability. Deterministic given `seed` (C1). +pub fn optimize_deflated( + family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64, +) -> Result<(SweepPoint, FamilySelection), RegistryError> { + let winner = optimize(family, metric)?; + let m = resolve_metric(metric)?; + let k = family.points.len(); + let raw = metric_value(&winner.report, m); + + let (deflated_score, overfit_probability) = if is_r_metric(m) { + let null_max = null_best_of_k(family, m, n_resamples, block_len, seed); + let p95 = MetricStats::from_values(&null_max).p95; + let over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64 + / (n_resamples + 1) as f64; + (raw - p95, Some(over)) + } else { + (raw - member_sd(family, m) * expected_max_of_normals(k), None) + }; + + Ok((winner.clone(), FamilySelection { + selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, + deflated_score, overfit_probability, mode: SelectionMode::Argmax, + n_resamples, block_len, seed, + })) +} +``` + +- [ ] **Step 5: Run the new + existing selection tests** + +Run: `cargo test -p aura-registry` +Expected: PASS — the four new `optimize_deflated*` tests pass; the existing `optimize_picks_the_max_metric_point_ties_to_earliest`, `rank_by_sqn_normalized_orders_members_descending`, `rank_r_metric_sorts_none_member_last`, `rank_by_unknown_metric_is_an_error` stay green (the `metric_cmp` refactor is behaviour-preserving). + +--- + +### Task 6: aura-cli — wire the call sites + the display line + +**Files:** +- Modify: `crates/aura-cli/src/main.rs:26-27` (import), constants block, `:1793-1813` (`walkforward_family`), `:2178-2204` (`runs_family`) +- Modify: `crates/aura-cli/tests/cli_run.rs` (assert the new display + keep behaviour goldens green) + +- [ ] **Step 1: Import `optimize_deflated`** — add it to the `use aura_registry::{… optimize, …}` list at `:26-27`. + +- [ ] **Step 2: Add the module constants** near the other `const`s at the top of `main.rs`: + +```rust +/// Trials-deflation resampling budget for walk-forward winner selection (recorded +/// on each winner's manifest, so the figure is reproducible by re-run). CLI flags +/// for these are a deferred refinement. +const DEFLATION_N_RESAMPLES: usize = 1000; +const DEFLATION_BLOCK_LEN: usize = 5; +const DEFLATION_SEED: u64 = 0xDEF1_A7ED; +``` + +- [ ] **Step 3: Wire the `total_pips` arm** (`:1793-1803`) + +```rust + walk_forward(roller, space, |w: WindowBounds| { + let is_family = sweep_over(w.is.0, w.is.1, data); + let (best, selection) = optimize_deflated( + &is_family, "total_pips", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, + ).expect("total_pips is a known metric"); + let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); + oos_report.manifest.selection = Some(selection); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) +``` + +- [ ] **Step 4: Wire the `sqn_normalized` arm** (`:1808-1812`) + +```rust + walk_forward(roller, space, |w: WindowBounds| { + let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); + let (best, selection) = optimize_deflated( + &is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, + ).expect("sqn_normalized is a known metric"); + let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); + oos_report.manifest.selection = Some(selection); + WindowRun { chosen_params: best.params, oos_equity, oos_report } + }) +``` + +- [ ] **Step 5: Add the display line** in `runs_family`, right after the per-member `println!("{}", report.to_json())` (`:2202`) + +```rust + if let Some(sel) = &report.manifest.selection { + match sel.overfit_probability { + Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score, p), + None => println!(" deflated={:.4}", sel.deflated_score), + } + } +``` + +- [ ] **Step 6: Build the CLI** + +Run: `cargo build -p aura-cli` +Expected: 0 errors. + +- [ ] **Step 7: Run the CLI suite; reconcile goldens** + +Run: `cargo test -p aura-cli` +Expected: PASS, with this reconciliation rule applied: +- **Unstamped families stay byte-identical** — `sweep_*` and `mc_*` members carry no `selection` (the selector runs only in walk-forward), so `sweep_strategy_stage1_r_ranks_a_family_by_sqn`/`_by_sqn_normalized`, the `mc_*` goldens, and any standalone-run golden MUST stay green unchanged. If one of these changed, it is a bug — stop and fix (do not edit the golden). +- **Stamped walk-forward output may legitimately gain the block** — `walkforward_strategy_stage1_r_reports_oos_r`, `walkforward_bare_sma_summary_has_no_oos_r`, and `runs_families_list_and_per_family_rank_across_invocations`: if (and only if) their asserted output now includes an OOS winner's `selection`/`deflated=` line, update the expected output to the new correct value — this is the feature's intended addition, not a regression. The OOS metric numbers (R, pips) must be unchanged (C23 — same winner). + +- [ ] **Step 8: Add an explicit display assertion** to `cli_run.rs`, named `runs_family_rank_shows_deflated_line` (the name makes the filter below resolve) — run a small synthetic `walkforward --strategy stage1-r`, persist it, and assert the `runs family rank sqn_normalized` stdout contains both `deflated=` and `P(overfit)=`. Model it on the existing `runs_families_list_and_per_family_rank_across_invocations` harness. + +Run: `cargo test -p aura-cli runs_family_rank_shows_deflated` +Expected: PASS — the filter matches exactly the new test; the deflated/P(overfit) line is asserted present for a stamped walk-forward family. + +- [ ] **Step 9: Workspace gate + lint** + +Run: `cargo test --workspace` +Expected: PASS (all goldens reconciled). + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: 0 warnings.