From 9c6b9c7ea9cc13aad637dce587db3bb5a6bb2c77 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 26 Jun 2026 15:01:51 +0200 Subject: [PATCH] =?UTF-8?q?audit(0076):=20cycle=20close=20=E2=80=94=20drif?= =?UTF-8?q?t=20addressed;=20ledger=20inferential-thread=20synced;=20drop?= =?UTF-8?q?=20ephemera?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architect drift review (c192dfd..a295905): drift_found, medium/low only — no ledger or hot-path contract crossed. C23-additive, C1-deterministic, C14/C18 back-compat all verified green (full workspace suite + clippy). No regression scripts configured (test+clippy are the gate). Resolution: - [medium] three spec-promised optimize_deflated tests were missing (the centring control, the all-noise high-p companion, the C2 IS-only read) — the centring was unprotected ("removing it would pass every test"). Plus an unguarded dispersion-floor sign assumption. FIXED in 0076.tidy (ecc9541). - [low, RATIFY] the ledger's "Open architectural threads" inferential entry was stale on one fact (it described optimize as having no trials penalty). Synced: it now records that #144's trials-deflation landed this cycle (additive optimize_deflated, winner unchanged — C23; optimize/rank_by stay bare argmax), that #139's per-candidate OOS bootstrap landed in 0075, and that #145/#146 remain open. This ratifies the cycle-0076 intent; the gap description is preserved for the still-open default-ranker path. Deferred (spec 0076 Out of scope, not drift): n_eff effective-trials advisory, reload-time recompute of the R-arm statistic, deflated-score re-ranking, CLI flags for the resampling config. Ephemeral spec + plan (docs/specs/0076-*, docs/plans/0076-*) removed per the cycle-close convention. refs #144 --- docs/design/INDEX.md | 17 +- .../plans/0076-deflate-sweep-winner-trials.md | 674 ------------------ .../specs/0076-deflate-sweep-winner-trials.md | 366 ---------- 3 files changed, 13 insertions(+), 1044 deletions(-) delete mode 100644 docs/plans/0076-deflate-sweep-winner-trials.md delete mode 100644 docs/specs/0076-deflate-sweep-winner-trials.md diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index 6e96105..8d3d3db 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -1419,10 +1419,19 @@ load-bearing in the flat graph. selection/aggregation layer must deflate a winner for the family size (#144), prefer a plateau over a peak (#145), and score cross-instrument generalization (#146), with a per-candidate out-of-sample bootstrap as the adjacent - significance read (#139). Distinct from the two threads above — neither - orchestration *composability* (#109) nor a search *policy* (Bayesian/genetic), - but the statistical *validity* of the selection itself. Not a C-invariant until - built (the engine today violates it); recorded as the World's third unbuilt half. + significance read (#139, landed cycle 0075). Distinct from the two threads above — + neither orchestration *composability* (#109) nor a search *policy* (Bayesian/ + genetic), but the statistical *validity* of the selection itself. Not a + C-invariant until built; recorded as the World's third (now partly built) half. + **Status (cycle 0076):** the trials-deflation piece (#144) landed — + `optimize_deflated` (aura-registry) wraps `optimize` and records, additively on + the winning member's manifest (C18, the new `RunManifest.selection`), a deflated + score + an overfit probability for the family size, **without changing which + member wins** (C23; `optimize` / `rank_by` stay a bare argmax — the deflation is + recorded provenance, not a re-ranking). The R arm is a centred moving-block + reality-check (reusing the `r_bootstrap` kernel); the `total_pips` arm a + closed-form expected-max-of-K dispersion floor. #145 (plateau-over-peak) and #146 + (cross-instrument generalization) remain open. - **`aura-std` contents** — the crate exists (doc-only); which universal blocks land first follows the walking-skeleton's needs. - **`strategies/` split** — a later split, *inside a project*, of top-level diff --git a/docs/plans/0076-deflate-sweep-winner-trials.md b/docs/plans/0076-deflate-sweep-winner-trials.md deleted file mode 100644 index 269b849..0000000 --- a/docs/plans/0076-deflate-sweep-winner-trials.md +++ /dev/null @@ -1,674 +0,0 @@ -# Deflate the sweep winner for trials — Implementation Plan - -> **Parent spec:** `docs/specs/0076-deflate-sweep-winner-trials.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run -> this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Stamp an additive trials-deflation record (`deflated_score` + -`overfit_probability`) on each walk-forward OOS winner's manifest, without changing -which member wins. - -**Architecture:** A new `optimize_deflated` sits beside `optimize` in -`aura-registry`; it returns `optimize`'s byte-identical argmax winner plus a -`FamilySelection` record (an `aura-engine` type on `RunManifest`). The R arm runs a -centred moving-block reality-check reusing the `r_bootstrap` kernel; the `total_pips` -arm uses a closed-form expected-max-of-K dispersion floor. The two `walkforward_family` -seams call it and stamp the record on the chosen OOS report. - -**Tech Stack:** `aura-engine` (report.rs types + mc.rs kernel + stats helpers), -`aura-registry` (selector + compat mirror), `aura-cli` (wiring + display). - ---- - -## Files this plan creates or modifies - -- Modify: `crates/aura-engine/src/report.rs` — `FamilySelection`/`SelectionMode` types, `RunManifest.selection` field, `inv_norm_cdf`/`expected_max_of_normals`, thread engine-internal fixtures -- Modify: `crates/aura-engine/src/lib.rs:64-68` — export the new public items -- Modify: `crates/aura-engine/src/mc.rs` — extract `resample_block` from `r_bootstrap` -- Modify: `crates/aura-engine/src/harness.rs:143` — make `SplitMix64` `pub` -- Modify: `crates/aura-engine/src/{blueprint.rs,mc.rs,sweep.rs,walkforward.rs}` + test fixtures — thread `selection: None` -- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112` — thread `selection: None` -- Modify: `crates/aura-registry/src/compat.rs:27,65,67` — mirror + destructure + build -- Modify: `crates/aura-registry/src/lib.rs` — `resolve_metric`/`metric_value` refactor, `optimize_deflated` + helpers; thread fixture `:239` -- Modify: `crates/aura-registry/src/trace_store.rs:250` — thread `selection: None` -- Modify: `crates/aura-cli/src/main.rs:26,163,1795,1810,2178` — import, `make_manifest`, wiring, display -- Modify: `crates/aura-ingest/{examples,tests}/*` — thread `selection: None` (6 sites) -- Test: new serde / extraction / golden / statistic tests in the files above - ---- - -### Task 1: aura-engine — record types, manifest field, engine-internal threading - -**Files:** -- Modify: `crates/aura-engine/src/report.rs` (types beside `RMetrics` ~:44-86; `RunManifest` :419-434; fixtures :779,:1154,:1182,:1202) -- Modify: `crates/aura-engine/src/lib.rs:64-68` -- Modify: `crates/aura-engine/src/{blueprint.rs:929, mc.rs:224, mc.rs:247, sweep.rs:643, walkforward.rs:288}` -- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112` - -- [ ] **Step 1: Add the record types** in `report.rs`, immediately before `pub struct RMetrics` (~:44) - -```rust -/// Which selection objective produced the record (additive provenance, C23). -/// `Argmax` is the bare-best pick (cycle 0076). The enum is deliberately left -/// open: cycle 0145 adds a `Plateau*` variant on this same field, so peak-vs- -/// plateau runs stay distinguishable. -#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub enum SelectionMode { - Argmax, -} - -/// Selection-provenance for a sweep winner: how its metric was deflated for the -/// number of configurations it beat. Additive — recorded, never re-ranking (C23). -/// `overfit_probability` is the empirical data-snooping p-value (R arm only); -/// `None` on the `total_pips` dispersion-floor arm. -#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub struct FamilySelection { - pub selection_metric: String, - pub n_trials: usize, - pub raw_winner_metric: f64, - pub deflated_score: f64, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub overfit_probability: Option, - pub mode: SelectionMode, - pub n_resamples: usize, - pub block_len: usize, - pub seed: u64, -} -``` - -- [ ] **Step 2: Add the `selection` field** to `RunManifest` (after `pub broker: String,`, ~:433) - -```rust - /// Selection provenance, present only on a sweep/walk-forward winner; a - /// standalone run and every pre-0076 line read back as `None`. One-directional - /// serde widening (C14/C18), identical idiom to `RunMetrics.r`. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub selection: Option, -``` - -- [ ] **Step 3: Thread `selection: None` into every engine-internal `RunManifest { … }` literal.** Add `selection: None,` to the struct literal at each of: `blueprint.rs:929`, `mc.rs:224`, `mc.rs:247`, `report.rs:779`, `report.rs:1154`, `report.rs:1182`, `report.rs:1202`, `sweep.rs:643`, `walkforward.rs:288`, `tests/random_sweep_e2e.rs:112`. (Each is a `RunManifest { commit: …, …, broker: … }` body; append `selection: None,`.) - -- [ ] **Step 4: Export the new types** — extend the `pub use report::{…}` list at `lib.rs:64-68` with `FamilySelection, SelectionMode` (keep alphabetical/existing order). - -- [ ] **Step 5: Compile-gate the engine** - -Run: `cargo test -p aura-engine --no-run` -Expected: builds with 0 errors (all engine-internal literals threaded). - -- [ ] **Step 6: Add serde tests** in `report.rs`'s `#[cfg(test)] mod tests` (Timestamp is already in scope there) - -```rust - #[test] - fn runmanifest_without_selection_field_deserialises_to_none() { - let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#; - let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads"); - assert!(m.selection.is_none()); - } - - #[test] - fn runmanifest_without_selection_serialises_without_the_key() { - let m = RunManifest { - commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), - seed: 0, broker: "b".into(), selection: None, - }; - assert!(!serde_json::to_string(&m).unwrap().contains("selection")); - } - - #[test] - fn family_selection_round_trips_on_the_manifest() { - let sel = FamilySelection { - selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83, - deflated_score: 0.21, overfit_probability: Some(0.06), mode: SelectionMode::Argmax, - n_resamples: 1000, block_len: 5, seed: 42, - }; - let m = RunManifest { - commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), - seed: 0, broker: "b".into(), selection: Some(sel.clone()), - }; - let back: RunManifest = serde_json::from_str(&serde_json::to_string(&m).unwrap()).unwrap(); - assert_eq!(back.selection, Some(sel)); - } -``` - -- [ ] **Step 7: Run the engine tests (incl. the existing serde/round-trip goldens)** - -Run: `cargo test -p aura-engine` -Expected: PASS — the three new tests pass; the existing `to_json_equals_serde_disk_shape`, `runreport_serde_round_trips`, `to_json_renders_the_canonical_form` stay green (unstamped manifests emit no `selection` key, byte-identical). - ---- - -### Task 2: Downstream compile completion — compat mirror + remaining literals - -**Files:** -- Modify: `crates/aura-registry/src/compat.rs:13,27,65,67` -- Modify: `crates/aura-registry/src/lib.rs:239` (fixture); `crates/aura-registry/src/trace_store.rs:250` -- Modify: `crates/aura-cli/src/main.rs:163` (`make_manifest`) -- Modify: `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`, `examples/ger40_breakout_walkforward.rs:73`, `examples/shared/breakout_real.rs:390`, `tests/ger40_breakout_world.rs:65`, `tests/real_bars.rs:82`, `tests/streaming_seam.rs:82` - -- [ ] **Step 1: Extend the compat import** — `compat.rs:13`, add `FamilySelection`: - -```rust -use aura_engine::{FamilySelection, RunManifest, RunMetrics, RunReport, Scalar, Timestamp}; -``` - -- [ ] **Step 2: Mirror the field** in `RunManifestRead` (`compat.rs:27-33`), after `broker: String,`: - -```rust - #[serde(default)] - selection: Option, -``` - -- [ ] **Step 3: Fix the destructure + build** (`compat.rs:65-72`) — this is the load-bearing compile tripwire: - -```rust - let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest; - RunReport { - manifest: RunManifest { - commit, - params: params.into_iter().map(|(name, v)| (name, v.0)).collect(), - window, - seed, - broker, - selection, - }, - metrics: r.metrics, - } -``` - -- [ ] **Step 4: Thread `selection: None`** into the remaining literals: `aura-registry/src/lib.rs:239`, `aura-registry/src/trace_store.rs:250`, `aura-cli/src/main.rs:163`, and the six `aura-ingest` sites listed in **Files**. - -- [ ] **Step 5: Workspace compile-gate** - -Run: `cargo check --workspace --all-targets` -Expected: 0 errors (all 21 literals + the compat destructure threaded). - -- [ ] **Step 6: Add a compat lift test** in `compat.rs`'s test module (or `lib.rs` tests) — a legacy line *with* a selection block lifts it: - -```rust - #[test] - fn load_lifts_a_selection_block_through_the_compat_mirror() { - let line = r#"{"manifest":{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b","selection":{"selection_metric":"sqn_normalized","n_trials":4,"raw_winner_metric":1.8,"deflated_score":0.2,"overfit_probability":0.06,"mode":"Argmax","n_resamples":1000,"block_len":5,"seed":42}},"metrics":{"total_pips":0.0,"max_drawdown":0.0,"bias_sign_flips":0}}"#; - let rep: RunReport = serde_json::from_str::(line).unwrap().into(); - let sel = rep.manifest.selection.expect("selection lifted"); - assert_eq!(sel.n_trials, 4); - assert_eq!(sel.mode, aura_engine::SelectionMode::Argmax); - } -``` - -(If `RunReportRead` is not reachable from the chosen test module, place this test inside `compat.rs`'s own `#[cfg(test)] mod tests`. The existing `append_then_load_round_trips_in_order` and `load_reads_a_legacy_bare_float_params_line` must stay green.) - -- [ ] **Step 7: Run registry tests** - -Run: `cargo test -p aura-registry` -Expected: PASS — the new lift test passes; `append_then_load_round_trips_in_order`, `load_reads_a_legacy_bare_float_params_line`, `lineage_round_trips_one_family` stay green. - ---- - -### Task 3: aura-engine — extract `resample_block`, make `SplitMix64` public - -**Files:** -- Modify: `crates/aura-engine/src/mc.rs:173-199` (extract), `:14` (import unchanged) -- Modify: `crates/aura-engine/src/harness.rs:143` (visibility) -- Modify: `crates/aura-engine/src/lib.rs` (re-export `SplitMix64`, `resample_block`) - -- [ ] **Step 1: Make `SplitMix64` public** — at `harness.rs:143`, change `pub(crate) struct SplitMix64` to `pub struct SplitMix64`, and ensure its `new` and `next_u64` methods are `pub` (widen any `pub(crate)` on them). - -- [ ] **Step 2: Re-export** from `lib.rs` — add `pub use harness::SplitMix64;` (beside the existing `harness` re-exports) and `pub use mc::{… resample_block …}` once Step 3 lands. - -- [ ] **Step 3: Extract the kernel** — insert `resample_block` above `r_bootstrap` (~:169) and call it from the loop: - -```rust -/// One moving-block resample of `rs` to length `rs.len()` (non-circular; the final -/// block is truncated so the resample has exactly `n` values). `block_len` must be -/// pre-clamped to `[1, rs.len()]` by the caller. Pure given the `rng` state — the -/// shared kernel behind `r_bootstrap` and the trials-deflation reality-check (C1). -pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec { - let n = rs.len(); - let mut sample: Vec = Vec::with_capacity(n); - while sample.len() < n { - let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize; - let take = block_len.min(n - sample.len()); - sample.extend_from_slice(&rs[start..start + take]); - } - sample -} -``` - -Then replace the loop body at `:187-194` so `r_bootstrap` calls it: - -```rust - for _ in 0..n_resamples { - let sample = resample_block(rs, block_len, &mut rng); - means.push(sample.iter().sum::() / n as f64); - } -``` - -- [ ] **Step 4: Verify the kernel is byte-identical** - -Run: `cargo test -p aura-engine r_bootstrap` -Expected: PASS — `r_bootstrap_is_deterministic_given_seed`, `r_bootstrap_single_block_equals_full_series_mean`, `r_bootstrap_block_len_is_clamped_to_series_len`, `r_bootstrap_empty_series_is_all_zero` all green (proves extraction, not change). - ---- - -### Task 4: aura-engine — `inv_norm_cdf` + `expected_max_of_normals` - -**Files:** -- Modify: `crates/aura-engine/src/report.rs` (module-level fns beside `summarize`/`r_metrics_from_rs`) -- Modify: `crates/aura-engine/src/lib.rs:64-68` (export) - -- [ ] **Step 1: Write the failing test** in `report.rs` tests - -```rust - #[test] - fn inv_norm_cdf_matches_known_quantiles() { - assert!((inv_norm_cdf(0.975) - 1.959964).abs() < 1e-3); - assert!(inv_norm_cdf(0.5).abs() < 1e-9); - assert!((inv_norm_cdf(0.1) + inv_norm_cdf(0.9)).abs() < 1e-9); // symmetry - } - - #[test] - fn expected_max_of_normals_is_zero_at_one_and_monotone() { - assert_eq!(expected_max_of_normals(1), 0.0); - assert_eq!(expected_max_of_normals(0), 0.0); - let (e2, e4, e10, e100) = ( - expected_max_of_normals(2), expected_max_of_normals(4), - expected_max_of_normals(10), expected_max_of_normals(100), - ); - assert!(e2 < e4 && e4 < e10 && e10 < e100); // strictly increasing - assert!((1.0..1.1).contains(&e4)); // ~1.05 (not the √(2 ln 4)=1.66 asymptote) - assert!((2.4..2.7).contains(&e100)); // ~2.53 - } -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `cargo test -p aura-engine expected_max_of_normals` -Expected: FAIL to compile — `cannot find function 'expected_max_of_normals'`. - -- [ ] **Step 3: Write the implementation** (module-level in `report.rs`) - -```rust -/// Inverse standard-normal CDF (quantile function), Acklam's rational -/// approximation — absolute error < ~1.2e-9 over `p ∈ (0,1)`. Pure (C1). -/// Callers pass strictly-interior `p` (`k >= 2` keeps the argument off the -/// boundaries); the boundary branches return ±inf as a defined limit. -pub fn inv_norm_cdf(p: f64) -> f64 { - const A: [f64; 6] = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02, - 1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00]; - const B: [f64; 5] = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02, - 6.680131188771972e+01, -1.328068155288572e+01]; - const C: [f64; 6] = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00, - -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00]; - const D: [f64; 4] = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00, - 3.754408661907416e+00]; - const P_LOW: f64 = 0.02425; - if p <= 0.0 { return f64::NEG_INFINITY; } - if p >= 1.0 { return f64::INFINITY; } - if p < P_LOW { - let q = (-2.0 * p.ln()).sqrt(); - (((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5]) - / ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0) - } else if p <= 1.0 - P_LOW { - let q = p - 0.5; - let r = q * q; - (((((A[0]*r + A[1])*r + A[2])*r + A[3])*r + A[4])*r + A[5]) * q - / (((((B[0]*r + B[1])*r + B[2])*r + B[3])*r + B[4])*r + 1.0) - } else { - let q = (-2.0 * (1.0 - p).ln()).sqrt(); - -(((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5]) - / ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0) - } -} - -/// Expected maximum of `k` i.i.d. standard normals — the hurdle a search of `k` -/// configurations clears by chance alone. `k <= 1` -> `0.0` (a single trial has no -/// multiple-comparison inflation, and guards the `Φ⁻¹(1 - 1/k) -> Φ⁻¹(0)` -/// divergence). For `k >= 2`: -/// `(1 - γ)·Φ⁻¹(1 - 1/k) + γ·Φ⁻¹(1 - 1/(k·e))`, `γ = 0.5772156649…`. Pure (C1). -pub fn expected_max_of_normals(k: usize) -> f64 { - if k <= 1 { - return 0.0; - } - const GAMMA: f64 = 0.577_215_664_901_532_9; - let kf = k as f64; - (1.0 - GAMMA) * inv_norm_cdf(1.0 - 1.0 / kf) - + GAMMA * inv_norm_cdf(1.0 - 1.0 / (kf * std::f64::consts::E)) -} -``` - -- [ ] **Step 4: Export** — add `expected_max_of_normals, inv_norm_cdf` to the `pub use report::{…}` list at `lib.rs:64-68`. - -- [ ] **Step 5: Run to verify it passes** - -Run: `cargo test -p aura-engine expected_max_of_normals` -Expected: PASS. Then `cargo test -p aura-engine inv_norm_cdf` — PASS. - ---- - -### Task 5: aura-registry — `optimize_deflated` + the value-read refactor - -**Files:** -- Modify: `crates/aura-registry/src/lib.rs:20` (imports), `:125-165` (refactor), beside `:186` (`optimize_deflated` + helpers) - -- [ ] **Step 1: Extend imports** (`lib.rs:20`) - -```rust -use aura_engine::{ - expected_max_of_normals, r_metrics_from_rs, resample_block, FamilySelection, MetricStats, - RunReport, SelectionMode, SplitMix64, SweepFamily, SweepPoint, -}; -``` - -- [ ] **Step 2: Add fixture helpers** to `lib.rs`'s `#[cfg(test)] mod tests` (import what is missing — `use aura_engine::{RunManifest, RunMetrics, RunReport, Timestamp};`). Members are built from a trade-R slice via `r_metrics_from_rs`, so every `RMetrics` field is populated consistently: - -```rust - fn member(total_pips: f64, trade_rs: Vec) -> SweepPoint { - SweepPoint { - params: vec![], - report: RunReport { - manifest: RunManifest { - commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), - seed: 0, broker: "b".into(), selection: None, - }, - metrics: RunMetrics { - total_pips, max_drawdown: 0.0, bias_sign_flips: 0, - r: Some(aura_engine::r_metrics_from_rs(&trade_rs)), - }, - }, - } - } - - fn fixture_family_with_r() -> SweepFamily { - SweepFamily { space: vec![], points: vec![ - member(10.0, vec![1.0, -0.5, 0.8, -0.2, 0.6]), - member(30.0, vec![2.0, 1.5, -0.3, 1.0, 0.9]), // best by every metric here - member(5.0, vec![-0.5, 0.2, -0.8, 0.1, -0.3]), - ]} - } - - fn fixture_family_one_strong_edge() -> SweepFamily { - SweepFamily { space: vec![], points: vec![ - member(1.0, vec![2.0, 2.5, 1.8, 2.2, 2.1, 1.9]), // strong, consistent +R edge - member(1.0, vec![0.05, -0.1, 0.0, 0.1, -0.05, 0.02]), // ~zero-edge noise - member(1.0, vec![-0.1, 0.1, 0.0, -0.05, 0.05, 0.0]), - ]} - } -``` - -- [ ] **Step 2a: Write the failing tests** (same module) — the C23 tripwire + determinism + arm behaviour: - -```rust - #[test] - fn optimize_deflated_winner_is_byte_identical_to_optimize() { - let fam = fixture_family_with_r(); // helper: ≥3 members, varied sqn_normalized + trade_rs - for metric in ["total_pips", "sqn_normalized", "expectancy_r"] { - let plain = optimize(&fam, metric).unwrap(); - let (defl, _) = optimize_deflated(&fam, metric, 200, 3, 7).unwrap(); - assert_eq!(defl, plain, "deflation must not change the winner ({metric})"); - } - } - - #[test] - fn optimize_deflated_is_reproducible_from_its_seed() { - let fam = fixture_family_with_r(); - let a = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1; - let b = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1; - assert_eq!(a, b); - } - - #[test] - fn optimize_deflated_total_pips_arm_floors_without_a_probability() { - let fam = fixture_family_with_r(); - let sel = optimize_deflated(&fam, "total_pips", 100, 3, 1).unwrap().1; - assert!(sel.overfit_probability.is_none()); - assert_eq!(sel.selection_metric, "total_pips"); - assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw - } - - #[test] - fn optimize_deflated_overfit_probability_is_low_for_a_clear_edge() { - // A family whose winner has a strong real R-edge over near-zero-edge peers: - // the centred best-of-K null rarely reaches the winner's raw metric. - let fam = fixture_family_one_strong_edge(); - let sel = optimize_deflated(&fam, "expectancy_r", 1000, 1, 9).unwrap().1; - assert!(sel.overfit_probability.unwrap() < 0.2, "p={:?}", sel.overfit_probability); - assert!(sel.deflated_score > 0.0); - } -``` - -- [ ] **Step 2b: Run to verify it fails** - -Run: `cargo test -p aura-registry optimize_deflated` -Expected: FAIL to compile — `cannot find function 'optimize_deflated'`. - -- [ ] **Step 3: Refactor `metric_cmp`'s value-read into shared helpers** — extract the name→enum match (`:128-138`) and the value read (`:141-164`): - -```rust -/// Resolve a metric NAME to the closed `Metric` kind (the single name→kind map). -fn resolve_metric(name: &str) -> Result { - Ok(match name { - "total_pips" => Metric::TotalPips, - "max_drawdown" => Metric::MaxDrawdown, - "bias_sign_flips" | "exposure_sign_flips" => Metric::BiasSignFlips, - "sqn" => Metric::Sqn, - "sqn_normalized" => Metric::SqnNormalized, - "expectancy_r" => Metric::ExpectancyR, - "net_expectancy_r" => Metric::NetExpectancyR, - other => return Err(RegistryError::UnknownMetric(other.to_string())), - }) -} - -/// The metric's scalar value for one report. R metrics live in the optional `r` -/// block; a missing block reads `NEG_INFINITY` (the worst rank for the -/// higher-is-better R keys), preserving `metric_cmp`'s documented contract. -fn metric_value(rep: &RunReport, m: Metric) -> f64 { - fn r_get(rep: &RunReport, f: impl Fn(&aura_engine::RMetrics) -> f64) -> f64 { - rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY) - } - match m { - Metric::TotalPips => rep.metrics.total_pips, - Metric::MaxDrawdown => rep.metrics.max_drawdown, - Metric::BiasSignFlips => rep.metrics.bias_sign_flips as f64, - Metric::Sqn => r_get(rep, |r| r.sqn), - Metric::SqnNormalized => r_get(rep, |r| r.sqn_normalized), - Metric::ExpectancyR => r_get(rep, |r| r.expectancy_r), - Metric::NetExpectancyR => r_get(rep, |r| r.net_expectancy_r), - } -} -``` - -Rewrite `metric_cmp` to resolve via `resolve_metric` and compare via `metric_value`, keeping the per-metric direction: - -```rust -fn metric_cmp(metric: &str) -> Result Ordering, RegistryError> { - let m = resolve_metric(metric)?; - Ok(move |a: &RunReport, b: &RunReport| { - let (va, vb) = (metric_value(a, m), metric_value(b, m)); - match m { - // lower-is-better - Metric::MaxDrawdown | Metric::BiasSignFlips => va.total_cmp(&vb), - // higher-is-better - _ => vb.total_cmp(&va), - } - }) -} -``` - -- [ ] **Step 4: Add `optimize_deflated` + helpers** beside `optimize` (~:194) - -```rust -fn is_r_metric(m: Metric) -> bool { - matches!(m, Metric::Sqn | Metric::SqnNormalized | Metric::ExpectancyR | Metric::NetExpectancyR) -} - -fn member_trade_rs(rep: &RunReport) -> &[f64] { - rep.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[]) -} - -/// Recompute the selected R metric from a trade-R slice (reuses the engine's -/// `r_metrics_from_rs`, the same arithmetic `summarize_r` uses). -fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64 { - let rm = r_metrics_from_rs(rs); - match m { - Metric::Sqn => rm.sqn, - Metric::SqnNormalized => rm.sqn_normalized, - Metric::ExpectancyR => rm.expectancy_r, - Metric::NetExpectancyR => rm.net_expectancy_r, - _ => unreachable!("member_metric_from_rs is R-only"), - } -} - -/// The centred best-of-K null-max distribution: each member's `trade_rs` is -/// mean-subtracted (the no-edge null), then for each resample every member is -/// moving-block-resampled (in odometer order, from a per-iteration seed) and the -/// max recomputed metric across members is taken. Deterministic given `seed` (C1). -fn null_best_of_k(family: &SweepFamily, m: Metric, n_resamples: usize, block_len: usize, seed: u64) -> Vec { - let centred: Vec> = family.points.iter().map(|p| { - let rs = member_trade_rs(&p.report); - if rs.is_empty() { return Vec::new(); } - let mean = rs.iter().sum::() / rs.len() as f64; - rs.iter().map(|x| x - mean).collect() - }).collect(); - (0..n_resamples).map(|i| { - let mut rng = SplitMix64::new(seed ^ i as u64); - centred.iter().fold(f64::NEG_INFINITY, |best, rs| { - if rs.is_empty() { return best; } - let bl = block_len.clamp(1, rs.len()); - let v = member_metric_from_rs(&resample_block(rs, bl, &mut rng), m); - best.max(v) - }) - }).collect() -} - -/// Sample standard deviation of the K members' metric values (the `total_pips` -/// dispersion-floor arm). `< 2` members -> `0.0`. -fn member_sd(family: &SweepFamily, m: Metric) -> f64 { - let vals: Vec = family.points.iter().map(|p| metric_value(&p.report, m)).collect(); - let n = vals.len(); - if n < 2 { return 0.0; } - let mean = vals.iter().sum::() / n as f64; - (vals.iter().map(|v| (v - mean).powi(2)).sum::() / (n - 1) as f64).sqrt() -} - -/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned -/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23). -/// R arm: a centred moving-block reality-check (`overfit_probability` = -/// `(count(null ≥ raw) + 1)/(n + 1)`, `deflated_score` = `raw − p95(null)`). -/// `total_pips` arm: a closed-form expected-max-of-K dispersion floor, no -/// probability. Deterministic given `seed` (C1). -pub fn optimize_deflated( - family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64, -) -> Result<(SweepPoint, FamilySelection), RegistryError> { - let winner = optimize(family, metric)?; - let m = resolve_metric(metric)?; - let k = family.points.len(); - let raw = metric_value(&winner.report, m); - - let (deflated_score, overfit_probability) = if is_r_metric(m) { - let null_max = null_best_of_k(family, m, n_resamples, block_len, seed); - let p95 = MetricStats::from_values(&null_max).p95; - let over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64 - / (n_resamples + 1) as f64; - (raw - p95, Some(over)) - } else { - (raw - member_sd(family, m) * expected_max_of_normals(k), None) - }; - - Ok((winner.clone(), FamilySelection { - selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, - deflated_score, overfit_probability, mode: SelectionMode::Argmax, - n_resamples, block_len, seed, - })) -} -``` - -- [ ] **Step 5: Run the new + existing selection tests** - -Run: `cargo test -p aura-registry` -Expected: PASS — the four new `optimize_deflated*` tests pass; the existing `optimize_picks_the_max_metric_point_ties_to_earliest`, `rank_by_sqn_normalized_orders_members_descending`, `rank_r_metric_sorts_none_member_last`, `rank_by_unknown_metric_is_an_error` stay green (the `metric_cmp` refactor is behaviour-preserving). - ---- - -### Task 6: aura-cli — wire the call sites + the display line - -**Files:** -- Modify: `crates/aura-cli/src/main.rs:26-27` (import), constants block, `:1793-1813` (`walkforward_family`), `:2178-2204` (`runs_family`) -- Modify: `crates/aura-cli/tests/cli_run.rs` (assert the new display + keep behaviour goldens green) - -- [ ] **Step 1: Import `optimize_deflated`** — add it to the `use aura_registry::{… optimize, …}` list at `:26-27`. - -- [ ] **Step 2: Add the module constants** near the other `const`s at the top of `main.rs`: - -```rust -/// Trials-deflation resampling budget for walk-forward winner selection (recorded -/// on each winner's manifest, so the figure is reproducible by re-run). CLI flags -/// for these are a deferred refinement. -const DEFLATION_N_RESAMPLES: usize = 1000; -const DEFLATION_BLOCK_LEN: usize = 5; -const DEFLATION_SEED: u64 = 0xDEF1_A7ED; -``` - -- [ ] **Step 3: Wire the `total_pips` arm** (`:1793-1803`) - -```rust - walk_forward(roller, space, |w: WindowBounds| { - let is_family = sweep_over(w.is.0, w.is.1, data); - let (best, selection) = optimize_deflated( - &is_family, "total_pips", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, - ).expect("total_pips is a known metric"); - let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data); - oos_report.manifest.selection = Some(selection); - WindowRun { chosen_params: best.params, oos_equity, oos_report } - }) -``` - -- [ ] **Step 4: Wire the `sqn_normalized` arm** (`:1808-1812`) - -```rust - walk_forward(roller, space, |w: WindowBounds| { - let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid); - let (best, selection) = optimize_deflated( - &is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED, - ).expect("sqn_normalized is a known metric"); - let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); - oos_report.manifest.selection = Some(selection); - WindowRun { chosen_params: best.params, oos_equity, oos_report } - }) -``` - -- [ ] **Step 5: Add the display line** in `runs_family`, right after the per-member `println!("{}", report.to_json())` (`:2202`) - -```rust - if let Some(sel) = &report.manifest.selection { - match sel.overfit_probability { - Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score, p), - None => println!(" deflated={:.4}", sel.deflated_score), - } - } -``` - -- [ ] **Step 6: Build the CLI** - -Run: `cargo build -p aura-cli` -Expected: 0 errors. - -- [ ] **Step 7: Run the CLI suite; reconcile goldens** - -Run: `cargo test -p aura-cli` -Expected: PASS, with this reconciliation rule applied: -- **Unstamped families stay byte-identical** — `sweep_*` and `mc_*` members carry no `selection` (the selector runs only in walk-forward), so `sweep_strategy_stage1_r_ranks_a_family_by_sqn`/`_by_sqn_normalized`, the `mc_*` goldens, and any standalone-run golden MUST stay green unchanged. If one of these changed, it is a bug — stop and fix (do not edit the golden). -- **Stamped walk-forward output may legitimately gain the block** — `walkforward_strategy_stage1_r_reports_oos_r`, `walkforward_bare_sma_summary_has_no_oos_r`, and `runs_families_list_and_per_family_rank_across_invocations`: if (and only if) their asserted output now includes an OOS winner's `selection`/`deflated=` line, update the expected output to the new correct value — this is the feature's intended addition, not a regression. The OOS metric numbers (R, pips) must be unchanged (C23 — same winner). - -- [ ] **Step 8: Add an explicit display assertion** to `cli_run.rs`, named `runs_family_rank_shows_deflated_line` (the name makes the filter below resolve) — run a small synthetic `walkforward --strategy stage1-r`, persist it, and assert the `runs family rank sqn_normalized` stdout contains both `deflated=` and `P(overfit)=`. Model it on the existing `runs_families_list_and_per_family_rank_across_invocations` harness. - -Run: `cargo test -p aura-cli runs_family_rank_shows_deflated` -Expected: PASS — the filter matches exactly the new test; the deflated/P(overfit) line is asserted present for a stamped walk-forward family. - -- [ ] **Step 9: Workspace gate + lint** - -Run: `cargo test --workspace` -Expected: PASS (all goldens reconciled). - -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: 0 warnings. diff --git a/docs/specs/0076-deflate-sweep-winner-trials.md b/docs/specs/0076-deflate-sweep-winner-trials.md deleted file mode 100644 index feadff5..0000000 --- a/docs/specs/0076-deflate-sweep-winner-trials.md +++ /dev/null @@ -1,366 +0,0 @@ -# Deflate the sweep winner's metric for the number of trials — Design Spec - -**Date:** 2026-06-26 -**Status:** Draft — awaiting user spec review -**Authors:** orchestrator + Claude - -Cycle 0076 — first cycle of the milestone *Inferential validation (defend against -false discovery at sweep scale)*. Closes #144 (refs the milestone). - -## Goal - -The World's in-sample sweep picks the single best member by a bare argmax -(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains, -the higher the best member's metric climbs by chance alone, so the winner's -`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many -configurations competed for the top slot — the family-scale false-discovery the -milestone exists to defend against. - -This cycle adds a **selection-provenance record** that, *without changing which -member wins* (additive only — C23), reports how inflated the winner's metric is by -the size of the search it won: - -- a **deflated score** — the winner's metric minus the edge a search of the same - size produces under a no-edge null; -- an **overfit probability** — the chance a search of `K` members yields the - observed winner under that null; - -both stamped on the surviving out-of-sample winner's manifest (C18), reproducible -from a recorded seed (C1). - -This is framed as aura's own selection-honesty discipline; it introduces no new -ranking key and never re-orders a family. - -## Architecture - -One new family-level selector sits **above** the closed `metric_cmp` / `Metric` -direction source, beside `optimize`, in the registry that already owns selection -(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then -computes the deflation as an additive companion: - -- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r, - net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each - member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle - 0075 already populates) is mean-subtracted to impose the no-edge null, then - resampled with the **same** moving-block kernel `r_bootstrap` uses (a - `SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K` - member metric forms the null-max distribution the winner is judged against. -- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a - closed-form **dispersion-floor** deflation — the winner minus the - expected-maximum-of-`K` inflation implied by the cross-member metric dispersion. - -The record rides inside each member's existing `RunReport.manifest` as a new -optional field, using the one-directional serde widening already proven by -`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every -legacy `runs.jsonl` / `families.jsonl` line are untouched. - -The two walk-forward in-sample selection seams (`walkforward_family`, -`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and -stamp the record onto each window's chosen OOS report. The read-only -`runs family … rank` display surfaces it. - -**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type -(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type -and `aura-registry` depends on `aura-engine` — defining it in the registry would -invert the dependency. The selector `optimize_deflated` lives in `aura-registry` -(which owns `optimize` / `metric_cmp`) and constructs the engine-side record. - -## Concrete code shapes - -### User-facing program (the acceptance evidence) - -A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample -grid per window — unchanged invocation, new provenance on the result: - -```console -$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \ - --fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0 -``` - -Each window's chosen OOS run now persists a `selection` block on its manifest: - -```jsonc -// one member line in runs/families/ … families.jsonl -"manifest": { - "commit": "c192dfd…", - "params": [["signals.trend.fast.length", {"I64":100}], …], - "window": [1704067200000000000, 1706745600000000000], - "seed": 0, - "broker": "sim-optimal(pip_size=1e-4)", - "selection": { - "selection_metric": "sqn_normalized", - "n_trials": 4, - "raw_winner_metric": 1.83, - "deflated_score": 0.21, // winner − p95(null best-of-K); > 0 ⇒ survives - "overfit_probability": 0.06, // (count(nullmax ≥ winner) + 1)/(n+1) - "mode": "Argmax", - "n_resamples": 1000, - "block_len": 5, - "seed": 42 - } -} -``` - -The legacy `total_pips` SmaCross arm records the same block with -`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the -dispersion-floor arm computes no empirical probability); `selection_metric` plus the -absent probability disambiguate the arm. A pre-0076 line carries no `selection` -field and reads back as `None`, byte-unchanged. - -The provenance is surfaced read-only beside the raw metric: - -```console -$ aura runs family stage1-r-3 rank sqn_normalized -# … each member, best-first, now with: sqn_normalized=1.83 deflated=0.21 P(overfit)=0.06 -``` - -### New record type (`aura-engine/src/report.rs`, beside `RMetrics`) - -```rust -/// Selection-provenance for a sweep winner: how its metric was deflated for the -/// number of configurations it beat. Additive (C23) — recorded, never re-ranking. -#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub struct FamilySelection { - pub selection_metric: String, // the metric the family was optimised against - pub n_trials: usize, // K — the family size the winner beat - pub raw_winner_metric: f64, // the argmax winner's face-value metric (audit) - pub deflated_score: f64, // trials-adjusted score (per-arm; see the selector) - /// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor - /// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub overfit_probability: Option, - pub mode: SelectionMode, - pub n_resamples: usize, // resample provenance — reproduces the statistic - pub block_len: usize, - pub seed: u64, -} - -/// Which selection objective produced the record. `Argmax` is the bare-best -/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*` -/// variant on this same field, so peak-vs-plateau runs stay distinguishable. -#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] -pub enum SelectionMode { - Argmax, -} -``` - -### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`) - -```rust -// before -pub struct RunManifest { - pub commit: String, - pub params: Vec<(String, Scalar)>, - pub window: (Timestamp, Timestamp), - pub seed: u64, - pub broker: String, -} -// after — one-directional widening, identical idiom to RunMetrics.r -pub struct RunManifest { - pub commit: String, - pub params: Vec<(String, Scalar)>, - pub window: (Timestamp, Timestamp), - pub seed: u64, - pub broker: String, - /// Selection provenance, present only on a sweep/walk-forward winner; a - /// standalone run and every pre-0076 line read back as `None`. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub selection: Option, -} -``` - -### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`) - -```rust -// RunManifestRead (compat.rs:27) gains the same optional field … -struct RunManifestRead { - commit: String, - params: Vec<(String, ScalarRead)>, - window: (Timestamp, Timestamp), - seed: u64, - broker: String, - #[serde(default)] - selection: Option, // legacy lines → None -} -// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it: -let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest; -RunManifest { commit, params: /* … */, window, seed, broker, selection } -``` - -(The hard 5-field destructure fails to compile the moment the 6th field exists — -this edit is the load-bearing, most-forgettable change; pinned by the existing -compat round-trip tests.) - -### The selector (`aura-registry/src/lib.rs`, beside `optimize`) - -```rust -/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned -/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23); -/// `FamilySelection` is the new companion record. Deterministic given `seed`. -pub fn optimize_deflated( - family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64, -) -> Result<(SweepPoint, FamilySelection), RegistryError> { - let winner = optimize(family, metric)?; // unchanged argmax - let m = resolve_metric(metric)?; // shared with metric_cmp - let k = family.points.len(); - let raw = metric_value(&winner.report, m); // extracted from metric_cmp - - let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) { - // R arm: centred moving-block reality-check over each member's trade_rs. - let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed); - let stats = MetricStats::from_values(&null_max); // reuse mc.rs - let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64 - / (n_resamples + 1) as f64; // (z+1)/(N+1) - (raw - stats.p95, Some(p_over)) - } else { - // total_pips arm: closed-form expected-max-of-K dispersion floor. - let sd = member_sd(family, m); - (raw - sd * expected_max_of_normals(k), None) - }; - - Ok((winner.clone(), FamilySelection { - selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, - deflated_score, overfit_probability, mode: SelectionMode::Argmax, - n_resamples, block_len, seed, - })) -} -``` - -Supporting shapes (planner fills the bodies): - -```rust -// aura-registry: refactor metric_cmp's value-read into a shared helper so the -// comparator and the deflation read one metric source (direction stays in metric_cmp). -fn resolve_metric(name: &str) -> Result; // the existing name→enum match -fn metric_value(rep: &RunReport, m: Metric) -> f64; // r_get / total_pips read, lifted -fn r_series_of(m: Metric) -> Option &[f64]>; // Some for R metrics → trade_rs; None for total_pips/dd/flips - -// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i); -// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k. -fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec; -fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64; // r_metrics_from_rs(rs) → project field - -// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared, -// byte-identical (r_bootstrap keeps its golden test). -pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec; - -// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient. -pub fn inv_norm_cdf(p: f64) -> f64; // rational approximation, golden-tested -pub fn expected_max_of_normals(k: usize) -> f64; // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772… -``` - -### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`) - -```rust -// :1810 R arm — was: let best = optimize(&is_family, "sqn_normalized")…; -let (best, selection) = - optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED) - .expect("sqn_normalized is a known metric"); -let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); -oos_report.manifest.selection = Some(selection); // stamp onto the OOS winner -WindowRun { chosen_params: best.params, oos_equity, oos_report } -// :1795 total_pips arm — identical shape with "total_pips" + run_oos. -``` - -`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module -constants this cycle (recorded in the manifest → reproducible); CLI flags for them -are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)` -beside the raw metric when `manifest.selection.is_some()`. - -## Components - -| Component | Crate / file | Change | -|---|---|---| -| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` | -| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened | -| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) | -| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested | -| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build | -| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor | -| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest | -| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present | - -## Data flow - -1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged). -2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the - winner's raw metric via `metric_value`. -3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives - `resample_block` over every member's **mean-centred** `trade_rs`; the metric is - recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K` - `M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`, - `overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`. -4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`, - `overfit_probability = None`. -5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`, - which flows through `append_family` unchanged and persists in `families.jsonl`. -6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family` - displays it. - -## Error handling - -- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs` - degrades exactly as `r_bootstrap` already does (all-zero kernel output); the - deflation then equals the raw metric (no spurious penalty). `block_len` is clamped - `[1, n]` by the shared kernel. -- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a - zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s - `NEG_INFINITY` treatment of a missing `r` block). -- A degenerate `K ≤ 1` family (a single member — no search) carries **no** - multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the - `Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and - `overfit_probability` reduces to the floor `1/(n+1)`. -- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name - (same as `optimize`), before any resampling. -- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds. - -## Testing strategy - -- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` → - bit-identical `overfit_probability` and `deflated_score`, across thread counts - (the best-of-`K` reduction collects in member/odometer order, never completion - order). A fixed-`trade_rs` fixture pins exact values. -- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for - every metric — the deflation never changes which point wins. -- **C2 IS-only:** the selector reads only the in-sample family's members; a test - asserts no OOS report is touched (the call site passes `is_family`). -- **Statistic correctness:** on a fabricated family where one member has a real edge - and the rest are zero-centred noise, `overfit_probability` is low and - `deflated_score > 0`; on an all-noise family (no member has edge), - `overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring - is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred - construction does not. -- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0, - symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing, - small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote). -- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads - as `None`; a stamped manifest round-trips byte-identically; the existing - `runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens - stay green (add, don't break). -- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays - bit-identical (proves the kernel was extracted, not changed). - -## Acceptance criteria - -- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest; - `aura runs family … rank` surfaces the deflated score and overfit probability - beside the raw metric — the worked program above runs and produces the shown block. -- The argmax winner is unchanged for every metric (C23 tripwire green). -- `overfit_probability` and `deflated_score` are reproducible from the recorded - `(seed, n_resamples, block_len)` (C1). -- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing - goldens stay green (C18/C14). -- The deflation is frictionless Stage-1 R; no Stage-2 cost enters. - -### Out of scope (deferred) - -- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries - no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a - full cartesian grid — deferred to a follow-on, addable by the same serde widening. -- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it - runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape - change). -- Re-ranking a family by the deflated score (a different design decision — would - change which member wins; belongs in the ledger, not this cycle). -- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is - reserved, not filled).