Aura/docs/specs/0076-deflate-sweep-winner-trials.md

Deflate the sweep winner's metric for the number of trials — Design Spec

Date: 2026-06-26 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Cycle 0076 — first cycle of the milestone Inferential validation (defend against false discovery at sweep scale). Closes #144 (refs the milestone).

Goal

The World's in-sample sweep picks the single best member by a bare argmax (optimize, aura-registry/src/lib.rs:186). The more members a family contains, the higher the best member's metric climbs by chance alone, so the winner's sqn_normalized / expectancy_r is reported at face value regardless of how many configurations competed for the top slot — the family-scale false-discovery the milestone exists to defend against.

This cycle adds a selection-provenance record that, without changing which member wins (additive only — C23), reports how inflated the winner's metric is by the size of the search it won:

• a deflated score — the winner's metric minus the edge a search of the same size produces under a no-edge null;
• an overfit probability — the chance a search of K members yields the observed winner under that null;

both stamped on the surviving out-of-sample winner's manifest (C18), reproducible from a recorded seed (C1).

This is framed as aura's own selection-honesty discipline; it introduces no new ranking key and never re-orders a family.

Architecture

One new family-level selector sits above the closed metric_cmp / Metric direction source, beside optimize, in the registry that already owns selection (aura-registry). It calls optimize for the byte-identical argmax winner, then computes the deflation as an additive companion:

• R arm (selection_metric ∈ {sqn, sqn_normalized, expectancy_r, net_expectancy_r}): an empirical centred moving-block reality-check. Each member's per-trade R series (metrics.r.trade_rs, the in-memory conduit cycle 0075 already populates) is mean-subtracted to impose the no-edge null, then resampled with the same moving-block kernel r_bootstrap uses (a SplitMix64 block draw — aura-engine/src/mc.rs); the per-resample best-of-K member metric forms the null-max distribution the winner is judged against.
• total_pips arm (the legacy SmaCross walk-forward, no trade_rs): a closed-form dispersion-floor deflation — the winner minus the expected-maximum-of-K inflation implied by the cross-member metric dispersion.

The record rides inside each member's existing RunReport.manifest as a new optional field, using the one-directional serde widening already proven by RunMetrics.r, so the family store (FamilyRunRecord / append_family) and every legacy runs.jsonl / families.jsonl line are untouched.

The two walk-forward in-sample selection seams (walkforward_family, aura-cli/src/main.rs:1795 total_pips arm, :1810 R arm) call the new selector and stamp the record onto each window's chosen OOS report. The read-only runs family … rank display surfaces it.

Crate placement (C9): the record type FamilySelection is an aura-engine type (beside RunManifest / RMetrics), because RunManifest is an aura-engine type and aura-registry depends on aura-engine — defining it in the registry would invert the dependency. The selector optimize_deflated lives in aura-registry (which owns optimize / metric_cmp) and constructs the engine-side record.

Concrete code shapes

User-facing program (the acceptance evidence)

A Stage-1 R walk-forward over a real symbol, sweeping a K = 2×2 = 4 in-sample grid per window — unchanged invocation, new provenance on the result:

$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \
      --fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0

Each window's chosen OOS run now persists a selection block on its manifest:

// one member line in runs/families/<id> … families.jsonl
"manifest": {
  "commit": "c192dfd…",
  "params": [["signals.trend.fast.length", {"I64":100}], …],
  "window": [1704067200000000000, 1706745600000000000],
  "seed": 0,
  "broker": "sim-optimal(pip_size=1e-4)",
  "selection": {
    "selection_metric": "sqn_normalized",
    "n_trials": 4,
    "raw_winner_metric": 1.83,
    "deflated_score": 0.21,          // winner − p95(null best-of-K); > 0 ⇒ survives
    "overfit_probability": 0.06,     // (count(nullmax ≥ winner) + 1)/(n+1)
    "mode": "Argmax",
    "n_resamples": 1000,
    "block_len": 5,
    "seed": 42
  }
}

The legacy total_pips SmaCross arm records the same block with "selection_metric":"total_pips" and overfit_probability omitted (the dispersion-floor arm computes no empirical probability); selection_metric plus the absent probability disambiguate the arm. A pre-0076 line carries no selection field and reads back as None, byte-unchanged.

The provenance is surfaced read-only beside the raw metric:

$ aura runs family stage1-r-3 rank sqn_normalized
# … each member, best-first, now with:  sqn_normalized=1.83  deflated=0.21  P(overfit)=0.06

New record type (aura-engine/src/report.rs, beside RMetrics)

/// Selection-provenance for a sweep winner: how its metric was deflated for the
/// number of configurations it beat. Additive (C23) — recorded, never re-ranking.
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct FamilySelection {
    pub selection_metric: String, // the metric the family was optimised against
    pub n_trials: usize,          // K — the family size the winner beat
    pub raw_winner_metric: f64,   // the argmax winner's face-value metric (audit)
    pub deflated_score: f64,      // trials-adjusted score (per-arm; see the selector)
    /// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor
    /// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub overfit_probability: Option<f64>,
    pub mode: SelectionMode,
    pub n_resamples: usize,       // resample provenance — reproduces the statistic
    pub block_len: usize,
    pub seed: u64,
}

/// Which selection objective produced the record. `Argmax` is the bare-best
/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*`
/// variant on this same field, so peak-vs-plateau runs stay distinguishable.
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub enum SelectionMode {
    Argmax,
}

RunManifest — 5 → 6 fields (aura-engine/src/report.rs:420)

// before
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
}
// after — one-directional widening, identical idiom to RunMetrics.r
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
    /// Selection provenance, present only on a sweep/walk-forward winner; a
    /// standalone run and every pre-0076 line read back as `None`.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub selection: Option<FamilySelection>,
}

compat.rs legacy read-path — mirror + destructure (aura-registry/src/compat.rs)

// RunManifestRead (compat.rs:27) gains the same optional field …
struct RunManifestRead {
    commit: String,
    params: Vec<(String, ScalarRead)>,
    window: (Timestamp, Timestamp),
    seed: u64,
    broker: String,
    #[serde(default)]
    selection: Option<FamilySelection>,   // legacy lines → None
}
// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it:
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
RunManifest { commit, params: /* … */, window, seed, broker, selection }

(The hard 5-field destructure fails to compile the moment the 6th field exists — this edit is the load-bearing, most-forgettable change; pinned by the existing compat round-trip tests.)

The selector (aura-registry/src/lib.rs, beside optimize)

/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23);
/// `FamilySelection` is the new companion record. Deterministic given `seed`.
pub fn optimize_deflated(
    family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
    let winner = optimize(family, metric)?;            // unchanged argmax
    let m = resolve_metric(metric)?;                   // shared with metric_cmp
    let k = family.points.len();
    let raw = metric_value(&winner.report, m);         // extracted from metric_cmp

    let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) {
        // R arm: centred moving-block reality-check over each member's trade_rs.
        let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed);
        let stats = MetricStats::from_values(&null_max);          // reuse mc.rs
        let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
            / (n_resamples + 1) as f64;                            // (z+1)/(N+1)
        (raw - stats.p95, Some(p_over))
    } else {
        // total_pips arm: closed-form expected-max-of-K dispersion floor.
        let sd = member_sd(family, m);
        (raw - sd * expected_max_of_normals(k), None)
    };

    Ok((winner.clone(), FamilySelection {
        selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
        deflated_score, overfit_probability, mode: SelectionMode::Argmax,
        n_resamples, block_len, seed,
    }))
}

Supporting shapes (planner fills the bodies):

// aura-registry: refactor metric_cmp's value-read into a shared helper so the
// comparator and the deflation read one metric source (direction stays in metric_cmp).
fn resolve_metric(name: &str) -> Result<Metric, RegistryError>;   // the existing name→enum match
fn metric_value(rep: &RunReport, m: Metric) -> f64;               // r_get / total_pips read, lifted
fn r_series_of(m: Metric) -> Option<fn(&RunReport) -> &[f64]>;    // Some for R metrics → trade_rs; None for total_pips/dd/flips

// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i);
// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k.
fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec<f64>;
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64;           // r_metrics_from_rs(rs) → project field

// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared,
// byte-identical (r_bootstrap keeps its golden test).
pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64>;

// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient.
pub fn inv_norm_cdf(p: f64) -> f64;                               // rational approximation, golden-tested
pub fn expected_max_of_normals(k: usize) -> f64;                  // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772…

Call-site stamping (aura-cli/src/main.rs, walkforward_family)

// :1810  R arm — was: let best = optimize(&is_family, "sqn_normalized")…;
let (best, selection) =
    optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED)
        .expect("sqn_normalized is a known metric");
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
oos_report.manifest.selection = Some(selection);          // stamp onto the OOS winner
WindowRun { chosen_params: best.params, oos_equity, oos_report }
// :1795  total_pips arm — identical shape with "total_pips" + run_oos.

DEFLATION_N_RESAMPLES / DEFLATION_BLOCK_LEN / DEFLATION_SEED are module constants this cycle (recorded in the manifest → reproducible); CLI flags for them are a deferred refinement. runs_family (:2192) prints deflated + P(overfit) beside the raw metric when manifest.selection.is_some().

Components

Component	Crate / file	Change
FamilySelection, SelectionMode	aura-engine/report.rs	new serde types beside RMetrics
RunManifest.selection	aura-engine/report.rs:420	new Option field, serde-widened
resample_block	aura-engine/mc.rs	extracted from r_bootstrap (kernel shared, byte-identical)
inv_norm_cdf, expected_max_of_normals	aura-engine (new stats item)	pure, deterministic, golden-tested
RunManifestRead.selection + destructure	aura-registry/compat.rs:27,65	mirror + 6-field destructure + build
optimize_deflated, metric_value, resolve_metric, null_best_of_k	aura-registry/lib.rs	new selector + metric_cmp value-read refactor
walkforward_family stamping	aura-cli/main.rs:1795,1810	call selector, stamp OOS manifest
runs_family display	aura-cli/main.rs:2192	print deflated + overfit when present

Data flow

1. walkforward_family builds the in-sample SweepFamily per window (unchanged).
2. optimize_deflated calls optimize → the same argmax SweepPoint; reads the winner's raw metric via metric_value.
3. R arm: for each resample i, SplitMix64::new(seed ^ i) drives resample_block over every member's mean-centred trade_rs; the metric is recomputed (member_metric_from_rs) per member and reduced to the best-of-K M_i. The winner is judged against {M_i}: deflated_score = raw − p95(M), overfit_probability = (count(M_i ≥ raw) + 1)/(n+1).
4. total_pips arm: deflated_score = raw − member_sd · expected_max_of_normals(K), overfit_probability = None.
5. The FamilySelection is stamped onto the chosen window's oos_report.manifest, which flows through append_family unchanged and persists in families.jsonl.
6. On read, compat.rs lifts the field (legacy lines → None); runs_family displays it.

Error handling

• An empty resample budget (n_resamples == 0) or a member with empty trade_rs degrades exactly as r_bootstrap already does (all-zero kernel output); the deflation then equals the raw metric (no spurious penalty). block_len is clamped [1, n] by the shared kernel.
• A family whose winner has r: None on the R arm (no trade_rs) contributes a zero null-max contribution for that member — never a panic (mirrors metric_cmp's NEG_INFINITY treatment of a missing r block).
• A degenerate K ≤ 1 family (a single member — no search) carries no multiple-comparison inflation: expected_max_of_normals(K ≤ 1) = 0.0 (guarding the Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞ divergence), so deflated_score == raw and overfit_probability reduces to the floor 1/(n+1).
• optimize_deflated returns RegistryError::UnknownMetric for an unknown name (same as optimize), before any resampling.
• No new exit path in the CLI: the selector cannot fail where optimize succeeds.

Testing strategy

• C1 determinism (primary): same family + (seed, n_resamples, block_len) → bit-identical overfit_probability and deflated_score, across thread counts (the best-of-K reduction collects in member/odometer order, never completion order). A fixed-trade_rs fixture pins exact values.
• C23 additive (tripwire): optimize_deflated(f, m, …).0 == optimize(f, m) for every metric — the deflation never changes which point wins.
• C2 IS-only: the selector reads only the in-sample family's members; a test asserts no OOS report is touched (the call site passes is_family).
• Statistic correctness: on a fabricated family where one member has a real edge and the rest are zero-centred noise, overfit_probability is low and deflated_score > 0; on an all-noise family (no member has edge), overfit_probability is high (≈ K/(K+…)) and deflated_score ≤ 0. The centring is pinned: an uncentred control would give ≈0.5 — the test asserts the centred construction does not.
• inv_norm_cdf golden: known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0, symmetry) to a tight tolerance; expected_max_of_normals(K) monotone increasing, small-K sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote).
• C14/C18 back-compat: a pre-0076 families.jsonl line (no selection) loads as None; a stamped manifest round-trips byte-identically; the existing runs.jsonl golden and the SMA/total_pips walk-forward + synthetic-MC goldens stay green (add, don't break).
• resample_block extraction: r_bootstrap's existing determinism golden stays bit-identical (proves the kernel was extracted, not changed).

Acceptance criteria

• A Stage-1 R walk-forward stamps a FamilySelection on each OOS winner's manifest; aura runs family … rank surfaces the deflated score and overfit probability beside the raw metric — the worked program above runs and produces the shown block.
• The argmax winner is unchanged for every metric (C23 tripwire green).
• overfit_probability and deflated_score are reproducible from the recorded (seed, n_resamples, block_len) (C1).
• Legacy runs.jsonl / families.jsonl lines load unchanged and all pre-existing goldens stay green (C18/C14).
• The deflation is frictionless Stage-1 R; no Stage-2 cost enters.

Out of scope (deferred)

• The effective-independent-trials advisory (n_eff_trials): SweepFamily carries no axis cardinalities (sweep.rs:291), so its per-axis inference mis-shapes off a full cartesian grid — deferred to a follow-on, addable by the same serde widening.
• A reload-time recompute of the R-arm statistic (trade_rs is serde(skip), so it runs only live in walkforward_family pre-persist — a recompute needs a wire-shape change).
• Re-ranking a family by the deflated score (a different design decision — would change which member wins; belongs in the ledger, not this cycle).
• CLI flags for the resampling config; #145's Plateau* variants (the enum slot is reserved, not filled).