Aura/docs/specs/0076-deflate-sweep-winner-trials.md

# Deflate the sweep winner's metric for the number of trials — Design Spec

**Date:** 2026-06-26
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

Cycle 0076 — first cycle of the milestone *Inferential validation (defend against
false discovery at sweep scale)*. Closes #144 (refs the milestone).

## Goal

The World's in-sample sweep picks the single best member by a bare argmax
(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains,
the higher the best member's metric climbs by chance alone, so the winner's
`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many
configurations competed for the top slot — the family-scale false-discovery the
milestone exists to defend against.

This cycle adds a **selection-provenance record** that, *without changing which
member wins* (additive only — C23), reports how inflated the winner's metric is by
the size of the search it won:

- a **deflated score** — the winner's metric minus the edge a search of the same
  size produces under a no-edge null;
- an **overfit probability** — the chance a search of `K` members yields the
  observed winner under that null;

both stamped on the surviving out-of-sample winner's manifest (C18), reproducible
from a recorded seed (C1).

This is framed as aura's own selection-honesty discipline; it introduces no new
ranking key and never re-orders a family.

## Architecture

One new family-level selector sits **above** the closed `metric_cmp` / `Metric`
direction source, beside `optimize`, in the registry that already owns selection
(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then
computes the deflation as an additive companion:

- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r,
  net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each
  member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle
  0075 already populates) is mean-subtracted to impose the no-edge null, then
  resampled with the **same** moving-block kernel `r_bootstrap` uses (a
  `SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K`
  member metric forms the null-max distribution the winner is judged against.
- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a
  closed-form **dispersion-floor** deflation — the winner minus the
  expected-maximum-of-`K` inflation implied by the cross-member metric dispersion.

The record rides inside each member's existing `RunReport.manifest` as a new
optional field, using the one-directional serde widening already proven by
`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every
legacy `runs.jsonl` / `families.jsonl` line are untouched.

The two walk-forward in-sample selection seams (`walkforward_family`,
`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and
stamp the record onto each window's chosen OOS report. The read-only
`runs family … rank` display surfaces it.

**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type
(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type
and `aura-registry` depends on `aura-engine` — defining it in the registry would
invert the dependency. The selector `optimize_deflated` lives in `aura-registry`
(which owns `optimize` / `metric_cmp`) and constructs the engine-side record.

## Concrete code shapes

### User-facing program (the acceptance evidence)

A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample
grid per window — unchanged invocation, new provenance on the result:

```console
$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \
      --fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0
```

Each window's chosen OOS run now persists a `selection` block on its manifest:

```jsonc
// one member line in runs/families/<id> … families.jsonl
"manifest": {
  "commit": "c192dfd…",
  "params": [["signals.trend.fast.length", {"I64":100}], …],
  "window": [1704067200000000000, 1706745600000000000],
  "seed": 0,
  "broker": "sim-optimal(pip_size=1e-4)",
  "selection": {
    "selection_metric": "sqn_normalized",
    "n_trials": 4,
    "raw_winner_metric": 1.83,
    "deflated_score": 0.21,          // winner − p95(null best-of-K); > 0 ⇒ survives
    "overfit_probability": 0.06,     // (count(nullmax ≥ winner) + 1)/(n+1)
    "mode": "Argmax",
    "n_resamples": 1000,
    "block_len": 5,
    "seed": 42
  }
}
```

The legacy `total_pips` SmaCross arm records the same block with
`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the
dispersion-floor arm computes no empirical probability); `selection_metric` plus the
absent probability disambiguate the arm. A pre-0076 line carries no `selection`
field and reads back as `None`, byte-unchanged.

The provenance is surfaced read-only beside the raw metric:

```console
$ aura runs family stage1-r-3 rank sqn_normalized
# … each member, best-first, now with:  sqn_normalized=1.83  deflated=0.21  P(overfit)=0.06
```

### New record type (`aura-engine/src/report.rs`, beside `RMetrics`)

```rust
/// Selection-provenance for a sweep winner: how its metric was deflated for the
/// number of configurations it beat. Additive (C23) — recorded, never re-ranking.
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub struct FamilySelection {
    pub selection_metric: String, // the metric the family was optimised against
    pub n_trials: usize,          // K — the family size the winner beat
    pub raw_winner_metric: f64,   // the argmax winner's face-value metric (audit)
    pub deflated_score: f64,      // trials-adjusted score (per-arm; see the selector)
    /// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor
    /// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub overfit_probability: Option<f64>,
    pub mode: SelectionMode,
    pub n_resamples: usize,       // resample provenance — reproduces the statistic
    pub block_len: usize,
    pub seed: u64,
}

/// Which selection objective produced the record. `Argmax` is the bare-best
/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*`
/// variant on this same field, so peak-vs-plateau runs stay distinguishable.
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
pub enum SelectionMode {
    Argmax,
}
```

### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`)

```rust
// before
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
}
// after — one-directional widening, identical idiom to RunMetrics.r
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
    /// Selection provenance, present only on a sweep/walk-forward winner; a
    /// standalone run and every pre-0076 line read back as `None`.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub selection: Option<FamilySelection>,
}
```

### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`)

```rust
// RunManifestRead (compat.rs:27) gains the same optional field …
struct RunManifestRead {
    commit: String,
    params: Vec<(String, ScalarRead)>,
    window: (Timestamp, Timestamp),
    seed: u64,
    broker: String,
    #[serde(default)]
    selection: Option<FamilySelection>,   // legacy lines → None
}
// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it:
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
RunManifest { commit, params: /* … */, window, seed, broker, selection }
```

(The hard 5-field destructure fails to compile the moment the 6th field exists —
this edit is the load-bearing, most-forgettable change; pinned by the existing
compat round-trip tests.)

### The selector (`aura-registry/src/lib.rs`, beside `optimize`)

```rust
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23);
/// `FamilySelection` is the new companion record. Deterministic given `seed`.
pub fn optimize_deflated(
    family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
    let winner = optimize(family, metric)?;            // unchanged argmax
    let m = resolve_metric(metric)?;                   // shared with metric_cmp
    let k = family.points.len();
    let raw = metric_value(&winner.report, m);         // extracted from metric_cmp

    let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) {
        // R arm: centred moving-block reality-check over each member's trade_rs.
        let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed);
        let stats = MetricStats::from_values(&null_max);          // reuse mc.rs
        let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
            / (n_resamples + 1) as f64;                            // (z+1)/(N+1)
        (raw - stats.p95, Some(p_over))
    } else {
        // total_pips arm: closed-form expected-max-of-K dispersion floor.
        let sd = member_sd(family, m);
        (raw - sd * expected_max_of_normals(k), None)
    };

    Ok((winner.clone(), FamilySelection {
        selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
        deflated_score, overfit_probability, mode: SelectionMode::Argmax,
        n_resamples, block_len, seed,
    }))
}
```

Supporting shapes (planner fills the bodies):

```rust
// aura-registry: refactor metric_cmp's value-read into a shared helper so the
// comparator and the deflation read one metric source (direction stays in metric_cmp).
fn resolve_metric(name: &str) -> Result<Metric, RegistryError>;   // the existing name→enum match
fn metric_value(rep: &RunReport, m: Metric) -> f64;               // r_get / total_pips read, lifted
fn r_series_of(m: Metric) -> Option<fn(&RunReport) -> &[f64]>;    // Some for R metrics → trade_rs; None for total_pips/dd/flips

// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i);
// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k.
fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec<f64>;
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64;           // r_metrics_from_rs(rs) → project field

// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared,
// byte-identical (r_bootstrap keeps its golden test).
pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64>;

// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient.
pub fn inv_norm_cdf(p: f64) -> f64;                               // rational approximation, golden-tested
pub fn expected_max_of_normals(k: usize) -> f64;                  // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772…
```

### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`)

```rust
// :1810  R arm — was: let best = optimize(&is_family, "sqn_normalized")…;
let (best, selection) =
    optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED)
        .expect("sqn_normalized is a known metric");
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
oos_report.manifest.selection = Some(selection);          // stamp onto the OOS winner
WindowRun { chosen_params: best.params, oos_equity, oos_report }
// :1795  total_pips arm — identical shape with "total_pips" + run_oos.
```

`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module
constants this cycle (recorded in the manifest → reproducible); CLI flags for them
are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)`
beside the raw metric when `manifest.selection.is_some()`.

## Components

| Component | Crate / file | Change |
|---|---|---|
| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` |
| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened |
| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) |
| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested |
| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build |
| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor |
| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest |
| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present |

## Data flow

1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged).
2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the
   winner's raw metric via `metric_value`.
3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives
   `resample_block` over every member's **mean-centred** `trade_rs`; the metric is
   recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K`
   `M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`,
   `overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`.
4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`,
   `overfit_probability = None`.
5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`,
   which flows through `append_family` unchanged and persists in `families.jsonl`.
6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family`
   displays it.

## Error handling

- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs`
  degrades exactly as `r_bootstrap` already does (all-zero kernel output); the
  deflation then equals the raw metric (no spurious penalty). `block_len` is clamped
  `[1, n]` by the shared kernel.
- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a
  zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s
  `NEG_INFINITY` treatment of a missing `r` block).
- A degenerate `K ≤ 1` family (a single member — no search) carries **no**
  multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the
  `Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and
  `overfit_probability` reduces to the floor `1/(n+1)`.
- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name
  (same as `optimize`), before any resampling.
- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds.

## Testing strategy

- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` →
  bit-identical `overfit_probability` and `deflated_score`, across thread counts
  (the best-of-`K` reduction collects in member/odometer order, never completion
  order). A fixed-`trade_rs` fixture pins exact values.
- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for
  every metric — the deflation never changes which point wins.
- **C2 IS-only:** the selector reads only the in-sample family's members; a test
  asserts no OOS report is touched (the call site passes `is_family`).
- **Statistic correctness:** on a fabricated family where one member has a real edge
  and the rest are zero-centred noise, `overfit_probability` is low and
  `deflated_score > 0`; on an all-noise family (no member has edge),
  `overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring
  is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred
  construction does not.
- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0,
  symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing,
  small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote).
- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads
  as `None`; a stamped manifest round-trips byte-identically; the existing
  `runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens
  stay green (add, don't break).
- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays
  bit-identical (proves the kernel was extracted, not changed).

## Acceptance criteria

- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest;
  `aura runs family … rank` surfaces the deflated score and overfit probability
  beside the raw metric — the worked program above runs and produces the shown block.
- The argmax winner is unchanged for every metric (C23 tripwire green).
- `overfit_probability` and `deflated_score` are reproducible from the recorded
  `(seed, n_resamples, block_len)` (C1).
- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing
  goldens stay green (C18/C14).
- The deflation is frictionless Stage-1 R; no Stage-2 cost enters.

### Out of scope (deferred)

- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries
  no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a
  full cartesian grid — deferred to a follow-on, addable by the same serde widening.
- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it
  runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape
  change).
- Re-ranking a family by the deflated score (a different design decision — would
  change which member wins; belongs in the ledger, not this cycle).
- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is
  reserved, not filled).