spec: 0076 deflate sweep winner for trials (boss-signed)

First cycle of the inferential-validation milestone. Adds optimize_deflated beside optimize: an additive selection-provenance record (deflated score + overfit probability) stamped on the OOS winner's manifest, never re-ranking (C23). R arm = centred moving-block reality-check reusing the r_bootstrap kernel; total_pips arm = expected-max-of-K dispersion floor. Shared RunManifest.selection carrier with a SelectionMode::Plateau slot reserved for #145. refs #144
2026-06-26 13:42:45 +02:00
parent c192dfd344
commit a1908bd15e
1 changed files with 366 additions and 0 deletions
@@ -0,0 +1,366 @@
+# Deflate the sweep winner's metric for the number of trials — Design Spec
+
+**Date:** 2026-06-26
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+Cycle 0076 — first cycle of the milestone *Inferential validation (defend against
+false discovery at sweep scale)*. Closes #144 (refs the milestone).
+
+## Goal
+
+The World's in-sample sweep picks the single best member by a bare argmax
+(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains,
+the higher the best member's metric climbs by chance alone, so the winner's
+`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many
+configurations competed for the top slot — the family-scale false-discovery the
+milestone exists to defend against.
+
+This cycle adds a **selection-provenance record** that, *without changing which
+member wins* (additive only — C23), reports how inflated the winner's metric is by
+the size of the search it won:
+
+- a **deflated score** — the winner's metric minus the edge a search of the same
+  size produces under a no-edge null;
+- an **overfit probability** — the chance a search of `K` members yields the
+  observed winner under that null;
+
+both stamped on the surviving out-of-sample winner's manifest (C18), reproducible
+from a recorded seed (C1).
+
+This is framed as aura's own selection-honesty discipline; it introduces no new
+ranking key and never re-orders a family.
+
+## Architecture
+
+One new family-level selector sits **above** the closed `metric_cmp` / `Metric`
+direction source, beside `optimize`, in the registry that already owns selection
+(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then
+computes the deflation as an additive companion:
+
+- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r,
+  net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each
+  member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle
+  0075 already populates) is mean-subtracted to impose the no-edge null, then
+  resampled with the **same** moving-block kernel `r_bootstrap` uses (a
+  `SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K`
+  member metric forms the null-max distribution the winner is judged against.
+- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a
+  closed-form **dispersion-floor** deflation — the winner minus the
+  expected-maximum-of-`K` inflation implied by the cross-member metric dispersion.
+
+The record rides inside each member's existing `RunReport.manifest` as a new
+optional field, using the one-directional serde widening already proven by
+`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every
+legacy `runs.jsonl` / `families.jsonl` line are untouched.
+
+The two walk-forward in-sample selection seams (`walkforward_family`,
+`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and
+stamp the record onto each window's chosen OOS report. The read-only
+`runs family … rank` display surfaces it.
+
+**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type
+(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type
+and `aura-registry` depends on `aura-engine` — defining it in the registry would
+invert the dependency. The selector `optimize_deflated` lives in `aura-registry`
+(which owns `optimize` / `metric_cmp`) and constructs the engine-side record.
+
+## Concrete code shapes
+
+### User-facing program (the acceptance evidence)
+
+A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample
+grid per window — unchanged invocation, new provenance on the result:
+
+```console
+$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \
+      --fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0
+```
+
+Each window's chosen OOS run now persists a `selection` block on its manifest:
+
+```jsonc
+// one member line in runs/families/<id> … families.jsonl
+"manifest": {
+  "commit": "c192dfd…",
+  "params": [["signals.trend.fast.length", {"I64":100}], …],
+  "window": [1704067200000000000, 1706745600000000000],
+  "seed": 0,
+  "broker": "sim-optimal(pip_size=1e-4)",
+  "selection": {
+    "selection_metric": "sqn_normalized",
+    "n_trials": 4,
+    "raw_winner_metric": 1.83,
+    "deflated_score": 0.21,          // winner − p95(null best-of-K); > 0 ⇒ survives
+    "overfit_probability": 0.06,     // (count(nullmax ≥ winner) + 1)/(n+1)
+    "mode": "Argmax",
+    "n_resamples": 1000,
+    "block_len": 5,
+    "seed": 42
+  }
+}
+```
+
+The legacy `total_pips` SmaCross arm records the same block with
+`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the
+dispersion-floor arm computes no empirical probability); `selection_metric` plus the
+absent probability disambiguate the arm. A pre-0076 line carries no `selection`
+field and reads back as `None`, byte-unchanged.
+
+The provenance is surfaced read-only beside the raw metric:
+
+```console
+$ aura runs family stage1-r-3 rank sqn_normalized
+# … each member, best-first, now with:  sqn_normalized=1.83  deflated=0.21  P(overfit)=0.06
+```
+
+### New record type (`aura-engine/src/report.rs`, beside `RMetrics`)
+
+```rust
+/// Selection-provenance for a sweep winner: how its metric was deflated for the
+/// number of configurations it beat. Additive (C23) — recorded, never re-ranking.
+#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
+pub struct FamilySelection {
+    pub selection_metric: String, // the metric the family was optimised against
+    pub n_trials: usize,          // K — the family size the winner beat
+    pub raw_winner_metric: f64,   // the argmax winner's face-value metric (audit)
+    pub deflated_score: f64,      // trials-adjusted score (per-arm; see the selector)
+    /// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor
+    /// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub overfit_probability: Option<f64>,
+    pub mode: SelectionMode,
+    pub n_resamples: usize,       // resample provenance — reproduces the statistic
+    pub block_len: usize,
+    pub seed: u64,
+}
+
+/// Which selection objective produced the record. `Argmax` is the bare-best
+/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*`
+/// variant on this same field, so peak-vs-plateau runs stay distinguishable.
+#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
+pub enum SelectionMode {
+    Argmax,
+}
+```
+
+### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`)
+
+```rust
+// before
+pub struct RunManifest {
+    pub commit: String,
+    pub params: Vec<(String, Scalar)>,
+    pub window: (Timestamp, Timestamp),
+    pub seed: u64,
+    pub broker: String,
+}
+// after — one-directional widening, identical idiom to RunMetrics.r
+pub struct RunManifest {
+    pub commit: String,
+    pub params: Vec<(String, Scalar)>,
+    pub window: (Timestamp, Timestamp),
+    pub seed: u64,
+    pub broker: String,
+    /// Selection provenance, present only on a sweep/walk-forward winner; a
+    /// standalone run and every pre-0076 line read back as `None`.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub selection: Option<FamilySelection>,
+}
+```
+
+### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`)
+
+```rust
+// RunManifestRead (compat.rs:27) gains the same optional field …
+struct RunManifestRead {
+    commit: String,
+    params: Vec<(String, ScalarRead)>,
+    window: (Timestamp, Timestamp),
+    seed: u64,
+    broker: String,
+    #[serde(default)]
+    selection: Option<FamilySelection>,   // legacy lines → None
+}
+// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it:
+let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
+RunManifest { commit, params: /* … */, window, seed, broker, selection }
+```
+
+(The hard 5-field destructure fails to compile the moment the 6th field exists —
+this edit is the load-bearing, most-forgettable change; pinned by the existing
+compat round-trip tests.)
+
+### The selector (`aura-registry/src/lib.rs`, beside `optimize`)
+
+```rust
+/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
+/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23);
+/// `FamilySelection` is the new companion record. Deterministic given `seed`.
+pub fn optimize_deflated(
+    family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
+) -> Result<(SweepPoint, FamilySelection), RegistryError> {
+    let winner = optimize(family, metric)?;            // unchanged argmax
+    let m = resolve_metric(metric)?;                   // shared with metric_cmp
+    let k = family.points.len();
+    let raw = metric_value(&winner.report, m);         // extracted from metric_cmp
+
+    let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) {
+        // R arm: centred moving-block reality-check over each member's trade_rs.
+        let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed);
+        let stats = MetricStats::from_values(&null_max);          // reuse mc.rs
+        let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
+            / (n_resamples + 1) as f64;                            // (z+1)/(N+1)
+        (raw - stats.p95, Some(p_over))
+    } else {
+        // total_pips arm: closed-form expected-max-of-K dispersion floor.
+        let sd = member_sd(family, m);
+        (raw - sd * expected_max_of_normals(k), None)
+    };
+
+    Ok((winner.clone(), FamilySelection {
+        selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
+        deflated_score, overfit_probability, mode: SelectionMode::Argmax,
+        n_resamples, block_len, seed,
+    }))
+}
+```
+
+Supporting shapes (planner fills the bodies):
+
+```rust
+// aura-registry: refactor metric_cmp's value-read into a shared helper so the
+// comparator and the deflation read one metric source (direction stays in metric_cmp).
+fn resolve_metric(name: &str) -> Result<Metric, RegistryError>;   // the existing name→enum match
+fn metric_value(rep: &RunReport, m: Metric) -> f64;               // r_get / total_pips read, lifted
+fn r_series_of(m: Metric) -> Option<fn(&RunReport) -> &[f64]>;    // Some for R metrics → trade_rs; None for total_pips/dd/flips
+
+// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i);
+// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k.
+fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec<f64>;
+fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64;           // r_metrics_from_rs(rs) → project field
+
+// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared,
+// byte-identical (r_bootstrap keeps its golden test).
+pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64>;
+
+// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient.
+pub fn inv_norm_cdf(p: f64) -> f64;                               // rational approximation, golden-tested
+pub fn expected_max_of_normals(k: usize) -> f64;                  // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772…
+```
+
+### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`)
+
+```rust
+// :1810  R arm — was: let best = optimize(&is_family, "sqn_normalized")…;
+let (best, selection) =
+    optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED)
+        .expect("sqn_normalized is a known metric");
+let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
+oos_report.manifest.selection = Some(selection);          // stamp onto the OOS winner
+WindowRun { chosen_params: best.params, oos_equity, oos_report }
+// :1795  total_pips arm — identical shape with "total_pips" + run_oos.
+```
+
+`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module
+constants this cycle (recorded in the manifest → reproducible); CLI flags for them
+are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)`
+beside the raw metric when `manifest.selection.is_some()`.
+
+## Components
+
+| Component | Crate / file | Change |
+|---|---|---|
+| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` |
+| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened |
+| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) |
+| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested |
+| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build |
+| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor |
+| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest |
+| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present |
+
+## Data flow
+
+1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged).
+2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the
+   winner's raw metric via `metric_value`.
+3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives
+   `resample_block` over every member's **mean-centred** `trade_rs`; the metric is
+   recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K`
+   `M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`,
+   `overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`.
+4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`,
+   `overfit_probability = None`.
+5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`,
+   which flows through `append_family` unchanged and persists in `families.jsonl`.
+6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family`
+   displays it.
+
+## Error handling
+
+- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs`
+  degrades exactly as `r_bootstrap` already does (all-zero kernel output); the
+  deflation then equals the raw metric (no spurious penalty). `block_len` is clamped
+  `[1, n]` by the shared kernel.
+- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a
+  zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s
+  `NEG_INFINITY` treatment of a missing `r` block).
+- A degenerate `K ≤ 1` family (a single member — no search) carries **no**
+  multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the
+  `Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and
+  `overfit_probability` reduces to the floor `1/(n+1)`.
+- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name
+  (same as `optimize`), before any resampling.
+- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds.
+
+## Testing strategy
+
+- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` →
+  bit-identical `overfit_probability` and `deflated_score`, across thread counts
+  (the best-of-`K` reduction collects in member/odometer order, never completion
+  order). A fixed-`trade_rs` fixture pins exact values.
+- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for
+  every metric — the deflation never changes which point wins.
+- **C2 IS-only:** the selector reads only the in-sample family's members; a test
+  asserts no OOS report is touched (the call site passes `is_family`).
+- **Statistic correctness:** on a fabricated family where one member has a real edge
+  and the rest are zero-centred noise, `overfit_probability` is low and
+  `deflated_score > 0`; on an all-noise family (no member has edge),
+  `overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring
+  is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred
+  construction does not.
+- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0,
+  symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing,
+  small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote).
+- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads
+  as `None`; a stamped manifest round-trips byte-identically; the existing
+  `runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens
+  stay green (add, don't break).
+- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays
+  bit-identical (proves the kernel was extracted, not changed).
+
+## Acceptance criteria
+
+- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest;
+  `aura runs family … rank` surfaces the deflated score and overfit probability
+  beside the raw metric — the worked program above runs and produces the shown block.
+- The argmax winner is unchanged for every metric (C23 tripwire green).
+- `overfit_probability` and `deflated_score` are reproducible from the recorded
+  `(seed, n_resamples, block_len)` (C1).
+- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing
+  goldens stay green (C18/C14).
+- The deflation is frictionless Stage-1 R; no Stage-2 cost enters.
+
+### Out of scope (deferred)
+
+- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries
+  no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a
+  full cartesian grid — deferred to a follow-on, addable by the same serde widening.
+- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it
+  runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape
+  change).
+- Re-ranking a family by the deflated score (a different design decision — would
+  change which member wins; belongs in the ledger, not this cycle).
+- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is
+  reserved, not filled).