diff --git a/docs/specs/0076-deflate-sweep-winner-trials.md b/docs/specs/0076-deflate-sweep-winner-trials.md new file mode 100644 index 0000000..feadff5 --- /dev/null +++ b/docs/specs/0076-deflate-sweep-winner-trials.md @@ -0,0 +1,366 @@ +# Deflate the sweep winner's metric for the number of trials — Design Spec + +**Date:** 2026-06-26 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +Cycle 0076 — first cycle of the milestone *Inferential validation (defend against +false discovery at sweep scale)*. Closes #144 (refs the milestone). + +## Goal + +The World's in-sample sweep picks the single best member by a bare argmax +(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains, +the higher the best member's metric climbs by chance alone, so the winner's +`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many +configurations competed for the top slot — the family-scale false-discovery the +milestone exists to defend against. + +This cycle adds a **selection-provenance record** that, *without changing which +member wins* (additive only — C23), reports how inflated the winner's metric is by +the size of the search it won: + +- a **deflated score** — the winner's metric minus the edge a search of the same + size produces under a no-edge null; +- an **overfit probability** — the chance a search of `K` members yields the + observed winner under that null; + +both stamped on the surviving out-of-sample winner's manifest (C18), reproducible +from a recorded seed (C1). + +This is framed as aura's own selection-honesty discipline; it introduces no new +ranking key and never re-orders a family. + +## Architecture + +One new family-level selector sits **above** the closed `metric_cmp` / `Metric` +direction source, beside `optimize`, in the registry that already owns selection +(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then +computes the deflation as an additive companion: + +- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r, + net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each + member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle + 0075 already populates) is mean-subtracted to impose the no-edge null, then + resampled with the **same** moving-block kernel `r_bootstrap` uses (a + `SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K` + member metric forms the null-max distribution the winner is judged against. +- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a + closed-form **dispersion-floor** deflation — the winner minus the + expected-maximum-of-`K` inflation implied by the cross-member metric dispersion. + +The record rides inside each member's existing `RunReport.manifest` as a new +optional field, using the one-directional serde widening already proven by +`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every +legacy `runs.jsonl` / `families.jsonl` line are untouched. + +The two walk-forward in-sample selection seams (`walkforward_family`, +`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and +stamp the record onto each window's chosen OOS report. The read-only +`runs family … rank` display surfaces it. + +**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type +(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type +and `aura-registry` depends on `aura-engine` — defining it in the registry would +invert the dependency. The selector `optimize_deflated` lives in `aura-registry` +(which owns `optimize` / `metric_cmp`) and constructs the engine-side record. + +## Concrete code shapes + +### User-facing program (the acceptance evidence) + +A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample +grid per window — unchanged invocation, new provenance on the result: + +```console +$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \ + --fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0 +``` + +Each window's chosen OOS run now persists a `selection` block on its manifest: + +```jsonc +// one member line in runs/families/ … families.jsonl +"manifest": { + "commit": "c192dfd…", + "params": [["signals.trend.fast.length", {"I64":100}], …], + "window": [1704067200000000000, 1706745600000000000], + "seed": 0, + "broker": "sim-optimal(pip_size=1e-4)", + "selection": { + "selection_metric": "sqn_normalized", + "n_trials": 4, + "raw_winner_metric": 1.83, + "deflated_score": 0.21, // winner − p95(null best-of-K); > 0 ⇒ survives + "overfit_probability": 0.06, // (count(nullmax ≥ winner) + 1)/(n+1) + "mode": "Argmax", + "n_resamples": 1000, + "block_len": 5, + "seed": 42 + } +} +``` + +The legacy `total_pips` SmaCross arm records the same block with +`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the +dispersion-floor arm computes no empirical probability); `selection_metric` plus the +absent probability disambiguate the arm. A pre-0076 line carries no `selection` +field and reads back as `None`, byte-unchanged. + +The provenance is surfaced read-only beside the raw metric: + +```console +$ aura runs family stage1-r-3 rank sqn_normalized +# … each member, best-first, now with: sqn_normalized=1.83 deflated=0.21 P(overfit)=0.06 +``` + +### New record type (`aura-engine/src/report.rs`, beside `RMetrics`) + +```rust +/// Selection-provenance for a sweep winner: how its metric was deflated for the +/// number of configurations it beat. Additive (C23) — recorded, never re-ranking. +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub struct FamilySelection { + pub selection_metric: String, // the metric the family was optimised against + pub n_trials: usize, // K — the family size the winner beat + pub raw_winner_metric: f64, // the argmax winner's face-value metric (audit) + pub deflated_score: f64, // trials-adjusted score (per-arm; see the selector) + /// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor + /// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub overfit_probability: Option, + pub mode: SelectionMode, + pub n_resamples: usize, // resample provenance — reproduces the statistic + pub block_len: usize, + pub seed: u64, +} + +/// Which selection objective produced the record. `Argmax` is the bare-best +/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*` +/// variant on this same field, so peak-vs-plateau runs stay distinguishable. +#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub enum SelectionMode { + Argmax, +} +``` + +### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`) + +```rust +// before +pub struct RunManifest { + pub commit: String, + pub params: Vec<(String, Scalar)>, + pub window: (Timestamp, Timestamp), + pub seed: u64, + pub broker: String, +} +// after — one-directional widening, identical idiom to RunMetrics.r +pub struct RunManifest { + pub commit: String, + pub params: Vec<(String, Scalar)>, + pub window: (Timestamp, Timestamp), + pub seed: u64, + pub broker: String, + /// Selection provenance, present only on a sweep/walk-forward winner; a + /// standalone run and every pre-0076 line read back as `None`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selection: Option, +} +``` + +### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`) + +```rust +// RunManifestRead (compat.rs:27) gains the same optional field … +struct RunManifestRead { + commit: String, + params: Vec<(String, ScalarRead)>, + window: (Timestamp, Timestamp), + seed: u64, + broker: String, + #[serde(default)] + selection: Option, // legacy lines → None +} +// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it: +let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest; +RunManifest { commit, params: /* … */, window, seed, broker, selection } +``` + +(The hard 5-field destructure fails to compile the moment the 6th field exists — +this edit is the load-bearing, most-forgettable change; pinned by the existing +compat round-trip tests.) + +### The selector (`aura-registry/src/lib.rs`, beside `optimize`) + +```rust +/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned +/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23); +/// `FamilySelection` is the new companion record. Deterministic given `seed`. +pub fn optimize_deflated( + family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64, +) -> Result<(SweepPoint, FamilySelection), RegistryError> { + let winner = optimize(family, metric)?; // unchanged argmax + let m = resolve_metric(metric)?; // shared with metric_cmp + let k = family.points.len(); + let raw = metric_value(&winner.report, m); // extracted from metric_cmp + + let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) { + // R arm: centred moving-block reality-check over each member's trade_rs. + let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed); + let stats = MetricStats::from_values(&null_max); // reuse mc.rs + let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64 + / (n_resamples + 1) as f64; // (z+1)/(N+1) + (raw - stats.p95, Some(p_over)) + } else { + // total_pips arm: closed-form expected-max-of-K dispersion floor. + let sd = member_sd(family, m); + (raw - sd * expected_max_of_normals(k), None) + }; + + Ok((winner.clone(), FamilySelection { + selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw, + deflated_score, overfit_probability, mode: SelectionMode::Argmax, + n_resamples, block_len, seed, + })) +} +``` + +Supporting shapes (planner fills the bodies): + +```rust +// aura-registry: refactor metric_cmp's value-read into a shared helper so the +// comparator and the deflation read one metric source (direction stays in metric_cmp). +fn resolve_metric(name: &str) -> Result; // the existing name→enum match +fn metric_value(rep: &RunReport, m: Metric) -> f64; // r_get / total_pips read, lifted +fn r_series_of(m: Metric) -> Option &[f64]>; // Some for R metrics → trade_rs; None for total_pips/dd/flips + +// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i); +// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k. +fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec; +fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64; // r_metrics_from_rs(rs) → project field + +// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared, +// byte-identical (r_bootstrap keeps its golden test). +pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec; + +// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient. +pub fn inv_norm_cdf(p: f64) -> f64; // rational approximation, golden-tested +pub fn expected_max_of_normals(k: usize) -> f64; // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772… +``` + +### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`) + +```rust +// :1810 R arm — was: let best = optimize(&is_family, "sqn_normalized")…; +let (best, selection) = + optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED) + .expect("sqn_normalized is a known metric"); +let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data); +oos_report.manifest.selection = Some(selection); // stamp onto the OOS winner +WindowRun { chosen_params: best.params, oos_equity, oos_report } +// :1795 total_pips arm — identical shape with "total_pips" + run_oos. +``` + +`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module +constants this cycle (recorded in the manifest → reproducible); CLI flags for them +are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)` +beside the raw metric when `manifest.selection.is_some()`. + +## Components + +| Component | Crate / file | Change | +|---|---|---| +| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` | +| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened | +| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) | +| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested | +| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build | +| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor | +| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest | +| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present | + +## Data flow + +1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged). +2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the + winner's raw metric via `metric_value`. +3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives + `resample_block` over every member's **mean-centred** `trade_rs`; the metric is + recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K` + `M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`, + `overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`. +4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`, + `overfit_probability = None`. +5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`, + which flows through `append_family` unchanged and persists in `families.jsonl`. +6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family` + displays it. + +## Error handling + +- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs` + degrades exactly as `r_bootstrap` already does (all-zero kernel output); the + deflation then equals the raw metric (no spurious penalty). `block_len` is clamped + `[1, n]` by the shared kernel. +- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a + zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s + `NEG_INFINITY` treatment of a missing `r` block). +- A degenerate `K ≤ 1` family (a single member — no search) carries **no** + multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the + `Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and + `overfit_probability` reduces to the floor `1/(n+1)`. +- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name + (same as `optimize`), before any resampling. +- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds. + +## Testing strategy + +- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` → + bit-identical `overfit_probability` and `deflated_score`, across thread counts + (the best-of-`K` reduction collects in member/odometer order, never completion + order). A fixed-`trade_rs` fixture pins exact values. +- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for + every metric — the deflation never changes which point wins. +- **C2 IS-only:** the selector reads only the in-sample family's members; a test + asserts no OOS report is touched (the call site passes `is_family`). +- **Statistic correctness:** on a fabricated family where one member has a real edge + and the rest are zero-centred noise, `overfit_probability` is low and + `deflated_score > 0`; on an all-noise family (no member has edge), + `overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring + is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred + construction does not. +- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0, + symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing, + small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote). +- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads + as `None`; a stamped manifest round-trips byte-identically; the existing + `runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens + stay green (add, don't break). +- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays + bit-identical (proves the kernel was extracted, not changed). + +## Acceptance criteria + +- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest; + `aura runs family … rank` surfaces the deflated score and overfit probability + beside the raw metric — the worked program above runs and produces the shown block. +- The argmax winner is unchanged for every metric (C23 tripwire green). +- `overfit_probability` and `deflated_score` are reproducible from the recorded + `(seed, n_resamples, block_len)` (C1). +- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing + goldens stay green (C18/C14). +- The deflation is frictionless Stage-1 R; no Stage-2 cost enters. + +### Out of scope (deferred) + +- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries + no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a + full cartesian grid — deferred to a follow-on, addable by the same serde widening. +- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it + runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape + change). +- Re-ranking a family by the deflated score (a different design decision — would + change which member wins; belongs in the ledger, not this cycle). +- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is + reserved, not filled).