spec: 0076 deflate sweep winner for trials (boss-signed)
First cycle of the inferential-validation milestone. Adds optimize_deflated beside optimize: an additive selection-provenance record (deflated score + overfit probability) stamped on the OOS winner's manifest, never re-ranking (C23). R arm = centred moving-block reality-check reusing the r_bootstrap kernel; total_pips arm = expected-max-of-K dispersion floor. Shared RunManifest.selection carrier with a SelectionMode::Plateau slot reserved for #145. refs #144
This commit is contained in:
@@ -0,0 +1,366 @@
|
||||
# Deflate the sweep winner's metric for the number of trials — Design Spec
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
Cycle 0076 — first cycle of the milestone *Inferential validation (defend against
|
||||
false discovery at sweep scale)*. Closes #144 (refs the milestone).
|
||||
|
||||
## Goal
|
||||
|
||||
The World's in-sample sweep picks the single best member by a bare argmax
|
||||
(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains,
|
||||
the higher the best member's metric climbs by chance alone, so the winner's
|
||||
`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many
|
||||
configurations competed for the top slot — the family-scale false-discovery the
|
||||
milestone exists to defend against.
|
||||
|
||||
This cycle adds a **selection-provenance record** that, *without changing which
|
||||
member wins* (additive only — C23), reports how inflated the winner's metric is by
|
||||
the size of the search it won:
|
||||
|
||||
- a **deflated score** — the winner's metric minus the edge a search of the same
|
||||
size produces under a no-edge null;
|
||||
- an **overfit probability** — the chance a search of `K` members yields the
|
||||
observed winner under that null;
|
||||
|
||||
both stamped on the surviving out-of-sample winner's manifest (C18), reproducible
|
||||
from a recorded seed (C1).
|
||||
|
||||
This is framed as aura's own selection-honesty discipline; it introduces no new
|
||||
ranking key and never re-orders a family.
|
||||
|
||||
## Architecture
|
||||
|
||||
One new family-level selector sits **above** the closed `metric_cmp` / `Metric`
|
||||
direction source, beside `optimize`, in the registry that already owns selection
|
||||
(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then
|
||||
computes the deflation as an additive companion:
|
||||
|
||||
- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r,
|
||||
net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each
|
||||
member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle
|
||||
0075 already populates) is mean-subtracted to impose the no-edge null, then
|
||||
resampled with the **same** moving-block kernel `r_bootstrap` uses (a
|
||||
`SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K`
|
||||
member metric forms the null-max distribution the winner is judged against.
|
||||
- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a
|
||||
closed-form **dispersion-floor** deflation — the winner minus the
|
||||
expected-maximum-of-`K` inflation implied by the cross-member metric dispersion.
|
||||
|
||||
The record rides inside each member's existing `RunReport.manifest` as a new
|
||||
optional field, using the one-directional serde widening already proven by
|
||||
`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every
|
||||
legacy `runs.jsonl` / `families.jsonl` line are untouched.
|
||||
|
||||
The two walk-forward in-sample selection seams (`walkforward_family`,
|
||||
`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and
|
||||
stamp the record onto each window's chosen OOS report. The read-only
|
||||
`runs family … rank` display surfaces it.
|
||||
|
||||
**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type
|
||||
(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type
|
||||
and `aura-registry` depends on `aura-engine` — defining it in the registry would
|
||||
invert the dependency. The selector `optimize_deflated` lives in `aura-registry`
|
||||
(which owns `optimize` / `metric_cmp`) and constructs the engine-side record.
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### User-facing program (the acceptance evidence)
|
||||
|
||||
A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample
|
||||
grid per window — unchanged invocation, new provenance on the result:
|
||||
|
||||
```console
|
||||
$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \
|
||||
--fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0
|
||||
```
|
||||
|
||||
Each window's chosen OOS run now persists a `selection` block on its manifest:
|
||||
|
||||
```jsonc
|
||||
// one member line in runs/families/<id> … families.jsonl
|
||||
"manifest": {
|
||||
"commit": "c192dfd…",
|
||||
"params": [["signals.trend.fast.length", {"I64":100}], …],
|
||||
"window": [1704067200000000000, 1706745600000000000],
|
||||
"seed": 0,
|
||||
"broker": "sim-optimal(pip_size=1e-4)",
|
||||
"selection": {
|
||||
"selection_metric": "sqn_normalized",
|
||||
"n_trials": 4,
|
||||
"raw_winner_metric": 1.83,
|
||||
"deflated_score": 0.21, // winner − p95(null best-of-K); > 0 ⇒ survives
|
||||
"overfit_probability": 0.06, // (count(nullmax ≥ winner) + 1)/(n+1)
|
||||
"mode": "Argmax",
|
||||
"n_resamples": 1000,
|
||||
"block_len": 5,
|
||||
"seed": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The legacy `total_pips` SmaCross arm records the same block with
|
||||
`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the
|
||||
dispersion-floor arm computes no empirical probability); `selection_metric` plus the
|
||||
absent probability disambiguate the arm. A pre-0076 line carries no `selection`
|
||||
field and reads back as `None`, byte-unchanged.
|
||||
|
||||
The provenance is surfaced read-only beside the raw metric:
|
||||
|
||||
```console
|
||||
$ aura runs family stage1-r-3 rank sqn_normalized
|
||||
# … each member, best-first, now with: sqn_normalized=1.83 deflated=0.21 P(overfit)=0.06
|
||||
```
|
||||
|
||||
### New record type (`aura-engine/src/report.rs`, beside `RMetrics`)
|
||||
|
||||
```rust
|
||||
/// Selection-provenance for a sweep winner: how its metric was deflated for the
|
||||
/// number of configurations it beat. Additive (C23) — recorded, never re-ranking.
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String, // the metric the family was optimised against
|
||||
pub n_trials: usize, // K — the family size the winner beat
|
||||
pub raw_winner_metric: f64, // the argmax winner's face-value metric (audit)
|
||||
pub deflated_score: f64, // trials-adjusted score (per-arm; see the selector)
|
||||
/// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor
|
||||
/// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
pub mode: SelectionMode,
|
||||
pub n_resamples: usize, // resample provenance — reproduces the statistic
|
||||
pub block_len: usize,
|
||||
pub seed: u64,
|
||||
}
|
||||
|
||||
/// Which selection objective produced the record. `Argmax` is the bare-best
|
||||
/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*`
|
||||
/// variant on this same field, so peak-vs-plateau runs stay distinguishable.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
}
|
||||
```
|
||||
|
||||
### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`)
|
||||
|
||||
```rust
|
||||
// before
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
}
|
||||
// after — one-directional widening, identical idiom to RunMetrics.r
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
/// Selection provenance, present only on a sweep/walk-forward winner; a
|
||||
/// standalone run and every pre-0076 line read back as `None`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
}
|
||||
```
|
||||
|
||||
### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`)
|
||||
|
||||
```rust
|
||||
// RunManifestRead (compat.rs:27) gains the same optional field …
|
||||
struct RunManifestRead {
|
||||
commit: String,
|
||||
params: Vec<(String, ScalarRead)>,
|
||||
window: (Timestamp, Timestamp),
|
||||
seed: u64,
|
||||
broker: String,
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>, // legacy lines → None
|
||||
}
|
||||
// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it:
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
|
||||
RunManifest { commit, params: /* … */, window, seed, broker, selection }
|
||||
```
|
||||
|
||||
(The hard 5-field destructure fails to compile the moment the 6th field exists —
|
||||
this edit is the load-bearing, most-forgettable change; pinned by the existing
|
||||
compat round-trip tests.)
|
||||
|
||||
### The selector (`aura-registry/src/lib.rs`, beside `optimize`)
|
||||
|
||||
```rust
|
||||
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
|
||||
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23);
|
||||
/// `FamilySelection` is the new companion record. Deterministic given `seed`.
|
||||
pub fn optimize_deflated(
|
||||
family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let winner = optimize(family, metric)?; // unchanged argmax
|
||||
let m = resolve_metric(metric)?; // shared with metric_cmp
|
||||
let k = family.points.len();
|
||||
let raw = metric_value(&winner.report, m); // extracted from metric_cmp
|
||||
|
||||
let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) {
|
||||
// R arm: centred moving-block reality-check over each member's trade_rs.
|
||||
let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed);
|
||||
let stats = MetricStats::from_values(&null_max); // reuse mc.rs
|
||||
let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
|
||||
/ (n_resamples + 1) as f64; // (z+1)/(N+1)
|
||||
(raw - stats.p95, Some(p_over))
|
||||
} else {
|
||||
// total_pips arm: closed-form expected-max-of-K dispersion floor.
|
||||
let sd = member_sd(family, m);
|
||||
(raw - sd * expected_max_of_normals(k), None)
|
||||
};
|
||||
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
deflated_score, overfit_probability, mode: SelectionMode::Argmax,
|
||||
n_resamples, block_len, seed,
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
Supporting shapes (planner fills the bodies):
|
||||
|
||||
```rust
|
||||
// aura-registry: refactor metric_cmp's value-read into a shared helper so the
|
||||
// comparator and the deflation read one metric source (direction stays in metric_cmp).
|
||||
fn resolve_metric(name: &str) -> Result<Metric, RegistryError>; // the existing name→enum match
|
||||
fn metric_value(rep: &RunReport, m: Metric) -> f64; // r_get / total_pips read, lifted
|
||||
fn r_series_of(m: Metric) -> Option<fn(&RunReport) -> &[f64]>; // Some for R metrics → trade_rs; None for total_pips/dd/flips
|
||||
|
||||
// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i);
|
||||
// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k.
|
||||
fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec<f64>;
|
||||
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64; // r_metrics_from_rs(rs) → project field
|
||||
|
||||
// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared,
|
||||
// byte-identical (r_bootstrap keeps its golden test).
|
||||
pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64>;
|
||||
|
||||
// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient.
|
||||
pub fn inv_norm_cdf(p: f64) -> f64; // rational approximation, golden-tested
|
||||
pub fn expected_max_of_normals(k: usize) -> f64; // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772…
|
||||
```
|
||||
|
||||
### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`)
|
||||
|
||||
```rust
|
||||
// :1810 R arm — was: let best = optimize(&is_family, "sqn_normalized")…;
|
||||
let (best, selection) =
|
||||
optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED)
|
||||
.expect("sqn_normalized is a known metric");
|
||||
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection); // stamp onto the OOS winner
|
||||
WindowRun { chosen_params: best.params, oos_equity, oos_report }
|
||||
// :1795 total_pips arm — identical shape with "total_pips" + run_oos.
|
||||
```
|
||||
|
||||
`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module
|
||||
constants this cycle (recorded in the manifest → reproducible); CLI flags for them
|
||||
are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)`
|
||||
beside the raw metric when `manifest.selection.is_some()`.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Crate / file | Change |
|
||||
|---|---|---|
|
||||
| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` |
|
||||
| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened |
|
||||
| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) |
|
||||
| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested |
|
||||
| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build |
|
||||
| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor |
|
||||
| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest |
|
||||
| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present |
|
||||
|
||||
## Data flow
|
||||
|
||||
1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged).
|
||||
2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the
|
||||
winner's raw metric via `metric_value`.
|
||||
3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives
|
||||
`resample_block` over every member's **mean-centred** `trade_rs`; the metric is
|
||||
recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K`
|
||||
`M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`,
|
||||
`overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`.
|
||||
4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`,
|
||||
`overfit_probability = None`.
|
||||
5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`,
|
||||
which flows through `append_family` unchanged and persists in `families.jsonl`.
|
||||
6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family`
|
||||
displays it.
|
||||
|
||||
## Error handling
|
||||
|
||||
- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs`
|
||||
degrades exactly as `r_bootstrap` already does (all-zero kernel output); the
|
||||
deflation then equals the raw metric (no spurious penalty). `block_len` is clamped
|
||||
`[1, n]` by the shared kernel.
|
||||
- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a
|
||||
zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s
|
||||
`NEG_INFINITY` treatment of a missing `r` block).
|
||||
- A degenerate `K ≤ 1` family (a single member — no search) carries **no**
|
||||
multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the
|
||||
`Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and
|
||||
`overfit_probability` reduces to the floor `1/(n+1)`.
|
||||
- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name
|
||||
(same as `optimize`), before any resampling.
|
||||
- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` →
|
||||
bit-identical `overfit_probability` and `deflated_score`, across thread counts
|
||||
(the best-of-`K` reduction collects in member/odometer order, never completion
|
||||
order). A fixed-`trade_rs` fixture pins exact values.
|
||||
- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for
|
||||
every metric — the deflation never changes which point wins.
|
||||
- **C2 IS-only:** the selector reads only the in-sample family's members; a test
|
||||
asserts no OOS report is touched (the call site passes `is_family`).
|
||||
- **Statistic correctness:** on a fabricated family where one member has a real edge
|
||||
and the rest are zero-centred noise, `overfit_probability` is low and
|
||||
`deflated_score > 0`; on an all-noise family (no member has edge),
|
||||
`overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring
|
||||
is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred
|
||||
construction does not.
|
||||
- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0,
|
||||
symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing,
|
||||
small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote).
|
||||
- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads
|
||||
as `None`; a stamped manifest round-trips byte-identically; the existing
|
||||
`runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens
|
||||
stay green (add, don't break).
|
||||
- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays
|
||||
bit-identical (proves the kernel was extracted, not changed).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest;
|
||||
`aura runs family … rank` surfaces the deflated score and overfit probability
|
||||
beside the raw metric — the worked program above runs and produces the shown block.
|
||||
- The argmax winner is unchanged for every metric (C23 tripwire green).
|
||||
- `overfit_probability` and `deflated_score` are reproducible from the recorded
|
||||
`(seed, n_resamples, block_len)` (C1).
|
||||
- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing
|
||||
goldens stay green (C18/C14).
|
||||
- The deflation is frictionless Stage-1 R; no Stage-2 cost enters.
|
||||
|
||||
### Out of scope (deferred)
|
||||
|
||||
- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries
|
||||
no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a
|
||||
full cartesian grid — deferred to a follow-on, addable by the same serde widening.
|
||||
- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it
|
||||
runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape
|
||||
change).
|
||||
- Re-ranking a family by the deflated score (a different design decision — would
|
||||
change which member wins; belongs in the ledger, not this cycle).
|
||||
- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is
|
||||
reserved, not filled).
|
||||
Reference in New Issue
Block a user