audit(0076): cycle close — drift addressed; ledger inferential-thread synced; drop ephemera
Architect drift review (c192dfd..a295905): drift_found, medium/low only — no
ledger or hot-path contract crossed. C23-additive, C1-deterministic, C14/C18
back-compat all verified green (full workspace suite + clippy). No regression
scripts configured (test+clippy are the gate).
Resolution:
- [medium] three spec-promised optimize_deflated tests were missing (the
centring control, the all-noise high-p companion, the C2 IS-only read) — the
centring was unprotected ("removing it would pass every test"). Plus an
unguarded dispersion-floor sign assumption. FIXED in 0076.tidy (ecc9541).
- [low, RATIFY] the ledger's "Open architectural threads" inferential entry was
stale on one fact (it described optimize as having no trials penalty). Synced:
it now records that #144's trials-deflation landed this cycle (additive
optimize_deflated, winner unchanged — C23; optimize/rank_by stay bare argmax),
that #139's per-candidate OOS bootstrap landed in 0075, and that #145/#146
remain open. This ratifies the cycle-0076 intent; the gap description is
preserved for the still-open default-ranker path.
Deferred (spec 0076 Out of scope, not drift): n_eff effective-trials advisory,
reload-time recompute of the R-arm statistic, deflated-score re-ranking, CLI
flags for the resampling config.
Ephemeral spec + plan (docs/specs/0076-*, docs/plans/0076-*) removed per the
cycle-close convention. refs #144
This commit is contained in:
+13
-4
@@ -1419,10 +1419,19 @@ load-bearing in the flat graph.
|
||||
selection/aggregation layer must deflate a winner for the family size (#144),
|
||||
prefer a plateau over a peak (#145), and score cross-instrument generalization
|
||||
(#146), with a per-candidate out-of-sample bootstrap as the adjacent
|
||||
significance read (#139). Distinct from the two threads above — neither
|
||||
orchestration *composability* (#109) nor a search *policy* (Bayesian/genetic),
|
||||
but the statistical *validity* of the selection itself. Not a C-invariant until
|
||||
built (the engine today violates it); recorded as the World's third unbuilt half.
|
||||
significance read (#139, landed cycle 0075). Distinct from the two threads above —
|
||||
neither orchestration *composability* (#109) nor a search *policy* (Bayesian/
|
||||
genetic), but the statistical *validity* of the selection itself. Not a
|
||||
C-invariant until built; recorded as the World's third (now partly built) half.
|
||||
**Status (cycle 0076):** the trials-deflation piece (#144) landed —
|
||||
`optimize_deflated` (aura-registry) wraps `optimize` and records, additively on
|
||||
the winning member's manifest (C18, the new `RunManifest.selection`), a deflated
|
||||
score + an overfit probability for the family size, **without changing which
|
||||
member wins** (C23; `optimize` / `rank_by` stay a bare argmax — the deflation is
|
||||
recorded provenance, not a re-ranking). The R arm is a centred moving-block
|
||||
reality-check (reusing the `r_bootstrap` kernel); the `total_pips` arm a
|
||||
closed-form expected-max-of-K dispersion floor. #145 (plateau-over-peak) and #146
|
||||
(cross-instrument generalization) remain open.
|
||||
- **`aura-std` contents** — the crate exists (doc-only); which universal blocks
|
||||
land first follows the walking-skeleton's needs.
|
||||
- **`strategies/` split** — a later split, *inside a project*, of top-level
|
||||
|
||||
@@ -1,674 +0,0 @@
|
||||
# Deflate the sweep winner for trials — Implementation Plan
|
||||
|
||||
> **Parent spec:** `docs/specs/0076-deflate-sweep-winner-trials.md`
|
||||
>
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run
|
||||
> this plan. Steps use `- [ ]` checkboxes for tracking.
|
||||
|
||||
**Goal:** Stamp an additive trials-deflation record (`deflated_score` +
|
||||
`overfit_probability`) on each walk-forward OOS winner's manifest, without changing
|
||||
which member wins.
|
||||
|
||||
**Architecture:** A new `optimize_deflated` sits beside `optimize` in
|
||||
`aura-registry`; it returns `optimize`'s byte-identical argmax winner plus a
|
||||
`FamilySelection` record (an `aura-engine` type on `RunManifest`). The R arm runs a
|
||||
centred moving-block reality-check reusing the `r_bootstrap` kernel; the `total_pips`
|
||||
arm uses a closed-form expected-max-of-K dispersion floor. The two `walkforward_family`
|
||||
seams call it and stamp the record on the chosen OOS report.
|
||||
|
||||
**Tech Stack:** `aura-engine` (report.rs types + mc.rs kernel + stats helpers),
|
||||
`aura-registry` (selector + compat mirror), `aura-cli` (wiring + display).
|
||||
|
||||
---
|
||||
|
||||
## Files this plan creates or modifies
|
||||
|
||||
- Modify: `crates/aura-engine/src/report.rs` — `FamilySelection`/`SelectionMode` types, `RunManifest.selection` field, `inv_norm_cdf`/`expected_max_of_normals`, thread engine-internal fixtures
|
||||
- Modify: `crates/aura-engine/src/lib.rs:64-68` — export the new public items
|
||||
- Modify: `crates/aura-engine/src/mc.rs` — extract `resample_block` from `r_bootstrap`
|
||||
- Modify: `crates/aura-engine/src/harness.rs:143` — make `SplitMix64` `pub`
|
||||
- Modify: `crates/aura-engine/src/{blueprint.rs,mc.rs,sweep.rs,walkforward.rs}` + test fixtures — thread `selection: None`
|
||||
- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112` — thread `selection: None`
|
||||
- Modify: `crates/aura-registry/src/compat.rs:27,65,67` — mirror + destructure + build
|
||||
- Modify: `crates/aura-registry/src/lib.rs` — `resolve_metric`/`metric_value` refactor, `optimize_deflated` + helpers; thread fixture `:239`
|
||||
- Modify: `crates/aura-registry/src/trace_store.rs:250` — thread `selection: None`
|
||||
- Modify: `crates/aura-cli/src/main.rs:26,163,1795,1810,2178` — import, `make_manifest`, wiring, display
|
||||
- Modify: `crates/aura-ingest/{examples,tests}/*` — thread `selection: None` (6 sites)
|
||||
- Test: new serde / extraction / golden / statistic tests in the files above
|
||||
|
||||
---
|
||||
|
||||
### Task 1: aura-engine — record types, manifest field, engine-internal threading
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs` (types beside `RMetrics` ~:44-86; `RunManifest` :419-434; fixtures :779,:1154,:1182,:1202)
|
||||
- Modify: `crates/aura-engine/src/lib.rs:64-68`
|
||||
- Modify: `crates/aura-engine/src/{blueprint.rs:929, mc.rs:224, mc.rs:247, sweep.rs:643, walkforward.rs:288}`
|
||||
- Modify: `crates/aura-engine/tests/random_sweep_e2e.rs:112`
|
||||
|
||||
- [ ] **Step 1: Add the record types** in `report.rs`, immediately before `pub struct RMetrics` (~:44)
|
||||
|
||||
```rust
|
||||
/// Which selection objective produced the record (additive provenance, C23).
|
||||
/// `Argmax` is the bare-best pick (cycle 0076). The enum is deliberately left
|
||||
/// open: cycle 0145 adds a `Plateau*` variant on this same field, so peak-vs-
|
||||
/// plateau runs stay distinguishable.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
}
|
||||
|
||||
/// Selection-provenance for a sweep winner: how its metric was deflated for the
|
||||
/// number of configurations it beat. Additive — recorded, never re-ranking (C23).
|
||||
/// `overfit_probability` is the empirical data-snooping p-value (R arm only);
|
||||
/// `None` on the `total_pips` dispersion-floor arm.
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String,
|
||||
pub n_trials: usize,
|
||||
pub raw_winner_metric: f64,
|
||||
pub deflated_score: f64,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
pub mode: SelectionMode,
|
||||
pub n_resamples: usize,
|
||||
pub block_len: usize,
|
||||
pub seed: u64,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the `selection` field** to `RunManifest` (after `pub broker: String,`, ~:433)
|
||||
|
||||
```rust
|
||||
/// Selection provenance, present only on a sweep/walk-forward winner; a
|
||||
/// standalone run and every pre-0076 line read back as `None`. One-directional
|
||||
/// serde widening (C14/C18), identical idiom to `RunMetrics.r`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Thread `selection: None` into every engine-internal `RunManifest { … }` literal.** Add `selection: None,` to the struct literal at each of: `blueprint.rs:929`, `mc.rs:224`, `mc.rs:247`, `report.rs:779`, `report.rs:1154`, `report.rs:1182`, `report.rs:1202`, `sweep.rs:643`, `walkforward.rs:288`, `tests/random_sweep_e2e.rs:112`. (Each is a `RunManifest { commit: …, …, broker: … }` body; append `selection: None,`.)
|
||||
|
||||
- [ ] **Step 4: Export the new types** — extend the `pub use report::{…}` list at `lib.rs:64-68` with `FamilySelection, SelectionMode` (keep alphabetical/existing order).
|
||||
|
||||
- [ ] **Step 5: Compile-gate the engine**
|
||||
|
||||
Run: `cargo test -p aura-engine --no-run`
|
||||
Expected: builds with 0 errors (all engine-internal literals threaded).
|
||||
|
||||
- [ ] **Step 6: Add serde tests** in `report.rs`'s `#[cfg(test)] mod tests` (Timestamp is already in scope there)
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn runmanifest_without_selection_field_deserialises_to_none() {
|
||||
let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#;
|
||||
let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads");
|
||||
assert!(m.selection.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runmanifest_without_selection_serialises_without_the_key() {
|
||||
let m = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None,
|
||||
};
|
||||
assert!(!serde_json::to_string(&m).unwrap().contains("selection"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn family_selection_round_trips_on_the_manifest() {
|
||||
let sel = FamilySelection {
|
||||
selection_metric: "sqn_normalized".into(), n_trials: 4, raw_winner_metric: 1.83,
|
||||
deflated_score: 0.21, overfit_probability: Some(0.06), mode: SelectionMode::Argmax,
|
||||
n_resamples: 1000, block_len: 5, seed: 42,
|
||||
};
|
||||
let m = RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: Some(sel.clone()),
|
||||
};
|
||||
let back: RunManifest = serde_json::from_str(&serde_json::to_string(&m).unwrap()).unwrap();
|
||||
assert_eq!(back.selection, Some(sel));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Run the engine tests (incl. the existing serde/round-trip goldens)**
|
||||
|
||||
Run: `cargo test -p aura-engine`
|
||||
Expected: PASS — the three new tests pass; the existing `to_json_equals_serde_disk_shape`, `runreport_serde_round_trips`, `to_json_renders_the_canonical_form` stay green (unstamped manifests emit no `selection` key, byte-identical).
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Downstream compile completion — compat mirror + remaining literals
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/compat.rs:13,27,65,67`
|
||||
- Modify: `crates/aura-registry/src/lib.rs:239` (fixture); `crates/aura-registry/src/trace_store.rs:250`
|
||||
- Modify: `crates/aura-cli/src/main.rs:163` (`make_manifest`)
|
||||
- Modify: `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`, `examples/ger40_breakout_walkforward.rs:73`, `examples/shared/breakout_real.rs:390`, `tests/ger40_breakout_world.rs:65`, `tests/real_bars.rs:82`, `tests/streaming_seam.rs:82`
|
||||
|
||||
- [ ] **Step 1: Extend the compat import** — `compat.rs:13`, add `FamilySelection`:
|
||||
|
||||
```rust
|
||||
use aura_engine::{FamilySelection, RunManifest, RunMetrics, RunReport, Scalar, Timestamp};
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Mirror the field** in `RunManifestRead` (`compat.rs:27-33`), after `broker: String,`:
|
||||
|
||||
```rust
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>,
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Fix the destructure + build** (`compat.rs:65-72`) — this is the load-bearing compile tripwire:
|
||||
|
||||
```rust
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
|
||||
RunReport {
|
||||
manifest: RunManifest {
|
||||
commit,
|
||||
params: params.into_iter().map(|(name, v)| (name, v.0)).collect(),
|
||||
window,
|
||||
seed,
|
||||
broker,
|
||||
selection,
|
||||
},
|
||||
metrics: r.metrics,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Thread `selection: None`** into the remaining literals: `aura-registry/src/lib.rs:239`, `aura-registry/src/trace_store.rs:250`, `aura-cli/src/main.rs:163`, and the six `aura-ingest` sites listed in **Files**.
|
||||
|
||||
- [ ] **Step 5: Workspace compile-gate**
|
||||
|
||||
Run: `cargo check --workspace --all-targets`
|
||||
Expected: 0 errors (all 21 literals + the compat destructure threaded).
|
||||
|
||||
- [ ] **Step 6: Add a compat lift test** in `compat.rs`'s test module (or `lib.rs` tests) — a legacy line *with* a selection block lifts it:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn load_lifts_a_selection_block_through_the_compat_mirror() {
|
||||
let line = r#"{"manifest":{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b","selection":{"selection_metric":"sqn_normalized","n_trials":4,"raw_winner_metric":1.8,"deflated_score":0.2,"overfit_probability":0.06,"mode":"Argmax","n_resamples":1000,"block_len":5,"seed":42}},"metrics":{"total_pips":0.0,"max_drawdown":0.0,"bias_sign_flips":0}}"#;
|
||||
let rep: RunReport = serde_json::from_str::<crate::compat::RunReportRead>(line).unwrap().into();
|
||||
let sel = rep.manifest.selection.expect("selection lifted");
|
||||
assert_eq!(sel.n_trials, 4);
|
||||
assert_eq!(sel.mode, aura_engine::SelectionMode::Argmax);
|
||||
}
|
||||
```
|
||||
|
||||
(If `RunReportRead` is not reachable from the chosen test module, place this test inside `compat.rs`'s own `#[cfg(test)] mod tests`. The existing `append_then_load_round_trips_in_order` and `load_reads_a_legacy_bare_float_params_line` must stay green.)
|
||||
|
||||
- [ ] **Step 7: Run registry tests**
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS — the new lift test passes; `append_then_load_round_trips_in_order`, `load_reads_a_legacy_bare_float_params_line`, `lineage_round_trips_one_family` stay green.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: aura-engine — extract `resample_block`, make `SplitMix64` public
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/mc.rs:173-199` (extract), `:14` (import unchanged)
|
||||
- Modify: `crates/aura-engine/src/harness.rs:143` (visibility)
|
||||
- Modify: `crates/aura-engine/src/lib.rs` (re-export `SplitMix64`, `resample_block`)
|
||||
|
||||
- [ ] **Step 1: Make `SplitMix64` public** — at `harness.rs:143`, change `pub(crate) struct SplitMix64` to `pub struct SplitMix64`, and ensure its `new` and `next_u64` methods are `pub` (widen any `pub(crate)` on them).
|
||||
|
||||
- [ ] **Step 2: Re-export** from `lib.rs` — add `pub use harness::SplitMix64;` (beside the existing `harness` re-exports) and `pub use mc::{… resample_block …}` once Step 3 lands.
|
||||
|
||||
- [ ] **Step 3: Extract the kernel** — insert `resample_block` above `r_bootstrap` (~:169) and call it from the loop:
|
||||
|
||||
```rust
|
||||
/// One moving-block resample of `rs` to length `rs.len()` (non-circular; the final
|
||||
/// block is truncated so the resample has exactly `n` values). `block_len` must be
|
||||
/// pre-clamped to `[1, rs.len()]` by the caller. Pure given the `rng` state — the
|
||||
/// shared kernel behind `r_bootstrap` and the trials-deflation reality-check (C1).
|
||||
pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64> {
|
||||
let n = rs.len();
|
||||
let mut sample: Vec<f64> = Vec::with_capacity(n);
|
||||
while sample.len() < n {
|
||||
let start = (rng.next_u64() % (n - block_len + 1) as u64) as usize;
|
||||
let take = block_len.min(n - sample.len());
|
||||
sample.extend_from_slice(&rs[start..start + take]);
|
||||
}
|
||||
sample
|
||||
}
|
||||
```
|
||||
|
||||
Then replace the loop body at `:187-194` so `r_bootstrap` calls it:
|
||||
|
||||
```rust
|
||||
for _ in 0..n_resamples {
|
||||
let sample = resample_block(rs, block_len, &mut rng);
|
||||
means.push(sample.iter().sum::<f64>() / n as f64);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify the kernel is byte-identical**
|
||||
|
||||
Run: `cargo test -p aura-engine r_bootstrap`
|
||||
Expected: PASS — `r_bootstrap_is_deterministic_given_seed`, `r_bootstrap_single_block_equals_full_series_mean`, `r_bootstrap_block_len_is_clamped_to_series_len`, `r_bootstrap_empty_series_is_all_zero` all green (proves extraction, not change).
|
||||
|
||||
---
|
||||
|
||||
### Task 4: aura-engine — `inv_norm_cdf` + `expected_max_of_normals`
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-engine/src/report.rs` (module-level fns beside `summarize`/`r_metrics_from_rs`)
|
||||
- Modify: `crates/aura-engine/src/lib.rs:64-68` (export)
|
||||
|
||||
- [ ] **Step 1: Write the failing test** in `report.rs` tests
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn inv_norm_cdf_matches_known_quantiles() {
|
||||
assert!((inv_norm_cdf(0.975) - 1.959964).abs() < 1e-3);
|
||||
assert!(inv_norm_cdf(0.5).abs() < 1e-9);
|
||||
assert!((inv_norm_cdf(0.1) + inv_norm_cdf(0.9)).abs() < 1e-9); // symmetry
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn expected_max_of_normals_is_zero_at_one_and_monotone() {
|
||||
assert_eq!(expected_max_of_normals(1), 0.0);
|
||||
assert_eq!(expected_max_of_normals(0), 0.0);
|
||||
let (e2, e4, e10, e100) = (
|
||||
expected_max_of_normals(2), expected_max_of_normals(4),
|
||||
expected_max_of_normals(10), expected_max_of_normals(100),
|
||||
);
|
||||
assert!(e2 < e4 && e4 < e10 && e10 < e100); // strictly increasing
|
||||
assert!((1.0..1.1).contains(&e4)); // ~1.05 (not the √(2 ln 4)=1.66 asymptote)
|
||||
assert!((2.4..2.7).contains(&e100)); // ~2.53
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `cargo test -p aura-engine expected_max_of_normals`
|
||||
Expected: FAIL to compile — `cannot find function 'expected_max_of_normals'`.
|
||||
|
||||
- [ ] **Step 3: Write the implementation** (module-level in `report.rs`)
|
||||
|
||||
```rust
|
||||
/// Inverse standard-normal CDF (quantile function), Acklam's rational
|
||||
/// approximation — absolute error < ~1.2e-9 over `p ∈ (0,1)`. Pure (C1).
|
||||
/// Callers pass strictly-interior `p` (`k >= 2` keeps the argument off the
|
||||
/// boundaries); the boundary branches return ±inf as a defined limit.
|
||||
pub fn inv_norm_cdf(p: f64) -> f64 {
|
||||
const A: [f64; 6] = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
|
||||
1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
|
||||
const B: [f64; 5] = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
|
||||
6.680131188771972e+01, -1.328068155288572e+01];
|
||||
const C: [f64; 6] = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
|
||||
-2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
|
||||
const D: [f64; 4] = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
|
||||
3.754408661907416e+00];
|
||||
const P_LOW: f64 = 0.02425;
|
||||
if p <= 0.0 { return f64::NEG_INFINITY; }
|
||||
if p >= 1.0 { return f64::INFINITY; }
|
||||
if p < P_LOW {
|
||||
let q = (-2.0 * p.ln()).sqrt();
|
||||
(((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5])
|
||||
/ ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0)
|
||||
} else if p <= 1.0 - P_LOW {
|
||||
let q = p - 0.5;
|
||||
let r = q * q;
|
||||
(((((A[0]*r + A[1])*r + A[2])*r + A[3])*r + A[4])*r + A[5]) * q
|
||||
/ (((((B[0]*r + B[1])*r + B[2])*r + B[3])*r + B[4])*r + 1.0)
|
||||
} else {
|
||||
let q = (-2.0 * (1.0 - p).ln()).sqrt();
|
||||
-(((((C[0]*q + C[1])*q + C[2])*q + C[3])*q + C[4])*q + C[5])
|
||||
/ ((((D[0]*q + D[1])*q + D[2])*q + D[3])*q + 1.0)
|
||||
}
|
||||
}
|
||||
|
||||
/// Expected maximum of `k` i.i.d. standard normals — the hurdle a search of `k`
|
||||
/// configurations clears by chance alone. `k <= 1` -> `0.0` (a single trial has no
|
||||
/// multiple-comparison inflation, and guards the `Φ⁻¹(1 - 1/k) -> Φ⁻¹(0)`
|
||||
/// divergence). For `k >= 2`:
|
||||
/// `(1 - γ)·Φ⁻¹(1 - 1/k) + γ·Φ⁻¹(1 - 1/(k·e))`, `γ = 0.5772156649…`. Pure (C1).
|
||||
pub fn expected_max_of_normals(k: usize) -> f64 {
|
||||
if k <= 1 {
|
||||
return 0.0;
|
||||
}
|
||||
const GAMMA: f64 = 0.577_215_664_901_532_9;
|
||||
let kf = k as f64;
|
||||
(1.0 - GAMMA) * inv_norm_cdf(1.0 - 1.0 / kf)
|
||||
+ GAMMA * inv_norm_cdf(1.0 - 1.0 / (kf * std::f64::consts::E))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Export** — add `expected_max_of_normals, inv_norm_cdf` to the `pub use report::{…}` list at `lib.rs:64-68`.
|
||||
|
||||
- [ ] **Step 5: Run to verify it passes**
|
||||
|
||||
Run: `cargo test -p aura-engine expected_max_of_normals`
|
||||
Expected: PASS. Then `cargo test -p aura-engine inv_norm_cdf` — PASS.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: aura-registry — `optimize_deflated` + the value-read refactor
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-registry/src/lib.rs:20` (imports), `:125-165` (refactor), beside `:186` (`optimize_deflated` + helpers)
|
||||
|
||||
- [ ] **Step 1: Extend imports** (`lib.rs:20`)
|
||||
|
||||
```rust
|
||||
use aura_engine::{
|
||||
expected_max_of_normals, r_metrics_from_rs, resample_block, FamilySelection, MetricStats,
|
||||
RunReport, SelectionMode, SplitMix64, SweepFamily, SweepPoint,
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add fixture helpers** to `lib.rs`'s `#[cfg(test)] mod tests` (import what is missing — `use aura_engine::{RunManifest, RunMetrics, RunReport, Timestamp};`). Members are built from a trade-R slice via `r_metrics_from_rs`, so every `RMetrics` field is populated consistently:
|
||||
|
||||
```rust
|
||||
fn member(total_pips: f64, trade_rs: Vec<f64>) -> SweepPoint {
|
||||
SweepPoint {
|
||||
params: vec![],
|
||||
report: RunReport {
|
||||
manifest: RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None,
|
||||
},
|
||||
metrics: RunMetrics {
|
||||
total_pips, max_drawdown: 0.0, bias_sign_flips: 0,
|
||||
r: Some(aura_engine::r_metrics_from_rs(&trade_rs)),
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn fixture_family_with_r() -> SweepFamily {
|
||||
SweepFamily { space: vec![], points: vec![
|
||||
member(10.0, vec![1.0, -0.5, 0.8, -0.2, 0.6]),
|
||||
member(30.0, vec![2.0, 1.5, -0.3, 1.0, 0.9]), // best by every metric here
|
||||
member(5.0, vec![-0.5, 0.2, -0.8, 0.1, -0.3]),
|
||||
]}
|
||||
}
|
||||
|
||||
fn fixture_family_one_strong_edge() -> SweepFamily {
|
||||
SweepFamily { space: vec![], points: vec![
|
||||
member(1.0, vec![2.0, 2.5, 1.8, 2.2, 2.1, 1.9]), // strong, consistent +R edge
|
||||
member(1.0, vec![0.05, -0.1, 0.0, 0.1, -0.05, 0.02]), // ~zero-edge noise
|
||||
member(1.0, vec![-0.1, 0.1, 0.0, -0.05, 0.05, 0.0]),
|
||||
]}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2a: Write the failing tests** (same module) — the C23 tripwire + determinism + arm behaviour:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn optimize_deflated_winner_is_byte_identical_to_optimize() {
|
||||
let fam = fixture_family_with_r(); // helper: ≥3 members, varied sqn_normalized + trade_rs
|
||||
for metric in ["total_pips", "sqn_normalized", "expectancy_r"] {
|
||||
let plain = optimize(&fam, metric).unwrap();
|
||||
let (defl, _) = optimize_deflated(&fam, metric, 200, 3, 7).unwrap();
|
||||
assert_eq!(defl, plain, "deflation must not change the winner ({metric})");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_deflated_is_reproducible_from_its_seed() {
|
||||
let fam = fixture_family_with_r();
|
||||
let a = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1;
|
||||
let b = optimize_deflated(&fam, "sqn_normalized", 500, 4, 42).unwrap().1;
|
||||
assert_eq!(a, b);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_deflated_total_pips_arm_floors_without_a_probability() {
|
||||
let fam = fixture_family_with_r();
|
||||
let sel = optimize_deflated(&fam, "total_pips", 100, 3, 1).unwrap().1;
|
||||
assert!(sel.overfit_probability.is_none());
|
||||
assert_eq!(sel.selection_metric, "total_pips");
|
||||
assert!(sel.deflated_score <= sel.raw_winner_metric); // dispersion floor ≤ raw
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimize_deflated_overfit_probability_is_low_for_a_clear_edge() {
|
||||
// A family whose winner has a strong real R-edge over near-zero-edge peers:
|
||||
// the centred best-of-K null rarely reaches the winner's raw metric.
|
||||
let fam = fixture_family_one_strong_edge();
|
||||
let sel = optimize_deflated(&fam, "expectancy_r", 1000, 1, 9).unwrap().1;
|
||||
assert!(sel.overfit_probability.unwrap() < 0.2, "p={:?}", sel.overfit_probability);
|
||||
assert!(sel.deflated_score > 0.0);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2b: Run to verify it fails**
|
||||
|
||||
Run: `cargo test -p aura-registry optimize_deflated`
|
||||
Expected: FAIL to compile — `cannot find function 'optimize_deflated'`.
|
||||
|
||||
- [ ] **Step 3: Refactor `metric_cmp`'s value-read into shared helpers** — extract the name→enum match (`:128-138`) and the value read (`:141-164`):
|
||||
|
||||
```rust
|
||||
/// Resolve a metric NAME to the closed `Metric` kind (the single name→kind map).
|
||||
fn resolve_metric(name: &str) -> Result<Metric, RegistryError> {
|
||||
Ok(match name {
|
||||
"total_pips" => Metric::TotalPips,
|
||||
"max_drawdown" => Metric::MaxDrawdown,
|
||||
"bias_sign_flips" | "exposure_sign_flips" => Metric::BiasSignFlips,
|
||||
"sqn" => Metric::Sqn,
|
||||
"sqn_normalized" => Metric::SqnNormalized,
|
||||
"expectancy_r" => Metric::ExpectancyR,
|
||||
"net_expectancy_r" => Metric::NetExpectancyR,
|
||||
other => return Err(RegistryError::UnknownMetric(other.to_string())),
|
||||
})
|
||||
}
|
||||
|
||||
/// The metric's scalar value for one report. R metrics live in the optional `r`
|
||||
/// block; a missing block reads `NEG_INFINITY` (the worst rank for the
|
||||
/// higher-is-better R keys), preserving `metric_cmp`'s documented contract.
|
||||
fn metric_value(rep: &RunReport, m: Metric) -> f64 {
|
||||
fn r_get(rep: &RunReport, f: impl Fn(&aura_engine::RMetrics) -> f64) -> f64 {
|
||||
rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY)
|
||||
}
|
||||
match m {
|
||||
Metric::TotalPips => rep.metrics.total_pips,
|
||||
Metric::MaxDrawdown => rep.metrics.max_drawdown,
|
||||
Metric::BiasSignFlips => rep.metrics.bias_sign_flips as f64,
|
||||
Metric::Sqn => r_get(rep, |r| r.sqn),
|
||||
Metric::SqnNormalized => r_get(rep, |r| r.sqn_normalized),
|
||||
Metric::ExpectancyR => r_get(rep, |r| r.expectancy_r),
|
||||
Metric::NetExpectancyR => r_get(rep, |r| r.net_expectancy_r),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Rewrite `metric_cmp` to resolve via `resolve_metric` and compare via `metric_value`, keeping the per-metric direction:
|
||||
|
||||
```rust
|
||||
fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Ordering, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
Ok(move |a: &RunReport, b: &RunReport| {
|
||||
let (va, vb) = (metric_value(a, m), metric_value(b, m));
|
||||
match m {
|
||||
// lower-is-better
|
||||
Metric::MaxDrawdown | Metric::BiasSignFlips => va.total_cmp(&vb),
|
||||
// higher-is-better
|
||||
_ => vb.total_cmp(&va),
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add `optimize_deflated` + helpers** beside `optimize` (~:194)
|
||||
|
||||
```rust
|
||||
fn is_r_metric(m: Metric) -> bool {
|
||||
matches!(m, Metric::Sqn | Metric::SqnNormalized | Metric::ExpectancyR | Metric::NetExpectancyR)
|
||||
}
|
||||
|
||||
fn member_trade_rs(rep: &RunReport) -> &[f64] {
|
||||
rep.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// Recompute the selected R metric from a trade-R slice (reuses the engine's
|
||||
/// `r_metrics_from_rs`, the same arithmetic `summarize_r` uses).
|
||||
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64 {
|
||||
let rm = r_metrics_from_rs(rs);
|
||||
match m {
|
||||
Metric::Sqn => rm.sqn,
|
||||
Metric::SqnNormalized => rm.sqn_normalized,
|
||||
Metric::ExpectancyR => rm.expectancy_r,
|
||||
Metric::NetExpectancyR => rm.net_expectancy_r,
|
||||
_ => unreachable!("member_metric_from_rs is R-only"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The centred best-of-K null-max distribution: each member's `trade_rs` is
|
||||
/// mean-subtracted (the no-edge null), then for each resample every member is
|
||||
/// moving-block-resampled (in odometer order, from a per-iteration seed) and the
|
||||
/// max recomputed metric across members is taken. Deterministic given `seed` (C1).
|
||||
fn null_best_of_k(family: &SweepFamily, m: Metric, n_resamples: usize, block_len: usize, seed: u64) -> Vec<f64> {
|
||||
let centred: Vec<Vec<f64>> = family.points.iter().map(|p| {
|
||||
let rs = member_trade_rs(&p.report);
|
||||
if rs.is_empty() { return Vec::new(); }
|
||||
let mean = rs.iter().sum::<f64>() / rs.len() as f64;
|
||||
rs.iter().map(|x| x - mean).collect()
|
||||
}).collect();
|
||||
(0..n_resamples).map(|i| {
|
||||
let mut rng = SplitMix64::new(seed ^ i as u64);
|
||||
centred.iter().fold(f64::NEG_INFINITY, |best, rs| {
|
||||
if rs.is_empty() { return best; }
|
||||
let bl = block_len.clamp(1, rs.len());
|
||||
let v = member_metric_from_rs(&resample_block(rs, bl, &mut rng), m);
|
||||
best.max(v)
|
||||
})
|
||||
}).collect()
|
||||
}
|
||||
|
||||
/// Sample standard deviation of the K members' metric values (the `total_pips`
|
||||
/// dispersion-floor arm). `< 2` members -> `0.0`.
|
||||
fn member_sd(family: &SweepFamily, m: Metric) -> f64 {
|
||||
let vals: Vec<f64> = family.points.iter().map(|p| metric_value(&p.report, m)).collect();
|
||||
let n = vals.len();
|
||||
if n < 2 { return 0.0; }
|
||||
let mean = vals.iter().sum::<f64>() / n as f64;
|
||||
(vals.iter().map(|v| (v - mean).powi(2)).sum::<f64>() / (n - 1) as f64).sqrt()
|
||||
}
|
||||
|
||||
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
|
||||
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23).
|
||||
/// R arm: a centred moving-block reality-check (`overfit_probability` =
|
||||
/// `(count(null ≥ raw) + 1)/(n + 1)`, `deflated_score` = `raw − p95(null)`).
|
||||
/// `total_pips` arm: a closed-form expected-max-of-K dispersion floor, no
|
||||
/// probability. Deterministic given `seed` (C1).
|
||||
pub fn optimize_deflated(
|
||||
family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let winner = optimize(family, metric)?;
|
||||
let m = resolve_metric(metric)?;
|
||||
let k = family.points.len();
|
||||
let raw = metric_value(&winner.report, m);
|
||||
|
||||
let (deflated_score, overfit_probability) = if is_r_metric(m) {
|
||||
let null_max = null_best_of_k(family, m, n_resamples, block_len, seed);
|
||||
let p95 = MetricStats::from_values(&null_max).p95;
|
||||
let over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
|
||||
/ (n_resamples + 1) as f64;
|
||||
(raw - p95, Some(over))
|
||||
} else {
|
||||
(raw - member_sd(family, m) * expected_max_of_normals(k), None)
|
||||
};
|
||||
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
deflated_score, overfit_probability, mode: SelectionMode::Argmax,
|
||||
n_resamples, block_len, seed,
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the new + existing selection tests**
|
||||
|
||||
Run: `cargo test -p aura-registry`
|
||||
Expected: PASS — the four new `optimize_deflated*` tests pass; the existing `optimize_picks_the_max_metric_point_ties_to_earliest`, `rank_by_sqn_normalized_orders_members_descending`, `rank_r_metric_sorts_none_member_last`, `rank_by_unknown_metric_is_an_error` stay green (the `metric_cmp` refactor is behaviour-preserving).
|
||||
|
||||
---
|
||||
|
||||
### Task 6: aura-cli — wire the call sites + the display line
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/aura-cli/src/main.rs:26-27` (import), constants block, `:1793-1813` (`walkforward_family`), `:2178-2204` (`runs_family`)
|
||||
- Modify: `crates/aura-cli/tests/cli_run.rs` (assert the new display + keep behaviour goldens green)
|
||||
|
||||
- [ ] **Step 1: Import `optimize_deflated`** — add it to the `use aura_registry::{… optimize, …}` list at `:26-27`.
|
||||
|
||||
- [ ] **Step 2: Add the module constants** near the other `const`s at the top of `main.rs`:
|
||||
|
||||
```rust
|
||||
/// Trials-deflation resampling budget for walk-forward winner selection (recorded
|
||||
/// on each winner's manifest, so the figure is reproducible by re-run). CLI flags
|
||||
/// for these are a deferred refinement.
|
||||
const DEFLATION_N_RESAMPLES: usize = 1000;
|
||||
const DEFLATION_BLOCK_LEN: usize = 5;
|
||||
const DEFLATION_SEED: u64 = 0xDEF1_A7ED;
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Wire the `total_pips` arm** (`:1793-1803`)
|
||||
|
||||
```rust
|
||||
walk_forward(roller, space, |w: WindowBounds| {
|
||||
let is_family = sweep_over(w.is.0, w.is.1, data);
|
||||
let (best, selection) = optimize_deflated(
|
||||
&is_family, "total_pips", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED,
|
||||
).expect("total_pips is a known metric");
|
||||
let (oos_equity, mut oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
WindowRun { chosen_params: best.params, oos_equity, oos_report }
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Wire the `sqn_normalized` arm** (`:1808-1812`)
|
||||
|
||||
```rust
|
||||
walk_forward(roller, space, |w: WindowBounds| {
|
||||
let is_family = stage1_r_sweep_over(w.is.0, w.is.1, data, grid);
|
||||
let (best, selection) = optimize_deflated(
|
||||
&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED,
|
||||
).expect("sqn_normalized is a known metric");
|
||||
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection);
|
||||
WindowRun { chosen_params: best.params, oos_equity, oos_report }
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add the display line** in `runs_family`, right after the per-member `println!("{}", report.to_json())` (`:2202`)
|
||||
|
||||
```rust
|
||||
if let Some(sel) = &report.manifest.selection {
|
||||
match sel.overfit_probability {
|
||||
Some(p) => println!(" deflated={:.4} P(overfit)={:.4}", sel.deflated_score, p),
|
||||
None => println!(" deflated={:.4}", sel.deflated_score),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Build the CLI**
|
||||
|
||||
Run: `cargo build -p aura-cli`
|
||||
Expected: 0 errors.
|
||||
|
||||
- [ ] **Step 7: Run the CLI suite; reconcile goldens**
|
||||
|
||||
Run: `cargo test -p aura-cli`
|
||||
Expected: PASS, with this reconciliation rule applied:
|
||||
- **Unstamped families stay byte-identical** — `sweep_*` and `mc_*` members carry no `selection` (the selector runs only in walk-forward), so `sweep_strategy_stage1_r_ranks_a_family_by_sqn`/`_by_sqn_normalized`, the `mc_*` goldens, and any standalone-run golden MUST stay green unchanged. If one of these changed, it is a bug — stop and fix (do not edit the golden).
|
||||
- **Stamped walk-forward output may legitimately gain the block** — `walkforward_strategy_stage1_r_reports_oos_r`, `walkforward_bare_sma_summary_has_no_oos_r`, and `runs_families_list_and_per_family_rank_across_invocations`: if (and only if) their asserted output now includes an OOS winner's `selection`/`deflated=` line, update the expected output to the new correct value — this is the feature's intended addition, not a regression. The OOS metric numbers (R, pips) must be unchanged (C23 — same winner).
|
||||
|
||||
- [ ] **Step 8: Add an explicit display assertion** to `cli_run.rs`, named `runs_family_rank_shows_deflated_line` (the name makes the filter below resolve) — run a small synthetic `walkforward --strategy stage1-r`, persist it, and assert the `runs family <id> rank sqn_normalized` stdout contains both `deflated=` and `P(overfit)=`. Model it on the existing `runs_families_list_and_per_family_rank_across_invocations` harness.
|
||||
|
||||
Run: `cargo test -p aura-cli runs_family_rank_shows_deflated`
|
||||
Expected: PASS — the filter matches exactly the new test; the deflated/P(overfit) line is asserted present for a stamped walk-forward family.
|
||||
|
||||
- [ ] **Step 9: Workspace gate + lint**
|
||||
|
||||
Run: `cargo test --workspace`
|
||||
Expected: PASS (all goldens reconciled).
|
||||
|
||||
Run: `cargo clippy --workspace --all-targets -- -D warnings`
|
||||
Expected: 0 warnings.
|
||||
@@ -1,366 +0,0 @@
|
||||
# Deflate the sweep winner's metric for the number of trials — Design Spec
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Draft — awaiting user spec review
|
||||
**Authors:** orchestrator + Claude
|
||||
|
||||
Cycle 0076 — first cycle of the milestone *Inferential validation (defend against
|
||||
false discovery at sweep scale)*. Closes #144 (refs the milestone).
|
||||
|
||||
## Goal
|
||||
|
||||
The World's in-sample sweep picks the single best member by a bare argmax
|
||||
(`optimize`, `aura-registry/src/lib.rs:186`). The more members a family contains,
|
||||
the higher the best member's metric climbs by chance alone, so the winner's
|
||||
`sqn_normalized` / `expectancy_r` is reported at face value regardless of how many
|
||||
configurations competed for the top slot — the family-scale false-discovery the
|
||||
milestone exists to defend against.
|
||||
|
||||
This cycle adds a **selection-provenance record** that, *without changing which
|
||||
member wins* (additive only — C23), reports how inflated the winner's metric is by
|
||||
the size of the search it won:
|
||||
|
||||
- a **deflated score** — the winner's metric minus the edge a search of the same
|
||||
size produces under a no-edge null;
|
||||
- an **overfit probability** — the chance a search of `K` members yields the
|
||||
observed winner under that null;
|
||||
|
||||
both stamped on the surviving out-of-sample winner's manifest (C18), reproducible
|
||||
from a recorded seed (C1).
|
||||
|
||||
This is framed as aura's own selection-honesty discipline; it introduces no new
|
||||
ranking key and never re-orders a family.
|
||||
|
||||
## Architecture
|
||||
|
||||
One new family-level selector sits **above** the closed `metric_cmp` / `Metric`
|
||||
direction source, beside `optimize`, in the registry that already owns selection
|
||||
(`aura-registry`). It calls `optimize` for the byte-identical argmax winner, then
|
||||
computes the deflation as an additive companion:
|
||||
|
||||
- **R arm** (`selection_metric ∈ {sqn, sqn_normalized, expectancy_r,
|
||||
net_expectancy_r}`): an empirical **centred moving-block reality-check**. Each
|
||||
member's per-trade R series (`metrics.r.trade_rs`, the in-memory conduit cycle
|
||||
0075 already populates) is mean-subtracted to impose the no-edge null, then
|
||||
resampled with the **same** moving-block kernel `r_bootstrap` uses (a
|
||||
`SplitMix64` block draw — `aura-engine/src/mc.rs`); the per-resample best-of-`K`
|
||||
member metric forms the null-max distribution the winner is judged against.
|
||||
- **`total_pips` arm** (the legacy SmaCross walk-forward, no `trade_rs`): a
|
||||
closed-form **dispersion-floor** deflation — the winner minus the
|
||||
expected-maximum-of-`K` inflation implied by the cross-member metric dispersion.
|
||||
|
||||
The record rides inside each member's existing `RunReport.manifest` as a new
|
||||
optional field, using the one-directional serde widening already proven by
|
||||
`RunMetrics.r`, so the family store (`FamilyRunRecord` / `append_family`) and every
|
||||
legacy `runs.jsonl` / `families.jsonl` line are untouched.
|
||||
|
||||
The two walk-forward in-sample selection seams (`walkforward_family`,
|
||||
`aura-cli/src/main.rs:1795` total_pips arm, `:1810` R arm) call the new selector and
|
||||
stamp the record onto each window's chosen OOS report. The read-only
|
||||
`runs family … rank` display surfaces it.
|
||||
|
||||
**Crate placement (C9):** the record type `FamilySelection` is an `aura-engine` type
|
||||
(beside `RunManifest` / `RMetrics`), because `RunManifest` is an `aura-engine` type
|
||||
and `aura-registry` depends on `aura-engine` — defining it in the registry would
|
||||
invert the dependency. The selector `optimize_deflated` lives in `aura-registry`
|
||||
(which owns `optimize` / `metric_cmp`) and constructs the engine-side record.
|
||||
|
||||
## Concrete code shapes
|
||||
|
||||
### User-facing program (the acceptance evidence)
|
||||
|
||||
A Stage-1 R walk-forward over a real symbol, sweeping a `K = 2×2 = 4` in-sample
|
||||
grid per window — unchanged invocation, new provenance on the result:
|
||||
|
||||
```console
|
||||
$ aura walkforward --strategy stage1-r --real EURUSD --from 1672531200000 --to 1735689600000 \
|
||||
--fast 50,100 --slow 200,400 --stop-length 14 --stop-k 2.0
|
||||
```
|
||||
|
||||
Each window's chosen OOS run now persists a `selection` block on its manifest:
|
||||
|
||||
```jsonc
|
||||
// one member line in runs/families/<id> … families.jsonl
|
||||
"manifest": {
|
||||
"commit": "c192dfd…",
|
||||
"params": [["signals.trend.fast.length", {"I64":100}], …],
|
||||
"window": [1704067200000000000, 1706745600000000000],
|
||||
"seed": 0,
|
||||
"broker": "sim-optimal(pip_size=1e-4)",
|
||||
"selection": {
|
||||
"selection_metric": "sqn_normalized",
|
||||
"n_trials": 4,
|
||||
"raw_winner_metric": 1.83,
|
||||
"deflated_score": 0.21, // winner − p95(null best-of-K); > 0 ⇒ survives
|
||||
"overfit_probability": 0.06, // (count(nullmax ≥ winner) + 1)/(n+1)
|
||||
"mode": "Argmax",
|
||||
"n_resamples": 1000,
|
||||
"block_len": 5,
|
||||
"seed": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The legacy `total_pips` SmaCross arm records the same block with
|
||||
`"selection_metric":"total_pips"` and **`overfit_probability` omitted** (the
|
||||
dispersion-floor arm computes no empirical probability); `selection_metric` plus the
|
||||
absent probability disambiguate the arm. A pre-0076 line carries no `selection`
|
||||
field and reads back as `None`, byte-unchanged.
|
||||
|
||||
The provenance is surfaced read-only beside the raw metric:
|
||||
|
||||
```console
|
||||
$ aura runs family stage1-r-3 rank sqn_normalized
|
||||
# … each member, best-first, now with: sqn_normalized=1.83 deflated=0.21 P(overfit)=0.06
|
||||
```
|
||||
|
||||
### New record type (`aura-engine/src/report.rs`, beside `RMetrics`)
|
||||
|
||||
```rust
|
||||
/// Selection-provenance for a sweep winner: how its metric was deflated for the
|
||||
/// number of configurations it beat. Additive (C23) — recorded, never re-ranking.
|
||||
#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct FamilySelection {
|
||||
pub selection_metric: String, // the metric the family was optimised against
|
||||
pub n_trials: usize, // K — the family size the winner beat
|
||||
pub raw_winner_metric: f64, // the argmax winner's face-value metric (audit)
|
||||
pub deflated_score: f64, // trials-adjusted score (per-arm; see the selector)
|
||||
/// Empirical data-snooping p-value, R arm only; `None` on the dispersion-floor
|
||||
/// (`total_pips`) arm. `Option` (not a sentinel) keeps `PartialEq`/serde clean.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub overfit_probability: Option<f64>,
|
||||
pub mode: SelectionMode,
|
||||
pub n_resamples: usize, // resample provenance — reproduces the statistic
|
||||
pub block_len: usize,
|
||||
pub seed: u64,
|
||||
}
|
||||
|
||||
/// Which selection objective produced the record. `Argmax` is the bare-best
|
||||
/// pick (this cycle). The enum is left **open**: cycle 0145 adds a `Plateau*`
|
||||
/// variant on this same field, so peak-vs-plateau runs stay distinguishable.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
|
||||
pub enum SelectionMode {
|
||||
Argmax,
|
||||
}
|
||||
```
|
||||
|
||||
### `RunManifest` — 5 → 6 fields (`aura-engine/src/report.rs:420`)
|
||||
|
||||
```rust
|
||||
// before
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
}
|
||||
// after — one-directional widening, identical idiom to RunMetrics.r
|
||||
pub struct RunManifest {
|
||||
pub commit: String,
|
||||
pub params: Vec<(String, Scalar)>,
|
||||
pub window: (Timestamp, Timestamp),
|
||||
pub seed: u64,
|
||||
pub broker: String,
|
||||
/// Selection provenance, present only on a sweep/walk-forward winner; a
|
||||
/// standalone run and every pre-0076 line read back as `None`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub selection: Option<FamilySelection>,
|
||||
}
|
||||
```
|
||||
|
||||
### `compat.rs` legacy read-path — mirror + destructure (`aura-registry/src/compat.rs`)
|
||||
|
||||
```rust
|
||||
// RunManifestRead (compat.rs:27) gains the same optional field …
|
||||
struct RunManifestRead {
|
||||
commit: String,
|
||||
params: Vec<(String, ScalarRead)>,
|
||||
window: (Timestamp, Timestamp),
|
||||
seed: u64,
|
||||
broker: String,
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>, // legacy lines → None
|
||||
}
|
||||
// … and BOTH the destructure (compat.rs:65) and the RunManifest build lift it:
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
|
||||
RunManifest { commit, params: /* … */, window, seed, broker, selection }
|
||||
```
|
||||
|
||||
(The hard 5-field destructure fails to compile the moment the 6th field exists —
|
||||
this edit is the load-bearing, most-forgettable change; pinned by the existing
|
||||
compat round-trip tests.)
|
||||
|
||||
### The selector (`aura-registry/src/lib.rs`, beside `optimize`)
|
||||
|
||||
```rust
|
||||
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
|
||||
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23);
|
||||
/// `FamilySelection` is the new companion record. Deterministic given `seed`.
|
||||
pub fn optimize_deflated(
|
||||
family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
|
||||
) -> Result<(SweepPoint, FamilySelection), RegistryError> {
|
||||
let winner = optimize(family, metric)?; // unchanged argmax
|
||||
let m = resolve_metric(metric)?; // shared with metric_cmp
|
||||
let k = family.points.len();
|
||||
let raw = metric_value(&winner.report, m); // extracted from metric_cmp
|
||||
|
||||
let (deflated_score, overfit_probability) = if let Some(get_rs) = r_series_of(m) {
|
||||
// R arm: centred moving-block reality-check over each member's trade_rs.
|
||||
let null_max = null_best_of_k(family, m, get_rs, n_resamples, block_len, seed);
|
||||
let stats = MetricStats::from_values(&null_max); // reuse mc.rs
|
||||
let p_over = (null_max.iter().filter(|&&x| x >= raw).count() + 1) as f64
|
||||
/ (n_resamples + 1) as f64; // (z+1)/(N+1)
|
||||
(raw - stats.p95, Some(p_over))
|
||||
} else {
|
||||
// total_pips arm: closed-form expected-max-of-K dispersion floor.
|
||||
let sd = member_sd(family, m);
|
||||
(raw - sd * expected_max_of_normals(k), None)
|
||||
};
|
||||
|
||||
Ok((winner.clone(), FamilySelection {
|
||||
selection_metric: metric.to_string(), n_trials: k, raw_winner_metric: raw,
|
||||
deflated_score, overfit_probability, mode: SelectionMode::Argmax,
|
||||
n_resamples, block_len, seed,
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
Supporting shapes (planner fills the bodies):
|
||||
|
||||
```rust
|
||||
// aura-registry: refactor metric_cmp's value-read into a shared helper so the
|
||||
// comparator and the deflation read one metric source (direction stays in metric_cmp).
|
||||
fn resolve_metric(name: &str) -> Result<Metric, RegistryError>; // the existing name→enum match
|
||||
fn metric_value(rep: &RunReport, m: Metric) -> f64; // r_get / total_pips read, lifted
|
||||
fn r_series_of(m: Metric) -> Option<fn(&RunReport) -> &[f64]>; // Some for R metrics → trade_rs; None for total_pips/dd/flips
|
||||
|
||||
// null-max distribution: for i in 0..n_resamples, rng_i = SplitMix64::new(seed ^ i);
|
||||
// per member k, resample the CENTRED series, recompute the same metric; M_i = max_k.
|
||||
fn null_best_of_k(family, m, get_rs, n_resamples, block_len, seed) -> Vec<f64>;
|
||||
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64; // r_metrics_from_rs(rs) → project field
|
||||
|
||||
// aura-engine/src/mc.rs: extract the moving-block draw so the kernel is shared,
|
||||
// byte-identical (r_bootstrap keeps its golden test).
|
||||
pub fn resample_block(rs: &[f64], block_len: usize, rng: &mut SplitMix64) -> Vec<f64>;
|
||||
|
||||
// aura-engine: a small pure inverse-normal-CDF + the expected-max-of-K coefficient.
|
||||
pub fn inv_norm_cdf(p: f64) -> f64; // rational approximation, golden-tested
|
||||
pub fn expected_max_of_normals(k: usize) -> f64; // (1−γ)Φ⁻¹(1−1/K)+γΦ⁻¹(1−1/(Ke)), γ=0.5772…
|
||||
```
|
||||
|
||||
### Call-site stamping (`aura-cli/src/main.rs`, `walkforward_family`)
|
||||
|
||||
```rust
|
||||
// :1810 R arm — was: let best = optimize(&is_family, "sqn_normalized")…;
|
||||
let (best, selection) =
|
||||
optimize_deflated(&is_family, "sqn_normalized", DEFLATION_N_RESAMPLES, DEFLATION_BLOCK_LEN, DEFLATION_SEED)
|
||||
.expect("sqn_normalized is a known metric");
|
||||
let (oos_equity, mut oos_report) = run_oos_r(&best.params, w.oos.0, w.oos.1, trace, data);
|
||||
oos_report.manifest.selection = Some(selection); // stamp onto the OOS winner
|
||||
WindowRun { chosen_params: best.params, oos_equity, oos_report }
|
||||
// :1795 total_pips arm — identical shape with "total_pips" + run_oos.
|
||||
```
|
||||
|
||||
`DEFLATION_N_RESAMPLES` / `DEFLATION_BLOCK_LEN` / `DEFLATION_SEED` are module
|
||||
constants this cycle (recorded in the manifest → reproducible); CLI flags for them
|
||||
are a deferred refinement. `runs_family` (`:2192`) prints `deflated` + `P(overfit)`
|
||||
beside the raw metric when `manifest.selection.is_some()`.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Crate / file | Change |
|
||||
|---|---|---|
|
||||
| `FamilySelection`, `SelectionMode` | `aura-engine/report.rs` | new serde types beside `RMetrics` |
|
||||
| `RunManifest.selection` | `aura-engine/report.rs:420` | new `Option` field, serde-widened |
|
||||
| `resample_block` | `aura-engine/mc.rs` | extracted from `r_bootstrap` (kernel shared, byte-identical) |
|
||||
| `inv_norm_cdf`, `expected_max_of_normals` | `aura-engine` (new `stats` item) | pure, deterministic, golden-tested |
|
||||
| `RunManifestRead.selection` + destructure | `aura-registry/compat.rs:27,65` | mirror + 6-field destructure + build |
|
||||
| `optimize_deflated`, `metric_value`, `resolve_metric`, `null_best_of_k` | `aura-registry/lib.rs` | new selector + metric_cmp value-read refactor |
|
||||
| `walkforward_family` stamping | `aura-cli/main.rs:1795,1810` | call selector, stamp OOS manifest |
|
||||
| `runs_family` display | `aura-cli/main.rs:2192` | print deflated + overfit when present |
|
||||
|
||||
## Data flow
|
||||
|
||||
1. `walkforward_family` builds the in-sample `SweepFamily` per window (unchanged).
|
||||
2. `optimize_deflated` calls `optimize` → the same argmax `SweepPoint`; reads the
|
||||
winner's raw metric via `metric_value`.
|
||||
3. **R arm:** for each resample `i`, `SplitMix64::new(seed ^ i)` drives
|
||||
`resample_block` over every member's **mean-centred** `trade_rs`; the metric is
|
||||
recomputed (`member_metric_from_rs`) per member and reduced to the best-of-`K`
|
||||
`M_i`. The winner is judged against `{M_i}`: `deflated_score = raw − p95(M)`,
|
||||
`overfit_probability = (count(M_i ≥ raw) + 1)/(n+1)`.
|
||||
4. **total_pips arm:** `deflated_score = raw − member_sd · expected_max_of_normals(K)`,
|
||||
`overfit_probability = None`.
|
||||
5. The `FamilySelection` is stamped onto the chosen window's `oos_report.manifest`,
|
||||
which flows through `append_family` unchanged and persists in `families.jsonl`.
|
||||
6. On read, `compat.rs` lifts the field (legacy lines → `None`); `runs_family`
|
||||
displays it.
|
||||
|
||||
## Error handling
|
||||
|
||||
- An empty resample budget (`n_resamples == 0`) or a member with empty `trade_rs`
|
||||
degrades exactly as `r_bootstrap` already does (all-zero kernel output); the
|
||||
deflation then equals the raw metric (no spurious penalty). `block_len` is clamped
|
||||
`[1, n]` by the shared kernel.
|
||||
- A family whose winner has `r: None` on the R arm (no `trade_rs`) contributes a
|
||||
zero null-max contribution for that member — never a panic (mirrors `metric_cmp`'s
|
||||
`NEG_INFINITY` treatment of a missing `r` block).
|
||||
- A degenerate `K ≤ 1` family (a single member — no search) carries **no**
|
||||
multiple-comparison inflation: `expected_max_of_normals(K ≤ 1) = 0.0` (guarding the
|
||||
`Φ⁻¹(1 − 1/K) → Φ⁻¹(0) = −∞` divergence), so `deflated_score == raw` and
|
||||
`overfit_probability` reduces to the floor `1/(n+1)`.
|
||||
- `optimize_deflated` returns `RegistryError::UnknownMetric` for an unknown name
|
||||
(same as `optimize`), before any resampling.
|
||||
- No new exit path in the CLI: the selector cannot fail where `optimize` succeeds.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **C1 determinism (primary):** same family + `(seed, n_resamples, block_len)` →
|
||||
bit-identical `overfit_probability` and `deflated_score`, across thread counts
|
||||
(the best-of-`K` reduction collects in member/odometer order, never completion
|
||||
order). A fixed-`trade_rs` fixture pins exact values.
|
||||
- **C23 additive (tripwire):** `optimize_deflated(f, m, …).0 == optimize(f, m)` for
|
||||
every metric — the deflation never changes which point wins.
|
||||
- **C2 IS-only:** the selector reads only the in-sample family's members; a test
|
||||
asserts no OOS report is touched (the call site passes `is_family`).
|
||||
- **Statistic correctness:** on a fabricated family where one member has a real edge
|
||||
and the rest are zero-centred noise, `overfit_probability` is low and
|
||||
`deflated_score > 0`; on an all-noise family (no member has edge),
|
||||
`overfit_probability` is high (≈ K/(K+…)) and `deflated_score ≤ 0`. The centring
|
||||
is pinned: an *uncentred* control would give ≈0.5 — the test asserts the centred
|
||||
construction does not.
|
||||
- **`inv_norm_cdf` golden:** known quantiles (Φ⁻¹(0.975)=1.959964…, Φ⁻¹(0.5)=0,
|
||||
symmetry) to a tight tolerance; `expected_max_of_normals(K)` monotone increasing,
|
||||
small-`K` sane (K=4 ≈ 1.03, not the √(2 ln 4)=1.66 asymptote).
|
||||
- **C14/C18 back-compat:** a pre-0076 `families.jsonl` line (no `selection`) loads
|
||||
as `None`; a stamped manifest round-trips byte-identically; the existing
|
||||
`runs.jsonl` golden and the SMA/`total_pips` walk-forward + synthetic-MC goldens
|
||||
stay green (add, don't break).
|
||||
- **`resample_block` extraction:** `r_bootstrap`'s existing determinism golden stays
|
||||
bit-identical (proves the kernel was extracted, not changed).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A Stage-1 R walk-forward stamps a `FamilySelection` on each OOS winner's manifest;
|
||||
`aura runs family … rank` surfaces the deflated score and overfit probability
|
||||
beside the raw metric — the worked program above runs and produces the shown block.
|
||||
- The argmax winner is unchanged for every metric (C23 tripwire green).
|
||||
- `overfit_probability` and `deflated_score` are reproducible from the recorded
|
||||
`(seed, n_resamples, block_len)` (C1).
|
||||
- Legacy `runs.jsonl` / `families.jsonl` lines load unchanged and all pre-existing
|
||||
goldens stay green (C18/C14).
|
||||
- The deflation is frictionless Stage-1 R; no Stage-2 cost enters.
|
||||
|
||||
### Out of scope (deferred)
|
||||
|
||||
- The effective-independent-trials advisory (`n_eff_trials`): `SweepFamily` carries
|
||||
no axis cardinalities (`sweep.rs:291`), so its per-axis inference mis-shapes off a
|
||||
full cartesian grid — deferred to a follow-on, addable by the same serde widening.
|
||||
- A reload-time recompute of the R-arm statistic (`trade_rs` is `serde(skip)`, so it
|
||||
runs only live in `walkforward_family` pre-persist — a recompute needs a wire-shape
|
||||
change).
|
||||
- Re-ranking a family by the deflated score (a different design decision — would
|
||||
change which member wins; belongs in the ledger, not this cycle).
|
||||
- CLI flags for the resampling config; #145's `Plateau*` variants (the enum slot is
|
||||
reserved, not filled).
|
||||
Reference in New Issue
Block a user