Deflate the sweep winner's metric for the number of trials #144

New Issue

2026-06-26T10:30:47+02:00

Brummel commented

2026-06-26 10:30:47 +02:00

Concern

optimize(family, metric) selects the single best sweep member by a bare argmax — crates/aura-registry/src/lib.rs:186, the reduce at :191 keeps a member only when it is strictly better by metric_cmp (:128). rank_by (:171) orders the same way. Neither applies any penalty for the number of configurations evaluated.

The World (C12) makes massively parallel sweeps aura's differentiator: GridSpace/RandomSpace enumerate K parameters, optionally across M instruments. The more members a family contains, the higher the best member's metric climbs by chance alone — a sweep over a large grid selects the best cell on noise. The current selection path has no defense against this: the winner's expectancy_r/sqn_normalized is reported and ranked at face value regardless of how many members competed for the top slot.

What to build

A multiple-comparisons-aware selection step that adjusts the selected member's score for the size of the family it won against, so the reported edge reflects the search effort:

A deflation applied to the winning metric as a function of the number of (effectively independent) trials in the family, surfaced alongside the raw metric rather than silently replacing it.
An accompanying estimate of the probability that the selected member is a false positive given the family size — an overfit-probability the registry can record per family.
Recorded on the family/run manifest (C18) so a selected candidate carries the trials count it beat and its deflated score, reproducibly.

Relationship to #139

Orthogonal. #139 validates a given candidate out-of-sample (walk-forward IS to OOS, plus a block-bootstrap confidence interval on its OOS per-trade R). That tests whether one candidate's OOS edge is separable from zero; it does not correct for the candidate having been chosen as the best of many. #139's own per-window step still picks the best-of-grid by an R metric with no trials-deflation. This issue addresses the selection-across-the-family bias that #139 leaves unguarded.

Notes

Stage-1 frictionless R is the unit (summarize_r, crates/aura-engine/src/report.rs:89); the deflation operates on the R-family metrics already ranked in rank_by.
Claim (unverified): a closed-form deflation is feasible from the family's metric dispersion without re-running members; a resampling-based estimate is the fallback if the closed form proves unreliable on small grids.

## Concern `optimize(family, metric)` selects the single best sweep member by a bare argmax — `crates/aura-registry/src/lib.rs:186`, the reduce at `:191` keeps a member only when it is strictly better by `metric_cmp` (`:128`). `rank_by` (`:171`) orders the same way. Neither applies any penalty for the number of configurations evaluated. The World (C12) makes massively parallel sweeps aura's differentiator: `GridSpace`/`RandomSpace` enumerate K parameters, optionally across M instruments. The more members a family contains, the higher the best member's metric climbs by chance alone — a sweep over a large grid selects the best cell on noise. The current selection path has no defense against this: the winner's `expectancy_r`/`sqn_normalized` is reported and ranked at face value regardless of how many members competed for the top slot. ## What to build A multiple-comparisons-aware selection step that adjusts the selected member's score for the size of the family it won against, so the reported edge reflects the search effort: - [ ] A deflation applied to the winning metric as a function of the number of (effectively independent) trials in the family, surfaced alongside the raw metric rather than silently replacing it. - [ ] An accompanying estimate of the probability that the selected member is a false positive given the family size — an overfit-probability the registry can record per family. - [ ] Recorded on the family/run manifest (C18) so a selected candidate carries the trials count it beat and its deflated score, reproducibly. ## Relationship to #139 Orthogonal. #139 validates a given candidate out-of-sample (walk-forward IS to OOS, plus a block-bootstrap confidence interval on its OOS per-trade R). That tests whether one candidate's OOS edge is separable from zero; it does not correct for the candidate having been chosen as the best of many. #139's own per-window step still picks the best-of-grid by an R metric with no trials-deflation. This issue addresses the selection-across-the-family bias that #139 leaves unguarded. ## Notes - Stage-1 frictionless R is the unit (`summarize_r`, `crates/aura-engine/src/report.rs:89`); the deflation operates on the R-family metrics already ranked in `rank_by`. - Claim (unverified): a closed-form deflation is feasible from the family's metric dispersion without re-running members; a resampling-based estimate is the fallback if the closed form proves unreliable on small grids.

Brummel added this to the Inferential validation (defend against false discovery at sweep scale) milestone 2026-06-26 10:30:47 +02:00

Brummel added the feature label 2026-06-26 10:30:47 +02:00

Brummel referenced this issue from a commit

2026-06-26 12:39:17 +02:00

docs(ledger): record the World's inferential-honesty gap as an open thread

Brummel commented

2026-06-26 13:36:40 +02:00

Design reconciliation (specify)

Spec: 0076-deflate-sweep-winner-trials (first cycle of the milestone).
The forks below were settled this session; this records their resolution
before the spec is written.

User decisions (ratified 2026-06-26 via a structured prompt):

Fork: total_pips arm asymmetry → the R / sqn_normalized arm gets the
full empirical overfit probability; the total_pips SmaCross arm gets only the
dispersion-floor deflation. Basis: user decision ("accept the asymmetry"); R is
the C7 yardstick and is risk-normalised, pips are not; selection_metric
self-documents which path ran.
Fork: recording scope → record on the surviving OOS winner's manifest only;
the in-sample family stays transient as today. Basis: user decision; the recorded
(seed, n_resamples, block_len) make the number reproducible-by-re-run without a
C18 wire-shape expansion.
Fork: overfit null model → moving-block resample (reusing the cycle-0075
r_bootstrap kernel), not sign/label permutation. Basis: user decision; one
determinism-tested mechanism (C1).
Fork: cycle sequencing → #144 first, then #145 (which inherits the
RunManifest.selection carrier + the SelectionMode enum with a reserved
Plateau slot), then #146 (the orthogonal across-instrument axis). Basis: user
decision.

Derived orchestrator decisions (rationale, not preference):

Trunk vs the issue's Notes → the issue floated a closed-form deflation as
primary with resampling as the fallback; this is inverted. The resampled
best-of-K is the trunk; the closed-form effective-trials (N_eff) is demoted to
an optional advisory and deferred out of this cycle. Basis: derived —
SweepFamily carries only space + odometer points and no axis cardinalities
(sweep.rs:291), so N_eff's per-axis-cardinality inference silently mis-shapes
off a full cartesian grid and is undefined on RandomSpace; the resampling
statistic rests on aura's own determinism-tested kernel and needs no grid topology.
FamilySelection lives in aura-engine (beside RunManifest/RMetrics),
constructed by aura-registry. Basis: derived — RunManifest is an aura-engine
type and aura-registry depends on aura-engine; defining the carrier in the
registry would invert the dependency (C9).
The block-resample is a centred reality-check → each member's trade_rs is
mean-subtracted (imposing the no-edge null) before the moving-block resample;
overfit_probability is the best-of-K max-statistic p-value in the
(count + 1) / (n_resamples + 1) form. Basis: derived — resampling each member's
own uncentred trades reproduces its own edge, so the best-of-K ≈ the winner and
the statistic degenerates to ≈0.5; centring to the null is what makes the
block-resample a false-positive test.
The total_pips dispersion floor uses the exact expected-max-of-K-normals
coefficient (via a small pure inverse-normal-CDF helper), not the
sqrt(2*ln K) asymptote. Basis: derived — the SmaCross grid is tiny (K≈4), where
sqrt(2*ln K) overstates the hurdle by ~60%; the exact order-statistic
coefficient is accurate at small K and reusable for the milestone.

Out of scope (deferred): the N_eff effective-trials advisory; any reload-time
recompute of the resampling statistic (trade_rs is serde(skip), so the variant
runs only live inside walkforward_family pre-persist — a reload recompute needs a
wire-shape change). Frictionless Stage-1 R; existing SMA/total_pips walk-forward
and synthetic-MC goldens preserved (add, don't break).

## Design reconciliation (specify) Spec: `0076-deflate-sweep-winner-trials` (first cycle of the milestone). The forks below were settled this session; this records their resolution before the spec is written. **User decisions (ratified 2026-06-26 via a structured prompt):** - **Fork: `total_pips` arm asymmetry** → the R / `sqn_normalized` arm gets the full empirical overfit probability; the `total_pips` SmaCross arm gets only the dispersion-floor deflation. Basis: user decision ("accept the asymmetry"); R is the C7 yardstick and is risk-normalised, pips are not; `selection_metric` self-documents which path ran. - **Fork: recording scope** → record on the surviving OOS winner's manifest only; the in-sample family stays transient as today. Basis: user decision; the recorded `(seed, n_resamples, block_len)` make the number reproducible-by-re-run without a C18 wire-shape expansion. - **Fork: overfit null model** → moving-block resample (reusing the cycle-0075 `r_bootstrap` kernel), not sign/label permutation. Basis: user decision; one determinism-tested mechanism (C1). - **Fork: cycle sequencing** → #144 first, then #145 (which inherits the `RunManifest.selection` carrier + the `SelectionMode` enum with a reserved `Plateau` slot), then #146 (the orthogonal across-instrument axis). Basis: user decision. **Derived orchestrator decisions (rationale, not preference):** - **Trunk vs the issue's Notes** → the issue floated a closed-form deflation as primary with resampling as the fallback; this is **inverted**. The resampled best-of-K is the trunk; the closed-form effective-trials (`N_eff`) is demoted to an optional advisory and **deferred out of this cycle**. Basis: derived — `SweepFamily` carries only `space` + odometer points and **no axis cardinalities** (`sweep.rs:291`), so `N_eff`'s per-axis-cardinality inference silently mis-shapes off a full cartesian grid and is undefined on `RandomSpace`; the resampling statistic rests on aura's own determinism-tested kernel and needs no grid topology. - **`FamilySelection` lives in `aura-engine`** (beside `RunManifest`/`RMetrics`), constructed by `aura-registry`. Basis: derived — `RunManifest` is an `aura-engine` type and `aura-registry` depends on `aura-engine`; defining the carrier in the registry would invert the dependency (C9). - **The block-resample is a centred reality-check** → each member's `trade_rs` is mean-subtracted (imposing the no-edge null) before the moving-block resample; `overfit_probability` is the best-of-K max-statistic p-value in the `(count + 1) / (n_resamples + 1)` form. Basis: derived — resampling each member's *own uncentred* trades reproduces its own edge, so the best-of-K ≈ the winner and the statistic degenerates to ≈0.5; centring to the null is what makes the block-resample a false-positive test. - **The `total_pips` dispersion floor uses the exact expected-max-of-K-normals coefficient** (via a small pure inverse-normal-CDF helper), not the `sqrt(2*ln K)` asymptote. Basis: derived — the SmaCross grid is tiny (K≈4), where `sqrt(2*ln K)` overstates the hurdle by ~60%; the exact order-statistic coefficient is accurate at small K and reusable for the milestone. Out of scope (deferred): the `N_eff` effective-trials advisory; any reload-time recompute of the resampling statistic (`trade_rs` is `serde(skip)`, so the variant runs only live inside `walkforward_family` pre-persist — a reload recompute needs a wire-shape change). Frictionless Stage-1 R; existing SMA/`total_pips` walk-forward and synthetic-MC goldens preserved (add, don't break).

Brummel commented

2026-06-26 13:54:01 +02:00

Derived decision (planner, /boss): SplitMix64 cross-crate visibility

While decomposing 0076, plan-recon surfaced that optimize_deflated
(aura-registry) must drive the moving-block resample deterministically, but
SplitMix64 is pub(crate) in aura-engine/src/harness.rs:143 and is not
re-exported — unreachable from aura-registry.

Decision → make SplitMix64 pub and re-export it from aura-engine
(beside the already-public r_bootstrap); aura-registry creates the per-resample
RNG (SplitMix64::new(seed ^ i)) and calls the extracted resample_block
directly. Basis: derived — this keeps the "which metric" selection logic in
aura-registry (where Metric/metric_cmp/optimize live) and the determinism
kernel + metric-from-series (resample_block, r_metrics_from_rs) in aura-engine,
rather than pushing metric selection into the engine via a duplicated metric enum
or a closure across the crate boundary. C1 is preserved (the RNG is the project's
canonical seeded primitive; exposing it changes no behaviour); the kernel stays
byte-identical (the extraction is pinned by r_bootstrap's determinism golden).

Two mechanical sub-points (not forks, recorded for completeness): the two
walkforward_family call sites rebind oos_report as mut to stamp the
selection; runs_family gains an additive human-readable deflated=… P(overfit)=…
line (only its own display golden is extended — sweep/mc/standalone manifests stay
byte-identical via skip_serializing_if = "Option::is_none").

## Derived decision (planner, /boss): SplitMix64 cross-crate visibility While decomposing 0076, plan-recon surfaced that `optimize_deflated` (aura-registry) must drive the moving-block resample deterministically, but `SplitMix64` is `pub(crate)` in `aura-engine/src/harness.rs:143` and is not re-exported — unreachable from aura-registry. **Decision** → make `SplitMix64` `pub` and re-export it from `aura-engine` (beside the already-public `r_bootstrap`); aura-registry creates the per-resample RNG (`SplitMix64::new(seed ^ i)`) and calls the extracted `resample_block` directly. Basis: derived — this keeps the "which metric" selection logic in aura-registry (where `Metric`/`metric_cmp`/`optimize` live) and the determinism kernel + metric-from-series (`resample_block`, `r_metrics_from_rs`) in aura-engine, rather than pushing metric selection into the engine via a duplicated metric enum or a closure across the crate boundary. C1 is preserved (the RNG is the project's canonical seeded primitive; exposing it changes no behaviour); the kernel stays byte-identical (the extraction is pinned by `r_bootstrap`'s determinism golden). Two mechanical sub-points (not forks, recorded for completeness): the two `walkforward_family` call sites rebind `oos_report` as `mut` to stamp the selection; `runs_family` gains an additive human-readable `deflated=… P(overfit)=…` line (only its own display golden is extended — sweep/mc/standalone manifests stay byte-identical via `skip_serializing_if = "Option::is_none"`).

Brummel referenced this issue

2026-06-26 15:09:11 +02:00

Prefer a robust parameter plateau over the in-sample peak in optimize() #145

Brummel referenced this issue

2026-06-26 18:10:39 +02:00

Aggregate cross-instrument generalization as a first-class validation step #146

Brummel referenced this issue

2026-06-26 19:13:47 +02:00

Aggregate cross-instrument generalization as a first-class validation step #146

Brummel referenced this issue from a commit

2026-06-26 21:50:15 +02:00

spec: 0076 deflate sweep winner for trials (boss-signed)