Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Aggregate cross-instrument generalization as a first-class validation step #146

New Issue
Closed
opened 2026-06-26 10:30:58 +02:00 by Brummel · 2 comments
Brummel commented 2026-06-26 10:30:58 +02:00
Owner
Copy Link
Copy Source

Concern

An edge that appears on a single instrument can be a fit to that series' noise; an edge that survives unchanged across several independent instruments is far less likely to be spurious. This cross-instrument agreement is one of the strongest available defenses against false discovery, but in aura it is run only by hand: aura sweep --strategy stage1-r --real <SYM> evaluates one symbol per invocation, and there is no built aggregator that scores a candidate by how consistently it holds across instruments.

#139 (decision log, comment of 2026-06-26) explicitly defers cross-symbol pooling: RealWindowGrammar accepts one symbol today, and cross-symbol aggregation is marked "a separable expansion, deferred to a follow-on, not built this cycle". This issue is that follow-on, scoped as a validation step rather than as edge research.

What to build

A World-layer aggregator that evaluates one candidate across a set of instruments and scores its generalization:

  • Run a candidate's evaluation (sweep and/or walk-forward) across a list of instruments and collect per-instrument R-metrics (summarize_r, crates/aura-engine/src/report.rs:89).
  • A pooled generalization score — consistency of the sign and magnitude of expectancy_r/sqn_normalized across instruments, not a single pooled average that one strong instrument could dominate.
  • Per-instrument and aggregate results recorded together on the family manifest (C18), with the instrument set as part of the lineage.

Relationship to the sweep-selection issues

This is the across-instrument axis of the same anti-false-discovery concern: the trials-deflation issue corrects for the number of parameter trials, the plateau issue corrects for peak-picking within one instrument, and this corrects for an edge that exists on one instrument but not others. Together they make the World's parallel sweep defensible rather than a false-discovery engine.

## Concern An edge that appears on a single instrument can be a fit to that series' noise; an edge that survives unchanged across several independent instruments is far less likely to be spurious. This cross-instrument agreement is one of the strongest available defenses against false discovery, but in aura it is run only by hand: `aura sweep --strategy stage1-r --real <SYM>` evaluates one symbol per invocation, and there is no built aggregator that scores a candidate by how consistently it holds across instruments. #139 (decision log, comment of 2026-06-26) explicitly defers cross-symbol pooling: `RealWindowGrammar` accepts one symbol today, and cross-symbol aggregation is marked "a separable expansion, deferred to a follow-on, not built this cycle". This issue is that follow-on, scoped as a validation step rather than as edge research. ## What to build A World-layer aggregator that evaluates one candidate across a set of instruments and scores its generalization: - [ ] Run a candidate's evaluation (sweep and/or walk-forward) across a list of instruments and collect per-instrument R-metrics (`summarize_r`, `crates/aura-engine/src/report.rs:89`). - [ ] A pooled generalization score — consistency of the sign and magnitude of `expectancy_r`/`sqn_normalized` across instruments, not a single pooled average that one strong instrument could dominate. - [ ] Per-instrument and aggregate results recorded together on the family manifest (C18), with the instrument set as part of the lineage. ## Relationship to the sweep-selection issues This is the across-instrument axis of the same anti-false-discovery concern: the trials-deflation issue corrects for the number of parameter trials, the plateau issue corrects for peak-picking within one instrument, and this corrects for an edge that exists on one instrument but not others. Together they make the World's parallel sweep defensible rather than a false-discovery engine.
Brummel added this to the Inferential validation (defend against false discovery at sweep scale) milestone 2026-06-26 10:30:58 +02:00
Brummel added the feature label 2026-06-26 10:30:58 +02:00
Brummel commented 2026-06-26 18:10:39 +02:00
Author
Owner
Copy Link
Copy Source

Design ratified (brainstorm → specify): #146 as a cross-instrument validator, not a third selector

Pivotal fork — aggregator/validator (A) vs cross-instrument selector (B) — decided A.
Basis: user decision, "mach es so" (@Brummel, 2026-06-26), ratifying the aggregator reading:
#146 grades a brought candidate's generalization across instruments and chains behind
deflation (#144) / plateau (#145); it does not select a winner across instruments.

Derived sub-decisions (orchestrator, with rationale):

  • R-only score. The generalization metric is an R-metric (expectancy_r default,
    sqn_normalized optional). Rationale: R is the only account/instrument-agnostic unit
    (C10); total_pips is meaningless across instruments (different pip scales / vol). The
    score is undefined for / refuses pip metrics.
  • Worst-case minimum as the headline score. score = min_i metric_i across instruments,
    recorded alongside a sign-agreement count (#{ i : metric_i > 0 }) and the full
    per-instrument breakdown. Rationale: exact structural sibling of PlateauMode::Worst
    (already min-over-neighbourhood); satisfies the issue's non-domination constraint by
    construction (a min cannot be lifted by one strong instrument); "consistency of sign and
    magnitude" is then fully carried by {min, sign-agreement, per-instrument vector}. Rejected
    alternative: a cross-instrument t-stat mean_i / std_i · sqrt(M) — reintroduces the pooled
    mean the issue warns against and adds knobs without clear benefit.
  • Honest-OOS for free. Because the candidate is fixed/external (not fit on each
    instrument), a single full-window R run per instrument is already an out-of-sample test for
    that instrument; no per-instrument walk-forward is needed for the first correct version
    (walk-forward-per-instrument is a possible later knob).
  • Aggregator, not selector -> recorded as its own family, not on FamilySelection. New
    FamilyKind::CrossInstrument; the M per-instrument runs are the family members. The
    aggregate (worst-case + sign-agreement) is a pure function of the members (recomputable,
    like McAggregate), printed live and derivable on reload — not stamped on the
    selector-annotation carrier (which has no meaning here: there is no within-family winner).
  • Instrument becomes first-class lineage. Add instrument: Option<String> to
    RunManifest (serde default / skip_serializing_if, same back-compat idiom as
    selection; C14/C18). Today a real-data run records the window/broker but not which
    symbol — a latent gap; this makes each member self-identifying and supplies the
    "instrument set as lineage" the issue asks for.
  • Score function home = aura-registry (beside optimize_deflated/optimize_plateau),
    using resolve_metric/metric_value. Rationale: C9 — the validation/selection-read layer
    is registry-side, the engine stays simulation. Consistent with where #144/#145 landed.
  • CLI: a new aura generalize subcommand reusing the Stage1RGrid flags as a
    single-value candidate and letting --real take an instrument list
    (--real EURUSD,GBPUSD,USDJPY). Maximal reuse of existing grid-parsing + per-instrument R
    evaluation; the new surface is just the instrument loop + the aggregate.

Next: handing the ratified design to specify.

## Design ratified (brainstorm → specify): #146 as a cross-instrument *validator*, not a third selector Pivotal fork — **aggregator/validator (A)** vs **cross-instrument selector (B)** — decided **A**. Basis: user decision, "mach es so" (@Brummel, 2026-06-26), ratifying the aggregator reading: #146 grades a *brought* candidate's generalization across instruments and chains behind deflation (#144) / plateau (#145); it does not select a winner across instruments. Derived sub-decisions (orchestrator, with rationale): - **R-only score.** The generalization metric is an R-metric (`expectancy_r` default, `sqn_normalized` optional). Rationale: R is the only account/instrument-agnostic unit (C10); `total_pips` is meaningless across instruments (different pip scales / vol). The score is undefined for / refuses pip metrics. - **Worst-case minimum as the headline score.** `score = min_i metric_i` across instruments, recorded alongside a sign-agreement count (`#{ i : metric_i > 0 }`) and the full per-instrument breakdown. Rationale: exact structural sibling of `PlateauMode::Worst` (already min-over-neighbourhood); satisfies the issue's non-domination constraint by construction (a min cannot be lifted by one strong instrument); "consistency of sign and magnitude" is then fully carried by {min, sign-agreement, per-instrument vector}. Rejected alternative: a cross-instrument t-stat `mean_i / std_i · sqrt(M)` — reintroduces the pooled `mean` the issue warns against and adds knobs without clear benefit. - **Honest-OOS for free.** Because the candidate is fixed/external (not fit on each instrument), a single full-window R run per instrument is already an out-of-sample test for that instrument; no per-instrument walk-forward is needed for the first correct version (walk-forward-per-instrument is a possible later knob). - **Aggregator, not selector -> recorded as its own family, not on `FamilySelection`.** New `FamilyKind::CrossInstrument`; the M per-instrument runs are the family members. The aggregate (worst-case + sign-agreement) is a pure function of the members (recomputable, like `McAggregate`), printed live and derivable on reload — not stamped on the selector-annotation carrier (which has no meaning here: there is no within-family winner). - **Instrument becomes first-class lineage.** Add `instrument: Option<String>` to `RunManifest` (serde `default` / `skip_serializing_if`, same back-compat idiom as `selection`; C14/C18). Today a real-data run records the window/broker but not *which* symbol — a latent gap; this makes each member self-identifying and supplies the "instrument set as lineage" the issue asks for. - **Score function home = aura-registry** (beside `optimize_deflated`/`optimize_plateau`), using `resolve_metric`/`metric_value`. Rationale: C9 — the validation/selection-read layer is registry-side, the engine stays simulation. Consistent with where #144/#145 landed. - **CLI: a new `aura generalize` subcommand** reusing the `Stage1RGrid` flags as a single-value *candidate* and letting `--real` take an instrument list (`--real EURUSD,GBPUSD,USDJPY`). Maximal reuse of existing grid-parsing + per-instrument R evaluation; the new surface is just the instrument loop + the aggregate. Next: handing the ratified design to `specify`.
Brummel commented 2026-06-26 19:13:47 +02:00
Author
Owner
Copy Link
Copy Source

Implemented & cycle-closed (cycle 0078)

Cross-instrument generalization landed as ratified (aggregator/validator, reading A).
The aura generalize subcommand runs a single-cell stage1-r candidate across a
--real SYM1,SYM2,... list and scores its generalization as a worst-case R floor
(min over instruments) + a sign-agreement count + the per-instrument breakdown
— R-only (C10), the cross-instrument sibling of plateau-worst, non-dominable by
construction (a strong instrument cannot lift a min).

New FamilyKind::CrossInstrument family + a first-class RunManifest.instrument
lineage field (serde-widened with skip_serializing_if, so legacy lines load and
every non-cross-instrument path stays byte-identical — C14/C23). The reduction
(generalization + check_r_metric) lives in aura-registry beside
optimize_deflated/optimize_plateau (C9); the score is a recomputable aggregate
over the family members, NOT stamped on FamilySelection (there is no within-family
winner here — that is the aggregator-vs-selector distinction from #144/#145).

All three milestone pieces — #144 trials-deflation, #145 plateau-over-peak, #146
cross-instrument — are now built; the inferential-validation half is structurally
complete. cargo test --workspace + clippy -D warnings green. Local commits:
spec (boss-signed) / plan / feat / audit-close; the ledger's inferential thread + the
C18 realization note are synced.

The Gitea milestone close (its end-to-end fieldtest) remains a deliberate manual step.
This issue auto-closes on push via the feat commit's closes #146.

## Implemented & cycle-closed (cycle 0078) Cross-instrument generalization landed as ratified (aggregator/validator, reading A). The `aura generalize` subcommand runs a single-cell stage1-r candidate across a `--real SYM1,SYM2,...` list and scores its generalization as a **worst-case R floor** (min over instruments) + a **sign-agreement count** + the **per-instrument breakdown** — R-only (C10), the cross-instrument sibling of plateau-worst, non-dominable by construction (a strong instrument cannot lift a min). New `FamilyKind::CrossInstrument` family + a first-class `RunManifest.instrument` lineage field (serde-widened with `skip_serializing_if`, so legacy lines load and every non-cross-instrument path stays byte-identical — C14/C23). The reduction (`generalization` + `check_r_metric`) lives in aura-registry beside `optimize_deflated`/`optimize_plateau` (C9); the score is a *recomputable aggregate* over the family members, NOT stamped on `FamilySelection` (there is no within-family winner here — that is the aggregator-vs-selector distinction from #144/#145). All three milestone pieces — #144 trials-deflation, #145 plateau-over-peak, #146 cross-instrument — are now built; the inferential-validation half is **structurally complete**. `cargo test --workspace` + `clippy -D warnings` green. Local commits: spec (boss-signed) / plan / feat / audit-close; the ledger's inferential thread + the C18 realization note are synced. The Gitea milestone close (its end-to-end fieldtest) remains a deliberate manual step. This issue auto-closes on push via the feat commit's `closes #146`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone Inferential validation (defend against false discovery at sweep scale)
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#146
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?