Aura/docs/specs/0078-cross-instrument-generalization.md

Cross-instrument generalization as a validation step — Design Spec

Date: 2026-06-26 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Goal

Add a first-class validation step that grades how consistently a single candidate holds across a set of instruments. This is the last piece of the "Inferential validation" milestone and the across-instrument axis of the same anti-false-discovery concern as trials-deflation (#144, within one family) and plateau-over-peak (#145, within one instrument's surface): an edge that appears on one instrument can be a fit to that series' noise; an edge that survives across several independent instruments is far less likely to be spurious.

The step is an aggregator / validator, not a third selector. It takes a brought candidate — one already chosen (e.g. by deflation or plateau on a reference instrument, or authored directly) — runs it across an instrument list, collects per-instrument R-metrics, and reduces them to a generalization grade. It does not select a winner across instruments (that is a heavier follow-on, out of scope below). A generalization grade is an aggregate (like McAggregate), not a within-family selection, so it is not recorded on FamilySelection (which annotates a within-family winner that does not exist here).

Architecture

A new aura generalize subcommand. Input is a single candidate (the four stage1-r grid knobs pinned to single values: --fast --slow --stop-length --stop-k) crossed with an instrument list (--real SYM1,SYM2,..., comma-separated; a shared optional --from/--to window, full available range per instrument if omitted). For each instrument the candidate runs as a single full-window stage1-r R evaluation, the M RunReports are collected, reduced to a generalization aggregate, the per-instrument breakdown and the aggregate are printed, and the M runs are persisted as a CrossInstrument family.

The per-instrument run is an honest out-of-sample test by construction: because the candidate is fixed/external — not fit on that instrument — a single full-window run on each instrument is already OOS for it. No per-instrument walk-forward is needed for this first correct version.

The reduction is R-only. R is the only account- and instrument-agnostic unit (C10); total_pips is meaningless across instruments (different pip scales and volatility). The headline score is the worst case across instruments — the minimum of the chosen R-metric — which a strong instrument cannot lift (the issue's non-domination constraint, satisfied by construction). It is the cross-instrument sibling of PlateauMode::Worst (which already takes the most-pessimistic neighbour). Because R-metrics are all higher-is-better, the worst case is unconditionally the minimum (no direction branch). Alongside it the aggregate carries the sign-agreement count (how many instruments are net-positive) and the full per-instrument breakdown — together these fully express "consistency of sign and magnitude".

This honours: C23 (additive — every existing path stays byte-identical; the subcommand is new surface), C9 (the reduction lives registry-side, the engine stays simulation), C12 (cross-instrument is the comparison orchestration axis → a new FamilyKind), C14/C18 (the new manifest field serde-widens with the same back-compat idiom as selection), C1 (a pure fold over the per-instrument results in input order), C2 (each per-instrument run is an ordinary causal sim), C10 (R-only).

Concrete code shapes

The user-facing program (the acceptance evidence)

The researcher has a candidate and asks "does it generalize across instruments?":

aura generalize --strategy stage1-r --real GER40,USDJPY \
                --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 \
                --from 1725148800000 --to 1727740799999

Expected stdout (the aggregate object + the persisted family handle; numbers illustrative):

{"generalize":{"metric":"expectancy_r","n_instruments":2,"worst_case":-0.07,"sign_agreement":1,"per_instrument":[["GER40",0.31],["USDJPY",-0.07]]}}
{"family_id":"generalize-0"}

The aggregate is the diagnosis: the candidate makes +0.31 R/trade on GER40 but loses on USDJPY, so its worst-case generalization is negative and only 1 of 2 instruments agrees in sign — a false-discovery flag the per-instrument view would have hidden behind GER40's strength.

A second --metric selects the R-metric (default expectancy_r):

aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 \
                --stop-length 14 --stop-k 2.0 --metric sqn_normalized

Must-fail surfaces (each exits 2 with a clear message):

aura generalize --strategy stage1-r --real GER40 --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
# refuse: a generalization needs >= 2 instruments

aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 --metric total_pips
# refuse: total_pips is not comparable across instruments; cross-instrument scoring is R-only

aura generalize --strategy stage1-r --real GER40,USDJPY --fast 2,3 --slow 12 --stop-length 14 --stop-k 2.0
# refuse: a candidate is a single cell; --fast got 2 values

aura generalize --strategy sma --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
# refuse: generalize requires --strategy stage1-r (the candidate must produce R)

Implementation shapes (before → after, secondary)

1. RunManifest gains a first-class instrument field — crates/aura-engine/src/report.rs. Same serde-widening idiom as selection (C14/C18): legacy lines without the key read back as None. Closes a latent gap (today a real run records the window/broker but not which symbol).

// before
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub selection: Option<FamilySelection>,
}

// after — add one field (every RunManifest literal threads `instrument: None`,
// the same compile-gate the #144 `selection` field was; the planner compile-drives
// the full literal set)
pub struct RunManifest {
    pub commit: String,
    pub params: Vec<(String, Scalar)>,
    pub window: (Timestamp, Timestamp),
    pub seed: u64,
    pub broker: String,
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub selection: Option<FamilySelection>,
    /// The instrument this run evaluated, when it is a real-data run; `None` for a
    /// synthetic run and for every pre-0078 line. The "instrument set as lineage"
    /// (C18) for a cross-instrument family is each member's stamped symbol.
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub instrument: Option<String>,
}

2. FamilyKind gains the comparison axis — crates/aura-registry/src/lineage.rs. A compile-gate: every match on FamilyKind (the runs family display, any kind-dispatch) must gain the arm.

// before
pub enum FamilyKind { Sweep, MonteCarlo, WalkForward }
// after
pub enum FamilyKind { Sweep, MonteCarlo, WalkForward, CrossInstrument }

3. The generalization reduction — crates/aura-registry/src/lib.rs, a sibling of optimize_deflated / optimize_plateau, reusing resolve_metric / metric_value / is_r_metric (all already present). Note the input is not a SweepFamily and the output is not (SweepPoint, FamilySelection) — this is an aggregator, so it has its own value type.

/// The cross-instrument generalization aggregate (#146): a brought candidate's
/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement
/// count + the full breakdown. R-only (C10): `total_pips` is not comparable across
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
/// order. The headline `worst_case` is `min_i metric_i`; since every R-metric is
/// higher-is-better, the min is unconditionally the conservative floor (no
/// direction branch, unlike `PlateauMode::Worst`).
pub struct Generalization {
    pub selection_metric: String,
    pub n_instruments: usize,
    pub worst_case: f64,           // min_i metric_i
    pub sign_agreement: usize,     // #{ i : metric_i > 0 }
    pub per_instrument: Vec<(String, f64)>,
}

pub fn generalization(
    per_instrument: &[(String, &RunReport)],
    metric: &str,
) -> Result<Generalization, RegistryError> {
    let m = resolve_metric(metric)?;
    if !is_r_metric(m) {
        return Err(RegistryError::NonRMetric(metric.to_string()));
    }
    if per_instrument.len() < 2 {
        return Err(RegistryError::TooFewInstruments(per_instrument.len()));
    }
    let vals: Vec<(String, f64)> = per_instrument
        .iter()
        .map(|(label, rep)| (label.clone(), metric_value(rep, m)))
        .collect();
    let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
    let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
    Ok(Generalization {
        selection_metric: metric.to_string(),
        n_instruments: vals.len(),
        worst_case,
        sign_agreement,
        per_instrument: vals,
    })
}

Two new RegistryError variants (in-style with the existing typed variants):

pub enum RegistryError {
    Io(std::io::Error),
    Parse { line: usize, source: serde_json::Error },
    UnknownMetric(String),
    NonRMetric(String),         // a known metric that is not R-based (cross-instrument is R-only)
    TooFewInstruments(usize),   // a generalization needs >= 2 instruments
}

4. The CLI subcommand — crates/aura-cli/src/main.rs. A new dispatch arm beside sweep / walkforward, a parse_generalize_args, and a run_generalize that loops the candidate over the instruments, stamps each manifest's instrument, reduces, prints, and persists.

// dispatch (beside the existing ["sweep", ..] / ["walkforward", ..] arms)
["generalize", rest @ ..] => match parse_generalize_args(rest) {
    Ok((name, symbols, grid, metric, from_ms, to_ms)) => {
        run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms)
    }
    Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
},

// run_generalize (shape): one full-window stage1-r R run per instrument, the
// candidate Stage1RGrid pinned to single values; stamp manifest.instrument; reduce.
fn run_generalize(name: &str, symbols: &[String], grid: &Stage1RGrid,
                  metric: &str, from_ms: Option<i64>, to_ms: Option<i64>) {
    let mut members: Vec<RunReport> = Vec::new();
    for symbol in symbols {
        // the existing real-source construction (per-instrument pip_or_refuse + has_symbol)
        let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms };
        let data = DataSource::from_choice(choice);
        let family = stage1_r_sweep_family(None, &data, grid);   // single-cell grid -> 1 member
        let mut report = family.points[0].report.clone();
        report.manifest.instrument = Some(symbol.clone());
        members.push(report);
    }
    let pairs: Vec<(String, &RunReport)> =
        symbols.iter().cloned().zip(members.iter()).collect();
    let agg = match aura_registry::generalization(&pairs, metric) {
        Ok(a) => a,
        Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
    };
    println!("{}", generalize_json(&agg));                       // {"generalize": { ... }}
    // persist the M runs as a CrossInstrument family (same Err-handling as `sweep`)
    match reg.append_family(name, FamilyKind::CrossInstrument, &members) {
        Ok(id) => println!("{{\"family_id\":\"{id}\"}}"),
        Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); }
    }
}

The candidate is a Stage1RGrid whose fast/slow/stop_length/stop_k each carry exactly one value; parse_generalize_args requires all four explicitly and refuses a multi-value flag (a candidate is one cell — it must not default to the multi-value sweep grid). --real is parsed as a comma-list into Vec<String>; the existing single-symbol RealWindowGrammar / sweep / walkforward parsers are untouched.

Components

1. RunManifest.instrument: Option<String> (aura-engine/report.rs) — the instrument lineage; serde default / skip_serializing_if. Every existing RunManifest literal threads instrument: None (compile-gate, same shape as #144's selection). The synthetic and existing real run paths leave it None (or stamp it — a non-load-bearing nicety; this cycle only requires the generalize path to stamp it).
2. FamilyKind::CrossInstrument (aura-registry/lineage.rs) — the comparison axis tag; persisted on each member record. Compile-gate on FamilyKind matches.
3. generalization + Generalization + two RegistryError variants (aura-registry/lib.rs) — the R-only reduction beside the sibling selectors, reusing resolve_metric / metric_value / is_r_metric.
4. CLI: generalize dispatch arm, parse_generalize_args, run_generalize, generalize_json (aura-cli/main.rs) — the candidate × instrument-list loop; reuses stage1_r_sweep_family (single-cell grid), the per-instrument DataSource::Real construction (pip_or_refuse + has_symbol refusals), and append_family. USAGE gains the aura generalize ... form.

Data flow

candidate (single-cell Stage1RGrid) + [SYM_1 .. SYM_M] + window
  -> for each SYM_i:
       DataSource::Real{SYM_i}  (pip_or_refuse + has_symbol)
       -> stage1_r_sweep_family(single-cell grid)  -> SweepFamily{ 1 point }
       -> report = points[0].report; report.manifest.instrument = Some(SYM_i)
  -> members: Vec<RunReport>  (length M, input order)
  -> generalization(zip(symbols, members), metric)
       -> Generalization{ worst_case = min_i, sign_agreement, per_instrument }
  -> print {"generalize": agg}
  -> append_family(name, CrossInstrument, members) -> family_id
  -> print {"family_id": id}

The aggregate is a pure function of the persisted members, so it is recomputable on reload (the McAggregate precedent) — no new persisted family-level record type is introduced.

Error handling

Every refusal is stderr + exit(2), the established CLI convention:

• < 2 instruments — a one-instrument "generalization" is meaningless. Enforced both at parse_generalize_args (early, on the parsed list) and in generalization (TooFewInstruments, the reusable-core guard).
• non-R / pip metric — cross-instrument scoring is R-only; generalization returns NonRMetric. The CLI surfaces it as exit 2.
• strategy not stage1-r — the candidate must produce R; parse_generalize_args refuses any --strategy other than stage1-r.
• a multi-value candidate flag — a candidate is a single cell; parse_generalize_args refuses when any of --fast/--slow/--stop-length/--stop-k carries more than one value, or is absent.
• empty / duplicate symbol in the list — refuse (a duplicate would double-count one instrument in the worst-case and sign-agreement).
• a symbol with no recorded geometry / no archive — the existing pip_or_refuse / no_real_data refusals fire per instrument before any run.

Testing strategy

RED-first where pinnable. Per-crate test command, single positional filter.

Registry unit tests (aura-registry):

• worst_case picks the floor: a fabricated 3-member set (one strongly positive, one negative, one mid) → worst_case == the negative value, not the mean; sign_agreement == 2.
• non-domination: appending a hugely-positive fourth instrument does not raise worst_case (the min is unchanged).
• R-only refusal: generalization(.., "total_pips") → Err(NonRMetric).
• too-few refusal: a 1-element input → Err(TooFewInstruments(1)).
• determinism (C1): the same inputs → an identical Generalization (it is a pure fold).

Engine serde tests (aura-engine):

• back-compat (C14/C18): a pre-0078 manifest JSON line without instrument deserializes with instrument == None; a round-trip with instrument == Some("GER40") preserves it; the existing RunManifest/FamilySelection serde and legacy-load tests stay green.

CLI E2E (aura-cli/tests/cli_run.rs), gated-skip on absent local data exactly like the existing --real tests:

• aura generalize --strategy stage1-r --real GER40,USDJPY --fast .. --slow .. --stop-length .. --stop-k .. --from .. --to .. over the shared Sept-2024 window → stdout carries a {"generalize": ...} object with n_instruments == 2, a per_instrument breakdown naming both symbols, a worst_case, a sign_agreement, and a following {"family_id":"generalize-0"}; when the archive is absent, the run hits the no-data refusal and the test skips cleanly (the established pattern).
• a must-fail E2E: --real GER40 (single instrument) exits 2 with the

  =2-instruments message (no data needed — the arity refusal precedes any run).

C23 preservation:

• the existing sweep / walkforward / mc / standalone-run goldens stay byte-identical — generalize is new surface and touches no existing path.

Acceptance criteria

Against aura's feature-acceptance criterion:

1. The intended audience reaches for it. The aura researcher validating a candidate naturally asks "does this hold across instruments?" — the last anti-false-discovery piece of the milestone. The worked aura generalize invocation above is the evidence: it is the command a researcher writes, and its output is the generalization verdict.
2. It measurably improves correctness. A candidate that looks good on one instrument but fails the worst-case-across-instruments floor (negative worst_case, low sign_agreement) is now caught — a real defense the per-instrument view hides. It composes downstream of #144/#145.
3. It reintroduces no failure class the core constraints eliminate. R-only (C10), behaviour-preserving (C23 — existing paths byte-identical), deterministic (C1), causal per-instrument sims (C2), registry-side policy (C9), serde-widened with full back-compat (C14/C18). No invariant is touched.

Out of scope (named, deferred)

• Cross-instrument selection (reading B): running a full sweep per instrument and picking the cell that generalizes best across instruments. A heavier follow-on that conflates selection with validation; this cycle is the validator.
• Per-instrument walk-forward evaluation: the first version runs a full-window R evaluation per instrument (honest OOS because the candidate is fixed). A per-instrument walk-forward knob is a later refinement.
• A persisted family-level aggregate record: the aggregate is recomputable from the members (the McAggregate precedent); no new persisted record type.
• Cross-instrument R-pooling of trade series: #139's deferred cross-symbol pooling stays deferred.
• Pip-metric generalization: undefined across instruments (C10); refused.