spec: 0078 cross-instrument generalization (boss-signed)

Last piece of the Inferential-validation milestone (#146): grade how consistently a brought candidate holds across a set of instruments. Aggregator/validator, not a third selector (user-ratified "mach es so", decision + derivations minuted on #146): a new `aura generalize` subcommand runs a single candidate (single-cell stage1-r grid) across an instrument list, collects per-instrument R-metrics, and reduces them to a worst-case floor (min over instruments) + a sign-agreement count + the per-instrument breakdown. R-only (C10 — R is the only cross-instrument-comparable unit); the worst-case min is the cross-instrument sibling of PlateauMode::Worst and satisfies the non-domination constraint by construction. New FamilyKind::CrossInstrument (C12) + a first-class RunManifest.instrument lineage field (serde-widened, C14/C18); the reduction lives registry-side beside optimize_deflated/optimize_plateau (C9). Additive — existing sweep/walkforward/mc paths stay byte-identical (C23). grounding-check PASS: all 9 current-behaviour assumptions ratified by green tests. refs #146
2026-06-26 18:19:51 +02:00
parent d50a4828ad
commit 428d092298
1 changed files with 396 additions and 0 deletions
@@ -0,0 +1,396 @@
+# Cross-instrument generalization as a validation step — Design Spec
+
+**Date:** 2026-06-26
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+## Goal
+
+Add a first-class validation step that grades how consistently a single
+**candidate** holds across a set of instruments. This is the last piece of the
+"Inferential validation" milestone and the across-instrument axis of the same
+anti-false-discovery concern as trials-deflation (#144, within one family) and
+plateau-over-peak (#145, within one instrument's surface): an edge that appears
+on one instrument can be a fit to that series' noise; an edge that survives
+across several independent instruments is far less likely to be spurious.
+
+The step is an **aggregator / validator**, not a third selector. It takes a
+*brought* candidate — one already chosen (e.g. by deflation or plateau on a
+reference instrument, or authored directly) — runs it across an instrument list,
+collects per-instrument R-metrics, and reduces them to a generalization grade.
+It does **not** select a winner across instruments (that is a heavier follow-on,
+out of scope below). A generalization grade is an aggregate (like `McAggregate`),
+not a within-family selection, so it is **not** recorded on `FamilySelection`
+(which annotates a within-family winner that does not exist here).
+
+## Architecture
+
+A new `aura generalize` subcommand. Input is a single **candidate** (the four
+stage1-r grid knobs pinned to single values: `--fast --slow --stop-length
+--stop-k`) crossed with an **instrument list** (`--real SYM1,SYM2,...`,
+comma-separated; a shared optional `--from`/`--to` window, full available range
+per instrument if omitted). For each instrument the candidate runs as a single
+full-window stage1-r R evaluation, the M `RunReport`s are collected, reduced to a
+generalization aggregate, the per-instrument breakdown and the aggregate are
+printed, and the M runs are persisted as a `CrossInstrument` family.
+
+The per-instrument run is an **honest out-of-sample test by construction**:
+because the candidate is fixed/external — not fit on that instrument — a single
+full-window run on each instrument is already OOS for it. No per-instrument
+walk-forward is needed for this first correct version.
+
+The reduction is **R-only**. R is the only account- and instrument-agnostic unit
+(C10); `total_pips` is meaningless across instruments (different pip scales and
+volatility). The headline score is the **worst case across instruments** — the
+minimum of the chosen R-metric — which a strong instrument cannot lift (the
+issue's non-domination constraint, satisfied by construction). It is the
+cross-instrument sibling of `PlateauMode::Worst` (which already takes the
+most-pessimistic neighbour). Because R-metrics are all higher-is-better, the
+worst case is unconditionally the minimum (no direction branch). Alongside it the
+aggregate carries the **sign-agreement count** (how many instruments are
+net-positive) and the **full per-instrument breakdown** — together these fully
+express "consistency of sign and magnitude".
+
+This honours: **C23** (additive — every existing path stays byte-identical; the
+subcommand is new surface), **C9** (the reduction lives registry-side, the engine
+stays simulation), **C12** (cross-instrument is the comparison orchestration
+axis → a new `FamilyKind`), **C14/C18** (the new manifest field serde-widens with
+the same back-compat idiom as `selection`), **C1** (a pure fold over the
+per-instrument results in input order), **C2** (each per-instrument run is an
+ordinary causal sim), **C10** (R-only).
+
+## Concrete code shapes
+
+### The user-facing program (the acceptance evidence)
+
+The researcher has a candidate and asks "does it generalize across instruments?":
+
+```
+aura generalize --strategy stage1-r --real GER40,USDJPY \
+                --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 \
+                --from 1725148800000 --to 1727740799999
+```
+
+Expected stdout (the aggregate object + the persisted family handle; numbers
+illustrative):
+
+```json
+{"generalize":{"metric":"expectancy_r","n_instruments":2,"worst_case":-0.07,"sign_agreement":1,"per_instrument":[["GER40",0.31],["USDJPY",-0.07]]}}
+{"family_id":"generalize-0"}
+```
+
+The aggregate is the diagnosis: the candidate makes `+0.31 R/trade` on GER40 but
+**loses** on USDJPY, so its worst-case generalization is negative and only 1 of 2
+instruments agrees in sign — a false-discovery flag the per-instrument view would
+have hidden behind GER40's strength.
+
+A second `--metric` selects the R-metric (default `expectancy_r`):
+
+```
+aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 \
+                --stop-length 14 --stop-k 2.0 --metric sqn_normalized
+```
+
+Must-fail surfaces (each exits 2 with a clear message):
+
+```
+aura generalize --strategy stage1-r --real GER40 --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
+# refuse: a generalization needs >= 2 instruments
+
+aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 --metric total_pips
+# refuse: total_pips is not comparable across instruments; cross-instrument scoring is R-only
+
+aura generalize --strategy stage1-r --real GER40,USDJPY --fast 2,3 --slow 12 --stop-length 14 --stop-k 2.0
+# refuse: a candidate is a single cell; --fast got 2 values
+
+aura generalize --strategy sma --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0
+# refuse: generalize requires --strategy stage1-r (the candidate must produce R)
+```
+
+### Implementation shapes (before → after, secondary)
+
+**1. `RunManifest` gains a first-class instrument field** — `crates/aura-engine/src/report.rs`.
+Same serde-widening idiom as `selection` (C14/C18): legacy lines without the key
+read back as `None`. Closes a latent gap (today a real run records the
+window/broker but not *which* symbol).
+
+```rust
+// before
+pub struct RunManifest {
+    pub commit: String,
+    pub params: Vec<(String, Scalar)>,
+    pub window: (Timestamp, Timestamp),
+    pub seed: u64,
+    pub broker: String,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub selection: Option<FamilySelection>,
+}
+
+// after — add one field (every RunManifest literal threads `instrument: None`,
+// the same compile-gate the #144 `selection` field was; the planner compile-drives
+// the full literal set)
+pub struct RunManifest {
+    pub commit: String,
+    pub params: Vec<(String, Scalar)>,
+    pub window: (Timestamp, Timestamp),
+    pub seed: u64,
+    pub broker: String,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub selection: Option<FamilySelection>,
+    /// The instrument this run evaluated, when it is a real-data run; `None` for a
+    /// synthetic run and for every pre-0078 line. The "instrument set as lineage"
+    /// (C18) for a cross-instrument family is each member's stamped symbol.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub instrument: Option<String>,
+}
+```
+
+**2. `FamilyKind` gains the comparison axis** — `crates/aura-registry/src/lineage.rs`.
+A compile-gate: every `match` on `FamilyKind` (the `runs family` display, any
+kind-dispatch) must gain the arm.
+
+```rust
+// before
+pub enum FamilyKind { Sweep, MonteCarlo, WalkForward }
+// after
+pub enum FamilyKind { Sweep, MonteCarlo, WalkForward, CrossInstrument }
+```
+
+**3. The generalization reduction** — `crates/aura-registry/src/lib.rs`, a sibling
+of `optimize_deflated` / `optimize_plateau`, reusing `resolve_metric` /
+`metric_value` / `is_r_metric` (all already present). Note the input is **not** a
+`SweepFamily` and the output is **not** `(SweepPoint, FamilySelection)` — this is
+an aggregator, so it has its own value type.
+
+```rust
+/// The cross-instrument generalization aggregate (#146): a brought candidate's
+/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement
+/// count + the full breakdown. R-only (C10): `total_pips` is not comparable across
+/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
+/// order. The headline `worst_case` is `min_i metric_i`; since every R-metric is
+/// higher-is-better, the min is unconditionally the conservative floor (no
+/// direction branch, unlike `PlateauMode::Worst`).
+pub struct Generalization {
+    pub selection_metric: String,
+    pub n_instruments: usize,
+    pub worst_case: f64,           // min_i metric_i
+    pub sign_agreement: usize,     // #{ i : metric_i > 0 }
+    pub per_instrument: Vec<(String, f64)>,
+}
+
+pub fn generalization(
+    per_instrument: &[(String, &RunReport)],
+    metric: &str,
+) -> Result<Generalization, RegistryError> {
+    let m = resolve_metric(metric)?;
+    if !is_r_metric(m) {
+        return Err(RegistryError::NonRMetric(metric.to_string()));
+    }
+    if per_instrument.len() < 2 {
+        return Err(RegistryError::TooFewInstruments(per_instrument.len()));
+    }
+    let vals: Vec<(String, f64)> = per_instrument
+        .iter()
+        .map(|(label, rep)| (label.clone(), metric_value(rep, m)))
+        .collect();
+    let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
+    let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
+    Ok(Generalization {
+        selection_metric: metric.to_string(),
+        n_instruments: vals.len(),
+        worst_case,
+        sign_agreement,
+        per_instrument: vals,
+    })
+}
+```
+
+Two new `RegistryError` variants (in-style with the existing typed variants):
+
+```rust
+pub enum RegistryError {
+    Io(std::io::Error),
+    Parse { line: usize, source: serde_json::Error },
+    UnknownMetric(String),
+    NonRMetric(String),         // a known metric that is not R-based (cross-instrument is R-only)
+    TooFewInstruments(usize),   // a generalization needs >= 2 instruments
+}
+```
+
+**4. The CLI subcommand** — `crates/aura-cli/src/main.rs`. A new dispatch arm beside
+`sweep` / `walkforward`, a `parse_generalize_args`, and a `run_generalize` that
+loops the candidate over the instruments, stamps each manifest's `instrument`,
+reduces, prints, and persists.
+
+```rust
+// dispatch (beside the existing ["sweep", ..] / ["walkforward", ..] arms)
+["generalize", rest @ ..] => match parse_generalize_args(rest) {
+    Ok((name, symbols, grid, metric, from_ms, to_ms)) => {
+        run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms)
+    }
+    Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); }
+},
+
+// run_generalize (shape): one full-window stage1-r R run per instrument, the
+// candidate Stage1RGrid pinned to single values; stamp manifest.instrument; reduce.
+fn run_generalize(name: &str, symbols: &[String], grid: &Stage1RGrid,
+                  metric: &str, from_ms: Option<i64>, to_ms: Option<i64>) {
+    let mut members: Vec<RunReport> = Vec::new();
+    for symbol in symbols {
+        // the existing real-source construction (per-instrument pip_or_refuse + has_symbol)
+        let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms };
+        let data = DataSource::from_choice(choice);
+        let family = stage1_r_sweep_family(None, &data, grid);   // single-cell grid -> 1 member
+        let mut report = family.points[0].report.clone();
+        report.manifest.instrument = Some(symbol.clone());
+        members.push(report);
+    }
+    let pairs: Vec<(String, &RunReport)> =
+        symbols.iter().cloned().zip(members.iter()).collect();
+    let agg = match aura_registry::generalization(&pairs, metric) {
+        Ok(a) => a,
+        Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); }
+    };
+    println!("{}", generalize_json(&agg));                       // {"generalize": { ... }}
+    // persist the M runs as a CrossInstrument family (same Err-handling as `sweep`)
+    match reg.append_family(name, FamilyKind::CrossInstrument, &members) {
+        Ok(id) => println!("{{\"family_id\":\"{id}\"}}"),
+        Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); }
+    }
+}
+```
+
+The candidate is a `Stage1RGrid` whose `fast`/`slow`/`stop_length`/`stop_k` each
+carry **exactly one** value; `parse_generalize_args` requires all four explicitly
+and refuses a multi-value flag (a candidate is one cell — it must not default to
+the multi-value sweep grid). `--real` is parsed as a comma-list into
+`Vec<String>`; the existing single-symbol `RealWindowGrammar` / `sweep` /
+`walkforward` parsers are untouched.
+
+## Components
+
+1. **`RunManifest.instrument: Option<String>`** (aura-engine/report.rs) — the
+   instrument lineage; serde `default` / `skip_serializing_if`. Every existing
+   `RunManifest` literal threads `instrument: None` (compile-gate, same shape as
+   #144's `selection`). The synthetic and existing real run paths leave it `None`
+   (or stamp it — a non-load-bearing nicety; this cycle only requires the
+   generalize path to stamp it).
+2. **`FamilyKind::CrossInstrument`** (aura-registry/lineage.rs) — the comparison
+   axis tag; persisted on each member record. Compile-gate on `FamilyKind`
+   matches.
+3. **`generalization` + `Generalization` + two `RegistryError` variants**
+   (aura-registry/lib.rs) — the R-only reduction beside the sibling selectors,
+   reusing `resolve_metric` / `metric_value` / `is_r_metric`.
+4. **CLI: `generalize` dispatch arm, `parse_generalize_args`, `run_generalize`,
+   `generalize_json`** (aura-cli/main.rs) — the candidate × instrument-list loop;
+   reuses `stage1_r_sweep_family` (single-cell grid), the per-instrument
+   `DataSource::Real` construction (`pip_or_refuse` + `has_symbol` refusals), and
+   `append_family`. `USAGE` gains the `aura generalize ...` form.
+
+## Data flow
+
+```
+candidate (single-cell Stage1RGrid) + [SYM_1 .. SYM_M] + window
+  -> for each SYM_i:
+       DataSource::Real{SYM_i}  (pip_or_refuse + has_symbol)
+       -> stage1_r_sweep_family(single-cell grid)  -> SweepFamily{ 1 point }
+       -> report = points[0].report; report.manifest.instrument = Some(SYM_i)
+  -> members: Vec<RunReport>  (length M, input order)
+  -> generalization(zip(symbols, members), metric)
+       -> Generalization{ worst_case = min_i, sign_agreement, per_instrument }
+  -> print {"generalize": agg}
+  -> append_family(name, CrossInstrument, members) -> family_id
+  -> print {"family_id": id}
+```
+
+The aggregate is a pure function of the persisted members, so it is recomputable
+on reload (the `McAggregate` precedent) — no new persisted family-level record
+type is introduced.
+
+## Error handling
+
+Every refusal is stderr + `exit(2)`, the established CLI convention:
+
+- **< 2 instruments** — a one-instrument "generalization" is meaningless.
+  Enforced both at `parse_generalize_args` (early, on the parsed list) and in
+  `generalization` (`TooFewInstruments`, the reusable-core guard).
+- **non-R / pip metric** — cross-instrument scoring is R-only;
+  `generalization` returns `NonRMetric`. The CLI surfaces it as exit 2.
+- **strategy not stage1-r** — the candidate must produce R; `parse_generalize_args`
+  refuses any `--strategy` other than `stage1-r`.
+- **a multi-value candidate flag** — a candidate is a single cell;
+  `parse_generalize_args` refuses when any of `--fast/--slow/--stop-length/--stop-k`
+  carries more than one value, or is absent.
+- **empty / duplicate symbol** in the list — refuse (a duplicate would
+  double-count one instrument in the worst-case and sign-agreement).
+- **a symbol with no recorded geometry / no archive** — the existing
+  `pip_or_refuse` / `no_real_data` refusals fire per instrument before any run.
+
+## Testing strategy
+
+RED-first where pinnable. Per-crate test command, single positional filter.
+
+Registry unit tests (aura-registry):
+- **worst_case picks the floor:** a fabricated 3-member set (one strongly
+  positive, one negative, one mid) → `worst_case == the negative value`, **not**
+  the mean; `sign_agreement == 2`.
+- **non-domination:** appending a hugely-positive fourth instrument does **not**
+  raise `worst_case` (the min is unchanged).
+- **R-only refusal:** `generalization(.., "total_pips")` → `Err(NonRMetric)`.
+- **too-few refusal:** a 1-element input → `Err(TooFewInstruments(1))`.
+- **determinism (C1):** the same inputs → an identical `Generalization` (it is a
+  pure fold).
+
+Engine serde tests (aura-engine):
+- **back-compat (C14/C18):** a pre-0078 manifest JSON line without `instrument`
+  deserializes with `instrument == None`; a round-trip with `instrument ==
+  Some("GER40")` preserves it; the existing `RunManifest`/`FamilySelection` serde
+  and legacy-load tests stay green.
+
+CLI E2E (aura-cli/tests/cli_run.rs), gated-skip on absent local data exactly like
+the existing `--real` tests:
+- `aura generalize --strategy stage1-r --real GER40,USDJPY --fast .. --slow ..
+  --stop-length .. --stop-k .. --from .. --to ..` over the shared Sept-2024 window
+  → stdout carries a `{"generalize": ...}` object with `n_instruments == 2`, a
+  `per_instrument` breakdown naming both symbols, a `worst_case`, a
+  `sign_agreement`, and a following `{"family_id":"generalize-0"}`; when the
+  archive is absent, the run hits the no-data refusal and the test skips cleanly
+  (the established pattern).
+- a must-fail E2E: `--real GER40` (single instrument) exits 2 with the
+  >=2-instruments message (no data needed — the arity refusal precedes any run).
+
+C23 preservation:
+- the existing `sweep` / `walkforward` / `mc` / standalone-run goldens stay
+  **byte-identical** — `generalize` is new surface and touches no existing path.
+
+## Acceptance criteria
+
+Against aura's feature-acceptance criterion:
+
+1. **The intended audience reaches for it.** The aura researcher validating a
+   candidate naturally asks "does this hold across instruments?" — the last
+   anti-false-discovery piece of the milestone. The worked `aura generalize`
+   invocation above is the evidence: it is the command a researcher writes, and
+   its output is the generalization verdict.
+2. **It measurably improves correctness.** A candidate that looks good on one
+   instrument but fails the worst-case-across-instruments floor (negative
+   `worst_case`, low `sign_agreement`) is now caught — a real defense the
+   per-instrument view hides. It composes downstream of #144/#145.
+3. **It reintroduces no failure class the core constraints eliminate.** R-only
+   (C10), behaviour-preserving (C23 — existing paths byte-identical), deterministic
+   (C1), causal per-instrument sims (C2), registry-side policy (C9), serde-widened
+   with full back-compat (C14/C18). No invariant is touched.
+
+## Out of scope (named, deferred)
+
+- **Cross-instrument selection (reading B):** running a full sweep per instrument
+  and picking the cell that generalizes best across instruments. A heavier
+  follow-on that conflates selection with validation; this cycle is the validator.
+- **Per-instrument walk-forward evaluation:** the first version runs a full-window
+  R evaluation per instrument (honest OOS because the candidate is fixed). A
+  per-instrument walk-forward knob is a later refinement.
+- **A persisted family-level aggregate record:** the aggregate is recomputable from
+  the members (the `McAggregate` precedent); no new persisted record type.
+- **Cross-instrument R-pooling of trade series:** #139's deferred cross-symbol
+  pooling stays deferred.
+- **Pip-metric generalization:** undefined across instruments (C10); refused.