From 428d0922984322a45047706d40067c598ce24966 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 26 Jun 2026 18:19:51 +0200 Subject: [PATCH] spec: 0078 cross-instrument generalization (boss-signed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Last piece of the Inferential-validation milestone (#146): grade how consistently a brought candidate holds across a set of instruments. Aggregator/validator, not a third selector (user-ratified "mach es so", decision + derivations minuted on #146): a new `aura generalize` subcommand runs a single candidate (single-cell stage1-r grid) across an instrument list, collects per-instrument R-metrics, and reduces them to a worst-case floor (min over instruments) + a sign-agreement count + the per-instrument breakdown. R-only (C10 — R is the only cross-instrument-comparable unit); the worst-case min is the cross-instrument sibling of PlateauMode::Worst and satisfies the non-domination constraint by construction. New FamilyKind::CrossInstrument (C12) + a first-class RunManifest.instrument lineage field (serde-widened, C14/C18); the reduction lives registry-side beside optimize_deflated/optimize_plateau (C9). Additive — existing sweep/walkforward/mc paths stay byte-identical (C23). grounding-check PASS: all 9 current-behaviour assumptions ratified by green tests. refs #146 --- .../0078-cross-instrument-generalization.md | 396 ++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 docs/specs/0078-cross-instrument-generalization.md diff --git a/docs/specs/0078-cross-instrument-generalization.md b/docs/specs/0078-cross-instrument-generalization.md new file mode 100644 index 0000000..e6a9ca6 --- /dev/null +++ b/docs/specs/0078-cross-instrument-generalization.md @@ -0,0 +1,396 @@ +# Cross-instrument generalization as a validation step — Design Spec + +**Date:** 2026-06-26 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Add a first-class validation step that grades how consistently a single +**candidate** holds across a set of instruments. This is the last piece of the +"Inferential validation" milestone and the across-instrument axis of the same +anti-false-discovery concern as trials-deflation (#144, within one family) and +plateau-over-peak (#145, within one instrument's surface): an edge that appears +on one instrument can be a fit to that series' noise; an edge that survives +across several independent instruments is far less likely to be spurious. + +The step is an **aggregator / validator**, not a third selector. It takes a +*brought* candidate — one already chosen (e.g. by deflation or plateau on a +reference instrument, or authored directly) — runs it across an instrument list, +collects per-instrument R-metrics, and reduces them to a generalization grade. +It does **not** select a winner across instruments (that is a heavier follow-on, +out of scope below). A generalization grade is an aggregate (like `McAggregate`), +not a within-family selection, so it is **not** recorded on `FamilySelection` +(which annotates a within-family winner that does not exist here). + +## Architecture + +A new `aura generalize` subcommand. Input is a single **candidate** (the four +stage1-r grid knobs pinned to single values: `--fast --slow --stop-length +--stop-k`) crossed with an **instrument list** (`--real SYM1,SYM2,...`, +comma-separated; a shared optional `--from`/`--to` window, full available range +per instrument if omitted). For each instrument the candidate runs as a single +full-window stage1-r R evaluation, the M `RunReport`s are collected, reduced to a +generalization aggregate, the per-instrument breakdown and the aggregate are +printed, and the M runs are persisted as a `CrossInstrument` family. + +The per-instrument run is an **honest out-of-sample test by construction**: +because the candidate is fixed/external — not fit on that instrument — a single +full-window run on each instrument is already OOS for it. No per-instrument +walk-forward is needed for this first correct version. + +The reduction is **R-only**. R is the only account- and instrument-agnostic unit +(C10); `total_pips` is meaningless across instruments (different pip scales and +volatility). The headline score is the **worst case across instruments** — the +minimum of the chosen R-metric — which a strong instrument cannot lift (the +issue's non-domination constraint, satisfied by construction). It is the +cross-instrument sibling of `PlateauMode::Worst` (which already takes the +most-pessimistic neighbour). Because R-metrics are all higher-is-better, the +worst case is unconditionally the minimum (no direction branch). Alongside it the +aggregate carries the **sign-agreement count** (how many instruments are +net-positive) and the **full per-instrument breakdown** — together these fully +express "consistency of sign and magnitude". + +This honours: **C23** (additive — every existing path stays byte-identical; the +subcommand is new surface), **C9** (the reduction lives registry-side, the engine +stays simulation), **C12** (cross-instrument is the comparison orchestration +axis → a new `FamilyKind`), **C14/C18** (the new manifest field serde-widens with +the same back-compat idiom as `selection`), **C1** (a pure fold over the +per-instrument results in input order), **C2** (each per-instrument run is an +ordinary causal sim), **C10** (R-only). + +## Concrete code shapes + +### The user-facing program (the acceptance evidence) + +The researcher has a candidate and asks "does it generalize across instruments?": + +``` +aura generalize --strategy stage1-r --real GER40,USDJPY \ + --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 \ + --from 1725148800000 --to 1727740799999 +``` + +Expected stdout (the aggregate object + the persisted family handle; numbers +illustrative): + +```json +{"generalize":{"metric":"expectancy_r","n_instruments":2,"worst_case":-0.07,"sign_agreement":1,"per_instrument":[["GER40",0.31],["USDJPY",-0.07]]}} +{"family_id":"generalize-0"} +``` + +The aggregate is the diagnosis: the candidate makes `+0.31 R/trade` on GER40 but +**loses** on USDJPY, so its worst-case generalization is negative and only 1 of 2 +instruments agrees in sign — a false-discovery flag the per-instrument view would +have hidden behind GER40's strength. + +A second `--metric` selects the R-metric (default `expectancy_r`): + +``` +aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 \ + --stop-length 14 --stop-k 2.0 --metric sqn_normalized +``` + +Must-fail surfaces (each exits 2 with a clear message): + +``` +aura generalize --strategy stage1-r --real GER40 --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 +# refuse: a generalization needs >= 2 instruments + +aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 --metric total_pips +# refuse: total_pips is not comparable across instruments; cross-instrument scoring is R-only + +aura generalize --strategy stage1-r --real GER40,USDJPY --fast 2,3 --slow 12 --stop-length 14 --stop-k 2.0 +# refuse: a candidate is a single cell; --fast got 2 values + +aura generalize --strategy sma --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 +# refuse: generalize requires --strategy stage1-r (the candidate must produce R) +``` + +### Implementation shapes (before → after, secondary) + +**1. `RunManifest` gains a first-class instrument field** — `crates/aura-engine/src/report.rs`. +Same serde-widening idiom as `selection` (C14/C18): legacy lines without the key +read back as `None`. Closes a latent gap (today a real run records the +window/broker but not *which* symbol). + +```rust +// before +pub struct RunManifest { + pub commit: String, + pub params: Vec<(String, Scalar)>, + pub window: (Timestamp, Timestamp), + pub seed: u64, + pub broker: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selection: Option, +} + +// after — add one field (every RunManifest literal threads `instrument: None`, +// the same compile-gate the #144 `selection` field was; the planner compile-drives +// the full literal set) +pub struct RunManifest { + pub commit: String, + pub params: Vec<(String, Scalar)>, + pub window: (Timestamp, Timestamp), + pub seed: u64, + pub broker: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub selection: Option, + /// The instrument this run evaluated, when it is a real-data run; `None` for a + /// synthetic run and for every pre-0078 line. The "instrument set as lineage" + /// (C18) for a cross-instrument family is each member's stamped symbol. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub instrument: Option, +} +``` + +**2. `FamilyKind` gains the comparison axis** — `crates/aura-registry/src/lineage.rs`. +A compile-gate: every `match` on `FamilyKind` (the `runs family` display, any +kind-dispatch) must gain the arm. + +```rust +// before +pub enum FamilyKind { Sweep, MonteCarlo, WalkForward } +// after +pub enum FamilyKind { Sweep, MonteCarlo, WalkForward, CrossInstrument } +``` + +**3. The generalization reduction** — `crates/aura-registry/src/lib.rs`, a sibling +of `optimize_deflated` / `optimize_plateau`, reusing `resolve_metric` / +`metric_value` / `is_r_metric` (all already present). Note the input is **not** a +`SweepFamily` and the output is **not** `(SweepPoint, FamilySelection)` — this is +an aggregator, so it has its own value type. + +```rust +/// The cross-instrument generalization aggregate (#146): a brought candidate's +/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement +/// count + the full breakdown. R-only (C10): `total_pips` is not comparable across +/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input +/// order. The headline `worst_case` is `min_i metric_i`; since every R-metric is +/// higher-is-better, the min is unconditionally the conservative floor (no +/// direction branch, unlike `PlateauMode::Worst`). +pub struct Generalization { + pub selection_metric: String, + pub n_instruments: usize, + pub worst_case: f64, // min_i metric_i + pub sign_agreement: usize, // #{ i : metric_i > 0 } + pub per_instrument: Vec<(String, f64)>, +} + +pub fn generalization( + per_instrument: &[(String, &RunReport)], + metric: &str, +) -> Result { + let m = resolve_metric(metric)?; + if !is_r_metric(m) { + return Err(RegistryError::NonRMetric(metric.to_string())); + } + if per_instrument.len() < 2 { + return Err(RegistryError::TooFewInstruments(per_instrument.len())); + } + let vals: Vec<(String, f64)> = per_instrument + .iter() + .map(|(label, rep)| (label.clone(), metric_value(rep, m))) + .collect(); + let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min); + let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count(); + Ok(Generalization { + selection_metric: metric.to_string(), + n_instruments: vals.len(), + worst_case, + sign_agreement, + per_instrument: vals, + }) +} +``` + +Two new `RegistryError` variants (in-style with the existing typed variants): + +```rust +pub enum RegistryError { + Io(std::io::Error), + Parse { line: usize, source: serde_json::Error }, + UnknownMetric(String), + NonRMetric(String), // a known metric that is not R-based (cross-instrument is R-only) + TooFewInstruments(usize), // a generalization needs >= 2 instruments +} +``` + +**4. The CLI subcommand** — `crates/aura-cli/src/main.rs`. A new dispatch arm beside +`sweep` / `walkforward`, a `parse_generalize_args`, and a `run_generalize` that +loops the candidate over the instruments, stamps each manifest's `instrument`, +reduces, prints, and persists. + +```rust +// dispatch (beside the existing ["sweep", ..] / ["walkforward", ..] arms) +["generalize", rest @ ..] => match parse_generalize_args(rest) { + Ok((name, symbols, grid, metric, from_ms, to_ms)) => { + run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms) + } + Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } +}, + +// run_generalize (shape): one full-window stage1-r R run per instrument, the +// candidate Stage1RGrid pinned to single values; stamp manifest.instrument; reduce. +fn run_generalize(name: &str, symbols: &[String], grid: &Stage1RGrid, + metric: &str, from_ms: Option, to_ms: Option) { + let mut members: Vec = Vec::new(); + for symbol in symbols { + // the existing real-source construction (per-instrument pip_or_refuse + has_symbol) + let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms }; + let data = DataSource::from_choice(choice); + let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member + let mut report = family.points[0].report.clone(); + report.manifest.instrument = Some(symbol.clone()); + members.push(report); + } + let pairs: Vec<(String, &RunReport)> = + symbols.iter().cloned().zip(members.iter()).collect(); + let agg = match aura_registry::generalization(&pairs, metric) { + Ok(a) => a, + Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); } + }; + println!("{}", generalize_json(&agg)); // {"generalize": { ... }} + // persist the M runs as a CrossInstrument family (same Err-handling as `sweep`) + match reg.append_family(name, FamilyKind::CrossInstrument, &members) { + Ok(id) => println!("{{\"family_id\":\"{id}\"}}"), + Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); } + } +} +``` + +The candidate is a `Stage1RGrid` whose `fast`/`slow`/`stop_length`/`stop_k` each +carry **exactly one** value; `parse_generalize_args` requires all four explicitly +and refuses a multi-value flag (a candidate is one cell — it must not default to +the multi-value sweep grid). `--real` is parsed as a comma-list into +`Vec`; the existing single-symbol `RealWindowGrammar` / `sweep` / +`walkforward` parsers are untouched. + +## Components + +1. **`RunManifest.instrument: Option`** (aura-engine/report.rs) — the + instrument lineage; serde `default` / `skip_serializing_if`. Every existing + `RunManifest` literal threads `instrument: None` (compile-gate, same shape as + #144's `selection`). The synthetic and existing real run paths leave it `None` + (or stamp it — a non-load-bearing nicety; this cycle only requires the + generalize path to stamp it). +2. **`FamilyKind::CrossInstrument`** (aura-registry/lineage.rs) — the comparison + axis tag; persisted on each member record. Compile-gate on `FamilyKind` + matches. +3. **`generalization` + `Generalization` + two `RegistryError` variants** + (aura-registry/lib.rs) — the R-only reduction beside the sibling selectors, + reusing `resolve_metric` / `metric_value` / `is_r_metric`. +4. **CLI: `generalize` dispatch arm, `parse_generalize_args`, `run_generalize`, + `generalize_json`** (aura-cli/main.rs) — the candidate × instrument-list loop; + reuses `stage1_r_sweep_family` (single-cell grid), the per-instrument + `DataSource::Real` construction (`pip_or_refuse` + `has_symbol` refusals), and + `append_family`. `USAGE` gains the `aura generalize ...` form. + +## Data flow + +``` +candidate (single-cell Stage1RGrid) + [SYM_1 .. SYM_M] + window + -> for each SYM_i: + DataSource::Real{SYM_i} (pip_or_refuse + has_symbol) + -> stage1_r_sweep_family(single-cell grid) -> SweepFamily{ 1 point } + -> report = points[0].report; report.manifest.instrument = Some(SYM_i) + -> members: Vec (length M, input order) + -> generalization(zip(symbols, members), metric) + -> Generalization{ worst_case = min_i, sign_agreement, per_instrument } + -> print {"generalize": agg} + -> append_family(name, CrossInstrument, members) -> family_id + -> print {"family_id": id} +``` + +The aggregate is a pure function of the persisted members, so it is recomputable +on reload (the `McAggregate` precedent) — no new persisted family-level record +type is introduced. + +## Error handling + +Every refusal is stderr + `exit(2)`, the established CLI convention: + +- **< 2 instruments** — a one-instrument "generalization" is meaningless. + Enforced both at `parse_generalize_args` (early, on the parsed list) and in + `generalization` (`TooFewInstruments`, the reusable-core guard). +- **non-R / pip metric** — cross-instrument scoring is R-only; + `generalization` returns `NonRMetric`. The CLI surfaces it as exit 2. +- **strategy not stage1-r** — the candidate must produce R; `parse_generalize_args` + refuses any `--strategy` other than `stage1-r`. +- **a multi-value candidate flag** — a candidate is a single cell; + `parse_generalize_args` refuses when any of `--fast/--slow/--stop-length/--stop-k` + carries more than one value, or is absent. +- **empty / duplicate symbol** in the list — refuse (a duplicate would + double-count one instrument in the worst-case and sign-agreement). +- **a symbol with no recorded geometry / no archive** — the existing + `pip_or_refuse` / `no_real_data` refusals fire per instrument before any run. + +## Testing strategy + +RED-first where pinnable. Per-crate test command, single positional filter. + +Registry unit tests (aura-registry): +- **worst_case picks the floor:** a fabricated 3-member set (one strongly + positive, one negative, one mid) → `worst_case == the negative value`, **not** + the mean; `sign_agreement == 2`. +- **non-domination:** appending a hugely-positive fourth instrument does **not** + raise `worst_case` (the min is unchanged). +- **R-only refusal:** `generalization(.., "total_pips")` → `Err(NonRMetric)`. +- **too-few refusal:** a 1-element input → `Err(TooFewInstruments(1))`. +- **determinism (C1):** the same inputs → an identical `Generalization` (it is a + pure fold). + +Engine serde tests (aura-engine): +- **back-compat (C14/C18):** a pre-0078 manifest JSON line without `instrument` + deserializes with `instrument == None`; a round-trip with `instrument == + Some("GER40")` preserves it; the existing `RunManifest`/`FamilySelection` serde + and legacy-load tests stay green. + +CLI E2E (aura-cli/tests/cli_run.rs), gated-skip on absent local data exactly like +the existing `--real` tests: +- `aura generalize --strategy stage1-r --real GER40,USDJPY --fast .. --slow .. + --stop-length .. --stop-k .. --from .. --to ..` over the shared Sept-2024 window + → stdout carries a `{"generalize": ...}` object with `n_instruments == 2`, a + `per_instrument` breakdown naming both symbols, a `worst_case`, a + `sign_agreement`, and a following `{"family_id":"generalize-0"}`; when the + archive is absent, the run hits the no-data refusal and the test skips cleanly + (the established pattern). +- a must-fail E2E: `--real GER40` (single instrument) exits 2 with the + >=2-instruments message (no data needed — the arity refusal precedes any run). + +C23 preservation: +- the existing `sweep` / `walkforward` / `mc` / standalone-run goldens stay + **byte-identical** — `generalize` is new surface and touches no existing path. + +## Acceptance criteria + +Against aura's feature-acceptance criterion: + +1. **The intended audience reaches for it.** The aura researcher validating a + candidate naturally asks "does this hold across instruments?" — the last + anti-false-discovery piece of the milestone. The worked `aura generalize` + invocation above is the evidence: it is the command a researcher writes, and + its output is the generalization verdict. +2. **It measurably improves correctness.** A candidate that looks good on one + instrument but fails the worst-case-across-instruments floor (negative + `worst_case`, low `sign_agreement`) is now caught — a real defense the + per-instrument view hides. It composes downstream of #144/#145. +3. **It reintroduces no failure class the core constraints eliminate.** R-only + (C10), behaviour-preserving (C23 — existing paths byte-identical), deterministic + (C1), causal per-instrument sims (C2), registry-side policy (C9), serde-widened + with full back-compat (C14/C18). No invariant is touched. + +## Out of scope (named, deferred) + +- **Cross-instrument selection (reading B):** running a full sweep per instrument + and picking the cell that generalizes best across instruments. A heavier + follow-on that conflates selection with validation; this cycle is the validator. +- **Per-instrument walk-forward evaluation:** the first version runs a full-window + R evaluation per instrument (honest OOS because the candidate is fixed). A + per-instrument walk-forward knob is a later refinement. +- **A persisted family-level aggregate record:** the aggregate is recomputable from + the members (the `McAggregate` precedent); no new persisted record type. +- **Cross-instrument R-pooling of trade series:** #139's deferred cross-symbol + pooling stays deferred. +- **Pip-metric generalization:** undefined across instruments (C10); refused.