diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index 201f5ab..ea99733 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -929,6 +929,16 @@ record, and a member's window is **producer-supplied** via never a materialized-`Vec` scan at the call site. CLI surface: `aura mc`, `aura runs families`, `aura runs family [rank ]`; `aura sweep` / `walkforward` / `mc` persist via `append_family` with an optional `--name`. +**Realization (cycle 0078 — cross-instrument family + instrument lineage, #146).** +The comparison axis (C12) is realized as a `FamilyKind::CrossInstrument` family: +`aura generalize` runs one candidate across an instrument list and persists the M +per-instrument runs via `append_family`, each member self-identifying through a new +first-class `RunManifest.instrument` lineage field (serde-widened with +`skip_serializing_if`, so legacy lines and every non-cross-instrument path stay +byte-identical — C14/C23). The cross-instrument *generalization score* (worst-case R +floor + sign-agreement + per-instrument breakdown) is a **recomputable aggregate** +over those members, not a persisted family-level record — distinct from #144/#145's +per-winner selection annotation on `RunManifest.selection`. Deferred (Non-goals): content-addressed identity + replay-dedup; the "run-diff" depth and ranking families against each other (cross-family, vs. within-family); and a live producer for the flat `runs.jsonl` standalone-run path — no CLI command @@ -1422,8 +1432,8 @@ load-bearing in the flat graph. significance read (#139, landed cycle 0075). Distinct from the two threads above — neither orchestration *composability* (#109) nor a search *policy* (Bayesian/ genetic), but the statistical *validity* of the selection itself. Not a - C-invariant until built; recorded as the World's third (now two of three pieces - built) half. + C-invariant until built; recorded as the World's third half — **now structurally + complete** (all three pieces built, cycle 0078). **Status (cycle 0076):** the trials-deflation piece (#144) landed — `optimize_deflated` (aura-registry) wraps `optimize` and records, additively on the winning member's manifest (C18, the new `RunManifest.selection`), a deflated @@ -1442,8 +1452,20 @@ load-bearing in the flat graph. byte-identical (C23). The grid lattice surfaces from the engine (`SweepBinder::sweep_with_lattice`); the policy stays in aura-registry with `walk_forward` selection-agnostic (C9). Selection is now a *pluggable objective* - (bare argmax → trials-deflated → plateau), not a single hard-wired argmax. #146 - (cross-instrument generalization) remains the last open piece of the milestone. + (bare argmax → trials-deflated → plateau), not a single hard-wired argmax. + **Status (cycle 0078):** the last piece — cross-instrument generalization (#146) — + landed, completing the inferential half. Unlike #144/#145 (which *select + annotate* + a within-family winner on `RunManifest.selection`), #146 is an **aggregator/ + validator**, not a selector: the `aura generalize` subcommand runs a *brought* + candidate across an instrument list and `generalization` (aura-registry) reduces the + per-instrument R-metrics to a **worst-case floor** (min over instruments — the + cross-instrument sibling of `PlateauMode::Worst`, R-only per C10) + a sign-agreement + count + the per-instrument breakdown. It is a *recomputable aggregate* over a new + `FamilyKind::CrossInstrument` family (C12's comparison axis, realized), each member + self-identifying via the new `RunManifest.instrument` lineage field (C18) — *not* + stamped on `FamilySelection` (there is no within-family winner to annotate). The + inferential half is now structurally built; the per-candidate OOS bootstrap (#139, + cycle 0075) is its adjacent significance read. - **`aura-std` contents** — the crate exists (doc-only); which universal blocks land first follows the walking-skeleton's needs. - **`strategies/` split** — a later split, *inside a project*, of top-level diff --git a/docs/plans/0078-cross-instrument-generalization.md b/docs/plans/0078-cross-instrument-generalization.md deleted file mode 100644 index cdcb049..0000000 --- a/docs/plans/0078-cross-instrument-generalization.md +++ /dev/null @@ -1,789 +0,0 @@ -# Cross-instrument generalization — Implementation Plan - -> **Parent spec:** `docs/specs/0078-cross-instrument-generalization.md` -> -> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run -> this plan. Steps use `- [ ]` checkboxes for tracking. - -**Goal:** Add an `aura generalize` subcommand that grades how consistently a -single candidate holds across a set of instruments (worst-case R floor + -sign-agreement + per-instrument breakdown), as the last anti-false-discovery -piece of the Inferential-validation milestone (#146). - -**Architecture:** A first-class `RunManifest.instrument` lineage field -(serde-widened, C14/C18) + a new `FamilyKind::CrossInstrument` (C12) + an R-only -`generalization` reduction in aura-registry beside `optimize_deflated`/ -`optimize_plateau` (C9) + a CLI `generalize` subcommand that runs a single-cell -stage1-r candidate across an instrument list, stamps each member's instrument, -reduces, prints, and persists a CrossInstrument family. Additive — existing -sweep/walkforward/mc/standalone paths stay byte-identical (C23). - -**Tech Stack:** aura-engine (`RunManifest`), aura-registry (`FamilyKind`, -`RegistryError`, `generalization`/`Generalization`/`check_r_metric`), aura-cli -(`generalize` subcommand). - ---- - -**Files this plan creates or modifies:** - -- Modify: `crates/aura-engine/src/report.rs:463-482` — add `RunManifest.instrument` field + back-compat serde tests -- Modify: `crates/aura-registry/src/compat.rs:26-80` — the `RunManifestRead` mirror + destructure + literal (lockstep with the engine field) -- Modify: `crates/aura-registry/src/lineage.rs:22-27` — `FamilyKind::CrossInstrument` variant -- Modify: `crates/aura-registry/src/lib.rs:429-455` — `Generalization` struct + `generalization` fn + `check_r_metric` fn + two `RegistryError` variants + Display arms + unit tests -- Modify: `crates/aura-cli/src/main.rs` — imports (`:26`), `sim_optimal_manifest` literal (`:170`), `parse_generalize_args` + `run_generalize` + `generalize_json` (new fns near the sweep helpers), dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests -- Modify (thread `instrument: None`): every in-workspace `RunManifest { .. }` literal — enumerated set in Task 1, compiler-driven by `cargo check --workspace --all-targets` -- Test: `crates/aura-cli/tests/cli_run.rs` — gated-skip success E2E + two data-free must-fail E2Es - -**Out of scope (named in the spec):** reading-B cross-instrument *selection*; -per-instrument walk-forward; a persisted family-aggregate record; cross-symbol -R-pooling (#139 stays deferred); pip-metric generalization. `fieldtests/` is NOT -a workspace member (verified: workspace members are the 7 `crates/` only), so its -16 `RunManifest` literals are out of the compile-gate. - ---- - -## Task 1: `RunManifest.instrument` field + compat mirror + thread all literals + back-compat tests - -**Files:** -- Modify: `crates/aura-engine/src/report.rs:463-482` (struct) + the `#[cfg(test)] mod tests` (new serde tests) -- Modify: `crates/aura-registry/src/compat.rs:26-80` (mirror struct + destructure + literal — lockstep) -- Modify: every in-workspace `RunManifest { .. }` literal (enumerated below) - -> **RED-shape note for the implementer:** a serde-widening struct field is a -> *compile-gated additive* change — the back-compat test that asserts -> `m.instrument == None` / round-trips `Some` cannot exist until the field does, -> so the field and its pin land in one diff (the #144 `selection`-field -> precedent). The "RED" is the field's compile-absence, not a separate failing -> commit. Do not block as spec-ambiguous: add the field, thread the literals, -> add the tests, verify green. - -- [ ] **Step 1: Add the `instrument` field to `RunManifest`** - -In `crates/aura-engine/src/report.rs`, after the `selection` field (`:480-481`), -add: - -```rust - #[serde(default, skip_serializing_if = "Option::is_none")] - pub selection: Option, - /// The instrument this run evaluated, when it is a real-data run; `None` for a - /// synthetic run and for every pre-0078 line. The "instrument set as lineage" - /// (C18) for a cross-instrument family is each member's stamped symbol. Same - /// one-directional serde widening as `selection` (C14/C18). - #[serde(default, skip_serializing_if = "Option::is_none")] - pub instrument: Option, -} -``` - -- [ ] **Step 2: Thread the field through the compat mirror (lockstep)** - -In `crates/aura-registry/src/compat.rs`, the read-side mirror `RunManifestRead` -(`:27-35`), its destructure (`:67`), and the `From` literal -(`:69-76`) must all gain the field, or the registry will not compile. - -Mirror struct — after `selection` (`:33-34`): -```rust - #[serde(default)] - selection: Option, - #[serde(default)] - instrument: Option, -} -``` -Destructure (`:67`): -```rust - let RunManifestRead { commit, params, window, seed, broker, selection, instrument } = r.manifest; -``` -Literal (`:69-76`) — add `instrument,` after `selection,`: -```rust - manifest: RunManifest { - commit, - params: params.into_iter().map(|(name, v)| (name, v.0)).collect(), - window, - seed, - broker, - selection, - instrument, - }, -``` - -- [ ] **Step 3: Thread `instrument: None` into the production CLI literal** - -In `crates/aura-cli/src/main.rs`, `sim_optimal_manifest` (`:170-177`) — add -`instrument: None,` after `selection: None,`: -```rust - broker: format!("sim-optimal(pip_size={pip_size})"), - selection: None, - instrument: None, - } -``` - -- [ ] **Step 4: Thread `instrument: None` into every remaining in-workspace literal** - -A struct-field addition compile-breaks every `RunManifest { .. }` literal until -threaded. There is **no `Default` impl / builder** for `RunManifest` (verified), -so each literal must add the field explicitly. Add `instrument: None,` to each of -the literals below (the value is always `None` here — the only `Some(...)` stamp -is in Task 3's `run_generalize`). Line numbers are pre-edit anchors and will -drift as Step 1 shifts `report.rs`; the **authoritative enumerator is the -compiler** — run `cargo check --workspace --all-targets` and add the field to any -site it flags as missing `instrument`. - -Known in-workspace set (test-`src`, integration tests, examples): -- `crates/aura-engine/src/blueprint.rs:942` (test) -- `crates/aura-engine/src/mc.rs:239`, `:263` (test) -- `crates/aura-engine/src/report.rs:968`, `:1344`, `:1373`, `:1394` (test — the - pre-existing test literals; the new tests in Step 5 already include the field) -- `crates/aura-engine/src/sweep.rs:664` (test) -- `crates/aura-engine/src/walkforward.rs:288` (test) -- `crates/aura-engine/tests/random_sweep_e2e.rs:112` (integration test) -- `crates/aura-registry/src/lib.rs:474` (the `report_with` fixture — threading it - here covers every registry test that builds a report via `report_with`/ - `report_with_r`), `:813` (test) -- `crates/aura-registry/src/trace_store.rs:250` (test) -- `crates/aura-ingest/tests/ger40_breakout_world.rs:65`, `real_bars.rs:82`, - `streaming_seam.rs:82` (integration tests) -- `crates/aura-ingest/examples/ger40_breakout_sweep.rs:63`, - `ger40_breakout_walkforward.rs:73`, `shared/breakout_real.rs:390` (examples) - -Plus the two pre-existing `report.rs` test literals at the top of the test module -(`runmanifest_without_selection_serialises_without_the_key` `:771`, -`family_selection_round_trips_on_the_manifest` `:787`) — add `instrument: None,` -after their `selection:` field. - -- [ ] **Step 5: Add the back-compat serde tests** - -In `crates/aura-engine/src/report.rs`, inside `#[cfg(test)] mod tests` (beside the -`runmanifest_without_selection_*` tests near `:762`), add: - -```rust - #[test] - fn runmanifest_without_instrument_field_deserialises_to_none() { - // A pre-0078 line carries no `instrument` key; serde `default` -> None. - let json = r#"{"commit":"c","params":[],"window":[0,0],"seed":0,"broker":"b"}"#; - let m: RunManifest = serde_json::from_str(json).expect("legacy manifest loads"); - assert!(m.instrument.is_none()); - } - - #[test] - fn runmanifest_instrument_round_trips_and_omits_when_none() { - let with = RunManifest { - commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), - seed: 0, broker: "b".into(), selection: None, instrument: Some("GER40".into()), - }; - let json = serde_json::to_string(&with).unwrap(); - assert!(json.contains("\"instrument\":\"GER40\""), "json: {json}"); - let back: RunManifest = serde_json::from_str(&json).unwrap(); - assert_eq!(back.instrument, Some("GER40".to_string())); - - let without = RunManifest { - commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)), - seed: 0, broker: "b".into(), selection: None, instrument: None, - }; - // skip_serializing_if omits the key when None -> existing manifest bytes unchanged (C23). - assert!(!serde_json::to_string(&without).unwrap().contains("instrument")); - } -``` - -- [ ] **Step 6: Verify the new tests pass and the workspace compiles** - -Run: `cargo test -p aura-engine runmanifest_` -Expected: PASS — `runmanifest_without_instrument_field_deserialises_to_none`, -`runmanifest_instrument_round_trips_and_omits_when_none`, and the pre-existing -`runmanifest_without_selection_*` all green (4+ tests run, 0 failed). - -- [ ] **Step 7: Verify every literal is threaded (workspace gate)** - -Run: `cargo check --workspace --all-targets` -Expected: clean (0 errors). Any "missing field `instrument`" error names a literal -Step 4 missed — add `instrument: None,` there and re-run. - -- [ ] **Step 8: Verify the whole workspace suite stays green** - -Run: `cargo test --workspace` -Expected: PASS (0 failed) — the threaded literals compile and every existing test, -including the registry compat-load tests and the #144/#145 serde tests, stays -green. - ---- - -## Task 2: aura-registry — `FamilyKind::CrossInstrument`, the `generalization` reduction, `check_r_metric`, error variants - -**Files:** -- Modify: `crates/aura-registry/src/lineage.rs:22-27` (`FamilyKind`) -- Modify: `crates/aura-registry/src/lib.rs:429-455` (struct + fns + error variants + Display) + `#[cfg(test)] mod tests` (unit tests) - -- [ ] **Step 1: Add the `CrossInstrument` family kind** - -In `crates/aura-registry/src/lineage.rs` (`:22-27`): -```rust -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] -pub enum FamilyKind { - Sweep, - MonteCarlo, - WalkForward, - CrossInstrument, -} -``` -(No exhaustive `match` over `FamilyKind` exists workspace-wide — it serializes via -the derive — so this breaks no call site; the `runs families` display picks up -`"CrossInstrument"` automatically.) - -- [ ] **Step 2: Add the two `RegistryError` variants + their Display arms** - -In `crates/aura-registry/src/lib.rs`, the enum (`:432-440`): -```rust -#[derive(Debug)] -pub enum RegistryError { - /// An I/O error reading or writing the JSONL file. - Io(std::io::Error), - /// A stored line did not parse as a `RunReport` (1-based line number). - Parse { line: usize, source: serde_json::Error }, - /// `rank_by` was given a metric name it does not know. - UnknownMetric(String), - /// A known metric that is not R-based — cross-instrument generalization is - /// R-only (R is the only account/instrument-agnostic unit, C10). - NonRMetric(String), - /// A generalization was asked for with fewer than two instruments. - TooFewInstruments(usize), -} -``` -The Display match (`:444-453`) — add two arms before the closing brace: -```rust - RegistryError::UnknownMetric(m) => write!( - f, - "unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)" - ), - RegistryError::NonRMetric(m) => write!( - f, - "metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)" - ), - RegistryError::TooFewInstruments(n) => write!( - f, - "a generalization needs >= 2 instruments, got {n}" - ), - } - } -} -``` - -- [ ] **Step 3: Write the failing unit tests for `generalization` + `check_r_metric`** - -In `crates/aura-registry/src/lib.rs`, inside `#[cfg(test)] mod tests` (the module -that already defines `report_with`/`report_with_r`), add. These reuse -`report_with_r(sqn, expectancy_r, net_expectancy_r)` (sets `metrics.r = Some` with -the given `expectancy_r`); the default metric is `expectancy_r`. - -```rust - #[test] - fn generalization_worst_case_is_the_floor_not_the_mean() { - let a = report_with_r(0.0, 0.40, 0.0); // +0.40 - let b = report_with_r(0.0, -0.20, 0.0); // -0.20 (the floor) - let c = report_with_r(0.0, 0.10, 0.0); // +0.10 - let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("C".to_string(), &c)]; - let g = generalization(&pairs, "expectancy_r").expect("R metric"); - assert_eq!(g.n_instruments, 3); - assert!((g.worst_case - (-0.20)).abs() < 1e-12, "worst_case must be the min, got {}", g.worst_case); - assert_eq!(g.sign_agreement, 2, "two instruments are net-positive"); - assert_eq!(g.per_instrument, vec![ - ("A".to_string(), 0.40), ("B".to_string(), -0.20), ("C".to_string(), 0.10), - ]); - } - - #[test] - fn generalization_is_not_lifted_by_a_strong_instrument() { - let a = report_with_r(0.0, 0.40, 0.0); - let b = report_with_r(0.0, -0.20, 0.0); - let strong = report_with_r(0.0, 99.0, 0.0); - let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("S".to_string(), &strong)]; - let g = generalization(&pairs, "expectancy_r").expect("R metric"); - assert!((g.worst_case - (-0.20)).abs() < 1e-12, "a strong instrument must not raise the floor"); - } - - #[test] - fn generalization_refuses_a_non_r_metric() { - let a = report_with_r(0.0, 0.4, 0.0); - let b = report_with_r(0.0, 0.2, 0.0); - let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)]; - match generalization(&pairs, "total_pips") { - Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"), - other => panic!("expected NonRMetric, got {other:?}"), - } - } - - #[test] - fn generalization_refuses_fewer_than_two_instruments() { - let a = report_with_r(0.0, 0.4, 0.0); - let pairs = vec![("A".to_string(), &a)]; - match generalization(&pairs, "expectancy_r") { - Err(RegistryError::TooFewInstruments(n)) => assert_eq!(n, 1), - other => panic!("expected TooFewInstruments(1), got {other:?}"), - } - } - - #[test] - fn generalization_is_deterministic() { - let a = report_with_r(0.0, 0.40, 0.0); - let b = report_with_r(0.0, -0.20, 0.0); - let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)]; - let g1 = generalization(&pairs, "expectancy_r").expect("R metric"); - let g2 = generalization(&pairs, "expectancy_r").expect("R metric"); - assert_eq!(g1, g2, "a pure fold must be deterministic (C1)"); - } - - #[test] - fn check_r_metric_accepts_r_and_refuses_pip() { - assert!(check_r_metric("expectancy_r").is_ok()); - assert!(check_r_metric("sqn_normalized").is_ok()); - match check_r_metric("total_pips") { - Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"), - other => panic!("expected NonRMetric, got {other:?}"), - } - match check_r_metric("nope") { - Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"), - other => panic!("expected UnknownMetric, got {other:?}"), - } - } -``` - -- [ ] **Step 4: Run the new tests to verify they fail** - -Run: `cargo test -p aura-registry generalization` -Expected: FAIL to compile — `cannot find function `generalization` / `check_r_metric`` -and `cannot find type `Generalization`` (the symbols do not exist yet). This is the -RED state (compile-absence). - -- [ ] **Step 5: Implement `Generalization` + `generalization` + `check_r_metric`** - -In `crates/aura-registry/src/lib.rs`, after `optimize_deflated` ends (`:429`), add: - -```rust -/// The cross-instrument generalization aggregate (#146): a brought candidate's -/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement count -/// + the full breakdown. R-only (C10): `total_pips` is not comparable across -/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input -/// order. `worst_case` is `min_i metric_i`; since every R-metric is -/// higher-is-better, the min is unconditionally the conservative floor (no -/// direction branch, unlike `PlateauMode::Worst`). -#[derive(Clone, Debug, PartialEq)] -pub struct Generalization { - pub selection_metric: String, - pub n_instruments: usize, - pub worst_case: f64, - pub sign_agreement: usize, - pub per_instrument: Vec<(String, f64)>, -} - -/// Validate that `metric` is a known, R-based ranking key — the data-free pre-check -/// the CLI runs before evaluating any instrument. The registry owns metric truth -/// (C9), so the CLI never duplicates the R-set. -pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> { - let m = resolve_metric(metric)?; - if is_r_metric(m) { - Ok(()) - } else { - Err(RegistryError::NonRMetric(metric.to_string())) - } -} - -/// Grade how consistently one candidate holds across instruments: read the chosen -/// R-metric from each per-instrument report and reduce to the worst-case floor + -/// the sign-agreement count + the per-instrument breakdown. R-only and `>= 2` -/// instruments (the metric check precedes the arity check so a bad metric is -/// reported even with an empty slice). Additive — reads, never re-ranks (C23). -pub fn generalization( - per_instrument: &[(String, &RunReport)], - metric: &str, -) -> Result { - let m = resolve_metric(metric)?; - if !is_r_metric(m) { - return Err(RegistryError::NonRMetric(metric.to_string())); - } - if per_instrument.len() < 2 { - return Err(RegistryError::TooFewInstruments(per_instrument.len())); - } - let vals: Vec<(String, f64)> = per_instrument - .iter() - .map(|(label, rep)| (label.clone(), metric_value(rep, m))) - .collect(); - let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min); - let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count(); - Ok(Generalization { - selection_metric: metric.to_string(), - n_instruments: vals.len(), - worst_case, - sign_agreement, - per_instrument: vals, - }) -} -``` - -- [ ] **Step 6: Run the new tests to verify they pass** - -Run: `cargo test -p aura-registry generalization` -Expected: PASS — the five `generalization_*` tests green. - -Run: `cargo test -p aura-registry check_r_metric` -Expected: PASS — `check_r_metric_accepts_r_and_refuses_pip` green. - -- [ ] **Step 7: Verify the registry suite stays green (preservation gate)** - -Run: `cargo test -p aura-registry` -Expected: PASS (0 failed) — the #144/#145 selector tests and the compat-load tests -all stay green. - ---- - -## Task 3: aura-cli — the `generalize` subcommand - -**Files:** -- Modify: `crates/aura-cli/src/main.rs` — imports (`:26-30`), new fns, dispatch arm (`:2993`), `USAGE` (`:2956`), parser unit tests -- Test: `crates/aura-cli/tests/cli_run.rs` — E2Es - -- [ ] **Step 1: Import the new registry items** - -In `crates/aura-cli/src/main.rs`, extend the `use aura_registry::{ ... }` block -(`:26-30`) to add `generalization, Generalization, check_r_metric`: -```rust -use aura_registry::{ - check_r_metric, generalization, group_families, mc_member_reports, optimize_deflated, - optimize_plateau, rank_by, sweep_member_reports, walkforward_member_reports, FamilyKind, - FamilyMember, Generalization, NameKind, PlateauMode, Registry, RunTraces, TraceStore, WriteKind, -}; -``` - -- [ ] **Step 2: Write the failing parser unit tests** - -In `crates/aura-cli/src/main.rs`, inside `#[cfg(test)] mod tests` (where -`parse_sweep_args` tests live), add: - -```rust - #[test] - fn parse_generalize_requires_a_single_value_candidate() { - // a candidate is one cell: a multi-value grid flag is refused - assert!(parse_generalize_args(&[ - "--real", "GER40,USDJPY", "--fast", "2,3", "--slow", "12", - "--stop-length", "14", "--stop-k", "2.0", - ]).is_err()); - } - - #[test] - fn parse_generalize_requires_all_four_knobs() { - // --slow absent -> refuse (no defaulting to the multi-value sweep grid) - assert!(parse_generalize_args(&[ - "--real", "GER40,USDJPY", "--fast", "3", - "--stop-length", "14", "--stop-k", "2.0", - ]).is_err()); - } - - #[test] - fn parse_generalize_refuses_fewer_than_two_instruments() { - assert!(parse_generalize_args(&[ - "--real", "GER40", "--fast", "3", "--slow", "12", - "--stop-length", "14", "--stop-k", "2.0", - ]).is_err()); - } - - #[test] - fn parse_generalize_refuses_a_duplicate_instrument() { - assert!(parse_generalize_args(&[ - "--real", "GER40,GER40", "--fast", "3", "--slow", "12", - "--stop-length", "14", "--stop-k", "2.0", - ]).is_err()); - } - - #[test] - fn parse_generalize_refuses_a_non_stage1r_strategy() { - assert!(parse_generalize_args(&[ - "--strategy", "sma", "--real", "GER40,USDJPY", "--fast", "3", "--slow", "12", - "--stop-length", "14", "--stop-k", "2.0", - ]).is_err()); - } - - #[test] - fn parse_generalize_defaults_name_and_metric() { - let (name, symbols, grid, metric, _from, _to) = parse_generalize_args(&[ - "--real", "GER40,USDJPY", "--fast", "3", "--slow", "12", - "--stop-length", "14", "--stop-k", "2.0", - ]).expect("valid generalize args"); - assert_eq!(name, "generalize"); - assert_eq!(metric, "expectancy_r"); - assert_eq!(symbols, vec!["GER40".to_string(), "USDJPY".to_string()]); - assert_eq!(grid.fast, vec![3]); - assert_eq!(grid.slow, vec![12]); - assert_eq!(grid.stop_length, vec![14]); - assert_eq!(grid.stop_k, vec![2.0]); - } -``` - -- [ ] **Step 3: Run the parser tests to verify they fail** - -Run: `cargo test -p aura-cli parse_generalize` -Expected: FAIL to compile — `cannot find function `parse_generalize_args`` -(does not exist yet). RED state. - -- [ ] **Step 4: Implement `parse_generalize_args`** - -In `crates/aura-cli/src/main.rs`, near `parse_sweep_args` (`:1622`), add. It mirrors -`parse_sweep_args` but: defaults strategy to stage1-r and refuses any other; -`--real` is a comma-list of `>= 2` distinct non-empty symbols; the four grid knobs -are each required and single-valued; `--metric` defaults to `expectancy_r`; -`--name` defaults to `generalize`. - -```rust -/// Parse the `generalize` tail: -/// `[--strategy stage1-r] --real --fast --slow --stop-length -/// --stop-k [--from ] [--to ] [--metric ] [--name ]`. -/// The candidate is a single cell: each grid knob takes exactly one value and all -/// four are required. `--real` is a comma list of >= 2 distinct non-empty symbols. -/// Pure (no I/O / exit) so the grammar is unit-testable. -fn parse_generalize_args( - rest: &[&str], -) -> Result<(String, Vec, Stage1RGrid, String, Option, Option), String> { - let usage = || "generalize [--strategy stage1-r] --real --fast --slow --stop-length --stop-k [--from ] [--to ] [--metric ] [--name ]".to_string(); - let mut symbols: Option> = None; - let mut from_ms: Option = None; - let mut to_ms: Option = None; - let mut fast: Option = None; - let mut slow: Option = None; - let mut stop_length: Option = None; - let mut stop_k: Option = None; - let mut metric = "expectancy_r".to_string(); - let mut name = "generalize".to_string(); - // a single grid knob, required and single-valued (csv with exactly one item) - let one = |value: &str, usage: &dyn Fn() -> String| -> Result, String> { - let items: Vec<&str> = value.split(',').collect(); - if items.len() != 1 || items[0].is_empty() { return Err(usage()); } - Ok(items) - }; - let mut tail = rest; - while let Some((flag, t)) = tail.split_first() { - let (value, t) = t.split_first().ok_or_else(usage)?; - match *flag { - "--strategy" => { if *value != "stage1-r" { return Err(usage()); } } - "--real" => { - let parts: Vec = value.split(',').map(|s| s.to_string()).collect(); - if parts.iter().any(|s| s.is_empty()) { return Err(usage()); } - symbols = Some(parts); - } - "--from" => from_ms = Some(value.parse().map_err(|_| usage())?), - "--to" => to_ms = Some(value.parse().map_err(|_| usage())?), - "--fast" => { one(value, &usage)?; fast = Some(value.parse().map_err(|_| usage())?); } - "--slow" => { one(value, &usage)?; slow = Some(value.parse().map_err(|_| usage())?); } - "--stop-length" => { one(value, &usage)?; stop_length = Some(value.parse().map_err(|_| usage())?); } - "--stop-k" => { one(value, &usage)?; stop_k = Some(value.parse().map_err(|_| usage())?); } - "--metric" => metric = (*value).to_string(), - "--name" => name = (*value).to_string(), - _ => return Err(usage()), - } - tail = t; - } - let symbols = symbols.ok_or_else(usage)?; - if symbols.len() < 2 { return Err(usage()); } - // distinct: a duplicate would double-count one instrument in the floor / sign count - let mut seen = std::collections::HashSet::new(); - if !symbols.iter().all(|s| seen.insert(s.clone())) { return Err(usage()); } - let grid = Stage1RGrid { - fast: vec![fast.ok_or_else(usage)?], - slow: vec![slow.ok_or_else(usage)?], - stop_length: vec![stop_length.ok_or_else(usage)?], - stop_k: vec![stop_k.ok_or_else(usage)?], - ..Stage1RGrid::default() - }; - Ok((name, symbols, grid, metric, from_ms, to_ms)) -} -``` - -- [ ] **Step 5: Run the parser tests to verify they pass** - -Run: `cargo test -p aura-cli parse_generalize` -Expected: PASS — all six `parse_generalize_*` tests green. - -- [ ] **Step 6: Implement `generalize_json` and `run_generalize`** - -In `crates/aura-cli/src/main.rs`, near `walkforward_summary_json` (`:1969`), add the -JSON builder (mirrors the `{"walkforward": obj}` shape; renders `selection_metric` -as `"metric"`): - -```rust -fn generalize_json(agg: &Generalization) -> String { - let per: Vec = agg - .per_instrument - .iter() - .map(|(sym, v)| serde_json::json!([sym, v])) - .collect(); - let obj = serde_json::json!({ - "metric": agg.selection_metric, - "n_instruments": agg.n_instruments, - "worst_case": agg.worst_case, - "sign_agreement": agg.sign_agreement, - "per_instrument": per, - }); - serde_json::json!({ "generalize": obj }).to_string() -} -``` - -And the run handler (near `run_sweep`, `:1750`). It pre-checks the metric -(data-free refusal), runs the single-cell candidate per instrument, stamps the -instrument, reduces, prints, and persists a `CrossInstrument` family: - -```rust -fn run_generalize( - name: &str, - symbols: &[String], - grid: &Stage1RGrid, - metric: &str, - from_ms: Option, - to_ms: Option, -) { - // data-free metric pre-check: refuse a non-R / unknown metric before any run. - if let Err(e) = check_r_metric(metric) { - eprintln!("aura: {e}"); - std::process::exit(2); - } - let mut members: Vec = Vec::new(); - for symbol in symbols { - let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms }; - let data = DataSource::from_choice(choice); // per-instrument pip_or_refuse + has_symbol - let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member - let mut report = family.points[0].report.clone(); - report.manifest.instrument = Some(symbol.clone()); - members.push(report); - } - let pairs: Vec<(String, &RunReport)> = symbols.iter().cloned().zip(members.iter()).collect(); - let agg = match generalization(&pairs, metric) { - Ok(a) => a, - Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); } - }; - println!("{}", generalize_json(&agg)); - let reg = default_registry(); - match reg.append_family(name, FamilyKind::CrossInstrument, &members) { - Ok(id) => println!("{{\"family_id\":\"{id}\"}}"), - Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); } - } -} -``` - -- [ ] **Step 7: Wire the dispatch arm and extend USAGE** - -In `crates/aura-cli/src/main.rs`, add a `["generalize", rest @ ..]` arm beside -`["sweep", ..]` (`:2993`): -```rust - ["generalize", rest @ ..] => match parse_generalize_args(rest) { - Ok((name, symbols, grid, metric, from_ms, to_ms)) => { - run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms) - } - Err(msg) => { - eprintln!("aura: {msg}"); - std::process::exit(2); - } - }, -``` -Append the form to the `USAGE` const (`:2956`), before the closing `aura runs ...`: -``` - | aura generalize [--strategy stage1-r] --real --fast --slow --stop-length --stop-k [--from ] [--to ] [--metric ] [--name ] -``` - -- [ ] **Step 8: Write the E2E tests** - -In `crates/aura-cli/tests/cli_run.rs`, add (model the gated-skip on -`run_real_sidecar_index_pip_reaches_the_emitted_manifest`, `:250`): - -```rust -/// Property: `aura generalize` grades a single stage1-r candidate across two -/// instruments — its stdout carries the per-instrument breakdown + the worst-case -/// floor + the sign-agreement, then the persisted CrossInstrument family handle. -/// Gated on local GER40/USDJPY data (the shared Sept-2024 window), skips cleanly -/// when the archive is absent. -#[test] -fn generalize_grades_a_candidate_across_two_instruments() { - const FROM_MS: &str = "1725148800000"; - const TO_MS: &str = "1727740799999"; - let cwd = temp_cwd("generalize-two"); - let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura")) - .args([ - "generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY", - "--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0", - "--from", FROM_MS, "--to", TO_MS, - ]) - .current_dir(&cwd) - .output() - .expect("spawn aura"); - if out.status.code() == Some(2) { - let stderr = String::from_utf8_lossy(&out.stderr); - assert!( - stderr.contains("no local data") || stderr.contains("no recorded geometry"), - "exit 2 must be a data refusal, got: {stderr}" - ); - eprintln!("skip: no local GER40/USDJPY data"); - return; - } - assert_eq!(out.status.code(), Some(0), "exit: {:?}", out.status); - let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout"); - assert!(stdout.contains("\"generalize\":"), "aggregate object: {stdout}"); - assert!(stdout.contains("\"n_instruments\":2"), "two instruments: {stdout}"); - assert!(stdout.contains("\"worst_case\":"), "worst_case present: {stdout}"); - assert!(stdout.contains("\"sign_agreement\":"), "sign_agreement present: {stdout}"); - assert!(stdout.contains("GER40") && stdout.contains("USDJPY"), "per-instrument breakdown: {stdout}"); - assert!(stdout.contains("\"family_id\":\"generalize-0\""), "family handle: {stdout}"); -} - -/// Property: a single-instrument generalize is refused at parse time (exit 2), -/// before any data access — so it asserts the arity refusal on any machine. -#[test] -fn generalize_refuses_a_single_instrument() { - let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura")) - .args([ - "generalize", "--strategy", "stage1-r", "--real", "GER40", - "--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0", - ]) - .output() - .expect("spawn aura"); - assert_eq!(out.status.code(), Some(2), "single instrument must exit 2"); -} - -/// Property: a non-R metric is refused before any instrument runs (the data-free -/// `check_r_metric` pre-check), so this asserts the R-only refusal on any machine. -#[test] -fn generalize_refuses_a_non_r_metric() { - let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura")) - .args([ - "generalize", "--strategy", "stage1-r", "--real", "GER40,USDJPY", - "--fast", "3", "--slow", "12", "--stop-length", "14", "--stop-k", "2.0", - "--metric", "total_pips", - ]) - .output() - .expect("spawn aura"); - assert_eq!(out.status.code(), Some(2), "a non-R metric must exit 2"); - let stderr = String::from_utf8_lossy(&out.stderr); - assert!(stderr.contains("R-only"), "must name the R-only refusal, got: {stderr}"); -} -``` - -> `temp_cwd(name) -> PathBuf` (cli_run.rs:13) is the confirmed registry-isolation -> helper (a fresh per-test cwd, so the family-store run index starts at 0 → -> `generalize-0`); `env!("CARGO_BIN_EXE_aura")` is the binary handle (the same the -> real-data E2Es at cli_run.rs:256 use). Both already exist in the file. - -- [ ] **Step 9: Run the E2E tests** - -Run: `cargo test -p aura-cli generalize` -Expected: PASS — `generalize_refuses_a_single_instrument` and -`generalize_refuses_a_non_r_metric` assert exit 2 (data-free); -`generalize_grades_a_candidate_across_two_instruments` passes on a machine with -GER40/USDJPY data or skips cleanly otherwise; plus the six `parse_generalize_*` -unit tests green. - -- [ ] **Step 10: Verify the CLI suite stays green (C23 preservation)** - -Run: `cargo test -p aura-cli` -Expected: PASS (0 failed) — every existing sweep/walkforward/mc/run/`runs` test -stays green; the existing manifest goldens are byte-identical (instrument is -omitted when None — `skip_serializing_if`), so no sweep/walkforward/standalone -output changed. - -- [ ] **Step 11: Final workspace gate** - -Run: `cargo test --workspace` -Expected: PASS (0 failed). - -Run: `cargo clippy --workspace --all-targets -- -D warnings` -Expected: clean (0 warnings). diff --git a/docs/specs/0078-cross-instrument-generalization.md b/docs/specs/0078-cross-instrument-generalization.md deleted file mode 100644 index e6a9ca6..0000000 --- a/docs/specs/0078-cross-instrument-generalization.md +++ /dev/null @@ -1,396 +0,0 @@ -# Cross-instrument generalization as a validation step — Design Spec - -**Date:** 2026-06-26 -**Status:** Draft — awaiting user spec review -**Authors:** orchestrator + Claude - -## Goal - -Add a first-class validation step that grades how consistently a single -**candidate** holds across a set of instruments. This is the last piece of the -"Inferential validation" milestone and the across-instrument axis of the same -anti-false-discovery concern as trials-deflation (#144, within one family) and -plateau-over-peak (#145, within one instrument's surface): an edge that appears -on one instrument can be a fit to that series' noise; an edge that survives -across several independent instruments is far less likely to be spurious. - -The step is an **aggregator / validator**, not a third selector. It takes a -*brought* candidate — one already chosen (e.g. by deflation or plateau on a -reference instrument, or authored directly) — runs it across an instrument list, -collects per-instrument R-metrics, and reduces them to a generalization grade. -It does **not** select a winner across instruments (that is a heavier follow-on, -out of scope below). A generalization grade is an aggregate (like `McAggregate`), -not a within-family selection, so it is **not** recorded on `FamilySelection` -(which annotates a within-family winner that does not exist here). - -## Architecture - -A new `aura generalize` subcommand. Input is a single **candidate** (the four -stage1-r grid knobs pinned to single values: `--fast --slow --stop-length ---stop-k`) crossed with an **instrument list** (`--real SYM1,SYM2,...`, -comma-separated; a shared optional `--from`/`--to` window, full available range -per instrument if omitted). For each instrument the candidate runs as a single -full-window stage1-r R evaluation, the M `RunReport`s are collected, reduced to a -generalization aggregate, the per-instrument breakdown and the aggregate are -printed, and the M runs are persisted as a `CrossInstrument` family. - -The per-instrument run is an **honest out-of-sample test by construction**: -because the candidate is fixed/external — not fit on that instrument — a single -full-window run on each instrument is already OOS for it. No per-instrument -walk-forward is needed for this first correct version. - -The reduction is **R-only**. R is the only account- and instrument-agnostic unit -(C10); `total_pips` is meaningless across instruments (different pip scales and -volatility). The headline score is the **worst case across instruments** — the -minimum of the chosen R-metric — which a strong instrument cannot lift (the -issue's non-domination constraint, satisfied by construction). It is the -cross-instrument sibling of `PlateauMode::Worst` (which already takes the -most-pessimistic neighbour). Because R-metrics are all higher-is-better, the -worst case is unconditionally the minimum (no direction branch). Alongside it the -aggregate carries the **sign-agreement count** (how many instruments are -net-positive) and the **full per-instrument breakdown** — together these fully -express "consistency of sign and magnitude". - -This honours: **C23** (additive — every existing path stays byte-identical; the -subcommand is new surface), **C9** (the reduction lives registry-side, the engine -stays simulation), **C12** (cross-instrument is the comparison orchestration -axis → a new `FamilyKind`), **C14/C18** (the new manifest field serde-widens with -the same back-compat idiom as `selection`), **C1** (a pure fold over the -per-instrument results in input order), **C2** (each per-instrument run is an -ordinary causal sim), **C10** (R-only). - -## Concrete code shapes - -### The user-facing program (the acceptance evidence) - -The researcher has a candidate and asks "does it generalize across instruments?": - -``` -aura generalize --strategy stage1-r --real GER40,USDJPY \ - --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 \ - --from 1725148800000 --to 1727740799999 -``` - -Expected stdout (the aggregate object + the persisted family handle; numbers -illustrative): - -```json -{"generalize":{"metric":"expectancy_r","n_instruments":2,"worst_case":-0.07,"sign_agreement":1,"per_instrument":[["GER40",0.31],["USDJPY",-0.07]]}} -{"family_id":"generalize-0"} -``` - -The aggregate is the diagnosis: the candidate makes `+0.31 R/trade` on GER40 but -**loses** on USDJPY, so its worst-case generalization is negative and only 1 of 2 -instruments agrees in sign — a false-discovery flag the per-instrument view would -have hidden behind GER40's strength. - -A second `--metric` selects the R-metric (default `expectancy_r`): - -``` -aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 \ - --stop-length 14 --stop-k 2.0 --metric sqn_normalized -``` - -Must-fail surfaces (each exits 2 with a clear message): - -``` -aura generalize --strategy stage1-r --real GER40 --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 -# refuse: a generalization needs >= 2 instruments - -aura generalize --strategy stage1-r --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 --metric total_pips -# refuse: total_pips is not comparable across instruments; cross-instrument scoring is R-only - -aura generalize --strategy stage1-r --real GER40,USDJPY --fast 2,3 --slow 12 --stop-length 14 --stop-k 2.0 -# refuse: a candidate is a single cell; --fast got 2 values - -aura generalize --strategy sma --real GER40,USDJPY --fast 3 --slow 12 --stop-length 14 --stop-k 2.0 -# refuse: generalize requires --strategy stage1-r (the candidate must produce R) -``` - -### Implementation shapes (before → after, secondary) - -**1. `RunManifest` gains a first-class instrument field** — `crates/aura-engine/src/report.rs`. -Same serde-widening idiom as `selection` (C14/C18): legacy lines without the key -read back as `None`. Closes a latent gap (today a real run records the -window/broker but not *which* symbol). - -```rust -// before -pub struct RunManifest { - pub commit: String, - pub params: Vec<(String, Scalar)>, - pub window: (Timestamp, Timestamp), - pub seed: u64, - pub broker: String, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub selection: Option, -} - -// after — add one field (every RunManifest literal threads `instrument: None`, -// the same compile-gate the #144 `selection` field was; the planner compile-drives -// the full literal set) -pub struct RunManifest { - pub commit: String, - pub params: Vec<(String, Scalar)>, - pub window: (Timestamp, Timestamp), - pub seed: u64, - pub broker: String, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub selection: Option, - /// The instrument this run evaluated, when it is a real-data run; `None` for a - /// synthetic run and for every pre-0078 line. The "instrument set as lineage" - /// (C18) for a cross-instrument family is each member's stamped symbol. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub instrument: Option, -} -``` - -**2. `FamilyKind` gains the comparison axis** — `crates/aura-registry/src/lineage.rs`. -A compile-gate: every `match` on `FamilyKind` (the `runs family` display, any -kind-dispatch) must gain the arm. - -```rust -// before -pub enum FamilyKind { Sweep, MonteCarlo, WalkForward } -// after -pub enum FamilyKind { Sweep, MonteCarlo, WalkForward, CrossInstrument } -``` - -**3. The generalization reduction** — `crates/aura-registry/src/lib.rs`, a sibling -of `optimize_deflated` / `optimize_plateau`, reusing `resolve_metric` / -`metric_value` / `is_r_metric` (all already present). Note the input is **not** a -`SweepFamily` and the output is **not** `(SweepPoint, FamilySelection)` — this is -an aggregator, so it has its own value type. - -```rust -/// The cross-instrument generalization aggregate (#146): a brought candidate's -/// per-instrument R-metric, reduced to a worst-case floor + a sign-agreement -/// count + the full breakdown. R-only (C10): `total_pips` is not comparable across -/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input -/// order. The headline `worst_case` is `min_i metric_i`; since every R-metric is -/// higher-is-better, the min is unconditionally the conservative floor (no -/// direction branch, unlike `PlateauMode::Worst`). -pub struct Generalization { - pub selection_metric: String, - pub n_instruments: usize, - pub worst_case: f64, // min_i metric_i - pub sign_agreement: usize, // #{ i : metric_i > 0 } - pub per_instrument: Vec<(String, f64)>, -} - -pub fn generalization( - per_instrument: &[(String, &RunReport)], - metric: &str, -) -> Result { - let m = resolve_metric(metric)?; - if !is_r_metric(m) { - return Err(RegistryError::NonRMetric(metric.to_string())); - } - if per_instrument.len() < 2 { - return Err(RegistryError::TooFewInstruments(per_instrument.len())); - } - let vals: Vec<(String, f64)> = per_instrument - .iter() - .map(|(label, rep)| (label.clone(), metric_value(rep, m))) - .collect(); - let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min); - let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count(); - Ok(Generalization { - selection_metric: metric.to_string(), - n_instruments: vals.len(), - worst_case, - sign_agreement, - per_instrument: vals, - }) -} -``` - -Two new `RegistryError` variants (in-style with the existing typed variants): - -```rust -pub enum RegistryError { - Io(std::io::Error), - Parse { line: usize, source: serde_json::Error }, - UnknownMetric(String), - NonRMetric(String), // a known metric that is not R-based (cross-instrument is R-only) - TooFewInstruments(usize), // a generalization needs >= 2 instruments -} -``` - -**4. The CLI subcommand** — `crates/aura-cli/src/main.rs`. A new dispatch arm beside -`sweep` / `walkforward`, a `parse_generalize_args`, and a `run_generalize` that -loops the candidate over the instruments, stamps each manifest's `instrument`, -reduces, prints, and persists. - -```rust -// dispatch (beside the existing ["sweep", ..] / ["walkforward", ..] arms) -["generalize", rest @ ..] => match parse_generalize_args(rest) { - Ok((name, symbols, grid, metric, from_ms, to_ms)) => { - run_generalize(&name, &symbols, &grid, &metric, from_ms, to_ms) - } - Err(msg) => { eprintln!("aura: {msg}"); std::process::exit(2); } -}, - -// run_generalize (shape): one full-window stage1-r R run per instrument, the -// candidate Stage1RGrid pinned to single values; stamp manifest.instrument; reduce. -fn run_generalize(name: &str, symbols: &[String], grid: &Stage1RGrid, - metric: &str, from_ms: Option, to_ms: Option) { - let mut members: Vec = Vec::new(); - for symbol in symbols { - // the existing real-source construction (per-instrument pip_or_refuse + has_symbol) - let choice = DataChoice::Real { symbol: symbol.clone(), from_ms, to_ms }; - let data = DataSource::from_choice(choice); - let family = stage1_r_sweep_family(None, &data, grid); // single-cell grid -> 1 member - let mut report = family.points[0].report.clone(); - report.manifest.instrument = Some(symbol.clone()); - members.push(report); - } - let pairs: Vec<(String, &RunReport)> = - symbols.iter().cloned().zip(members.iter()).collect(); - let agg = match aura_registry::generalization(&pairs, metric) { - Ok(a) => a, - Err(e) => { eprintln!("aura: {e}"); std::process::exit(2); } - }; - println!("{}", generalize_json(&agg)); // {"generalize": { ... }} - // persist the M runs as a CrossInstrument family (same Err-handling as `sweep`) - match reg.append_family(name, FamilyKind::CrossInstrument, &members) { - Ok(id) => println!("{{\"family_id\":\"{id}\"}}"), - Err(e) => { eprintln!("aura: failed to persist family: {e}"); std::process::exit(1); } - } -} -``` - -The candidate is a `Stage1RGrid` whose `fast`/`slow`/`stop_length`/`stop_k` each -carry **exactly one** value; `parse_generalize_args` requires all four explicitly -and refuses a multi-value flag (a candidate is one cell — it must not default to -the multi-value sweep grid). `--real` is parsed as a comma-list into -`Vec`; the existing single-symbol `RealWindowGrammar` / `sweep` / -`walkforward` parsers are untouched. - -## Components - -1. **`RunManifest.instrument: Option`** (aura-engine/report.rs) — the - instrument lineage; serde `default` / `skip_serializing_if`. Every existing - `RunManifest` literal threads `instrument: None` (compile-gate, same shape as - #144's `selection`). The synthetic and existing real run paths leave it `None` - (or stamp it — a non-load-bearing nicety; this cycle only requires the - generalize path to stamp it). -2. **`FamilyKind::CrossInstrument`** (aura-registry/lineage.rs) — the comparison - axis tag; persisted on each member record. Compile-gate on `FamilyKind` - matches. -3. **`generalization` + `Generalization` + two `RegistryError` variants** - (aura-registry/lib.rs) — the R-only reduction beside the sibling selectors, - reusing `resolve_metric` / `metric_value` / `is_r_metric`. -4. **CLI: `generalize` dispatch arm, `parse_generalize_args`, `run_generalize`, - `generalize_json`** (aura-cli/main.rs) — the candidate × instrument-list loop; - reuses `stage1_r_sweep_family` (single-cell grid), the per-instrument - `DataSource::Real` construction (`pip_or_refuse` + `has_symbol` refusals), and - `append_family`. `USAGE` gains the `aura generalize ...` form. - -## Data flow - -``` -candidate (single-cell Stage1RGrid) + [SYM_1 .. SYM_M] + window - -> for each SYM_i: - DataSource::Real{SYM_i} (pip_or_refuse + has_symbol) - -> stage1_r_sweep_family(single-cell grid) -> SweepFamily{ 1 point } - -> report = points[0].report; report.manifest.instrument = Some(SYM_i) - -> members: Vec (length M, input order) - -> generalization(zip(symbols, members), metric) - -> Generalization{ worst_case = min_i, sign_agreement, per_instrument } - -> print {"generalize": agg} - -> append_family(name, CrossInstrument, members) -> family_id - -> print {"family_id": id} -``` - -The aggregate is a pure function of the persisted members, so it is recomputable -on reload (the `McAggregate` precedent) — no new persisted family-level record -type is introduced. - -## Error handling - -Every refusal is stderr + `exit(2)`, the established CLI convention: - -- **< 2 instruments** — a one-instrument "generalization" is meaningless. - Enforced both at `parse_generalize_args` (early, on the parsed list) and in - `generalization` (`TooFewInstruments`, the reusable-core guard). -- **non-R / pip metric** — cross-instrument scoring is R-only; - `generalization` returns `NonRMetric`. The CLI surfaces it as exit 2. -- **strategy not stage1-r** — the candidate must produce R; `parse_generalize_args` - refuses any `--strategy` other than `stage1-r`. -- **a multi-value candidate flag** — a candidate is a single cell; - `parse_generalize_args` refuses when any of `--fast/--slow/--stop-length/--stop-k` - carries more than one value, or is absent. -- **empty / duplicate symbol** in the list — refuse (a duplicate would - double-count one instrument in the worst-case and sign-agreement). -- **a symbol with no recorded geometry / no archive** — the existing - `pip_or_refuse` / `no_real_data` refusals fire per instrument before any run. - -## Testing strategy - -RED-first where pinnable. Per-crate test command, single positional filter. - -Registry unit tests (aura-registry): -- **worst_case picks the floor:** a fabricated 3-member set (one strongly - positive, one negative, one mid) → `worst_case == the negative value`, **not** - the mean; `sign_agreement == 2`. -- **non-domination:** appending a hugely-positive fourth instrument does **not** - raise `worst_case` (the min is unchanged). -- **R-only refusal:** `generalization(.., "total_pips")` → `Err(NonRMetric)`. -- **too-few refusal:** a 1-element input → `Err(TooFewInstruments(1))`. -- **determinism (C1):** the same inputs → an identical `Generalization` (it is a - pure fold). - -Engine serde tests (aura-engine): -- **back-compat (C14/C18):** a pre-0078 manifest JSON line without `instrument` - deserializes with `instrument == None`; a round-trip with `instrument == - Some("GER40")` preserves it; the existing `RunManifest`/`FamilySelection` serde - and legacy-load tests stay green. - -CLI E2E (aura-cli/tests/cli_run.rs), gated-skip on absent local data exactly like -the existing `--real` tests: -- `aura generalize --strategy stage1-r --real GER40,USDJPY --fast .. --slow .. - --stop-length .. --stop-k .. --from .. --to ..` over the shared Sept-2024 window - → stdout carries a `{"generalize": ...}` object with `n_instruments == 2`, a - `per_instrument` breakdown naming both symbols, a `worst_case`, a - `sign_agreement`, and a following `{"family_id":"generalize-0"}`; when the - archive is absent, the run hits the no-data refusal and the test skips cleanly - (the established pattern). -- a must-fail E2E: `--real GER40` (single instrument) exits 2 with the - >=2-instruments message (no data needed — the arity refusal precedes any run). - -C23 preservation: -- the existing `sweep` / `walkforward` / `mc` / standalone-run goldens stay - **byte-identical** — `generalize` is new surface and touches no existing path. - -## Acceptance criteria - -Against aura's feature-acceptance criterion: - -1. **The intended audience reaches for it.** The aura researcher validating a - candidate naturally asks "does this hold across instruments?" — the last - anti-false-discovery piece of the milestone. The worked `aura generalize` - invocation above is the evidence: it is the command a researcher writes, and - its output is the generalization verdict. -2. **It measurably improves correctness.** A candidate that looks good on one - instrument but fails the worst-case-across-instruments floor (negative - `worst_case`, low `sign_agreement`) is now caught — a real defense the - per-instrument view hides. It composes downstream of #144/#145. -3. **It reintroduces no failure class the core constraints eliminate.** R-only - (C10), behaviour-preserving (C23 — existing paths byte-identical), deterministic - (C1), causal per-instrument sims (C2), registry-side policy (C9), serde-widened - with full back-compat (C14/C18). No invariant is touched. - -## Out of scope (named, deferred) - -- **Cross-instrument selection (reading B):** running a full sweep per instrument - and picking the cell that generalizes best across instruments. A heavier - follow-on that conflates selection with validation; this cycle is the validator. -- **Per-instrument walk-forward evaluation:** the first version runs a full-window - R evaluation per instrument (honest OOS because the candidate is fixed). A - per-instrument walk-forward knob is a later refinement. -- **A persisted family-level aggregate record:** the aggregate is recomputable from - the members (the `McAggregate` precedent); no new persisted record type. -- **Cross-instrument R-pooling of trade series:** #139's deferred cross-symbol - pooling stays deferred. -- **Pip-metric generalization:** undefined across instruments (C10); refused.