feat(0078): cross-instrument generalization as a validation step

Last piece of the Inferential-validation milestone (#146): a new
`aura generalize` subcommand grades how consistently a single candidate
holds across a set of instruments — the across-instrument axis of the
same anti-false-discovery concern as trials-deflation (#144, within one
family) and plateau-over-peak (#145, within one instrument's surface).

Aggregator/validator, not a third selector (user-ratified): it runs a
brought candidate (a single-cell stage1-r grid — --fast/--slow/--stop-
length/--stop-k pinned to single values) across an instrument list
(--real SYM1,SYM2,...), collects per-instrument R-metrics, and reduces
them to a worst-case floor (min over instruments) + a sign-agreement
count + the per-instrument breakdown. R-only (C10 — R is the only
account/instrument-agnostic unit); the worst-case min is the cross-
instrument sibling of PlateauMode::Worst and satisfies the non-
domination constraint by construction (a strong instrument cannot lift
a min). Rejected: a cross-instrument t-stat mean/std*sqrt(M), which
reintroduces the pooled mean the milestone warns against.

- RunManifest gains a first-class `instrument` lineage field (serde-
  widened, C14/C18 — legacy lines load as None, existing manifest bytes
  unchanged); the compat read-mirror threads it in lockstep.
- FamilyKind::CrossInstrument (C12 comparison axis); the M per-instrument
  runs persist as a CrossInstrument family, each member self-identifying
  via its stamped instrument. The aggregate is recomputable from the
  members (McAggregate precedent), printed live.
- The generalization reduction lives registry-side beside optimize_
  deflated/optimize_plateau (C9), reusing resolve_metric/metric_value/
  is_r_metric; check_r_metric is the data-free pre-check the CLI runs
  before evaluating any instrument.
- Additive: existing sweep/walkforward/mc/standalone paths stay byte-
  identical (C23 — instrument omitted when None via skip_serializing_if).

Verified by the orchestrator: cargo test --workspace green (0 failed —
the new generalization_*, check_r_metric, parse_generalize_*, and the
gated cross-instrument E2E all pass; the E2E ran the real GER40/USDJPY
path on this host, exit 0); cargo clippy --workspace --all-targets
-D warnings clean.

closes #146
This commit is contained in:
2026-06-26 19:04:53 +02:00
parent cffc297e9a
commit 0a06afed29
18 changed files with 604 additions and 8 deletions
+4 -1
View File
@@ -32,6 +32,8 @@ struct RunManifestRead {
broker: String,
#[serde(default)]
selection: Option<FamilySelection>,
#[serde(default)]
instrument: Option<String>,
}
/// A param value that accepts BOTH the current tagged [`Scalar`] form
@@ -64,7 +66,7 @@ impl<'de> Deserialize<'de> for ScalarRead {
impl From<RunReportRead> for RunReport {
fn from(r: RunReportRead) -> Self {
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
let RunManifestRead { commit, params, window, seed, broker, selection, instrument } = r.manifest;
RunReport {
manifest: RunManifest {
commit,
@@ -73,6 +75,7 @@ impl From<RunReportRead> for RunReport {
seed,
broker,
selection,
instrument,
},
metrics: r.metrics,
}
+158 -1
View File
@@ -428,6 +428,65 @@ pub fn optimize_deflated(
}))
}
/// The cross-instrument generalization aggregate (#146): a brought candidate's
/// per-instrument R-metric, reduced to a worst-case floor, a sign-agreement count,
/// and the full breakdown. R-only (C10): `total_pips` is not comparable across
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
/// order. `worst_case` is `min_i metric_i`; since every R-metric is
/// higher-is-better, the min is unconditionally the conservative floor (no
/// direction branch, unlike `PlateauMode::Worst`).
#[derive(Clone, Debug, PartialEq)]
pub struct Generalization {
pub selection_metric: String,
pub n_instruments: usize,
pub worst_case: f64,
pub sign_agreement: usize,
pub per_instrument: Vec<(String, f64)>,
}
/// Validate that `metric` is a known, R-based ranking key — the data-free pre-check
/// the CLI runs before evaluating any instrument. The registry owns metric truth
/// (C9), so the CLI never duplicates the R-set.
pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> {
let m = resolve_metric(metric)?;
if is_r_metric(m) {
Ok(())
} else {
Err(RegistryError::NonRMetric(metric.to_string()))
}
}
/// Grade how consistently one candidate holds across instruments: read the chosen
/// R-metric from each per-instrument report and reduce to the worst-case floor +
/// the sign-agreement count + the per-instrument breakdown. R-only and `>= 2`
/// instruments (the metric check precedes the arity check so a bad metric is
/// reported even with an empty slice). Additive — reads, never re-ranks (C23).
pub fn generalization(
per_instrument: &[(String, &RunReport)],
metric: &str,
) -> Result<Generalization, RegistryError> {
let m = resolve_metric(metric)?;
if !is_r_metric(m) {
return Err(RegistryError::NonRMetric(metric.to_string()));
}
if per_instrument.len() < 2 {
return Err(RegistryError::TooFewInstruments(per_instrument.len()));
}
let vals: Vec<(String, f64)> = per_instrument
.iter()
.map(|(label, rep)| (label.clone(), metric_value(rep, m)))
.collect();
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
Ok(Generalization {
selection_metric: metric.to_string(),
n_instruments: vals.len(),
worst_case,
sign_agreement,
per_instrument: vals,
})
}
/// What can go wrong reading or ranking the registry.
#[derive(Debug)]
pub enum RegistryError {
@@ -437,6 +496,11 @@ pub enum RegistryError {
Parse { line: usize, source: serde_json::Error },
/// `rank_by` was given a metric name it does not know.
UnknownMetric(String),
/// A known metric that is not R-based — cross-instrument generalization is
/// R-only (R is the only account/instrument-agnostic unit, C10).
NonRMetric(String),
/// A generalization was asked for with fewer than two instruments.
TooFewInstruments(usize),
}
impl fmt::Display for RegistryError {
@@ -450,6 +514,14 @@ impl fmt::Display for RegistryError {
f,
"unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
),
RegistryError::NonRMetric(m) => write!(
f,
"metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
),
RegistryError::TooFewInstruments(n) => write!(
f,
"a generalization needs >= 2 instruments, got {n}"
),
}
}
}
@@ -478,6 +550,7 @@ mod tests {
seed: 0,
broker: "b".to_string(),
selection: None,
instrument: None,
},
metrics: RunMetrics { total_pips, max_drawdown, bias_sign_flips: flips, r: None },
}
@@ -812,7 +885,7 @@ mod tests {
report: RunReport {
manifest: RunManifest {
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
seed: 0, broker: "b".into(), selection: None,
seed: 0, broker: "b".into(), selection: None, instrument: None,
},
metrics: RunMetrics {
total_pips, max_drawdown: 0.0, bias_sign_flips: 0,
@@ -1095,4 +1168,88 @@ mod tests {
"the plateau interior's worst neighbour is below the dip's 60.0 shoulder",
);
}
#[test]
fn generalization_worst_case_is_the_floor_not_the_mean() {
let a = report_with_r(0.0, 0.40, 0.0); // +0.40
let b = report_with_r(0.0, -0.20, 0.0); // -0.20 (the floor)
let c = report_with_r(0.0, 0.10, 0.0); // +0.10
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("C".to_string(), &c)];
let g = generalization(&pairs, "expectancy_r").expect("R metric");
assert_eq!(g.n_instruments, 3);
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "worst_case must be the min, got {}", g.worst_case);
assert_eq!(g.sign_agreement, 2, "two instruments are net-positive");
assert_eq!(g.per_instrument, vec![
("A".to_string(), 0.40), ("B".to_string(), -0.20), ("C".to_string(), 0.10),
]);
}
#[test]
fn generalization_is_not_lifted_by_a_strong_instrument() {
let a = report_with_r(0.0, 0.40, 0.0);
let b = report_with_r(0.0, -0.20, 0.0);
let strong = report_with_r(0.0, 99.0, 0.0);
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("S".to_string(), &strong)];
let g = generalization(&pairs, "expectancy_r").expect("R metric");
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "a strong instrument must not raise the floor");
}
#[test]
fn generalization_refuses_a_non_r_metric() {
let a = report_with_r(0.0, 0.4, 0.0);
let b = report_with_r(0.0, 0.2, 0.0);
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
match generalization(&pairs, "total_pips") {
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
other => panic!("expected NonRMetric, got {other:?}"),
}
}
#[test]
fn generalization_refuses_fewer_than_two_instruments() {
let a = report_with_r(0.0, 0.4, 0.0);
let pairs = vec![("A".to_string(), &a)];
match generalization(&pairs, "expectancy_r") {
Err(RegistryError::TooFewInstruments(n)) => assert_eq!(n, 1),
other => panic!("expected TooFewInstruments(1), got {other:?}"),
}
}
#[test]
fn generalization_reports_a_bad_metric_before_arity() {
// The doc-comment pins this ordering: the metric check precedes the
// arity check, so a bad metric is surfaced even on an empty slice
// (which would otherwise yield TooFewInstruments(0)). This decides
// which error the CLI shows when both are wrong; reversing the two
// guards in `generalization` would flip this to TooFewInstruments.
let pairs: Vec<(String, &RunReport)> = vec![];
match generalization(&pairs, "total_pips") {
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
other => panic!("expected NonRMetric before arity, got {other:?}"),
}
}
#[test]
fn generalization_is_deterministic() {
let a = report_with_r(0.0, 0.40, 0.0);
let b = report_with_r(0.0, -0.20, 0.0);
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
let g1 = generalization(&pairs, "expectancy_r").expect("R metric");
let g2 = generalization(&pairs, "expectancy_r").expect("R metric");
assert_eq!(g1, g2, "a pure fold must be deterministic (C1)");
}
#[test]
fn check_r_metric_accepts_r_and_refuses_pip() {
assert!(check_r_metric("expectancy_r").is_ok());
assert!(check_r_metric("sqn_normalized").is_ok());
match check_r_metric("total_pips") {
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
other => panic!("expected NonRMetric, got {other:?}"),
}
match check_r_metric("nope") {
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"),
other => panic!("expected UnknownMetric, got {other:?}"),
}
}
}
+1
View File
@@ -24,6 +24,7 @@ pub enum FamilyKind {
Sweep,
MonteCarlo,
WalkForward,
CrossInstrument,
}
/// One persisted family member: a run record stamped with its lineage link.
+1
View File
@@ -254,6 +254,7 @@ mod tests {
seed: 0,
broker: "sim-optimal(pip_size=0.0001)".to_string(),
selection: None,
instrument: None,
}
}