feat(0078): cross-instrument generalization as a validation step
Last piece of the Inferential-validation milestone (#146): a new `aura generalize` subcommand grades how consistently a single candidate holds across a set of instruments — the across-instrument axis of the same anti-false-discovery concern as trials-deflation (#144, within one family) and plateau-over-peak (#145, within one instrument's surface). Aggregator/validator, not a third selector (user-ratified): it runs a brought candidate (a single-cell stage1-r grid — --fast/--slow/--stop- length/--stop-k pinned to single values) across an instrument list (--real SYM1,SYM2,...), collects per-instrument R-metrics, and reduces them to a worst-case floor (min over instruments) + a sign-agreement count + the per-instrument breakdown. R-only (C10 — R is the only account/instrument-agnostic unit); the worst-case min is the cross- instrument sibling of PlateauMode::Worst and satisfies the non- domination constraint by construction (a strong instrument cannot lift a min). Rejected: a cross-instrument t-stat mean/std*sqrt(M), which reintroduces the pooled mean the milestone warns against. - RunManifest gains a first-class `instrument` lineage field (serde- widened, C14/C18 — legacy lines load as None, existing manifest bytes unchanged); the compat read-mirror threads it in lockstep. - FamilyKind::CrossInstrument (C12 comparison axis); the M per-instrument runs persist as a CrossInstrument family, each member self-identifying via its stamped instrument. The aggregate is recomputable from the members (McAggregate precedent), printed live. - The generalization reduction lives registry-side beside optimize_ deflated/optimize_plateau (C9), reusing resolve_metric/metric_value/ is_r_metric; check_r_metric is the data-free pre-check the CLI runs before evaluating any instrument. - Additive: existing sweep/walkforward/mc/standalone paths stay byte- identical (C23 — instrument omitted when None via skip_serializing_if). Verified by the orchestrator: cargo test --workspace green (0 failed — the new generalization_*, check_r_metric, parse_generalize_*, and the gated cross-instrument E2E all pass; the E2E ran the real GER40/USDJPY path on this host, exit 0); cargo clippy --workspace --all-targets -D warnings clean. closes #146
This commit is contained in:
@@ -32,6 +32,8 @@ struct RunManifestRead {
|
||||
broker: String,
|
||||
#[serde(default)]
|
||||
selection: Option<FamilySelection>,
|
||||
#[serde(default)]
|
||||
instrument: Option<String>,
|
||||
}
|
||||
|
||||
/// A param value that accepts BOTH the current tagged [`Scalar`] form
|
||||
@@ -64,7 +66,7 @@ impl<'de> Deserialize<'de> for ScalarRead {
|
||||
|
||||
impl From<RunReportRead> for RunReport {
|
||||
fn from(r: RunReportRead) -> Self {
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection } = r.manifest;
|
||||
let RunManifestRead { commit, params, window, seed, broker, selection, instrument } = r.manifest;
|
||||
RunReport {
|
||||
manifest: RunManifest {
|
||||
commit,
|
||||
@@ -73,6 +75,7 @@ impl From<RunReportRead> for RunReport {
|
||||
seed,
|
||||
broker,
|
||||
selection,
|
||||
instrument,
|
||||
},
|
||||
metrics: r.metrics,
|
||||
}
|
||||
|
||||
@@ -428,6 +428,65 @@ pub fn optimize_deflated(
|
||||
}))
|
||||
}
|
||||
|
||||
/// The cross-instrument generalization aggregate (#146): a brought candidate's
|
||||
/// per-instrument R-metric, reduced to a worst-case floor, a sign-agreement count,
|
||||
/// and the full breakdown. R-only (C10): `total_pips` is not comparable across
|
||||
/// instruments. Pure, deterministic (C1) — a fold over `per_instrument` in input
|
||||
/// order. `worst_case` is `min_i metric_i`; since every R-metric is
|
||||
/// higher-is-better, the min is unconditionally the conservative floor (no
|
||||
/// direction branch, unlike `PlateauMode::Worst`).
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct Generalization {
|
||||
pub selection_metric: String,
|
||||
pub n_instruments: usize,
|
||||
pub worst_case: f64,
|
||||
pub sign_agreement: usize,
|
||||
pub per_instrument: Vec<(String, f64)>,
|
||||
}
|
||||
|
||||
/// Validate that `metric` is a known, R-based ranking key — the data-free pre-check
|
||||
/// the CLI runs before evaluating any instrument. The registry owns metric truth
|
||||
/// (C9), so the CLI never duplicates the R-set.
|
||||
pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if is_r_metric(m) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(RegistryError::NonRMetric(metric.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Grade how consistently one candidate holds across instruments: read the chosen
|
||||
/// R-metric from each per-instrument report and reduce to the worst-case floor +
|
||||
/// the sign-agreement count + the per-instrument breakdown. R-only and `>= 2`
|
||||
/// instruments (the metric check precedes the arity check so a bad metric is
|
||||
/// reported even with an empty slice). Additive — reads, never re-ranks (C23).
|
||||
pub fn generalization(
|
||||
per_instrument: &[(String, &RunReport)],
|
||||
metric: &str,
|
||||
) -> Result<Generalization, RegistryError> {
|
||||
let m = resolve_metric(metric)?;
|
||||
if !is_r_metric(m) {
|
||||
return Err(RegistryError::NonRMetric(metric.to_string()));
|
||||
}
|
||||
if per_instrument.len() < 2 {
|
||||
return Err(RegistryError::TooFewInstruments(per_instrument.len()));
|
||||
}
|
||||
let vals: Vec<(String, f64)> = per_instrument
|
||||
.iter()
|
||||
.map(|(label, rep)| (label.clone(), metric_value(rep, m)))
|
||||
.collect();
|
||||
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
|
||||
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
|
||||
Ok(Generalization {
|
||||
selection_metric: metric.to_string(),
|
||||
n_instruments: vals.len(),
|
||||
worst_case,
|
||||
sign_agreement,
|
||||
per_instrument: vals,
|
||||
})
|
||||
}
|
||||
|
||||
/// What can go wrong reading or ranking the registry.
|
||||
#[derive(Debug)]
|
||||
pub enum RegistryError {
|
||||
@@ -437,6 +496,11 @@ pub enum RegistryError {
|
||||
Parse { line: usize, source: serde_json::Error },
|
||||
/// `rank_by` was given a metric name it does not know.
|
||||
UnknownMetric(String),
|
||||
/// A known metric that is not R-based — cross-instrument generalization is
|
||||
/// R-only (R is the only account/instrument-agnostic unit, C10).
|
||||
NonRMetric(String),
|
||||
/// A generalization was asked for with fewer than two instruments.
|
||||
TooFewInstruments(usize),
|
||||
}
|
||||
|
||||
impl fmt::Display for RegistryError {
|
||||
@@ -450,6 +514,14 @@ impl fmt::Display for RegistryError {
|
||||
f,
|
||||
"unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::NonRMetric(m) => write!(
|
||||
f,
|
||||
"metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)"
|
||||
),
|
||||
RegistryError::TooFewInstruments(n) => write!(
|
||||
f,
|
||||
"a generalization needs >= 2 instruments, got {n}"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -478,6 +550,7 @@ mod tests {
|
||||
seed: 0,
|
||||
broker: "b".to_string(),
|
||||
selection: None,
|
||||
instrument: None,
|
||||
},
|
||||
metrics: RunMetrics { total_pips, max_drawdown, bias_sign_flips: flips, r: None },
|
||||
}
|
||||
@@ -812,7 +885,7 @@ mod tests {
|
||||
report: RunReport {
|
||||
manifest: RunManifest {
|
||||
commit: "c".into(), params: vec![], window: (Timestamp(0), Timestamp(0)),
|
||||
seed: 0, broker: "b".into(), selection: None,
|
||||
seed: 0, broker: "b".into(), selection: None, instrument: None,
|
||||
},
|
||||
metrics: RunMetrics {
|
||||
total_pips, max_drawdown: 0.0, bias_sign_flips: 0,
|
||||
@@ -1095,4 +1168,88 @@ mod tests {
|
||||
"the plateau interior's worst neighbour is below the dip's 60.0 shoulder",
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_worst_case_is_the_floor_not_the_mean() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0); // +0.40
|
||||
let b = report_with_r(0.0, -0.20, 0.0); // -0.20 (the floor)
|
||||
let c = report_with_r(0.0, 0.10, 0.0); // +0.10
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("C".to_string(), &c)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g.n_instruments, 3);
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "worst_case must be the min, got {}", g.worst_case);
|
||||
assert_eq!(g.sign_agreement, 2, "two instruments are net-positive");
|
||||
assert_eq!(g.per_instrument, vec![
|
||||
("A".to_string(), 0.40), ("B".to_string(), -0.20), ("C".to_string(), 0.10),
|
||||
]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_not_lifted_by_a_strong_instrument() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let strong = report_with_r(0.0, 99.0, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b), ("S".to_string(), &strong)];
|
||||
let g = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert!((g.worst_case - (-0.20)).abs() < 1e-12, "a strong instrument must not raise the floor");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_a_non_r_metric() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let b = report_with_r(0.0, 0.2, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
match generalization(&pairs, "total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_refuses_fewer_than_two_instruments() {
|
||||
let a = report_with_r(0.0, 0.4, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a)];
|
||||
match generalization(&pairs, "expectancy_r") {
|
||||
Err(RegistryError::TooFewInstruments(n)) => assert_eq!(n, 1),
|
||||
other => panic!("expected TooFewInstruments(1), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_reports_a_bad_metric_before_arity() {
|
||||
// The doc-comment pins this ordering: the metric check precedes the
|
||||
// arity check, so a bad metric is surfaced even on an empty slice
|
||||
// (which would otherwise yield TooFewInstruments(0)). This decides
|
||||
// which error the CLI shows when both are wrong; reversing the two
|
||||
// guards in `generalization` would flip this to TooFewInstruments.
|
||||
let pairs: Vec<(String, &RunReport)> = vec![];
|
||||
match generalization(&pairs, "total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric before arity, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generalization_is_deterministic() {
|
||||
let a = report_with_r(0.0, 0.40, 0.0);
|
||||
let b = report_with_r(0.0, -0.20, 0.0);
|
||||
let pairs = vec![("A".to_string(), &a), ("B".to_string(), &b)];
|
||||
let g1 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
let g2 = generalization(&pairs, "expectancy_r").expect("R metric");
|
||||
assert_eq!(g1, g2, "a pure fold must be deterministic (C1)");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_r_metric_accepts_r_and_refuses_pip() {
|
||||
assert!(check_r_metric("expectancy_r").is_ok());
|
||||
assert!(check_r_metric("sqn_normalized").is_ok());
|
||||
match check_r_metric("total_pips") {
|
||||
Err(RegistryError::NonRMetric(m)) => assert_eq!(m, "total_pips"),
|
||||
other => panic!("expected NonRMetric, got {other:?}"),
|
||||
}
|
||||
match check_r_metric("nope") {
|
||||
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"),
|
||||
other => panic!("expected UnknownMetric, got {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ pub enum FamilyKind {
|
||||
Sweep,
|
||||
MonteCarlo,
|
||||
WalkForward,
|
||||
CrossInstrument,
|
||||
}
|
||||
|
||||
/// One persisted family member: a run record stamped with its lineage link.
|
||||
|
||||
@@ -254,6 +254,7 @@ mod tests {
|
||||
seed: 0,
|
||||
broker: "sim-optimal(pip_size=0.0001)".to_string(),
|
||||
selection: None,
|
||||
instrument: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user