refactor: registry deflation dispatch generic over the MetricVocabulary trait

The A1 cut of #147 item 2, part 1 of 2 (the substrate): the registry's
ranking/deflation machinery no longer bakes in the R metric vocabulary.

- aura-analysis gains the deliberately narrow MetricVocabulary trait (name
  resolution, roster, direction, value read, one null draw) and the
  single-sourced one_sided_p_laplace estimator; re-exported through
  aura-engine's existing seam (no Cargo.toml edge changes anywhere).
- aura-backtest now OWNS the R vocabulary: RunMetricKey (was the registry's
  private Metric enum, exposure_sign_flips alias preserved), r_based (the
  C10 classification), R_BASED_METRICS/RANKABLE_METRICS rosters, the
  centred moving-block null_draw, and member_metric_from_rs.
- aura-registry: metric_cmp/rank_by/null_best_of_k/member_sd generic over
  M: MetricVocabulary; optimize/optimize_plateau/optimize_deflated over
  M: MetricVocabulary + Clone (SweepPoint<M>: Clone bounds on M: Clone);
  the private vocabulary deleted; check_r_metric/generalization stay
  monomorphic on the RunMetrics vocabulary (the C10 wall by type);
  RegistryError::UnknownMetric carries the vocabulary's roster so the
  refusal prose derives instead of being hand-strung (byte-identical for R).

Bit-identity (C18): rng consumption order, centring, Laplace arithmetic,
floors, tie rules, and refusal prose are all unchanged for RunMetrics
inputs — the full registry suite (75 tests) passes unchanged except three
syntax-only edits for the UnknownMetric variant reshape; clippy -D warnings
clean on the three touched crates. Alternatives considered: a vocabulary
table in aura-core (rejected: inverts C28's supplied-from-outer-rungs
direction and cannot host the null model); a parallel measurement-side
deflation path (rejected: duplicates the machinery and adds a fifth
roster). Part 2 lands the consumers (campaign roster single-sourcing, the
IC implementor, acceptance tests).

refs #147
This commit is contained in:
2026-07-20 19:10:39 +02:00
parent a9d36ddd70
commit 6744f670b1
5 changed files with 396 additions and 175 deletions
+48
View File
@@ -234,6 +234,45 @@ pub fn permute<T>(xs: &mut [T], rng: &mut SplitMix64) {
} }
} }
/// Per-metric vocabulary a report payload supplies to the registry's
/// ranking/deflation machinery (#147). Narrow by design (#136): exactly the
/// surface the backtest metrics (the sole implementor so far) and the
/// measurement IC (deflatable today via its own permutation-null path, not
/// yet through this trait) are expected to share. The registry owns the
/// family fold — best-of-K, p95, Laplace, dispersion floor — the payload
/// owns what each metric IS: its name, direction, value read, and what one
/// draw of its null statistic means.
pub trait MetricVocabulary: Sized {
/// Resolved metric identity — cheap to copy, resolved once per call.
type Key: Copy;
/// Name → key; `None` for a name outside this vocabulary.
fn resolve(name: &str) -> Option<Self::Key>;
/// The canonical accepted-name roster, in refusal-prose order.
fn known() -> &'static [&'static str];
/// The key's optimisation direction: `true` when larger is better.
fn higher_is_better(key: Self::Key) -> bool;
/// The member's scalar under the key. A missing optional block reads
/// `f64::NEG_INFINITY` (the worst rank for higher-is-better keys).
fn value(&self, key: Self::Key) -> f64;
/// Whether the key deflates via a resampled null (the best-of-K arm) or
/// the dispersion floor.
fn has_resampling_null(key: Self::Key) -> bool;
/// ONE draw of this member's null statistic under H0, advancing `rng`.
/// `None` means no usable null input for this member, and consumes no
/// rng. `block_len` parameterises resampling vocabularies; permutation
/// vocabularies ignore it.
fn null_draw(&self, key: Self::Key, block_len: usize, rng: &mut SplitMix64) -> Option<f64>;
}
/// One-sided Laplace-smoothed tail estimate `(ge + 1) / (n + 1)`: the
/// probability mass at-or-above an observed statistic given that `ge` of `n`
/// null draws reached it. Intended as the single source of the formula the
/// registry's R deflation arm and the CLI's IC reduction each compute inline
/// today; wiring those call sites to this function is deferred follow-up.
pub fn one_sided_p_laplace(ge: usize, n: usize) -> f64 {
(ge + 1) as f64 / (n + 1) as f64
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -356,4 +395,13 @@ mod tests {
filled += take; filled += take;
} }
} }
#[test]
fn one_sided_p_laplace_known_values() {
assert_eq!(one_sided_p_laplace(0, 3), 0.25);
assert_eq!(one_sided_p_laplace(3, 3), 1.0);
assert_eq!(one_sided_p_laplace(1, 999), 2.0 / 1000.0);
// the registry's R-arm shape: count-of-ge over usable draws
assert_eq!(one_sided_p_laplace(0, 999), 1.0 / 1000.0);
}
} }
+1 -1
View File
@@ -17,7 +17,7 @@ mod sim_broker;
pub use mc::{monte_carlo, r_bootstrap, McAggregate, McDraw, McFamily, RBootstrap}; pub use mc::{monte_carlo, r_bootstrap, McAggregate, McDraw, McFamily, RBootstrap};
pub use metrics::{ pub use metrics::{
derive_position_events, r_metrics_from_rs, summarize, summarize_r, PositionAction, derive_position_events, r_metrics_from_rs, summarize, summarize_r, PositionAction,
PositionEvent, RMetrics, RunMetrics, PositionEvent, RMetrics, RunMetricKey, RunMetrics, RANKABLE_METRICS, R_BASED_METRICS,
}; };
pub use position_management::{ pub use position_management::{
ExitReason, FIELD_NAMES as PM_FIELD_NAMES, PositionManagement, ExitReason, FIELD_NAMES as PM_FIELD_NAMES, PositionManagement,
+218
View File
@@ -9,6 +9,7 @@
//! re-associated. //! re-associated.
use aura_core::{Scalar, SeriesFold, Timestamp}; use aura_core::{Scalar, SeriesFold, Timestamp};
use aura_engine::{resample_block, MetricVocabulary, SplitMix64};
/// Reduce a run's recorded pip-equity + exposure streams into summary metrics. /// Reduce a run's recorded pip-equity + exposure streams into summary metrics.
/// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are /// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are
@@ -476,6 +477,125 @@ pub struct PositionEvent {
pub volume: f64, pub volume: f64,
} }
/// The backtest metric vocabulary's resolved keys (was aura-registry's
/// private `Metric` enum — the vocabulary is supplied by backtest now, not
/// baked into the registry; #147). The runtime hot path branches on this
/// rather than re-parsing the name.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum RunMetricKey {
TotalPips,
MaxDrawdown,
BiasSignFlips,
Sqn,
SqnNormalized,
ExpectancyR,
NetExpectancyR,
}
impl RunMetricKey {
/// C10: the four R-denominated keys — the account/instrument-agnostic
/// set the R-only registry gates (`check_r_metric`, `generalization`)
/// accept.
pub fn r_based(self) -> bool {
matches!(
self,
Self::Sqn | Self::SqnNormalized | Self::ExpectancyR | Self::NetExpectancyR
)
}
}
/// The four R-based names, in refusal-prose order (the NonRMetric message).
pub const R_BASED_METRICS: &[&str] =
&["sqn", "sqn_normalized", "expectancy_r", "net_expectancy_r"];
/// The rankable roster, in refusal-prose order — the roster this vocabulary
/// resolves against (#147). The registry's own refusal prose and
/// aura-campaign's re-export still hand-copy this list (not yet single-
/// sourced from here); wiring them to read from here is deferred follow-up.
pub const RANKABLE_METRICS: &[&str] = &[
"total_pips",
"max_drawdown",
"bias_sign_flips",
"sqn",
"sqn_normalized",
"expectancy_r",
"net_expectancy_r",
];
/// Recompute the selected R metric from a trade-R slice (reuses
/// `r_metrics_from_rs`, the same arithmetic `summarize_r` uses) — the
/// deflation null's member reduction (was aura-registry-private, #147).
fn member_metric_from_rs(rs: &[f64], key: RunMetricKey) -> f64 {
let rm = r_metrics_from_rs(rs);
match key {
RunMetricKey::Sqn => rm.sqn,
RunMetricKey::SqnNormalized => rm.sqn_normalized,
RunMetricKey::ExpectancyR => rm.expectancy_r,
RunMetricKey::NetExpectancyR => rm.net_expectancy_r,
_ => unreachable!("member_metric_from_rs is R-only"),
}
}
impl MetricVocabulary for RunMetrics {
type Key = RunMetricKey;
fn resolve(name: &str) -> Option<RunMetricKey> {
Some(match name {
"total_pips" => RunMetricKey::TotalPips,
"max_drawdown" => RunMetricKey::MaxDrawdown,
// accept the new name AND the pre-rename one (CLI back-compat).
"bias_sign_flips" | "exposure_sign_flips" => RunMetricKey::BiasSignFlips,
"sqn" => RunMetricKey::Sqn,
"sqn_normalized" => RunMetricKey::SqnNormalized,
"expectancy_r" => RunMetricKey::ExpectancyR,
"net_expectancy_r" => RunMetricKey::NetExpectancyR,
_ => return None,
})
}
fn known() -> &'static [&'static str] {
RANKABLE_METRICS
}
fn higher_is_better(key: RunMetricKey) -> bool {
!matches!(key, RunMetricKey::MaxDrawdown | RunMetricKey::BiasSignFlips)
}
fn value(&self, key: RunMetricKey) -> f64 {
fn r_get(m: &RunMetrics, f: impl Fn(&RMetrics) -> f64) -> f64 {
m.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY)
}
match key {
RunMetricKey::TotalPips => self.total_pips,
RunMetricKey::MaxDrawdown => self.max_drawdown,
RunMetricKey::BiasSignFlips => self.bias_sign_flips as f64,
RunMetricKey::Sqn => r_get(self, |r| r.sqn),
RunMetricKey::SqnNormalized => r_get(self, |r| r.sqn_normalized),
RunMetricKey::ExpectancyR => r_get(self, |r| r.expectancy_r),
RunMetricKey::NetExpectancyR => r_get(self, |r| r.net_expectancy_r),
}
}
fn has_resampling_null(key: RunMetricKey) -> bool {
key.r_based()
}
/// # Panics
/// Panics (via `member_metric_from_rs`'s `unreachable!`) if `key` is not
/// R-based — callers must gate on `has_resampling_null(key)` before
/// drawing, exactly as documented at the trait level.
fn null_draw(&self, key: RunMetricKey, block_len: usize, rng: &mut SplitMix64) -> Option<f64> {
let rs: &[f64] = self.r.as_ref().map(|r| r.net_trade_rs.as_slice()).unwrap_or(&[]);
if rs.is_empty() {
return None; // consumes no rng — bit-parity with the pre-#147 fold
}
let mean = rs.iter().sum::<f64>() / rs.len() as f64;
let centred: Vec<f64> = rs.iter().map(|x| x - mean).collect();
let bl = block_len.clamp(1, rs.len());
Some(member_metric_from_rs(&resample_block(&centred, bl, rng), key))
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -1252,4 +1372,102 @@ mod tests {
let m = summarize(&[], &exposure); let m = summarize(&[], &exposure);
assert_eq!(m.bias_sign_flips, 0); assert_eq!(m.bias_sign_flips, 0);
} }
#[test]
fn run_metrics_vocabulary_resolves_and_classifies() {
use aura_engine::MetricVocabulary;
assert!(matches!(
<RunMetrics as MetricVocabulary>::resolve("sqn"),
Some(RunMetricKey::Sqn)
));
// the pre-rename CLI alias must keep resolving (back-compat).
assert!(matches!(
<RunMetrics as MetricVocabulary>::resolve("exposure_sign_flips"),
Some(RunMetricKey::BiasSignFlips)
));
assert!(<RunMetrics as MetricVocabulary>::resolve("sharpe").is_none());
assert_eq!(<RunMetrics as MetricVocabulary>::known(), RANKABLE_METRICS);
assert!(RunMetricKey::Sqn.r_based());
assert!(!RunMetricKey::TotalPips.r_based());
}
/// Property: an R-based key on a member with no `r` block ranks worst for
/// a higher-is-better metric (`f64::NEG_INFINITY`), never a silent default
/// like `0.0` that could out-rank a genuinely poor member; non-R keys are
/// read straight from the pip-only fields regardless of the `r` block.
#[test]
fn value_r_based_key_reads_neg_infinity_when_r_block_absent() {
let m = RunMetrics { total_pips: 5.0, max_drawdown: 2.0, bias_sign_flips: 1, r: None };
for key in [
RunMetricKey::Sqn,
RunMetricKey::SqnNormalized,
RunMetricKey::ExpectancyR,
RunMetricKey::NetExpectancyR,
] {
assert_eq!(m.value(key), f64::NEG_INFINITY, "missing r block must rank worst");
}
assert_eq!(m.value(RunMetricKey::TotalPips), 5.0);
assert_eq!(m.value(RunMetricKey::MaxDrawdown), 2.0);
assert_eq!(m.value(RunMetricKey::BiasSignFlips), 1.0);
}
/// Property: a member with no trade-R input (no `r` block) yields no null
/// draw AND leaves the rng untouched — the "consumes no rng" contract the
/// implementation comment claims. Verified indirectly (SplitMix64 carries
/// no Clone/PartialEq): the rng's next draw after the no-op call matches a
/// freshly, identically-seeded rng's first draw.
#[test]
fn null_draw_returns_none_and_consumes_no_rng_when_no_trades() {
let m = RunMetrics { total_pips: 0.0, max_drawdown: 0.0, bias_sign_flips: 0, r: None };
let mut rng = SplitMix64::new(11);
let got = m.null_draw(RunMetricKey::Sqn, 2, &mut rng);
assert_eq!(got, None);
let mut fresh = SplitMix64::new(11);
assert_eq!(
rng.next_u64(),
fresh.next_u64(),
"null_draw's None path must consume no rng state"
);
}
/// Property: `null_draw` recomputes the selected key by mean-centring
/// `net_trade_rs`, block-resampling it under the given rng, and reducing
/// via `member_metric_from_rs` — exactly the recipe reproduced here by
/// hand with an identically-seeded rng (C1: same seed -> same draw).
#[test]
fn null_draw_recomputes_the_selected_key_from_a_seeded_block_resample() {
let net_trade_rs = vec![1.0, 2.0, -1.0, 3.0];
let m = RunMetrics {
total_pips: 0.0,
max_drawdown: 0.0,
bias_sign_flips: 0,
r: Some(RMetrics {
expectancy_r: 0.0,
n_trades: 4,
win_rate: 0.0,
avg_win_r: 0.0,
avg_loss_r: 0.0,
profit_factor: 0.0,
max_r_drawdown: 0.0,
n_open_at_end: 0,
sqn: 0.0,
sqn_normalized: 0.0,
net_expectancy_r: 0.0,
conviction_terciles_r: [0.0; 3],
net_trade_rs: net_trade_rs.clone(),
}),
};
let mut rng_a = SplitMix64::new(7);
let got = m
.null_draw(RunMetricKey::ExpectancyR, 2, &mut rng_a)
.expect("nonempty net_trade_rs yields a draw");
let mean = net_trade_rs.iter().sum::<f64>() / net_trade_rs.len() as f64;
let centred: Vec<f64> = net_trade_rs.iter().map(|x| x - mean).collect();
let mut rng_b = SplitMix64::new(7);
let resampled = resample_block(&centred, 2, &mut rng_b);
let want = member_metric_from_rs(&resampled, RunMetricKey::ExpectancyR);
assert_eq!(got, want);
}
} }
+1 -1
View File
@@ -79,7 +79,7 @@ pub use sweep::{
// `resample_block`/`MetricStats` are foundation-grade (aura-analysis, #291); // `resample_block`/`MetricStats` are foundation-grade (aura-analysis, #291);
// re-exported here (unchanged exported name set) so existing // re-exported here (unchanged exported name set) so existing
// `aura_engine::{resample_block, MetricStats}` consumers stay source-compatible. // `aura_engine::{resample_block, MetricStats}` consumers stay source-compatible.
pub use aura_analysis::{resample_block, MetricStats}; pub use aura_analysis::{one_sided_p_laplace, resample_block, MetricStats, MetricVocabulary};
pub use walkforward::{ pub use walkforward::{
param_stability, walk_forward, RollMode, WalkForwardError, WalkForwardResult, param_stability, walk_forward, RollMode, WalkForwardError, WalkForwardResult,
WindowBounds, WindowOutcome, WindowRoller, WindowRun, WindowBounds, WindowOutcome, WindowRoller, WindowRun,
+128 -173
View File
@@ -18,8 +18,8 @@ use std::io::Write;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use aura_core::PrimitiveBuilder; use aura_core::PrimitiveBuilder;
use aura_analysis::{resample_block, MetricStats, SplitMix64}; use aura_analysis::{one_sided_p_laplace, MetricStats, MetricVocabulary, SplitMix64};
use aura_backtest::{r_metrics_from_rs, RunReport, SweepFamily, SweepPoint}; use aura_backtest::{RunMetrics, RunReport};
use aura_engine::{ use aura_engine::{
blueprint_from_json, blueprint_identity_json, expected_max_of_normals, FamilySelection, blueprint_from_json, blueprint_identity_json, expected_max_of_normals, FamilySelection,
SelectionMode, SelectionMode,
@@ -495,77 +495,30 @@ impl Registry {
} }
} }
/// Which metric a best-first comparison keys on. Resolves a metric *name* (a
/// string at the API boundary) to a closed kind once, so the per-comparison hot
/// path branches on this rather than re-parsing the name.
#[derive(Clone, Copy)]
enum Metric {
TotalPips,
MaxDrawdown,
BiasSignFlips,
Sqn,
SqnNormalized,
ExpectancyR,
NetExpectancyR,
}
/// Resolve a metric NAME to the closed `Metric` kind (the single name→kind map).
/// An unknown name is a `RegistryError::UnknownMetric`.
fn resolve_metric(name: &str) -> Result<Metric, RegistryError> {
Ok(match name {
"total_pips" => Metric::TotalPips,
"max_drawdown" => Metric::MaxDrawdown,
// accept the new name AND the pre-rename one (CLI back-compat).
"bias_sign_flips" | "exposure_sign_flips" => Metric::BiasSignFlips,
"sqn" => Metric::Sqn,
"sqn_normalized" => Metric::SqnNormalized,
"expectancy_r" => Metric::ExpectancyR,
"net_expectancy_r" => Metric::NetExpectancyR,
other => return Err(RegistryError::UnknownMetric(other.to_string())),
})
}
/// The metric's scalar value for one report. R metrics live in the optional `r`
/// block; a missing block reads `NEG_INFINITY` (the worst rank for the
/// higher-is-better R keys), preserving `metric_cmp`'s documented contract.
fn metric_value(rep: &RunReport, m: Metric) -> f64 {
fn r_get(rep: &RunReport, f: impl Fn(&aura_backtest::RMetrics) -> f64) -> f64 {
rep.metrics.r.as_ref().map(f).unwrap_or(f64::NEG_INFINITY)
}
match m {
Metric::TotalPips => rep.metrics.total_pips,
Metric::MaxDrawdown => rep.metrics.max_drawdown,
Metric::BiasSignFlips => rep.metrics.bias_sign_flips as f64,
Metric::Sqn => r_get(rep, |r| r.sqn),
Metric::SqnNormalized => r_get(rep, |r| r.sqn_normalized),
Metric::ExpectancyR => r_get(rep, |r| r.expectancy_r),
Metric::NetExpectancyR => r_get(rep, |r| r.net_expectancy_r),
}
}
/// The per-metric **best-first ordering** of two reports: `Less` means `a` is /// The per-metric **best-first ordering** of two reports: `Less` means `a` is
/// better than `b`. "Best" is fixed by each metric's meaning: `total_pips` and /// better than `b`. "Best" is fixed by each metric's meaning as declared by
/// the four R keys (`sqn`, `sqn_normalized`, `expectancy_r`, `net_expectancy_r`) /// the payload's `MetricVocabulary` (direction via `higher_is_better`; a
/// higher-is-better; `max_drawdown` and `bias_sign_flips` lower-is-better. Every /// missing optional block reads `f64::NEG_INFINITY`, the worst rank for
/// metric compares via `total_cmp` (a total order over `f64`): top-level keys are /// higher-is-better keys). Every metric compares via `total_cmp` (a total
/// finite by construction, while the R keys live in an optional `metrics.r` block /// order over `f64`), so strays order deterministically without panicking. An
/// where a member with `r: None` reads `f64::NEG_INFINITY` (the worst rank for /// unknown metric name is a `RegistryError::UnknownMetric` carrying the
/// higher-is-better) — `total_cmp` orders that, and any stray `NaN`, /// vocabulary's own roster.
/// deterministically without panicking. An unknown metric name is a
/// `RegistryError::UnknownMetric`.
/// ///
/// The single source of best-first *ordering* — both [`rank_by`] (sort by it) and /// The single source of best-first *ordering* — both [`rank_by`] (sort by it)
/// [`optimize`] (argmax by it) call this. The per-metric direction itself lives in /// and [`optimize`] (argmax by it) call this. The name is resolved once via
/// `higher_is_better` (which this and `optimize_plateau` both read); `metric_cmp` /// `M::resolve`; the returned closure carries the resolved key, so
/// turns that direction into a `total_cmp` over the read values. The metric name is /// per-comparison work is just reading each value and the key compare.
/// resolved once via `resolve_metric`; the returned closure carries the resolved fn metric_cmp<M: MetricVocabulary>(
/// `Metric`, so per-comparison work is just reading each value (`metric_value`) and metric: &str,
/// the key compare. ) -> Result<impl Fn(&aura_engine::RunReport<M>, &aura_engine::RunReport<M>) -> Ordering, RegistryError>
fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Ordering, RegistryError> { {
let m = resolve_metric(metric)?; let key = M::resolve(metric).ok_or_else(|| RegistryError::UnknownMetric {
let hib = higher_is_better(m); name: metric.to_string(),
Ok(move |a: &RunReport, b: &RunReport| { known: M::known(),
let (va, vb) = (metric_value(a, m), metric_value(b, m)); })?;
let hib = M::higher_is_better(key);
Ok(move |a: &aura_engine::RunReport<M>, b: &aura_engine::RunReport<M>| {
let (va, vb) = (a.metrics.value(key), b.metrics.value(key));
// higher-is-better ranks the larger value first; lower-is-better the smaller. // higher-is-better ranks the larger value first; lower-is-better the smaller.
if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) } if hib { vb.total_cmp(&va) } else { va.total_cmp(&vb) }
}) })
@@ -575,8 +528,11 @@ fn metric_cmp(metric: &str) -> Result<impl Fn(&RunReport, &RunReport) -> Orderin
/// by each metric's meaning (see `metric_cmp`). A stable sort keeps file /// by each metric's meaning (see `metric_cmp`). A stable sort keeps file
/// (insertion) order among ties. An unknown metric name is a /// (insertion) order among ties. An unknown metric name is a
/// `RegistryError::UnknownMetric`. /// `RegistryError::UnknownMetric`.
pub fn rank_by(mut reports: Vec<RunReport>, metric: &str) -> Result<Vec<RunReport>, RegistryError> { pub fn rank_by<M: MetricVocabulary>(
let cmp = metric_cmp(metric)?; mut reports: Vec<aura_engine::RunReport<M>>,
metric: &str,
) -> Result<Vec<aura_engine::RunReport<M>>, RegistryError> {
let cmp = metric_cmp::<M>(metric)?;
reports.sort_by(|a, b| cmp(a, b)); reports.sort_by(|a, b| cmp(a, b));
Ok(reports) Ok(reports)
} }
@@ -590,8 +546,11 @@ pub fn rank_by(mut reports: Vec<RunReport>, metric: &str) -> Result<Vec<RunRepor
/// An unknown metric is a `RegistryError::UnknownMetric`. A `SweepFamily` is /// An unknown metric is a `RegistryError::UnknownMetric`. A `SweepFamily` is
/// non-empty by construction (a sweep over a zero-point grid is rejected upstream /// non-empty by construction (a sweep over a zero-point grid is rejected upstream
/// by `EmptyAxis`), so `reduce` always yields a winner. /// by `EmptyAxis`), so `reduce` always yields a winner.
pub fn optimize(family: &SweepFamily, metric: &str) -> Result<SweepPoint, RegistryError> { pub fn optimize<M: MetricVocabulary + Clone>(
let cmp = metric_cmp(metric)?; family: &aura_engine::SweepFamily<M>,
metric: &str,
) -> Result<aura_engine::SweepPoint<M>, RegistryError> {
let cmp = metric_cmp::<M>(metric)?;
let winner = family let winner = family
.points .points
.iter() .iter()
@@ -600,19 +559,6 @@ pub fn optimize(family: &SweepFamily, metric: &str) -> Result<SweepPoint, Regist
Ok(winner.clone()) Ok(winner.clone())
} }
fn is_r_metric(m: Metric) -> bool {
matches!(m, Metric::Sqn | Metric::SqnNormalized | Metric::ExpectancyR | Metric::NetExpectancyR)
}
/// The metric's optimisation direction: `true` if a larger value is better
/// (`total_pips`, the four R keys), `false` for the lower-is-better keys
/// (`max_drawdown`, `bias_sign_flips`). The single direction source — `metric_cmp`
/// (best-first ordering) and `optimize_plateau` (smoothed argmax + worst-neighbour)
/// both read it, so the per-metric sense lives in exactly one place.
fn higher_is_better(m: Metric) -> bool {
!matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips)
}
/// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice /// The closed grid neighbourhood of flat index `i` over a mixed-radix lattice
/// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}` /// (`axis_lens`, last axis fastest — the `GridSpace` odometer convention): `{i}`
/// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order /// plus each in-range ±1-per-axis cell. Pure index math, deterministic. The order
@@ -641,49 +587,36 @@ fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> {
out out
} }
fn member_net_trade_rs(rep: &RunReport) -> &[f64] { /// The best-of-K null-max distribution, vocabulary-generic: for each resample
rep.metrics.r.as_ref().map(|r| r.net_trade_rs.as_slice()).unwrap_or(&[]) /// every member draws ONE null statistic from its own vocabulary (in odometer
} /// order, from a per-iteration seed) and the max across members is taken. A
/// member with no usable null input contributes nothing and consumes no rng
/// Recompute the selected R metric from a trade-R slice (reuses the engine's /// (bit-parity with the pre-#147 R fold, whose centring is now inside
/// `r_metrics_from_rs`, the same arithmetic `summarize_r` uses). /// `RunMetrics::null_draw`). Deterministic given `seed` (C1).
fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64 { fn null_best_of_k<M: MetricVocabulary>(
let rm = r_metrics_from_rs(rs); family: &aura_engine::SweepFamily<M>,
match m { key: M::Key,
Metric::Sqn => rm.sqn, n_resamples: usize,
Metric::SqnNormalized => rm.sqn_normalized, block_len: usize,
Metric::ExpectancyR => rm.expectancy_r, seed: u64,
Metric::NetExpectancyR => rm.net_expectancy_r, ) -> Vec<f64> {
_ => unreachable!("member_metric_from_rs is R-only"), (0..n_resamples)
} .map(|i| {
} let mut rng = SplitMix64::new(seed ^ i as u64);
family.points.iter().fold(f64::NEG_INFINITY, |best, p| {
/// The centred best-of-K null-max distribution: each member's `net_trade_rs` is match p.report.metrics.null_draw(key, block_len, &mut rng) {
/// mean-subtracted (the no-edge null), then for each resample every member is Some(v) => best.max(v),
/// moving-block-resampled (in odometer order, from a per-iteration seed) and the None => best,
/// max recomputed metric across members is taken. Deterministic given `seed` (C1). }
fn null_best_of_k(family: &SweepFamily, m: Metric, n_resamples: usize, block_len: usize, seed: u64) -> Vec<f64> { })
let centred: Vec<Vec<f64>> = family.points.iter().map(|p| {
let rs = member_net_trade_rs(&p.report);
if rs.is_empty() { return Vec::new(); }
let mean = rs.iter().sum::<f64>() / rs.len() as f64;
rs.iter().map(|x| x - mean).collect()
}).collect();
(0..n_resamples).map(|i| {
let mut rng = SplitMix64::new(seed ^ i as u64);
centred.iter().fold(f64::NEG_INFINITY, |best, rs| {
if rs.is_empty() { return best; }
let bl = block_len.clamp(1, rs.len());
let v = member_metric_from_rs(&resample_block(rs, bl, &mut rng), m);
best.max(v)
}) })
}).collect() .collect()
} }
/// Sample standard deviation of the K members' metric values (the `total_pips` /// Sample standard deviation of the K members' metric values (the `total_pips`
/// dispersion-floor arm). `< 2` members -> `0.0`. /// dispersion-floor arm). `< 2` members -> `0.0`.
fn member_sd(family: &SweepFamily, m: Metric) -> f64 { fn member_sd<M: MetricVocabulary>(family: &aura_engine::SweepFamily<M>, key: M::Key) -> f64 {
let vals: Vec<f64> = family.points.iter().map(|p| metric_value(&p.report, m)).collect(); let vals: Vec<f64> = family.points.iter().map(|p| p.report.metrics.value(key)).collect();
let n = vals.len(); let n = vals.len();
if n < 2 { return 0.0; } if n < 2 { return 0.0; }
let mean = vals.iter().sum::<f64>() / n as f64; let mean = vals.iter().sum::<f64>() / n as f64;
@@ -702,27 +635,30 @@ pub enum PlateauMode {
/// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau /// `optimize`'s argmax over the NEIGHBOURHOOD-SMOOTHED surface, plus its plateau
/// provenance. Each member scores as the mean (or worst-case) of its closed grid /// provenance. Each member scores as the mean (or worst-case) of its closed grid
/// neighbourhood's `metric_value`; the winner is the best smoothed score by the /// neighbourhood's metric value; the winner is the best smoothed score by the
/// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are /// metric's own direction (earliest-odometer tie, as `optimize`). `axis_lens` are
/// the grid radixes (`param_space()` order, last axis fastest). Pure, /// the grid radixes (`param_space()` order, last axis fastest). Pure,
/// deterministic (C1). The returned `SweepPoint` is the winning grid CELL; /// deterministic (C1). The returned `SweepPoint` is the winning grid CELL;
/// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the /// `raw_winner_metric` is that cell's own metric, `neighbourhood_score` the
/// smoothed value the argmax maximised. /// smoothed value the argmax maximised.
pub fn optimize_plateau( pub fn optimize_plateau<M: MetricVocabulary + Clone>(
family: &SweepFamily, axis_lens: &[usize], metric: &str, mode: PlateauMode, family: &aura_engine::SweepFamily<M>, axis_lens: &[usize], metric: &str, mode: PlateauMode,
) -> Result<(SweepPoint, FamilySelection), RegistryError> { ) -> Result<(aura_engine::SweepPoint<M>, FamilySelection), RegistryError> {
let m = resolve_metric(metric)?; let key = M::resolve(metric).ok_or_else(|| RegistryError::UnknownMetric {
name: metric.to_string(),
known: M::known(),
})?;
let n = family.points.len(); let n = family.points.len();
debug_assert_eq!( debug_assert_eq!(
axis_lens.iter().product::<usize>(), n, axis_lens.iter().product::<usize>(), n,
"axis_lens product must equal the family size (a valid grid lattice)", "axis_lens product must equal the family size (a valid grid lattice)",
); );
let hib = higher_is_better(m); let hib = M::higher_is_better(key);
// smoothed score + closed-neighbourhood size per member, in odometer order // smoothed score + closed-neighbourhood size per member, in odometer order
let scored: Vec<(usize, f64, usize)> = (0..n) let scored: Vec<(usize, f64, usize)> = (0..n)
.map(|i| { .map(|i| {
let nbrs = closed_neighbourhood(i, axis_lens); let nbrs = closed_neighbourhood(i, axis_lens);
let vals: Vec<f64> = nbrs.iter().map(|&j| metric_value(&family.points[j].report, m)).collect(); let vals: Vec<f64> = nbrs.iter().map(|&j| family.points[j].report.metrics.value(key)).collect();
let score = match mode { let score = match mode {
PlateauMode::Mean => vals.iter().sum::<f64>() / vals.len() as f64, PlateauMode::Mean => vals.iter().sum::<f64>() / vals.len() as f64,
PlateauMode::Worst => { PlateauMode::Worst => {
@@ -746,7 +682,7 @@ pub fn optimize_plateau(
}) })
.expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)"); .expect("a SweepFamily is non-empty by construction (EmptyAxis is rejected upstream)");
let winner = family.points[wi].clone(); let winner = family.points[wi].clone();
let raw = metric_value(&winner.report, m); let raw = winner.report.metrics.value(key);
Ok(( Ok((
winner, winner,
FamilySelection { FamilySelection {
@@ -777,28 +713,34 @@ pub const DEFLATION_BLOCK_LEN: usize = 5;
/// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned /// `optimize`'s argmax winner PLUS its trials-deflation provenance. The returned
/// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23). /// `SweepPoint` is byte-identical to `optimize(family, metric)` (additive, C23).
/// R arm: a centred moving-block reality-check (`overfit_probability` = /// Which arm runs is vocabulary-declared resampling null vs dispersion floor
/// `(count(null ≥ raw) + 1)/(n + 1)`, `deflated_score` = `raw p95(null)`). /// (`M::has_resampling_null`): a resampling-null key gets a centred moving-block
/// `total_pips` arm: a closed-form expected-max-of-K dispersion floor, no /// reality-check (`overfit_probability` = `(count(null ≥ raw) + 1)/(n + 1)`,
/// probability. Deterministic given `seed` (C1). /// `deflated_score` = `raw p95(null)`); otherwise a closed-form
pub fn optimize_deflated( /// expected-max-of-K dispersion floor, no probability. Deterministic given
family: &SweepFamily, metric: &str, n_resamples: usize, block_len: usize, seed: u64, /// `seed` (C1).
) -> Result<(SweepPoint, FamilySelection), RegistryError> { pub fn optimize_deflated<M: MetricVocabulary + Clone>(
family: &aura_engine::SweepFamily<M>, metric: &str, n_resamples: usize, block_len: usize, seed: u64,
) -> Result<(aura_engine::SweepPoint<M>, FamilySelection), RegistryError> {
let winner = optimize(family, metric)?; let winner = optimize(family, metric)?;
let m = resolve_metric(metric)?; let key = M::resolve(metric).ok_or_else(|| RegistryError::UnknownMetric {
name: metric.to_string(),
known: M::known(),
})?;
let k = family.points.len(); let k = family.points.len();
let raw = metric_value(&winner.report, m); let raw = winner.report.metrics.value(key);
let (deflated_score, overfit_probability) = if is_r_metric(m) { let (deflated_score, overfit_probability) = if M::has_resampling_null(key) {
let null_max = null_best_of_k(family, m, n_resamples, block_len, seed); let null_max = null_best_of_k(family, key, n_resamples, block_len, seed);
// Keep only the finite best-of-K draws. The null is *not computable* in two // Keep only the finite best-of-K draws. The null is *not computable* in two
// degenerate cases: zero resamples (`null_max` empty), or no member carries // degenerate cases: zero resamples (`null_max` empty), or no member yields a
// `net_trade_rs` (every member's centred series is empty, so each iteration's // usable draw (`null_draw` returns `None` for every member, so each
// best-of-K folds to `NEG_INFINITY`). The latter is reachable — `net_trade_rs` // iteration's best-of-K folds to `NEG_INFINITY`). The latter is reachable —
// is `#[serde(skip)]`, so a family loaded back from the registry has empty // the R conduit `net_trade_rs` is `#[serde(skip)]`, so a family loaded back
// conduits even when its `r` block (hence `raw`) is finite. Filtering to // from the registry has empty conduits even when its `r` block (hence `raw`)
// finite values collapses both into one "no usable null" branch, instead of // is finite. Filtering to finite values collapses both into one "no usable
// letting `p95 = NEG_INFINITY` turn `deflated_score` into `+inf`. // null" branch, instead of letting `p95 = NEG_INFINITY` turn
// `deflated_score` into `+inf`.
let usable: Vec<f64> = null_max.into_iter().filter(|x| x.is_finite()).collect(); let usable: Vec<f64> = null_max.into_iter().filter(|x| x.is_finite()).collect();
if usable.is_empty() { if usable.is_empty() {
// No reality-check is computable: floor to a degenerate-but-defined // No reality-check is computable: floor to a degenerate-but-defined
@@ -807,22 +749,22 @@ pub fn optimize_deflated(
(raw, Some(1.0)) (raw, Some(1.0))
} else { } else {
let p95 = MetricStats::from_values(&usable).p95; let p95 = MetricStats::from_values(&usable).p95;
let over = (usable.iter().filter(|&&x| x >= raw).count() + 1) as f64 let ge = usable.iter().filter(|&&x| x >= raw).count();
/ (usable.len() + 1) as f64; let over = one_sided_p_laplace(ge, usable.len());
(raw - p95, Some(over)) (raw - p95, Some(over))
} }
} else { } else {
// The dispersion floor SUBTRACTS the expected-max inflation, so it is valid // The dispersion floor SUBTRACTS the expected-max inflation, so it is valid
// only for a higher-is-better metric (a larger value is the better one the // only for a higher-is-better metric (a larger value is the better one the
// search inflated). A lower-is-better metric would deflate wrong-signed; the // search inflated). A lower-is-better metric would deflate wrong-signed; the
// two shipped call sites only pass `total_pips`, so this is unreached, but // shipped call sites only pass `total_pips`, so this is unreached, but
// `optimize_deflated` is public — guard the assumption rather than leave it // `optimize_deflated` is public — guard the assumption rather than leave it
// implicit. // implicit.
debug_assert!( debug_assert!(
!matches!(m, Metric::MaxDrawdown | Metric::BiasSignFlips), M::higher_is_better(key),
"dispersion-floor arm is for higher-is-better metrics only", "dispersion-floor arm is for higher-is-better metrics only",
); );
(raw - member_sd(family, m) * expected_max_of_normals(k), None) (raw - member_sd(family, key) * expected_max_of_normals(k), None)
}; };
Ok((winner.clone(), FamilySelection { Ok((winner.clone(), FamilySelection {
@@ -854,8 +796,13 @@ pub struct Generalization {
/// the CLI runs before evaluating any instrument. The registry owns metric truth /// the CLI runs before evaluating any instrument. The registry owns metric truth
/// (C9), so the CLI never duplicates the R-set. /// (C9), so the CLI never duplicates the R-set.
pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> { pub fn check_r_metric(metric: &str) -> Result<(), RegistryError> {
let m = resolve_metric(metric)?; let key = <RunMetrics as MetricVocabulary>::resolve(metric).ok_or_else(|| {
if is_r_metric(m) { RegistryError::UnknownMetric {
name: metric.to_string(),
known: <RunMetrics as MetricVocabulary>::known(),
}
})?;
if key.r_based() {
Ok(()) Ok(())
} else { } else {
Err(RegistryError::NonRMetric(metric.to_string())) Err(RegistryError::NonRMetric(metric.to_string()))
@@ -871,8 +818,13 @@ pub fn generalization(
per_instrument: &[(String, &RunReport)], per_instrument: &[(String, &RunReport)],
metric: &str, metric: &str,
) -> Result<Generalization, RegistryError> { ) -> Result<Generalization, RegistryError> {
let m = resolve_metric(metric)?; let key = <RunMetrics as MetricVocabulary>::resolve(metric).ok_or_else(|| {
if !is_r_metric(m) { RegistryError::UnknownMetric {
name: metric.to_string(),
known: <RunMetrics as MetricVocabulary>::known(),
}
})?;
if !key.r_based() {
return Err(RegistryError::NonRMetric(metric.to_string())); return Err(RegistryError::NonRMetric(metric.to_string()));
} }
if per_instrument.len() < 2 { if per_instrument.len() < 2 {
@@ -880,7 +832,7 @@ pub fn generalization(
} }
let vals: Vec<(String, f64)> = per_instrument let vals: Vec<(String, f64)> = per_instrument
.iter() .iter()
.map(|(label, rep)| (label.clone(), metric_value(rep, m))) .map(|(label, rep)| (label.clone(), rep.metrics.value(key)))
.collect(); .collect();
let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min); let worst_case = vals.iter().map(|(_, v)| *v).fold(f64::INFINITY, f64::min);
let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count(); let sign_agreement = vals.iter().filter(|(_, v)| *v > 0.0).count();
@@ -900,8 +852,8 @@ pub enum RegistryError {
Io(std::io::Error), Io(std::io::Error),
/// A stored line did not parse as a `RunReport` (1-based line number). /// A stored line did not parse as a `RunReport` (1-based line number).
Parse { line: usize, source: serde_json::Error }, Parse { line: usize, source: serde_json::Error },
/// `rank_by` was given a metric name it does not know. /// `rank_by` was given a metric name outside the family's vocabulary.
UnknownMetric(String), UnknownMetric { name: String, known: &'static [&'static str] },
/// A known metric that is not R-based — cross-instrument generalization is /// A known metric that is not R-based — cross-instrument generalization is
/// R-only (R is the only account/instrument-agnostic unit, C10). /// R-only (R is the only account/instrument-agnostic unit, C10).
NonRMetric(String), NonRMetric(String),
@@ -916,13 +868,15 @@ impl fmt::Display for RegistryError {
RegistryError::Parse { line, source } => { RegistryError::Parse { line, source } => {
write!(f, "registry parse error at line {line}: {source}") write!(f, "registry parse error at line {line}: {source}")
} }
RegistryError::UnknownMetric(m) => write!( RegistryError::UnknownMetric { name, known } => write!(
f, f,
"unknown metric '{m}' (known: total_pips, max_drawdown, bias_sign_flips, sqn, sqn_normalized, expectancy_r, net_expectancy_r)" "unknown metric '{name}' (known: {})",
known.join(", ")
), ),
RegistryError::NonRMetric(m) => write!( RegistryError::NonRMetric(m) => write!(
f, f,
"metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r)" "metric '{m}' is not comparable across instruments; cross-instrument scoring is R-only ({})",
aura_backtest::R_BASED_METRICS.join(", ")
), ),
RegistryError::TooFewInstruments(n) => write!( RegistryError::TooFewInstruments(n) => write!(
f, f,
@@ -944,7 +898,7 @@ impl From<std::io::Error> for RegistryError {
mod tests { mod tests {
use super::*; use super::*;
use aura_core::{Cell, Scalar, Timestamp}; use aura_core::{Cell, Scalar, Timestamp};
use aura_backtest::{RMetrics, RunMetrics}; use aura_backtest::{RMetrics, RunMetrics, SweepFamily, SweepPoint};
use aura_engine::RunManifest; use aura_engine::RunManifest;
fn report_with(total_pips: f64, max_drawdown: f64, flips: u64) -> RunReport { fn report_with(total_pips: f64, max_drawdown: f64, flips: u64) -> RunReport {
@@ -1037,7 +991,8 @@ mod tests {
#[test] #[test]
fn unknown_metric_message_lists_r_metrics() { fn unknown_metric_message_lists_r_metrics() {
let msg = RegistryError::UnknownMetric("nope".to_string()).to_string(); let msg = RegistryError::UnknownMetric { name: "nope".to_string(), known: aura_backtest::RANKABLE_METRICS }
.to_string();
assert!(msg.contains("sqn"), "msg: {msg}"); assert!(msg.contains("sqn"), "msg: {msg}");
assert!(msg.contains("sqn_normalized"), "msg: {msg}"); assert!(msg.contains("sqn_normalized"), "msg: {msg}");
assert!(msg.contains("expectancy_r"), "msg: {msg}"); assert!(msg.contains("expectancy_r"), "msg: {msg}");
@@ -1176,7 +1131,7 @@ mod tests {
fn rank_by_unknown_metric_is_an_error() { fn rank_by_unknown_metric_is_an_error() {
let reports = vec![report_with(1.0, 0.5, 1)]; let reports = vec![report_with(1.0, 0.5, 1)];
match rank_by(reports, "sharpe") { match rank_by(reports, "sharpe") {
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "sharpe"), Err(RegistryError::UnknownMetric { name, .. }) => assert_eq!(name, "sharpe"),
other => panic!("expected UnknownMetric, got {other:?}"), other => panic!("expected UnknownMetric, got {other:?}"),
} }
} }
@@ -1589,8 +1544,8 @@ mod tests {
let fam = fixture_family_with_r(); let fam = fixture_family_with_r();
let (winner, sel) = optimize_deflated(&fam, "sqn_normalized", 200, 3, 1).unwrap(); let (winner, sel) = optimize_deflated(&fam, "sqn_normalized", 200, 3, 1).unwrap();
assert_eq!(sel.n_trials, fam.points.len()); assert_eq!(sel.n_trials, fam.points.len());
let m = resolve_metric("sqn_normalized").unwrap(); let m = <RunMetrics as MetricVocabulary>::resolve("sqn_normalized").unwrap();
assert_eq!(sel.raw_winner_metric, metric_value(&winner.report, m)); assert_eq!(sel.raw_winner_metric, winner.report.metrics.value(m));
} }
/// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle /// A 1-D grid (7 cells, odometer order) with two end spikes and a broad middle
@@ -2373,7 +2328,7 @@ mod tests {
other => panic!("expected NonRMetric, got {other:?}"), other => panic!("expected NonRMetric, got {other:?}"),
} }
match check_r_metric("nope") { match check_r_metric("nope") {
Err(RegistryError::UnknownMetric(m)) => assert_eq!(m, "nope"), Err(RegistryError::UnknownMetric { name, .. }) => assert_eq!(name, "nope"),
other => panic!("expected UnknownMetric, got {other:?}"), other => panic!("expected UnknownMetric, got {other:?}"),
} }
} }