feat(analysis,campaign,registry,cli): bootstrap net R through the process pipeline
The OOS bootstrap conduit RMetrics.trade_rs becomes net_trade_rs and carries the COST-NETTED per-trade R (r − cost_in_r, trade order). Every conduit consumer is thereby net-when-costed: the monte-carlo pooled-OOS and per-survivor bootstraps, the walk-forward oos_r pooling, and the deflation null-max — which previously compared a net observed statistic against a gross-resampled null whenever a costed campaign selected on net_expectancy_r. An uncosted run is bit-identical (empty cost stream ⇒ cost 0.0 per trade), so every existing golden pin stays green. Design: one conduit, no knob — an explicit net: knob would let a costed campaign silently produce a gross headline again (the exact misreading of the issue's evidence). Per-member RMetrics scalar fields stay gross; net_expectancy_r keeps its meaning. Wire shape unchanged (serde(skip)). The net series is materialized as a separate expression in summarize_r; the pinned-float expressions (net_sum, the SQN pair, the lockstep r_metrics_from_rs copies) keep their tokens verbatim. Fork decisions and rationales: issue #259 comments. New coverage, both hostless over the synthetic SYMA archive: a costed campaign twin shifts the pooled-OOS bootstrap (sweep→gate→wf→mc) and every per-survivor bootstrap (sweep→mc) below its gross sibling, with the trade population unchanged; a unit test pins the conduit as the cost-netted series with the empty-stream degeneracy. Existing conduit tests renamed with the field. Verified: full workspace suite green (71 result blocks), clippy clean, zero bare trade_rs tokens remain, ledger conduit prose updated in place. closes #259
This commit is contained in:
@@ -100,12 +100,15 @@ pub struct RMetrics {
|
||||
pub sqn_normalized: f64,
|
||||
pub net_expectancy_r: f64, // mean(R - cost_in_r) folded from the cost-model stream; == gross when empty
|
||||
pub conviction_terciles_r: [f64; 3], // E[R] by conviction_at_entry tercile (asc); <3 trades -> [0,0,0]
|
||||
/// Realised R per closed trade, in trade order — an in-memory conduit for the
|
||||
/// OOS R-series bootstrap. Excluded from serde (`skip`) so the C18 wire
|
||||
/// shape is unchanged, and from `PartialEq` (below) so every existing
|
||||
/// Cost-netted realised R per closed trade (`r − cost_in_r`), in trade
|
||||
/// order — an in-memory conduit for the OOS R-series bootstrap. Equals
|
||||
/// the gross series bit-for-bit when no cost model is bound (an empty
|
||||
/// cost stream is the documented gross-R baseline: cost `0.0` per
|
||||
/// trade). Excluded from serde (`skip`) so the C18 wire shape is
|
||||
/// unchanged, and from `PartialEq` (below) so every existing
|
||||
/// `RMetrics`/`RunReport` equality assertion and round-trip stays green.
|
||||
#[serde(skip)]
|
||||
pub trade_rs: Vec<f64>,
|
||||
pub net_trade_rs: Vec<f64>,
|
||||
}
|
||||
|
||||
impl PartialEq for RMetrics {
|
||||
@@ -122,7 +125,7 @@ impl PartialEq for RMetrics {
|
||||
&& self.sqn_normalized == o.sqn_normalized
|
||||
&& self.net_expectancy_r == o.net_expectancy_r
|
||||
&& self.conviction_terciles_r == o.conviction_terciles_r
|
||||
// trade_rs deliberately excluded
|
||||
// net_trade_rs deliberately excluded
|
||||
}
|
||||
}
|
||||
|
||||
@@ -229,7 +232,7 @@ pub fn summarize_r(
|
||||
sqn_normalized: 0.0,
|
||||
net_expectancy_r: 0.0,
|
||||
conviction_terciles_r: [0.0; 3],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
};
|
||||
}
|
||||
let rs: Vec<f64> = trades.iter().map(|t| t.r).collect();
|
||||
@@ -277,6 +280,10 @@ pub fn summarize_r(
|
||||
// an empty stream is the gross-R baseline — every trade's cost is 0.0).
|
||||
let net_sum: f64 = trades.iter().map(|t| t.r - t.cost).sum();
|
||||
let net_expectancy_r = net_sum / n as f64;
|
||||
// The per-trade net series the OOS bootstrap conduit carries (#259). A
|
||||
// separate materialization, deliberately NOT factored through `net_sum`:
|
||||
// that expression's tokens are byte-pinned (see the SQN note above).
|
||||
let net_rs: Vec<f64> = trades.iter().map(|t| t.r - t.cost).collect();
|
||||
// conviction terciles: sort by conviction_at_entry ascending, split into three contiguous
|
||||
// near-equal-count buckets (floor boundaries i*n/3), E[R] per bucket. < 3 trades -> 0s.
|
||||
let conviction_terciles_r = if n < 3 {
|
||||
@@ -311,7 +318,7 @@ pub fn summarize_r(
|
||||
sqn_normalized,
|
||||
net_expectancy_r,
|
||||
conviction_terciles_r,
|
||||
trade_rs: rs,
|
||||
net_trade_rs: net_rs,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -324,7 +331,8 @@ pub fn summarize_r(
|
||||
/// `summarize_r_includes_open_trade_and_matches_r_metrics_from_rs` — touch one copy
|
||||
/// without the other and that test is the sole tripwire. The
|
||||
/// fields a flat R series cannot carry are set honestly: `n_open_at_end = 0`,
|
||||
/// `net_expectancy_r = expectancy_r` (exact under the cost = 0 invariant),
|
||||
/// `net_expectancy_r = expectancy_r` (exact: the input series is already
|
||||
/// cost-netted — the conduit carries `r − cost_in_r`),
|
||||
/// `conviction_terciles_r = [0,0,0]` (per-trade conviction is not pooled). Empty
|
||||
/// input -> a well-defined all-zero `RMetrics`.
|
||||
pub fn r_metrics_from_rs(rs: &[f64]) -> RMetrics {
|
||||
@@ -334,7 +342,7 @@ pub fn r_metrics_from_rs(rs: &[f64]) -> RMetrics {
|
||||
expectancy_r: 0.0, n_trades: 0, win_rate: 0.0, avg_win_r: 0.0,
|
||||
avg_loss_r: 0.0, profit_factor: 0.0, max_r_drawdown: 0.0,
|
||||
n_open_at_end: 0, sqn: 0.0, sqn_normalized: 0.0, net_expectancy_r: 0.0,
|
||||
conviction_terciles_r: [0.0; 3], trade_rs: Vec::new(),
|
||||
conviction_terciles_r: [0.0; 3], net_trade_rs: Vec::new(),
|
||||
};
|
||||
}
|
||||
let sum: f64 = rs.iter().sum();
|
||||
@@ -375,9 +383,9 @@ pub fn r_metrics_from_rs(rs: &[f64]) -> RMetrics {
|
||||
n_open_at_end: 0,
|
||||
sqn,
|
||||
sqn_normalized,
|
||||
net_expectancy_r: mean, // cost = 0 -> net == gross (frictionless)
|
||||
net_expectancy_r: mean, // input series is already net -> net == mean
|
||||
conviction_terciles_r: [0.0; 3],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -832,6 +840,30 @@ mod tests {
|
||||
assert!((m.net_expectancy_r - 0.0).abs() < 1e-12);
|
||||
}
|
||||
|
||||
/// Property (#259): the bootstrap conduit `net_trade_rs` carries the
|
||||
/// COST-NETTED per-trade R (`r − cost_in_r`) in trade order — the same
|
||||
/// per-trade values `net_expectancy_r` averages — while every gross
|
||||
/// scalar stays cost-free. With an empty cost stream the conduit equals
|
||||
/// the gross series bit-for-bit (cost 0.0 per trade), which is what
|
||||
/// keeps uncosted campaigns byte-identical.
|
||||
#[test]
|
||||
fn summarize_r_net_trade_rs_is_the_cost_netted_series_in_trade_order() {
|
||||
let c = 2.0_f64;
|
||||
let record = vec![
|
||||
(Timestamp(0), row14(true, 1.0, 100.0, 96.0, 1.0, false, 0.0)),
|
||||
(Timestamp(1), row14(false, 0.0, 100.0, 98.0, 1.0, true, 0.5)),
|
||||
];
|
||||
let cost = vec![
|
||||
(Timestamp(0), vec![Scalar::f64(c / 4.0), Scalar::f64(c / 4.0), Scalar::f64(0.0)]),
|
||||
(Timestamp(1), vec![Scalar::f64(0.0), Scalar::f64(c / 4.0), Scalar::f64(c / 2.0)]),
|
||||
];
|
||||
let m = summarize_r(&record, &cost);
|
||||
assert_eq!(m.net_trade_rs, vec![1.0 - 0.5, 0.5 - 1.0], "net conduit = r − cost, trade order");
|
||||
assert!((m.expectancy_r - 0.75).abs() < 1e-12, "gross scalars stay cost-free");
|
||||
let gross = summarize_r(&record, &[]);
|
||||
assert_eq!(gross.net_trade_rs, vec![1.0, 0.5], "empty cost stream ⇒ conduit == gross series");
|
||||
}
|
||||
|
||||
// One PositionManagement dense record row for the derive tests: only the columns
|
||||
// derive_position_events reads (closed, direction, size, open) are set; the rest
|
||||
// default to 0. Distinct per-row timestamps (the derive keys events on the row's
|
||||
@@ -949,7 +981,7 @@ mod tests {
|
||||
sqn_normalized: 1.0,
|
||||
net_expectancy_r: 0.4,
|
||||
conviction_terciles_r: [-0.5, 0.5, 1.5],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
}),
|
||||
};
|
||||
let json = serde_json::to_string(&m).expect("serialize");
|
||||
@@ -997,25 +1029,25 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn summarize_r_populates_trade_rs_in_trade_order() {
|
||||
fn summarize_r_populates_net_trade_rs_in_trade_order() {
|
||||
// two closed trades (R = +2.0, then -1.0) over a minimal PositionManagement
|
||||
// record; trade_rs must carry [2.0, -1.0] in trade order.
|
||||
// record; net_trade_rs must carry [2.0, -1.0] in trade order.
|
||||
let rec = pm_record_two_closed_trades(); // helper below
|
||||
let m = summarize_r(&rec, &[]);
|
||||
assert_eq!(m.trade_rs, vec![2.0, -1.0]);
|
||||
assert_eq!(m.net_trade_rs, vec![2.0, -1.0]);
|
||||
assert_eq!(m.n_trades, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn summarize_r_empty_record_has_empty_trade_rs() {
|
||||
fn summarize_r_empty_record_has_empty_net_trade_rs() {
|
||||
let m = summarize_r(&[], &[]);
|
||||
assert!(m.trade_rs.is_empty());
|
||||
assert!(m.net_trade_rs.is_empty());
|
||||
assert_eq!(m.n_trades, 0);
|
||||
}
|
||||
|
||||
/// Property: a position still open on the last row is folded into the trade
|
||||
/// ledger at its `unrealized_r` (a window-end trade), so `summarize_r`'s
|
||||
/// `trade_rs` carries that synthetic open trade's R and `n_trades` counts it.
|
||||
/// `net_trade_rs` carries that synthetic open trade's R and `n_trades` counts it.
|
||||
/// This is the one case where the two reducers' inputs differ in meaning — it
|
||||
/// is exactly the per-trade R series the OOS conduit hands `r_metrics_from_rs`,
|
||||
/// so the two must agree on the R-distribution arithmetic for an
|
||||
@@ -1024,12 +1056,12 @@ mod tests {
|
||||
fn summarize_r_includes_open_trade_and_matches_r_metrics_from_rs() {
|
||||
let rec = pm_record_closed_then_open_at_end();
|
||||
let m = summarize_r(&rec, &[]);
|
||||
assert_eq!(m.trade_rs, vec![2.0, 0.5]);
|
||||
assert_eq!(m.net_trade_rs, vec![2.0, 0.5]);
|
||||
assert_eq!(m.n_trades, 2);
|
||||
assert_eq!(m.n_open_at_end, 1);
|
||||
// Feed the open-at-end pooled series through the flat reducer: the
|
||||
// R-distribution fields (the verbatim-copied arithmetic) must agree.
|
||||
let pooled = r_metrics_from_rs(&m.trade_rs);
|
||||
let pooled = r_metrics_from_rs(&m.net_trade_rs);
|
||||
assert_eq!(pooled.n_trades, m.n_trades);
|
||||
assert_eq!(pooled.expectancy_r, m.expectancy_r);
|
||||
assert_eq!(pooled.win_rate, m.win_rate);
|
||||
@@ -1042,25 +1074,25 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rmetrics_partial_eq_ignores_trade_rs() {
|
||||
// two RMetrics equal in every metric but differing in trade_rs compare EQUAL
|
||||
// (trade_rs is an in-memory conduit, excluded from equality) — this is what
|
||||
// keeps serialize->deserialize round-trips equal (trade_rs is serde-skipped,
|
||||
fn rmetrics_partial_eq_ignores_net_trade_rs() {
|
||||
// two RMetrics equal in every metric but differing in net_trade_rs compare EQUAL
|
||||
// (net_trade_rs is an in-memory conduit, excluded from equality) — this is what
|
||||
// keeps serialize->deserialize round-trips equal (net_trade_rs is serde-skipped,
|
||||
// so it deserializes empty).
|
||||
let a = summarize_r(&pm_record_two_closed_trades(), &[]);
|
||||
let mut b = a.clone();
|
||||
b.trade_rs = Vec::new();
|
||||
b.net_trade_rs = Vec::new();
|
||||
assert_eq!(a, b);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn populated_trade_rs_is_absent_from_serialized_json() {
|
||||
// the in-memory conduit must never reach the wire: a populated trade_rs is
|
||||
fn populated_net_trade_rs_is_absent_from_serialized_json() {
|
||||
// the in-memory conduit must never reach the wire: a populated net_trade_rs is
|
||||
// dropped by #[serde(skip)], so the C18 on-disk shape is byte-unperturbed (#139).
|
||||
let m = summarize_r(&pm_record_two_closed_trades(), &[]);
|
||||
assert_eq!(m.trade_rs, vec![2.0, -1.0], "precondition: trade_rs is populated");
|
||||
assert_eq!(m.net_trade_rs, vec![2.0, -1.0], "precondition: net_trade_rs is populated");
|
||||
let json = serde_json::to_string(&m).expect("RMetrics serializes");
|
||||
assert!(!json.contains("trade_rs"), "trade_rs must not reach the wire: {json}");
|
||||
assert!(!json.contains("net_trade_rs"), "net_trade_rs must not reach the wire: {json}");
|
||||
}
|
||||
|
||||
/// A minimal dense PositionManagement record with two closed trades at R = +2, -1.
|
||||
|
||||
@@ -407,7 +407,7 @@ fn run_cell(
|
||||
let bootstrap =
|
||||
match families.iter().rev().find(|f| f.block == "std::walk_forward") {
|
||||
Some(fam) => StageBootstrap::PooledOos(r_bootstrap(
|
||||
&pooled_trade_rs(&fam.reports),
|
||||
&pooled_net_trade_rs(&fam.reports),
|
||||
*resamples as usize,
|
||||
*block_len as usize,
|
||||
seed,
|
||||
@@ -423,7 +423,7 @@ fn run_cell(
|
||||
.metrics
|
||||
.r
|
||||
.as_ref()
|
||||
.map(|r| r.trade_rs.as_slice())
|
||||
.map(|r| r.net_trade_rs.as_slice())
|
||||
.unwrap_or(&[]);
|
||||
(
|
||||
*ordinal,
|
||||
@@ -627,15 +627,15 @@ fn scalar_point(space: &[ParamSpec], point: &[Cell]) -> Vec<Scalar> {
|
||||
space.iter().zip(point).map(|(ps, c)| Scalar::from_cell(ps.kind, *c)).collect()
|
||||
}
|
||||
|
||||
/// The pooled walk-forward OOS trade-R series: the family reports' `trade_rs`
|
||||
/// The pooled walk-forward OOS trade-R series: the family reports' `net_trade_rs`
|
||||
/// concatenated in report order — which IS roll order (the wf stage builds its
|
||||
/// family via `walkforward_member_reports`, per-window OOS reports in roll
|
||||
/// order). Reports without an R block contribute nothing.
|
||||
fn pooled_trade_rs(reports: &[RunReport]) -> Vec<f64> {
|
||||
fn pooled_net_trade_rs(reports: &[RunReport]) -> Vec<f64> {
|
||||
let mut pooled = Vec::new();
|
||||
for report in reports {
|
||||
if let Some(r) = &report.metrics.r {
|
||||
pooled.extend_from_slice(&r.trade_rs);
|
||||
pooled.extend_from_slice(&r.net_trade_rs);
|
||||
}
|
||||
}
|
||||
pooled
|
||||
|
||||
@@ -365,7 +365,7 @@ mod tests {
|
||||
sqn: 13.0,
|
||||
net_expectancy_r: 14.0,
|
||||
conviction_terciles_r: [0.0; 3],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -41,9 +41,9 @@ fn param_i64(params: &[(String, Scalar)], name: &str) -> i64 {
|
||||
|
||||
/// The deterministic 4-trade R series a planted report carries (matches
|
||||
/// `n_trades: 4`; non-constant so block resampling has structure). In-memory
|
||||
/// only: `trade_rs` is serde-skipped and excluded from `PartialEq`, so every
|
||||
/// only: `net_trade_rs` is serde-skipped and excluded from `PartialEq`, so every
|
||||
/// existing equality and round-trip assertion stays green.
|
||||
fn planted_trade_rs(net: f64) -> Vec<f64> {
|
||||
fn planted_net_trade_rs(net: f64) -> Vec<f64> {
|
||||
vec![net, -net, 2.0 * net, net]
|
||||
}
|
||||
|
||||
@@ -84,7 +84,7 @@ fn planted_report(cell: &CellSpec, params: &[(String, Scalar)], window_ms: (i64,
|
||||
sqn_normalized: net,
|
||||
net_expectancy_r: net,
|
||||
conviction_terciles_r: [0.0, 0.0, 0.0],
|
||||
trade_rs: planted_trade_rs(net),
|
||||
net_trade_rs: planted_net_trade_rs(net),
|
||||
}),
|
||||
},
|
||||
}
|
||||
@@ -483,11 +483,11 @@ fn execute_mc_after_gate_bootstraps_each_survivor() {
|
||||
assert_eq!(mc.selection, None);
|
||||
|
||||
// each survivor's bootstrap == a hand-called r_bootstrap on its planted
|
||||
// trade_rs, seeded from the campaign doc (the deflation convention)
|
||||
// net_trade_rs, seeded from the campaign doc (the deflation convention)
|
||||
let net = |fast: i64, slow: i64| (fast * 10 + slow) as f64 / 100.0;
|
||||
let expected = StageBootstrap::PerSurvivor(vec![
|
||||
(2, r_bootstrap(&planted_trade_rs(net(3, 6)), 200, 2, 7)),
|
||||
(3, r_bootstrap(&planted_trade_rs(net(3, 9)), 200, 2, 7)),
|
||||
(2, r_bootstrap(&planted_net_trade_rs(net(3, 6)), 200, 2, 7)),
|
||||
(3, r_bootstrap(&planted_net_trade_rs(net(3, 9)), 200, 2, 7)),
|
||||
]);
|
||||
assert_eq!(mc.bootstrap, Some(expected));
|
||||
|
||||
@@ -513,15 +513,15 @@ fn execute_mc_after_wf_pools_the_oos_series() {
|
||||
assert_eq!(wf.block, "std::walk_forward");
|
||||
assert_eq!(wf.reports.len(), 2);
|
||||
|
||||
// pooled input = the wf family reports' trade_rs concatenated in report
|
||||
// pooled input = the wf family reports' net_trade_rs concatenated in report
|
||||
// order (roll order per walkforward_member_reports)
|
||||
let mut pooled: Vec<f64> = Vec::new();
|
||||
for report in &wf.reports {
|
||||
pooled.extend_from_slice(&report.metrics.r.as_ref().expect("planted R block").trade_rs);
|
||||
pooled.extend_from_slice(&report.metrics.r.as_ref().expect("planted R block").net_trade_rs);
|
||||
}
|
||||
let net = (3 * 10 + 9) as f64 / 100.0;
|
||||
let mut expected_pool = planted_trade_rs(net);
|
||||
expected_pool.extend(planted_trade_rs(net));
|
||||
let mut expected_pool = planted_net_trade_rs(net);
|
||||
expected_pool.extend(planted_net_trade_rs(net));
|
||||
assert_eq!(pooled, expected_pool);
|
||||
|
||||
let mc = &out.record.cells[0].stages[2];
|
||||
@@ -677,7 +677,7 @@ fn execute_generalize_only_after_sweep() {
|
||||
/// annotators compose in ONE pipeline without disturbing each other or the
|
||||
/// population stages' generalize-nominee. `std::monte_carlo` sits between
|
||||
/// `std::walk_forward` and `std::generalize` in the pipeline, yet its
|
||||
/// per-cell bootstrap still pools exactly the wf family's OOS `trade_rs`
|
||||
/// per-cell bootstrap still pools exactly the wf family's OOS `net_trade_rs`
|
||||
/// (unaffected by generalize running after it), and `std::generalize`'s
|
||||
/// campaign-scope nominee is still the wf stage's last-window winner
|
||||
/// (unaffected by mc having annotated the cell first). A regression that
|
||||
@@ -706,13 +706,13 @@ fn execute_mc_and_generalize_compose_in_one_pipeline() {
|
||||
assert_eq!(cell.stages[3].block, "std::monte_carlo");
|
||||
}
|
||||
|
||||
// mc's bootstrap pools the wf family's OOS trade_rs exactly as it does
|
||||
// mc's bootstrap pools the wf family's OOS net_trade_rs exactly as it does
|
||||
// with no generalize stage after it (execute_mc_after_wf_pools_the_oos_series)
|
||||
let wf = &out.cells[0].families[1];
|
||||
assert_eq!(wf.block, "std::walk_forward");
|
||||
let mut pooled: Vec<f64> = Vec::new();
|
||||
for report in &wf.reports {
|
||||
pooled.extend_from_slice(&report.metrics.r.as_ref().expect("planted R block").trade_rs);
|
||||
pooled.extend_from_slice(&report.metrics.r.as_ref().expect("planted R block").net_trade_rs);
|
||||
}
|
||||
let expected_mc = StageBootstrap::PooledOos(r_bootstrap(&pooled, 200, 2, 7));
|
||||
assert_eq!(out.record.cells[0].stages[3].bootstrap, Some(expected_mc));
|
||||
|
||||
@@ -125,7 +125,7 @@ fn rankable_metrics_are_all_per_member_resolvable() {
|
||||
sqn: 13.0,
|
||||
net_expectancy_r: 14.0,
|
||||
conviction_terciles_r: [0.0; 3],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
@@ -875,11 +875,11 @@ fn select_winner(
|
||||
/// (window order, then within-window trade order). Windows with no `r` block
|
||||
/// contribute nothing. The single home of the pooling-in-roll-order semantics —
|
||||
/// both the walk-forward `oos_r` summary and the `mc` R-bootstrap reduce this.
|
||||
fn pooled_oos_trade_rs(result: &WalkForwardResult) -> Vec<f64> {
|
||||
fn pooled_oos_net_trade_rs(result: &WalkForwardResult) -> Vec<f64> {
|
||||
result
|
||||
.windows
|
||||
.iter()
|
||||
.flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.trade_rs.clone()).unwrap_or_default())
|
||||
.flat_map(|w| w.run.oos_report.metrics.r.as_ref().map(|r| r.net_trade_rs.clone()).unwrap_or_default())
|
||||
.collect()
|
||||
}
|
||||
|
||||
@@ -888,14 +888,14 @@ fn pooled_oos_trade_rs(result: &WalkForwardResult) -> Vec<f64> {
|
||||
/// (C14).
|
||||
fn walkforward_summary_json(result: &WalkForwardResult) -> String {
|
||||
let total = result.stitched_oos_equity.last().map(|&(_, v)| v).unwrap_or(0.0);
|
||||
let pooled_rs = pooled_oos_trade_rs(result);
|
||||
let pooled_rs = pooled_oos_net_trade_rs(result);
|
||||
let mut obj = serde_json::json!({
|
||||
"windows": result.windows.len(),
|
||||
"stitched_total_pips": total,
|
||||
"param_stability": param_stability(result),
|
||||
});
|
||||
if result.windows.iter().any(|w| w.run.oos_report.metrics.r.is_some()) {
|
||||
// RMetrics serializes its scalar fields (trade_rs is serde-skipped, so the
|
||||
// RMetrics serializes its scalar fields (net_trade_rs is serde-skipped, so the
|
||||
// oos_r block is the clean R-metric summary of the pooled series).
|
||||
obj["oos_r"] = serde_json::to_value(r_metrics_from_rs(&pooled_rs))
|
||||
.expect("RMetrics serializes");
|
||||
@@ -914,7 +914,7 @@ fn walkforward_summary_json(result: &WalkForwardResult) -> String {
|
||||
/// (e.g. `sma_signal.fast.length`) by the strategy's own param space, while
|
||||
/// `stop_length`/`stop_k` ride the risk regime unwrapped, so an exact-name match
|
||||
/// would miss the wrapped ones) through the same `MetricStats::from_values`; the
|
||||
/// `oos_r` block pools the per-window `trade_rs` through `r_metrics_from_rs`.
|
||||
/// `oos_r` block pools the per-window `net_trade_rs` through `r_metrics_from_rs`.
|
||||
/// Canonical JSON (C14).
|
||||
///
|
||||
/// `axes` carries the invocation's raw axis names in argv order, followed by the
|
||||
@@ -954,7 +954,7 @@ fn walkforward_summary_json_from_reports(reports: &[RunReport], axes: &[String])
|
||||
.collect();
|
||||
let pooled_rs: Vec<f64> = reports
|
||||
.iter()
|
||||
.flat_map(|r| r.metrics.r.as_ref().map(|m| m.trade_rs.clone()).unwrap_or_default())
|
||||
.flat_map(|r| r.metrics.r.as_ref().map(|m| m.net_trade_rs.clone()).unwrap_or_default())
|
||||
.collect();
|
||||
let mut obj = serde_json::json!({
|
||||
"windows": reports.len(),
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
mod common;
|
||||
use common::{ScratchGuard, ScratchPath, fresh_project};
|
||||
use common::{ScratchGuard, ScratchPath, fresh_project, fresh_project_with_data};
|
||||
|
||||
/// A fresh, unique working directory for a process test that persists
|
||||
/// content-addressed documents under `./runs/` (so `aura process register`
|
||||
@@ -1010,7 +1010,10 @@ fn register_process_doc(dir: &Path, file: &str, doc: &str) -> String {
|
||||
/// the RAW `param_space` names (`fast.length` / `slow.length` — see the
|
||||
/// naming note in the referential test above). `persist_taps`/`emit` are
|
||||
/// spliced verbatim (pass `""` for empty, `"\"family_table\""` etc.).
|
||||
fn campaign_doc_json(
|
||||
/// [`campaign_doc_json`] with the instrument spliced — the synthetic-archive
|
||||
/// e2e targets `SYMA` (tests/common/mod.rs) instead of the hardcoded GER40.
|
||||
fn campaign_doc_json_for(
|
||||
instrument: &str,
|
||||
bp_id: &str,
|
||||
proc_id: &str,
|
||||
window: (i64, i64),
|
||||
@@ -1022,7 +1025,7 @@ fn campaign_doc_json(
|
||||
"format_version": 1,
|
||||
"kind": "campaign",
|
||||
"name": "run-seam",
|
||||
"data": {{ "instruments": ["GER40"], "windows": [ {{ "from_ms": {from}, "to_ms": {to} }} ] }},
|
||||
"data": {{ "instruments": ["{instrument}"], "windows": [ {{ "from_ms": {from}, "to_ms": {to} }} ] }},
|
||||
"strategies": [ {{ "ref": {{ "content_id": "{bp_id}" }},
|
||||
"axes": {{ "fast.length": {{ "kind": "I64", "values": [2, 4] }},
|
||||
"slow.length": {{ "kind": "I64", "values": [8, 16] }} }} }} ],
|
||||
@@ -1035,6 +1038,16 @@ fn campaign_doc_json(
|
||||
)
|
||||
}
|
||||
|
||||
fn campaign_doc_json(
|
||||
bp_id: &str,
|
||||
proc_id: &str,
|
||||
window: (i64, i64),
|
||||
persist_taps: &str,
|
||||
emit: &str,
|
||||
) -> String {
|
||||
campaign_doc_json_for("GER40", bp_id, proc_id, window, persist_taps, emit)
|
||||
}
|
||||
|
||||
/// An mc-bearing process: intrinsically valid AND (since the v2 executor) an
|
||||
/// executable pipeline shape — `sweep -> monte_carlo` annotates each sweep
|
||||
/// survivor. The two addressing-mode tests below run it over the [1, 2] 1970
|
||||
@@ -1990,6 +2003,163 @@ fn campaign_run_real_e2e_sweep_gate_walkforward() {
|
||||
);
|
||||
}
|
||||
|
||||
/// Property (#259): a campaign's cost block reaches the walk-forward →
|
||||
/// monte-carlo evidence chain — the pooled-OOS bootstrap resamples the
|
||||
/// COST-NETTED per-trade series, so the same matrix run with and without a
|
||||
/// cost block records DIFFERENT `pooled_oos` stats (today's bug: identical to
|
||||
/// every digit). Hostless: runs over the synthetic SYMA archive, no
|
||||
/// data-mount skip-guard.
|
||||
#[test]
|
||||
fn campaign_run_synthetic_e2e_cost_block_nets_the_pooled_oos_bootstrap() {
|
||||
let (dir, _fixture) = fresh_project_with_data();
|
||||
let runs_dir = dir.join("runs");
|
||||
std::fs::remove_dir_all(&runs_dir).ok();
|
||||
let _cleanup = ScratchGuard(vec![
|
||||
ScratchPath::Dir(runs_dir.clone()),
|
||||
ScratchPath::File(dir.join("netmc.process.json")),
|
||||
ScratchPath::File(dir.join("gross.campaign.json")),
|
||||
ScratchPath::File(dir.join("net.campaign.json")),
|
||||
]);
|
||||
let bp_id = seed_blueprint(&dir, "campaign-run-netmc-seed");
|
||||
let proc_id = register_process_doc(&dir, "netmc.process.json", WF_PROCESS_DOC);
|
||||
// SYMA's synthetic archive spans 2024-01..08; Mar-01..Jun-30 (~17 weeks)
|
||||
// gives the (14d, 7d, 7d) roller a comfortable tiling.
|
||||
let base = campaign_doc_json_for(
|
||||
"SYMA",
|
||||
&bp_id,
|
||||
&proc_id,
|
||||
(1709251200000, 1719791999999),
|
||||
"",
|
||||
"",
|
||||
);
|
||||
// The costed twin: the SAME document plus a constant cost block — the
|
||||
// only difference (the replacen pattern of the cost e2e siblings).
|
||||
let with_cost = base.replacen(
|
||||
"\"seed\": 7,",
|
||||
"\"seed\": 7,\n \"cost\": [ { \"constant\": { \"cost_per_trade\": 0.5 } } ],",
|
||||
1,
|
||||
);
|
||||
assert_ne!(with_cost, base, "replacen must actually match the seed field");
|
||||
write_doc(&dir, "gross.campaign.json", &base);
|
||||
write_doc(&dir, "net.campaign.json", &with_cost);
|
||||
|
||||
// Both runs share the store (the cost-e2e sibling pattern) — the record
|
||||
// line is read from each run's own stdout, so no isolation is needed.
|
||||
let pooled = |doc: &str| -> serde_json::Value {
|
||||
let (out, code) = run_code_in(&dir, &["campaign", "run", doc]);
|
||||
assert_eq!(code, Some(0), "campaign run failed: {out}");
|
||||
let line = out
|
||||
.lines()
|
||||
.find(|l| l.starts_with("{\"campaign_run\":"))
|
||||
.expect("the always-on final campaign_run line");
|
||||
let v: serde_json::Value = serde_json::from_str(line).expect("record parses");
|
||||
v["campaign_run"]["cells"][0]["stages"][3]["bootstrap"]["pooled_oos"].clone()
|
||||
};
|
||||
let gross = pooled("gross.campaign.json");
|
||||
let net = pooled("net.campaign.json");
|
||||
|
||||
let n_gross = gross["n_trades"].as_u64().expect("gross n_trades");
|
||||
let n_net = net["n_trades"].as_u64().expect("net n_trades");
|
||||
assert!(n_gross > 0, "the synthetic window must produce OOS trades: {gross}");
|
||||
assert_eq!(
|
||||
n_gross, n_net,
|
||||
"cost is a feed-forward subtraction — it must not change the trade population"
|
||||
);
|
||||
let mean_gross = gross["e_r"]["mean"].as_f64().expect("gross mean");
|
||||
let mean_net = net["e_r"]["mean"].as_f64().expect("net mean");
|
||||
assert!(
|
||||
mean_net < mean_gross,
|
||||
"the costed bootstrap must resample the NET series (#259): gross {mean_gross} vs net {mean_net}"
|
||||
);
|
||||
let p_gross = gross["prob_le_zero"].as_f64().expect("gross prob_le_zero");
|
||||
let p_net = net["prob_le_zero"].as_f64().expect("net prob_le_zero");
|
||||
assert!(
|
||||
p_net >= p_gross,
|
||||
"a strictly-lower resample mean cannot lower prob_le_zero: gross {p_gross} vs net {p_net}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Property (#259), the OTHER mc bootstrap input shape: with no
|
||||
/// `walk_forward` in the pipeline (`sweep -> monte_carlo`), each surviving
|
||||
/// sweep member gets its OWN bootstrap (`PerSurvivor`, keyed by ordinal) —
|
||||
/// a distinct code path from `PooledOos` above (`member_net_trade_rs` /
|
||||
/// `SweepPoint` in aura-registry, not the wf-family pooling in
|
||||
/// aura-campaign). That path must resample the same cost-netted
|
||||
/// `net_trade_rs` conduit: a cost block must shift EVERY surviving member's
|
||||
/// resampled mean down, never leave any member's bootstrap gross-identical.
|
||||
/// Hostless: runs over the synthetic SYMA archive, no data-mount skip-guard.
|
||||
/// The gated real-data sibling (`campaign_run_real_e2e_sweep_monte_carlo_per_survivor`)
|
||||
/// exercises this shape but never with a cost block, so it cannot pin this property.
|
||||
#[test]
|
||||
fn campaign_run_synthetic_e2e_cost_block_nets_the_per_survivor_bootstrap() {
|
||||
let (dir, _fixture) = fresh_project_with_data();
|
||||
let runs_dir = dir.join("runs");
|
||||
std::fs::remove_dir_all(&runs_dir).ok();
|
||||
let _cleanup = ScratchGuard(vec![
|
||||
ScratchPath::Dir(runs_dir.clone()),
|
||||
ScratchPath::File(dir.join("persurvivor.process.json")),
|
||||
ScratchPath::File(dir.join("gross_ps.campaign.json")),
|
||||
ScratchPath::File(dir.join("net_ps.campaign.json")),
|
||||
]);
|
||||
let bp_id = seed_blueprint(&dir, "campaign-run-persurvivor-seed");
|
||||
let proc_id = register_process_doc(&dir, "persurvivor.process.json", MC_PROCESS_DOC);
|
||||
// Same SYMA window as the pooled-OOS sibling above — known to produce
|
||||
// trades over the synthetic archive.
|
||||
let base = campaign_doc_json_for(
|
||||
"SYMA",
|
||||
&bp_id,
|
||||
&proc_id,
|
||||
(1709251200000, 1719791999999),
|
||||
"",
|
||||
"",
|
||||
);
|
||||
let with_cost = base.replacen(
|
||||
"\"seed\": 7,",
|
||||
"\"seed\": 7,\n \"cost\": [ { \"constant\": { \"cost_per_trade\": 0.5 } } ],",
|
||||
1,
|
||||
);
|
||||
assert_ne!(with_cost, base, "replacen must actually match the seed field");
|
||||
write_doc(&dir, "gross_ps.campaign.json", &base);
|
||||
write_doc(&dir, "net_ps.campaign.json", &with_cost);
|
||||
|
||||
let per_survivor = |doc: &str| -> Vec<(u64, serde_json::Value)> {
|
||||
let (out, code) = run_code_in(&dir, &["campaign", "run", doc]);
|
||||
assert_eq!(code, Some(0), "campaign run failed: {out}");
|
||||
let line = out
|
||||
.lines()
|
||||
.find(|l| l.starts_with("{\"campaign_run\":"))
|
||||
.expect("the always-on final campaign_run line");
|
||||
let v: serde_json::Value = serde_json::from_str(line).expect("record parses");
|
||||
v["campaign_run"]["cells"][0]["stages"][1]["bootstrap"]["per_survivor"]
|
||||
.as_array()
|
||||
.expect("no walk_forward precedes: the PerSurvivor arm, not pooled_oos")
|
||||
.iter()
|
||||
.map(|pair| (pair[0].as_u64().expect("ordinal"), pair[1].clone()))
|
||||
.collect()
|
||||
};
|
||||
let gross = per_survivor("gross_ps.campaign.json");
|
||||
let net = per_survivor("net_ps.campaign.json");
|
||||
|
||||
assert_eq!(gross.len(), 4, "2x2 axes -> four surviving members: {gross:?}");
|
||||
assert_eq!(gross.len(), net.len(), "cost must not change the surviving-member population");
|
||||
for ((g_ord, g), (n_ord, n)) in gross.iter().zip(net.iter()) {
|
||||
assert_eq!(g_ord, n_ord, "the two runs must enumerate members in the same ordinal order");
|
||||
let n_trades_g = g["n_trades"].as_u64().expect("gross n_trades");
|
||||
let n_trades_n = n["n_trades"].as_u64().expect("net n_trades");
|
||||
assert!(n_trades_g > 0, "member {g_ord} must have produced trades: {g}");
|
||||
assert_eq!(
|
||||
n_trades_g, n_trades_n,
|
||||
"member {g_ord}: cost is a feed-forward subtraction — it must not change the trade population"
|
||||
);
|
||||
let mean_g = g["e_r"]["mean"].as_f64().expect("gross mean");
|
||||
let mean_n = n["e_r"]["mean"].as_f64().expect("net mean");
|
||||
assert!(
|
||||
mean_n < mean_g,
|
||||
"member {g_ord}: the costed PerSurvivor bootstrap must resample the NET series (#259): gross {mean_g} vs net {mean_n}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The campaign `data.bindings` override end to end (#231, 6b): the same
|
||||
/// seeded strategy, window, and seed run twice — bare (price<-close default)
|
||||
/// and with `price` rebound to the open column. Both exit 0; the realized
|
||||
|
||||
@@ -505,8 +505,8 @@ fn closed_neighbourhood(i: usize, axis_lens: &[usize]) -> Vec<usize> {
|
||||
out
|
||||
}
|
||||
|
||||
fn member_trade_rs(rep: &RunReport) -> &[f64] {
|
||||
rep.metrics.r.as_ref().map(|r| r.trade_rs.as_slice()).unwrap_or(&[])
|
||||
fn member_net_trade_rs(rep: &RunReport) -> &[f64] {
|
||||
rep.metrics.r.as_ref().map(|r| r.net_trade_rs.as_slice()).unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// Recompute the selected R metric from a trade-R slice (reuses the engine's
|
||||
@@ -522,13 +522,13 @@ fn member_metric_from_rs(rs: &[f64], m: Metric) -> f64 {
|
||||
}
|
||||
}
|
||||
|
||||
/// The centred best-of-K null-max distribution: each member's `trade_rs` is
|
||||
/// The centred best-of-K null-max distribution: each member's `net_trade_rs` is
|
||||
/// mean-subtracted (the no-edge null), then for each resample every member is
|
||||
/// moving-block-resampled (in odometer order, from a per-iteration seed) and the
|
||||
/// max recomputed metric across members is taken. Deterministic given `seed` (C1).
|
||||
fn null_best_of_k(family: &SweepFamily, m: Metric, n_resamples: usize, block_len: usize, seed: u64) -> Vec<f64> {
|
||||
let centred: Vec<Vec<f64>> = family.points.iter().map(|p| {
|
||||
let rs = member_trade_rs(&p.report);
|
||||
let rs = member_net_trade_rs(&p.report);
|
||||
if rs.is_empty() { return Vec::new(); }
|
||||
let mean = rs.iter().sum::<f64>() / rs.len() as f64;
|
||||
rs.iter().map(|x| x - mean).collect()
|
||||
@@ -657,8 +657,8 @@ pub fn optimize_deflated(
|
||||
let null_max = null_best_of_k(family, m, n_resamples, block_len, seed);
|
||||
// Keep only the finite best-of-K draws. The null is *not computable* in two
|
||||
// degenerate cases: zero resamples (`null_max` empty), or no member carries
|
||||
// `trade_rs` (every member's centred series is empty, so each iteration's
|
||||
// best-of-K folds to `NEG_INFINITY`). The latter is reachable — `trade_rs`
|
||||
// `net_trade_rs` (every member's centred series is empty, so each iteration's
|
||||
// best-of-K folds to `NEG_INFINITY`). The latter is reachable — `net_trade_rs`
|
||||
// is `#[serde(skip)]`, so a family loaded back from the registry has empty
|
||||
// conduits even when its `r` block (hence `raw`) is finite. Filtering to
|
||||
// finite values collapses both into one "no usable null" branch, instead of
|
||||
@@ -847,7 +847,7 @@ mod tests {
|
||||
sqn_normalized: sqn, // mirror sqn: only the rank-key wiring is under test here
|
||||
net_expectancy_r,
|
||||
conviction_terciles_r: [0.0, 0.0, 0.0],
|
||||
trade_rs: Vec::new(),
|
||||
net_trade_rs: Vec::new(),
|
||||
});
|
||||
rep
|
||||
}
|
||||
@@ -1174,16 +1174,16 @@ mod tests {
|
||||
assert_eq!(ranked[0].metrics.total_pips, 3.0); // best-first within the family
|
||||
}
|
||||
|
||||
fn member(total_pips: f64, trade_rs: Vec<f64>) -> SweepPoint {
|
||||
fn member(total_pips: f64, net_trade_rs: Vec<f64>) -> SweepPoint {
|
||||
// Every RMetrics field is derived from the R slice by `r_metrics_from_rs`
|
||||
// (the same arithmetic production members carry), then the per-trade
|
||||
// `trade_rs` conduit is restored: `r_metrics_from_rs` empties it
|
||||
// (`trade_rs: Vec::new()`), whereas a production member is built by
|
||||
// `net_trade_rs` conduit is restored: `r_metrics_from_rs` empties it
|
||||
// (`net_trade_rs: Vec::new()`), whereas a production member is built by
|
||||
// `summarize_r`, which RETAINS it — and `optimize_deflated`'s R arm
|
||||
// resamples exactly that conduit, so a fixture without it cannot reach
|
||||
// the R-arm path under test.
|
||||
let mut r = aura_engine::r_metrics_from_rs(&trade_rs);
|
||||
r.trade_rs = trade_rs;
|
||||
let mut r = aura_engine::r_metrics_from_rs(&net_trade_rs);
|
||||
r.net_trade_rs = net_trade_rs;
|
||||
SweepPoint {
|
||||
params: vec![],
|
||||
report: RunReport {
|
||||
@@ -1219,7 +1219,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn optimize_deflated_winner_is_byte_identical_to_optimize() {
|
||||
let fam = fixture_family_with_r(); // helper: ≥3 members, varied sqn_normalized + trade_rs
|
||||
let fam = fixture_family_with_r(); // helper: ≥3 members, varied sqn_normalized + net_trade_rs
|
||||
for metric in ["total_pips", "sqn_normalized", "expectancy_r"] {
|
||||
let plain = optimize(&fam, metric).unwrap();
|
||||
let (defl, _) = optimize_deflated(&fam, metric, 200, 3, 7).unwrap();
|
||||
@@ -1259,18 +1259,18 @@ mod tests {
|
||||
assert_eq!(sel.overfit_probability, Some(1.0), "(0+1)/(0+1) Laplace floor");
|
||||
}
|
||||
|
||||
/// Regression: a family whose members carry an `r` block but NO `trade_rs` —
|
||||
/// Regression: a family whose members carry an `r` block but NO `net_trade_rs` —
|
||||
/// the serde-skipped state of a family `load`ed back from the registry — must
|
||||
/// NOT yield a `+inf` deflated score on the R arm with positive resamples. With
|
||||
/// no member contributing a centred series, every best-of-K draw is
|
||||
/// `NEG_INFINITY`; the finite-filter collapses this to the same degenerate floor
|
||||
/// as zero-resamples (`deflated_score == raw`, `overfit_probability == 1.0`).
|
||||
#[test]
|
||||
fn optimize_deflated_no_trade_rs_floors_instead_of_infinity() {
|
||||
fn optimize_deflated_no_net_trade_rs_floors_instead_of_infinity() {
|
||||
let mut fam = fixture_family_with_r();
|
||||
for p in &mut fam.points {
|
||||
if let Some(r) = p.report.metrics.r.as_mut() {
|
||||
r.trade_rs.clear(); // mimic a serde-loaded member (trade_rs is #[serde(skip)])
|
||||
r.net_trade_rs.clear(); // mimic a serde-loaded member (net_trade_rs is #[serde(skip)])
|
||||
}
|
||||
}
|
||||
let sel = optimize_deflated(&fam, "expectancy_r", 500, 3, 1).unwrap().1;
|
||||
|
||||
@@ -1564,11 +1564,13 @@ boundary is test-pinned on both sides), plus three new static guards
|
||||
(unanimous #200 triage): nothing flows out of them; filtering stays the gate's monopoly.
|
||||
`std::monte_carlo` bootstraps the stage's *incoming R-evidence* with one semantics,
|
||||
input-shaped by position — after a walk_forward, ONE `r_bootstrap` over the wf family's
|
||||
pooled per-window OOS `trade_rs` in roll order (`PooledOos`); after sweep/gates, one
|
||||
`r_bootstrap` per surviving member's fresh in-memory series (`PerSurvivor`, ordinals into
|
||||
the population family; a zero-trade member records the engine's defined all-zero
|
||||
degenerate) — seeded from the campaign doc's `seed` (C1; `trade_rs` is `#[serde(skip)]`,
|
||||
so annotators run in-executor or not at all). `std::generalize` executes at **campaign
|
||||
pooled per-window OOS `net_trade_rs` in roll order (`PooledOos`; #259 materialized this
|
||||
conduit as the cost-netted per-trade series `r − cost_in_r`, equal to the gross series
|
||||
bit-for-bit when no cost model is bound); after sweep/gates, one `r_bootstrap` per
|
||||
surviving member's fresh in-memory series (`PerSurvivor`, ordinals into the population
|
||||
family; a zero-trade member records the engine's defined all-zero degenerate) — seeded
|
||||
from the campaign doc's `seed` (C1; `net_trade_rs` is `#[serde(skip)]`, so annotators run
|
||||
in-executor or not at all). `std::generalize` executes at **campaign
|
||||
scope**: after all cells, per (strategy, window) the per-cell *nominees* (last wf
|
||||
window's OOS report, else the sweep winner; none on gate truncation) across instruments
|
||||
feed the shipped `generalization()` when ≥ 2 exist — divergent per-instrument winners
|
||||
|
||||
Reference in New Issue
Block a user