Brummel/Aura
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(cli): walkforward campaign translator + summary reducer (#210 T1-T2)

Browse Source 
The additive machinery for the walkforward dissolution, unwired this slice
(dispatch_walkforward untouched — `aura walkforward` behaviour byte-for-byte
unchanged). Two pieces, both mirroring the generalize seam:

- `translate_walkforward` + `register_generated_wf` + `GeneratedWalkforward`
  (verb_sugar.rs): argv -> a `[std::sweep(argmax), std::walk_forward]` process +
  a campaign carrying the multi-value fast/slow IS-refit grid over one instrument
  under a single Vol risk regime (the stop). The preceding sweep enumerates the
  survivor grid; the wf stage refits it per window.
- `walkforward_summary_json_from_reports` (main.rs): the pure-sugar reducer that
  reconstructs the verb's summary line from the campaign's recorded per-window OOS
  reports — `stitched_total_pips` = the per-window total_pips summed in roll order,
  `param_stability` over the four r-sma axes via the same `MetricStats::from_values`,
  pooled `oos_r` via `r_metrics_from_rs`. Byte-identical to the inline
  `walkforward_summary_json` (the committed anchor gates it once Task 3 wires it).

Both carry `#[allow(dead_code)]` until Task 3's dispatch split calls them.

The third `Generated*`/`register_generated_*` trio duplicates the sweep/generalize
shape by design (per-verb mirror); the bundle consolidation is deferred to #214.
Unit test note: `r_metrics_from_rs` clears `trade_rs` (it is the terminal pooled
reducer), so the reducer's characterization fixture rebuilds each window's
`RMetrics` with `trade_rs` restored, modelling what a real OOS window's report
carries (from `summarize_r`).

refs #210
This commit is contained in:
Brummel
2026-07-06 22:12:47 +02:00
parent f7835353de
commit a72cd6b115
2 changed files with 336 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/aura-cli/src/main.rs
+117
View File
@@ -1876,6 +1876,71 @@ fn walkforward_summary_json(result: &WalkForwardResult) -> String {
serde_json::json!({ "walkforward": obj }).to_string()
}
/// The walk-forward summary line reconstructed from the recorded per-window OOS
/// reports (the campaign path's `WalkForward` `StageFamily.reports`) rather than a
/// live `WalkForwardResult`. Byte-identical to `walkforward_summary_json`:
/// `stitched_total_pips` = the per-window `total_pips` summed left-to-right in roll
/// order (the engine `stitch` folds each segment's final cumulative value, and a
/// window's OOS segment ends at its `total_pips`); `param_stability` reduces each
/// r-sma IS-refit axis over the per-window chosen params (read by exact name from
/// each report's prefix-stripped `manifest.params`) through the same
/// `MetricStats::from_values`; the `oos_r` block pools the per-window `trade_rs`
/// through `r_metrics_from_rs`. Canonical JSON (C14).
///
/// The axis list + order mirror what `r_sma_space` yields (fast, slow, stop_length,
/// stop_k) — the same order + values the inline `param_stability` reduces. A wrong
/// order or axis would fail the committed exact-grade anchor loudly (it pins
/// `param_stability[0].mean` = the fast-MA refit mean), never silently.
#[allow(dead_code)]
fn walkforward_summary_json_from_reports(reports: &[RunReport]) -> String {
let total: f64 = reports.iter().map(|r| r.metrics.total_pips).sum();
// The four r-sma axis names, in `r_sma_space` order. Coercion to `f64` is
// decided per-value at runtime by the `Scalar` variant match below (i64 axes
// cast value-as-f64, the f64 axis passes through), exactly as
// `param_stability`'s per-`ScalarKind` coerce does — this list carries no
// per-axis type flag.
const AXES: [&str; 4] = ["fast.length", "slow.length", "stop_length", "stop_k"];
let param_stability: Vec<aura_engine::MetricStats> = AXES
.iter()
.map(|axis| {
let vals: Vec<f64> = reports
.iter()
.map(|r| {
let (_, v) = r
.manifest
.params
.iter()
.find(|(name, _)| name == axis)
.expect("each r-sma walk-forward window records its chosen axis");
match v {
Scalar::I64(i) => *i as f64,
Scalar::F64(f) => *f,
Scalar::Bool(b) => *b as i64 as f64,
Scalar::Timestamp(_) => {
unreachable!("a timestamp is a structural axis, never a param knob")
}
}
})
.collect();
aura_engine::MetricStats::from_values(&vals)
})
.collect();
let pooled_rs: Vec<f64> = reports
.iter()
.flat_map(|r| r.metrics.r.as_ref().map(|m| m.trade_rs.clone()).unwrap_or_default())
.collect();
let mut obj = serde_json::json!({
"windows": reports.len(),
"stitched_total_pips": total,
"param_stability": param_stability,
});
if reports.iter().any(|r| r.metrics.r.is_some()) {
obj["oos_r"] = serde_json::to_value(r_metrics_from_rs(&pooled_rs))
.expect("RMetrics serializes");
}
serde_json::json!({ "walkforward": obj }).to_string()
}
/// The cross-instrument generalization line: the chosen metric, instrument count,
/// worst-case floor, sign-agreement count, and the per-instrument breakdown. Canonical
/// JSON (C14), mirroring `walkforward_summary_json`'s `{"generalize": obj}` shape.
@@ -5224,6 +5289,58 @@ mod tests {
}
}
/// `walkforward_summary_json_from_reports` reduces a hand-built two-window
/// report set to the same `{"windows", "stitched_total_pips",
/// "param_stability", "oos_r"}` shape as the live-`WalkForwardResult` path:
/// pips sum in roll order, each r-sma axis reduces across windows, and the
/// pooled `trade_rs` yields the R-metric block.
#[test]
fn walkforward_summary_from_reports_reconstructs_the_summary() {
// Two windows, chosen params (3,12,14,2.0) then (5,20,14,2.0).
let mk = |fast: i64, slow: i64, pips: f64, rs: Vec<f64>| -> RunReport {
let mut rep = RunReport {
manifest: RunManifest {
commit: "t".into(),
params: vec![
("fast.length".into(), Scalar::I64(fast)),
("slow.length".into(), Scalar::I64(slow)),
("stop_length".into(), Scalar::I64(14)),
("stop_k".into(), Scalar::F64(2.0)),
("bias_scale".into(), Scalar::F64(0.5)),
],
window: (Timestamp(0), Timestamp(0)),
seed: 0,
broker: "t".into(),
selection: None,
instrument: None,
topology_hash: None,
project: None,
},
metrics: summarize(&[], &[]),
};
rep.metrics.total_pips = pips;
// `r_metrics_from_rs` deliberately clears `trade_rs` (it is the terminal
// pooled reducer, not a per-window record); a real OOS window's report
// gets `trade_rs` from `summarize_r` over the raw trade record instead,
// so restore it here to model that shape.
rep.metrics.r =
Some(aura_engine::RMetrics { trade_rs: rs.clone(), ..r_metrics_from_rs(&rs) });
rep
};
let reports = vec![mk(3, 12, 10.0, vec![1.0, -0.5]), mk(5, 20, -4.0, vec![0.25])];
let line = walkforward_summary_json_from_reports(&reports);
let v: serde_json::Value = serde_json::from_str(&line).unwrap();
let wf = &v["walkforward"];
assert_eq!(wf["windows"].as_u64(), Some(2));
assert_eq!(wf["stitched_total_pips"].as_f64(), Some(6.0)); // 10.0 + -4.0
let ps = wf["param_stability"].as_array().unwrap();
assert_eq!(ps.len(), 4, "four r-sma slots");
assert_eq!(ps[0]["mean"].as_f64(), Some(4.0)); // fast: (3+5)/2
assert_eq!(ps[1]["mean"].as_f64(), Some(16.0)); // slow: (12+20)/2
assert!(wf["oos_r"].is_object(), "pooled oos_r present when any window has r");
assert_eq!(wf["oos_r"]["n_trades"].as_u64(), Some(3)); // 2 + 1
}
/// The drained sink trace of a seeded run — the recorded rows of the equity
/// and exposure sinks. Compared row-for-row so the C1 seed-determinism
/// property is tested at the trace level (strictly stronger than the folded
crates/aura-cli/src/verb_sugar.rs
+219 -1
View File
@@ -13,7 +13,7 @@ use aura_core::{Scalar, ScalarKind};
use aura_research::{
campaign_to_json, content_id_of, process_to_json, Axis, CampaignDoc, DataSection, DocKind,
DocRef, Presentation, ProcessDoc, ProcessRef, RiskRegime, SelectRule, StageBlock,
StrategyEntry, SweepSelection, Window, FORMAT_VERSION,
StrategyEntry, SweepSelection, WfMode, Window, FORMAT_VERSION,
};
/// The two generated documents of one dissolved-verb invocation.
@@ -351,6 +351,109 @@ pub(crate) fn run_generalize_sugar(
Ok(())
}
/// The two generated documents of one dissolved `walkforward` invocation.
#[allow(dead_code)]
#[derive(Debug)]
pub(crate) struct GeneratedWalkforward {
pub process: ProcessDoc,
pub campaign: CampaignDoc,
}
/// Translate one `aura walkforward --strategy r-sma --real` invocation into its two
/// generated documents: a `[std::sweep(argmax), std::walk_forward]` process (the
/// sweep only enumerates the IS-refit survivor grid for the wf stage; its recorded
/// argmax selection is not part of the summary) and a campaign running the fast/slow
/// grid over one instrument under a single risk regime that carries the stop. The
/// roller sizes (`in_sample_ms`/`out_of_sample_ms`/`step_ms`) come from the caller
/// (ms). `blueprint_canonical` is the bare `sma_signal` blueprint already stored by
/// topology hash; its content id is the strategy ref.
#[allow(clippy::too_many_arguments)]
#[allow(dead_code)]
pub(crate) fn translate_walkforward(
fast: &[i64],
slow: &[i64],
stop_length: i64,
stop_k: f64,
metric: &str,
in_sample_ms: u64,
out_of_sample_ms: u64,
step_ms: u64,
name: &str,
symbol: &str,
from_ms: i64,
to_ms: i64,
blueprint_canonical: &str,
) -> Result<GeneratedWalkforward, String> {
let process = ProcessDoc {
format_version: FORMAT_VERSION,
kind: DocKind::Process,
name: "walkforward".to_string(),
description: None,
pipeline: vec![
StageBlock::Sweep {
selection: Some(SweepSelection {
metric: metric.to_string(),
select: SelectRule::Argmax,
deflate: false,
}),
},
StageBlock::WalkForward {
in_sample_ms,
out_of_sample_ms,
step_ms,
mode: WfMode::Rolling,
metric: metric.to_string(),
select: SelectRule::Argmax,
},
],
};
let mut doc_axes: BTreeMap<String, Axis> = BTreeMap::new();
doc_axes.insert(
"fast.length".to_string(),
axis_from_values("fast.length", &fast.iter().map(|&v| Scalar::i64(v)).collect::<Vec<_>>())?,
);
doc_axes.insert(
"slow.length".to_string(),
axis_from_values("slow.length", &slow.iter().map(|&v| Scalar::i64(v)).collect::<Vec<_>>())?,
);
let campaign = CampaignDoc {
format_version: FORMAT_VERSION,
kind: DocKind::Campaign,
name: name.to_string(),
description: None,
data: DataSection {
instruments: vec![symbol.to_string()],
windows: vec![Window { from_ms, to_ms }],
},
strategies: vec![StrategyEntry {
r#ref: DocRef::ContentId(content_id_of(blueprint_canonical)),
axes: doc_axes,
}],
process: ProcessRef { r#ref: DocRef::ContentId(content_id_of(&process_to_json(&process))) },
risk: vec![RiskRegime::Vol { length: stop_length, k: stop_k }],
seed: 0,
presentation: Presentation {
persist_taps: vec![],
emit: vec!["family_table".to_string()],
},
};
Ok(GeneratedWalkforward { process, campaign })
}
/// Auto-register both generated walkforward documents (content-addressed,
/// idempotent). Returns `(process_id, campaign_id)`.
#[allow(dead_code)]
pub(crate) fn register_generated_wf(
reg: &aura_registry::Registry,
generated: &GeneratedWalkforward,
) -> Result<(String, String), String> {
let process_id =
reg.put_process(&process_to_json(&generated.process)).map_err(|e| e.to_string())?;
let campaign_id =
reg.put_campaign(&campaign_to_json(&generated.campaign)).map_err(|e| e.to_string())?;
Ok((process_id, campaign_id))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -526,4 +629,119 @@ mod tests {
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn translate_walkforward_is_deterministic_and_carries_the_regime() {
let a = translate_walkforward(
&[3, 5], &[12, 20], 14, 2.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
let b = translate_walkforward(
&[3, 5], &[12, 20], 14, 2.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
assert_eq!(
content_id_of(&campaign_to_json(&a.campaign)),
content_id_of(&campaign_to_json(&b.campaign)),
"identical invocations must generate identical campaign ids"
);
// pipeline: sweep(argmax) then walk_forward(argmax, rolling)
assert_eq!(
a.process.pipeline,
vec![
StageBlock::Sweep {
selection: Some(SweepSelection {
metric: "sqn_normalized".to_string(),
select: SelectRule::Argmax,
deflate: false,
})
},
StageBlock::WalkForward {
in_sample_ms: 7_776_000_000,
out_of_sample_ms: 2_592_000_000,
step_ms: 2_592_000_000,
mode: WfMode::Rolling,
metric: "sqn_normalized".to_string(),
select: SelectRule::Argmax,
},
]
);
assert_eq!(a.campaign.risk, vec![RiskRegime::Vol { length: 14, k: 2.0 }]);
assert_eq!(a.campaign.data.instruments, vec!["GER40".to_string()]);
assert_eq!(a.campaign.strategies.len(), 1);
// multi-value IS-refit grid axes
assert_eq!(
a.campaign.strategies[0].axes["fast.length"].values,
vec![Scalar::i64(3), Scalar::i64(5)]
);
assert_eq!(
a.campaign.strategies[0].axes["slow.length"].values,
vec![Scalar::i64(12), Scalar::i64(20)]
);
assert_eq!(a.process.name, "walkforward");
assert!(aura_research::validate_campaign(&a.campaign).is_empty());
assert!(aura_research::validate_process(&a.process).is_empty());
assert!(aura_campaign::preflight(&a.process, &a.campaign).is_ok());
}
#[test]
fn translate_walkforward_diverging_regimes_do_not_collide_content_ids() {
let base = translate_walkforward(
&[3, 5], &[12, 20], 14, 2.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
let different_k = translate_walkforward(
&[3, 5], &[12, 20], 14, 3.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
let different_length = translate_walkforward(
&[3, 5], &[12, 20], 20, 2.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
let base_id = content_id_of(&campaign_to_json(&base.campaign));
let k_id = content_id_of(&campaign_to_json(&different_k.campaign));
let length_id = content_id_of(&campaign_to_json(&different_length.campaign));
assert_ne!(base_id, k_id, "a different stop_k must not collide onto the same campaign id");
assert_ne!(
base_id, length_id,
"a different stop_length must not collide onto the same campaign id"
);
assert_ne!(k_id, length_id, "the two diverging regimes must not collide with each other");
}
#[test]
fn register_generated_wf_stores_documents_the_campaign_can_actually_resolve() {
let dir = std::env::temp_dir().join(format!(
"aura-verb-sugar-walkforward-registry-test-{}",
std::process::id()
));
std::fs::create_dir_all(&dir).unwrap();
let reg = aura_registry::Registry::open(dir.join("runs.jsonl"));
let generated = translate_walkforward(
&[3, 5], &[12, 20], 14, 2.0, "sqn_normalized",
7_776_000_000, 2_592_000_000, 2_592_000_000,
"walkforward", "GER40", 100, 200, "{\"bp\":1}",
)
.unwrap();
let (process_id, campaign_id) = register_generated_wf(&reg, &generated).unwrap();
let stored_campaign = reg.get_campaign(&campaign_id).unwrap().expect("campaign stored");
assert_eq!(stored_campaign, campaign_to_json(&generated.campaign));
let DocRef::ContentId(ref_id) = &generated.campaign.process.r#ref else {
panic!("walkforward's process ref is a content id");
};
assert_eq!(ref_id, &process_id, "the stored process id must be the campaign's own ref");
let stored_process = reg.get_process(ref_id).unwrap().expect("process stored");
assert_eq!(stored_process, process_to_json(&generated.process));
let _ = std::fs::remove_dir_all(&dir);
}
}