feat(campaign,cli): permit --select plateau in the walk_forward stage (#215)
The campaign walk_forward stage picked its per-window in-sample winner by optimize_deflated (argmax) unconditionally, dropping the stage's `select` field, and both the preflight and the `walkforward` verb's `--real` campaign sub-branch refused plateau:* with a forward-pointer here (the #210 decision-4 deferral; the synthetic non-`--real` sub-branch has accepted plateau since #173). Honour plateau selection when no gate precedes the wf stage. - Preflight: the plateau-in-wf refusal is narrowed to gate-preceded stages. With no preceding gate the survivors are the full sweep grid, so the parameter lattice (grid.axis_lens) is exact and optimize_plateau is sound; a gate filters survivors below the grid, breaking the lattice, so plateau after any gate stays refused. The refusal is static — document validity must be data-independent (a gate's filtering depends on the data) — and keeps the existing "a gated survivor subset has no parameter lattice" message, which now describes exactly the only case it fires. - Executor: run_walk_forward_stage threads the per-cell grid.axis_lens and picks each window's in-sample winner via select_sweep_winner — the same dispatch the sweep stage uses. (Argmax, deflate=true) is byte-identical to the old optimize_deflated call, so the argmax grade and content-id pins are unchanged; the plateau arms read axis_lens and ignore deflate. - CLI: the walkforward verb threads --select through translate_walkforward / run_walkforward_sugar (a walkforward-local param, not the shared four-verb SugarInvocation), mapping the parsed Selection to the research SelectRule via select_rule_of; the refusal and its select_is_plateau helper are deleted. Scope is walkforward-only — the mc verb's wf-stage select is untouched. The plateau:worst grade anchor (stitched_total_pips = -9683776.67) is distinct from the argmax anchor (-10398606.67), proving the plateau path genuinely changes winner selection rather than falling back to argmax. A gate-free plateau process validates clean while a gate-preceded one is refused (a new campaign-validate e2e). Argmax defaults stay byte-identical. closes #215
This commit is contained in:
@@ -376,6 +376,7 @@ fn run_cell(
|
||||
stage_ordinal,
|
||||
stage,
|
||||
&grid.specs,
|
||||
&grid.axis_lens,
|
||||
&survivor_points,
|
||||
&family_name,
|
||||
runner,
|
||||
@@ -565,13 +566,17 @@ fn placeholder_report(params: &[(String, Scalar)], window_ms: (i64, i64)) -> Run
|
||||
}
|
||||
}
|
||||
|
||||
/// The sweep stage's winner + selection provenance. deflate=true composes only
|
||||
/// with argmax (preflighted: `DeflatePlateauConflict`), seeding the deflation
|
||||
/// nulls from the CAMPAIGN seed (C1: a campaign's realization is a pure
|
||||
/// function of the document). Bare argmax has no annotation to compute, so its
|
||||
/// `FamilySelection` is synthesized annotation-free; `raw_winner_metric`
|
||||
/// mirrors the registry's ranking read (an R metric against `r: None` ranks
|
||||
/// `NEG_INFINITY`).
|
||||
/// The shared in-sample-winner dispatch: called both for the sweep stage's
|
||||
/// own family and for each walk-forward window's in-sample family. deflate=
|
||||
/// true composes only with argmax (preflighted: `DeflatePlateauConflict`),
|
||||
/// seeding the deflation nulls from the CAMPAIGN seed (C1: a campaign's
|
||||
/// realization is a pure function of the document). Bare argmax has no
|
||||
/// annotation to compute, so its `FamilySelection` is synthesized annotation-
|
||||
/// free; `raw_winner_metric` mirrors the registry's ranking read (an R metric
|
||||
/// against `r: None` ranks `NEG_INFINITY`). The plateau arms (`PlateauMean`/
|
||||
/// `PlateauWorst`) ignore `deflate` outright (the `_` guard): they never
|
||||
/// deflate regardless of the caller's flag, so the walk-forward caller's
|
||||
/// hardcoded `deflate=true` only ever takes effect on the `Argmax` arm.
|
||||
fn select_sweep_winner(
|
||||
family: &SweepFamily,
|
||||
axis_lens: &[usize],
|
||||
@@ -650,12 +655,16 @@ fn placeholder_window_run(specs: &[ParamSpec]) -> WindowRun {
|
||||
/// Execute one `std::walk_forward` stage over the cell's surviving points:
|
||||
/// roll (IS, OOS) window splits over the cell window — entirely in ms
|
||||
/// (`Timestamp` is unit-agnostic `i64`; the driver owns ms->ns at its data
|
||||
/// seam) — then per window sweep the survivor points over the IS bounds, pick
|
||||
/// the winner (argmax with trials-deflation provenance, seeded from the
|
||||
/// campaign seed — the shipped select_winner convention; plateau in
|
||||
/// walk_forward is preflight-refused), run the winner over the OOS bounds
|
||||
/// with the selection stamped on its fresh report, and append the per-window
|
||||
/// OOS reports as a `FamilyKind::WalkForward` family.
|
||||
/// seam) — then per window sweep the survivor points over the IS bounds and
|
||||
/// pick the winner via `select_sweep_winner` (the same dispatch the sweep
|
||||
/// stage uses), seeded from the campaign seed: `Argmax` keeps the shipped
|
||||
/// trials-deflation provenance, and `PlateauMean`/`PlateauWorst` read
|
||||
/// `axis_lens` to score the closed neighbourhood — sound only because
|
||||
/// preflight refuses plateau whenever a gate precedes this stage (a gate
|
||||
/// filters survivors below the full grid, breaking the lattice `axis_lens`
|
||||
/// assumes intact). Then run the winner over the OOS bounds with the
|
||||
/// selection stamped on its fresh report, and append the per-window OOS
|
||||
/// reports as a `FamilyKind::WalkForward` family.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn run_walk_forward_stage(
|
||||
seed: u64,
|
||||
@@ -663,12 +672,13 @@ fn run_walk_forward_stage(
|
||||
stage: usize,
|
||||
block: &StageBlock,
|
||||
specs: &[ParamSpec],
|
||||
axis_lens: &[usize],
|
||||
survivors: &[Vec<Scalar>],
|
||||
family_name: &str,
|
||||
runner: &dyn MemberRunner,
|
||||
registry: &Registry,
|
||||
) -> Result<StageFamily, ExecFault> {
|
||||
let StageBlock::WalkForward { in_sample_ms, out_of_sample_ms, step_ms, mode, metric, .. } =
|
||||
let StageBlock::WalkForward { in_sample_ms, out_of_sample_ms, step_ms, mode, metric, select } =
|
||||
block
|
||||
else {
|
||||
return Err(ExecFault::PipelineShape {
|
||||
@@ -763,21 +773,18 @@ fn run_walk_forward_stage(
|
||||
faults.lock().unwrap().push((widx, ExecFault::Member(fault)));
|
||||
return placeholder_window_run(specs);
|
||||
}
|
||||
// IS winner: argmax with trials-deflation provenance, seeded from the
|
||||
// campaign seed (the shipped select_winner convention).
|
||||
let (winner, selection) = match optimize_deflated(
|
||||
&is_family,
|
||||
metric,
|
||||
DEFLATION_N_RESAMPLES,
|
||||
DEFLATION_BLOCK_LEN,
|
||||
seed,
|
||||
) {
|
||||
Ok(picked) => picked,
|
||||
Err(e) => {
|
||||
faults.lock().unwrap().push((widx, ExecFault::Registry(e)));
|
||||
return placeholder_window_run(specs);
|
||||
}
|
||||
};
|
||||
// IS winner: the same select_sweep_winner dispatch the sweep stage
|
||||
// uses, seeded from the campaign seed. deflate=true keeps Argmax's
|
||||
// shipped trials-deflation provenance identical to before this
|
||||
// dispatch existed; the Plateau arms ignore the deflate flag.
|
||||
let (winner, selection) =
|
||||
match select_sweep_winner(&is_family, axis_lens, metric, *select, true, seed) {
|
||||
Ok(picked) => picked,
|
||||
Err(e) => {
|
||||
faults.lock().unwrap().push((widx, e));
|
||||
return placeholder_window_run(specs);
|
||||
}
|
||||
};
|
||||
// OOS: run the winner over the out-of-sample bounds; the selection is
|
||||
// stamped on the fresh OOS report (a new record, never a mutation of
|
||||
// a stored one).
|
||||
|
||||
@@ -287,7 +287,9 @@ pub fn preflight(process: &ProcessDoc, campaign: &CampaignDoc) -> Result<(), Exe
|
||||
if !RANKABLE_METRICS.contains(&metric.as_str()) {
|
||||
return Err(ExecFault::UnrankableMetric { stage: i, metric: metric.clone() });
|
||||
}
|
||||
if matches!(select, SelectRule::PlateauMean | SelectRule::PlateauWorst) {
|
||||
if matches!(select, SelectRule::PlateauMean | SelectRule::PlateauWorst)
|
||||
&& process.pipeline[..i].iter().any(|s| matches!(s, StageBlock::Gate { .. }))
|
||||
{
|
||||
return Err(ExecFault::PlateauInWalkForward { stage: i });
|
||||
}
|
||||
for (field, len) in [
|
||||
@@ -795,17 +797,33 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
/// Plateau select in a walk_forward is only refused when a gate
|
||||
/// precedes it: a gate-free sweep -> wf keeps the survivors as the full
|
||||
/// grid (the parameter lattice is intact, so plateau is meaningful);
|
||||
/// a gate stage in between filters survivors below the full grid,
|
||||
/// breaking the lattice, so plateau is refused there.
|
||||
#[test]
|
||||
fn preflight_refuses_plateau_in_walk_forward() {
|
||||
fn preflight_permits_plateau_in_gate_free_walk_forward_refuses_after_a_gate() {
|
||||
let c = campaign();
|
||||
for select in [SelectRule::PlateauMean, SelectRule::PlateauWorst] {
|
||||
let p = process_of(vec![
|
||||
// gate-free: sweep -> wf(plateau) — survivors are the full grid,
|
||||
// the lattice is intact, so plateau is permitted.
|
||||
let ok = process_of(vec![
|
||||
sweep_stage("sqn", SelectRule::Argmax, false),
|
||||
wf_stage("sqn", select),
|
||||
]);
|
||||
assert!(preflight(&ok, &c).is_ok(), "gate-free plateau wf must pass preflight");
|
||||
|
||||
// gate-preceded: sweep -> gate -> wf(plateau) — the gate breaks
|
||||
// the lattice, so plateau is refused at stage 2.
|
||||
let gated = process_of(vec![
|
||||
sweep_stage("sqn", SelectRule::Argmax, false),
|
||||
gate_stage("sqn"),
|
||||
wf_stage("sqn", select),
|
||||
]);
|
||||
assert!(matches!(
|
||||
preflight(&p, &c),
|
||||
Err(ExecFault::PlateauInWalkForward { stage: 1 })
|
||||
preflight(&gated, &c),
|
||||
Err(ExecFault::PlateauInWalkForward { stage: 2 })
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -1248,6 +1266,70 @@ mod wf_tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// The walk_forward stage's IS-winner selection dispatches through
|
||||
/// `select_sweep_winner` too (exec.rs `run_walk_forward_stage`), not just
|
||||
/// the sweep stage: a gate-free `sweep -> wf(plateau)` is
|
||||
/// preflight-permitted (the lattice stays intact with no gate), and this
|
||||
/// pins that the runtime actually SELECTS via plateau rather than
|
||||
/// silently falling back to the deflated argmax the stage used
|
||||
/// unconditionally before this dispatch existed.
|
||||
#[test]
|
||||
fn wf_walk_forward_stage_dispatches_plateau_select_modes() {
|
||||
for (i, (select, expected_mode)) in [
|
||||
(SelectRule::PlateauMean, SelectionMode::PlateauMean),
|
||||
(SelectRule::PlateauWorst, SelectionMode::PlateauWorst),
|
||||
]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
let registry = wf_registry(&format!("wf-plateau-{i}"));
|
||||
let campaign = wf_campaign((0, 99));
|
||||
let process = ProcessDoc {
|
||||
format_version: 1,
|
||||
kind: DocKind::Process,
|
||||
name: "wf-plateau-proc".to_string(),
|
||||
description: None,
|
||||
pipeline: vec![
|
||||
StageBlock::Sweep {
|
||||
selection: Some(SweepSelection {
|
||||
metric: "total_pips".to_string(),
|
||||
select: SelectRule::Argmax,
|
||||
deflate: false,
|
||||
}),
|
||||
},
|
||||
StageBlock::WalkForward {
|
||||
in_sample_ms: 40,
|
||||
out_of_sample_ms: 20,
|
||||
step_ms: 20,
|
||||
mode: WfMode::Rolling,
|
||||
metric: "total_pips".to_string(),
|
||||
select,
|
||||
},
|
||||
],
|
||||
};
|
||||
let runner = WfFakeRunner::new();
|
||||
let outcome = run_wf(&campaign, &process, &runner, ®istry)
|
||||
.expect("gate-free plateau walk_forward executes");
|
||||
let fam = outcome.cells[0]
|
||||
.families
|
||||
.iter()
|
||||
.find(|f| f.block == "std::walk_forward")
|
||||
.expect("wf family present");
|
||||
assert!(!fam.reports.is_empty());
|
||||
for report in &fam.reports {
|
||||
let sel = report
|
||||
.manifest
|
||||
.selection
|
||||
.as_ref()
|
||||
.expect("every OOS member carries its IS selection");
|
||||
assert_eq!(
|
||||
sel.mode, expected_mode,
|
||||
"SelectRule::{select:?} on a walk_forward stage must dispatch to SelectionMode::{expected_mode:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `WfMode::Anchored` maps to the engine's `RollMode::Anchored` (only
|
||||
/// Rolling is exercised by the other wf_* tests) — the oracle is the same
|
||||
/// roller construction the stage itself uses, just with `RollMode::Anchored`.
|
||||
|
||||
+21
-17
@@ -686,6 +686,16 @@ fn parse_select(s: &str) -> Result<Selection, ()> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Map the CLI `--select` value to the research selection rule the campaign
|
||||
/// document carries.
|
||||
fn select_rule_of(sel: Selection) -> aura_research::SelectRule {
|
||||
match sel {
|
||||
Selection::Argmax => aura_research::SelectRule::Argmax,
|
||||
Selection::Plateau(PlateauMode::Mean) => aura_research::SelectRule::PlateauMean,
|
||||
Selection::Plateau(PlateauMode::Worst) => aura_research::SelectRule::PlateauWorst,
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a comma-separated list of `T` (each item parsed via `FromStr`), rejecting
|
||||
/// any item that fails to parse — the shared validator for the r-sma grid flags
|
||||
/// (#137). The caller folds the `Err` into the subcommand `usage()` so a malformed
|
||||
@@ -2300,13 +2310,6 @@ fn wf_ms_sizes() -> (u64, u64, u64) {
|
||||
)
|
||||
}
|
||||
|
||||
/// True iff `--select` names a plateau mode (refused on the dissolved path, Fork B,
|
||||
/// until #215 ships the wf-stage plateau relaxation). Reuses `parse_select` so the
|
||||
/// vocabulary stays single-sourced.
|
||||
fn select_is_plateau(select: Option<&str>) -> bool {
|
||||
matches!(select.map(parse_select), Some(Ok(Selection::Plateau(_))))
|
||||
}
|
||||
|
||||
/// Resolve a single optional stop knob (`--stop-length`/`--stop-k`): an absent flag
|
||||
/// defaults to `default`; a present one parses via [`parse_csv_list`] and refuses
|
||||
/// unless it holds exactly one value (the stop is a risk regime, not a swept axis).
|
||||
@@ -2938,16 +2941,16 @@ fn dispatch_walkforward(a: WalkforwardCmd, env: &project::Env) {
|
||||
// The campaign path (#220): the real-archive execution routes
|
||||
// through the one campaign executor, over the user's own
|
||||
// blueprint and axes (formerly the welded r-sma branch).
|
||||
if let Some(s) = a.select.as_deref()
|
||||
&& parse_select(s).is_err()
|
||||
{
|
||||
eprintln!("aura: {}", usage());
|
||||
std::process::exit(2);
|
||||
}
|
||||
if select_is_plateau(a.select.as_deref()) {
|
||||
eprintln!("aura: --select plateau is not yet available on the campaign path; see #215");
|
||||
std::process::exit(2);
|
||||
}
|
||||
let select = match a.select.as_deref() {
|
||||
Some(s) => match parse_select(s) {
|
||||
Ok(sel) => select_rule_of(sel),
|
||||
Err(()) => {
|
||||
eprintln!("aura: {}", usage());
|
||||
std::process::exit(2);
|
||||
}
|
||||
},
|
||||
None => aura_research::SelectRule::Argmax,
|
||||
};
|
||||
let (name, symbol, stop_length, stop_k, from, to) =
|
||||
walkforward_args_from(&a).unwrap_or_else(|m| {
|
||||
eprintln!("aura: {m}");
|
||||
@@ -2988,6 +2991,7 @@ fn dispatch_walkforward(a: WalkforwardCmd, env: &project::Env) {
|
||||
out_of_sample_ms: oos_ms,
|
||||
step_ms,
|
||||
},
|
||||
select,
|
||||
env,
|
||||
)
|
||||
.unwrap_or_else(|m| {
|
||||
|
||||
@@ -375,6 +375,7 @@ pub(crate) fn translate_walkforward(
|
||||
inv: &SugarInvocation,
|
||||
metric: &str,
|
||||
w: WfWindows,
|
||||
select: SelectRule,
|
||||
) -> Result<GeneratedWalkforward, String> {
|
||||
let process = ProcessDoc {
|
||||
format_version: FORMAT_VERSION,
|
||||
@@ -395,7 +396,7 @@ pub(crate) fn translate_walkforward(
|
||||
step_ms: w.step_ms,
|
||||
mode: WfMode::Rolling,
|
||||
metric: metric.to_string(),
|
||||
select: SelectRule::Argmax,
|
||||
select,
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -445,9 +446,10 @@ pub(crate) fn run_walkforward_sugar(
|
||||
inv: &SugarInvocation,
|
||||
metric: &str,
|
||||
w: WfWindows,
|
||||
select: SelectRule,
|
||||
env: &crate::project::Env,
|
||||
) -> Result<(), String> {
|
||||
let generated = translate_walkforward(inv, metric, w)?;
|
||||
let generated = translate_walkforward(inv, metric, w, select)?;
|
||||
let probe_params = probe_params_from(inv.axes);
|
||||
validate_before_register(&generated.process, &generated.campaign, inv.blueprint_canonical, &probe_params, env)?;
|
||||
let reg = env.registry();
|
||||
@@ -817,8 +819,8 @@ mod tests {
|
||||
fn translate_walkforward_is_deterministic_and_carries_the_regime() {
|
||||
let ax = wf_axes();
|
||||
let i = inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 2.0 }), "{\"bp\":1}");
|
||||
let a = translate_walkforward(&i, "sqn_normalized", WF_W).unwrap();
|
||||
let b = translate_walkforward(&i, "sqn_normalized", WF_W).unwrap();
|
||||
let a = translate_walkforward(&i, "sqn_normalized", WF_W, SelectRule::Argmax).unwrap();
|
||||
let b = translate_walkforward(&i, "sqn_normalized", WF_W, SelectRule::Argmax).unwrap();
|
||||
assert_eq!(
|
||||
content_id_of(&campaign_to_json(&a.campaign)),
|
||||
content_id_of(&campaign_to_json(&b.campaign)),
|
||||
@@ -866,9 +868,9 @@ mod tests {
|
||||
#[test]
|
||||
fn translate_walkforward_diverging_regimes_do_not_collide_content_ids() {
|
||||
let ax = wf_axes();
|
||||
let base = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 2.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W).unwrap();
|
||||
let different_k = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 3.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W).unwrap();
|
||||
let different_length = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 20, k: 2.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W).unwrap();
|
||||
let base = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 2.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W, SelectRule::Argmax).unwrap();
|
||||
let different_k = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 3.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W, SelectRule::Argmax).unwrap();
|
||||
let different_length = translate_walkforward(&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 20, k: 2.0 }), "{\"bp\":1}"), "sqn_normalized", WF_W, SelectRule::Argmax).unwrap();
|
||||
let base_id = content_id_of(&campaign_to_json(&base.campaign));
|
||||
let k_id = content_id_of(&campaign_to_json(&different_k.campaign));
|
||||
let length_id = content_id_of(&campaign_to_json(&different_length.campaign));
|
||||
@@ -893,6 +895,7 @@ mod tests {
|
||||
&inv(&ax, "walkforward", &["GER40"], Some(VolStop { length: 14, k: 2.0 }), "{\"bp\":1}"),
|
||||
"sqn_normalized",
|
||||
WF_W,
|
||||
SelectRule::Argmax,
|
||||
)
|
||||
.unwrap();
|
||||
let (process_id, campaign_id) = register_generated_wf(®, &generated).unwrap();
|
||||
|
||||
@@ -1801,11 +1801,13 @@ fn walkforward_dissolved_refuses_a_multi_value_stop() {
|
||||
assert!(stderr.contains("single risk regime"), "stop-regime refusal: {stderr}");
|
||||
}
|
||||
|
||||
/// Fork B: --select plateau is refused on the dissolved path (exit 2) with a
|
||||
/// pointer to #215 until the wf-stage plateau relaxation ships.
|
||||
/// #215: --select plateau is no longer refused on the dissolved path — it now
|
||||
/// threads through to the wf stage exactly like the built-in synthetic path.
|
||||
/// Gated on the shared GER40 archive; skips cleanly on a data refusal (exit 1),
|
||||
/// same tolerance as the sibling stop-knob-defaulting test.
|
||||
#[test]
|
||||
fn walkforward_dissolved_refuses_select_plateau() {
|
||||
let cwd = temp_cwd("walkforward-plateau");
|
||||
fn walkforward_dissolved_accepts_select_plateau() {
|
||||
let (cwd, _g) = fresh_project();
|
||||
let fixture = format!("{}/examples/r_sma_open.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
@@ -1814,12 +1816,83 @@ fn walkforward_dissolved_refuses_select_plateau() {
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
"--select", "plateau:worst",
|
||||
])
|
||||
.current_dir(&cwd)
|
||||
.current_dir(cwd)
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
assert_eq!(out.status.code(), Some(2), "plateau is refused on the dissolved path");
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(stderr.contains("#215"), "forward pointer to the plateau follow-up: {stderr}");
|
||||
if out.status.code() == Some(1) {
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
stderr.contains("no local data") || stderr.contains("no recorded geometry"),
|
||||
"exit 1 must be a data refusal, got: {stderr}"
|
||||
);
|
||||
eprintln!("skip: no local GER40 data");
|
||||
return;
|
||||
}
|
||||
assert_eq!(
|
||||
out.status.code(),
|
||||
Some(0),
|
||||
"--select plateau:worst must no longer be refused on the dissolved path: {}",
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
/// Characterization pin (#215): the same identical invocation as
|
||||
/// `walkforward_real_e2e_pins_the_exact_current_grade` (fixture, real
|
||||
/// instrument, axes, stop, window), but selecting `plateau:worst` instead of
|
||||
/// the default argmax. Plateau scores the closed neighbourhood over the full
|
||||
/// IS grid rather than picking the single best point, so its winner (and
|
||||
/// therefore its OOS stitch) legitimately differs from the argmax anchor;
|
||||
/// this pins that divergence exactly, so a regression that silently
|
||||
/// collapsed plateau back to argmax (or vice versa) inside
|
||||
/// `run_walk_forward_stage`'s `select_sweep_winner` dispatch goes red here
|
||||
/// even though the argmax anchor stays green. Gated on the shared GER40
|
||||
/// archive; skips cleanly on a data refusal.
|
||||
#[test]
|
||||
fn walkforward_real_e2e_pins_the_exact_current_plateau_grade() {
|
||||
const FROM_MS: &str = "1735689600000";
|
||||
const TO_MS: &str = "1767225599000";
|
||||
let (cwd, _g) = fresh_project();
|
||||
let fixture = format!("{}/examples/r_sma_open.json", env!("CARGO_MANIFEST_DIR"));
|
||||
let out = std::process::Command::new(env!("CARGO_BIN_EXE_aura"))
|
||||
.args([
|
||||
"walkforward", &fixture, "--real", "GER40",
|
||||
"--axis", "sma_signal.fast.length=3,5", "--axis", "sma_signal.slow.length=12,20",
|
||||
"--stop-length", "14", "--stop-k", "2.0",
|
||||
"--from", FROM_MS, "--to", TO_MS,
|
||||
"--select", "plateau:worst",
|
||||
])
|
||||
.current_dir(cwd)
|
||||
.output()
|
||||
.expect("spawn aura");
|
||||
if out.status.code() == Some(1) {
|
||||
let stderr = String::from_utf8_lossy(&out.stderr);
|
||||
assert!(
|
||||
stderr.contains("no local data") || stderr.contains("no recorded geometry"),
|
||||
"exit 1 must be a data refusal, got: {stderr}"
|
||||
);
|
||||
eprintln!("skip: no local GER40 data");
|
||||
return;
|
||||
}
|
||||
assert_eq!(out.status.code(), Some(0), "exit: {:?}", out.status);
|
||||
let stdout = String::from_utf8(out.stdout).expect("utf-8 stdout");
|
||||
let grade_line = stdout
|
||||
.lines()
|
||||
.find(|l| l.starts_with("{\"walkforward\":"))
|
||||
.unwrap_or_else(|| panic!("the walkforward summary line: {stdout}"));
|
||||
let v: serde_json::Value = serde_json::from_str(grade_line).expect("summary line parses as JSON");
|
||||
let wf = &v["walkforward"];
|
||||
assert_eq!(wf["windows"].as_u64(), Some(9), "window count: {grade_line}");
|
||||
// EXACT floats -- the byte-identity anchor a plateau-vs-argmax regression must preserve.
|
||||
assert_eq!(wf["stitched_total_pips"].as_f64(), Some(-9683776.66663749), "stitched OOS pips: {grade_line}");
|
||||
let r = &wf["oos_r"];
|
||||
assert_eq!(r["n_trades"].as_u64(), Some(20667), "pooled OOS trade count: {grade_line}");
|
||||
assert_eq!(r["expectancy_r"].as_f64(), Some(0.007635120926372627), "pooled OOS expectancy R: {grade_line}");
|
||||
// param_stability varies across windows -> the per-window plateau refit picks
|
||||
// different winners than the argmax anchor; pinning the means locks the
|
||||
// plateau winner-selection path.
|
||||
let ps = wf["param_stability"].as_array().expect("param_stability is an array");
|
||||
assert_eq!(ps[0]["mean"].as_f64(), Some(4.777777777777778), "fast-MA refit mean: {grade_line}");
|
||||
assert_eq!(ps[1]["mean"].as_f64(), Some(16.444444444444443), "slow-MA refit mean: {grade_line}");
|
||||
}
|
||||
|
||||
/// Property (#217): `walkforward_args_from` no longer requires both stop knobs
|
||||
@@ -1916,7 +1989,7 @@ fn walkforward_dissolved_refuses_an_unknown_axis_name() {
|
||||
|
||||
/// Front-end parity: an unknown `--select` token is refused (exit 2) on the
|
||||
/// dissolved path exactly as it is on the inline path — it must not be silently
|
||||
/// swallowed by `select_is_plateau`'s plateau-only check and fall through to argmax.
|
||||
/// swallowed by `select_rule_of`'s conversion and fall through to argmax.
|
||||
#[test]
|
||||
fn walkforward_dissolved_refuses_an_unknown_select_token() {
|
||||
let cwd = temp_cwd("walkforward-bad-select");
|
||||
|
||||
@@ -1582,6 +1582,105 @@ fn campaign_validate_runs_the_executor_preflight_as_a_third_tier() {
|
||||
assert!(!bad_out.contains("PipelineShape"), "Debug leak: {bad_out}");
|
||||
}
|
||||
|
||||
/// #215: `walk_forward` selecting a plateau mode with NO gate ahead of it —
|
||||
/// the sweep survivors are the full parameter grid, so the plateau
|
||||
/// neighbourhood scoring `axis_lens` assumes is intact.
|
||||
const WF_PLATEAU_GATE_FREE_PROCESS_DOC: &str = r#"{
|
||||
"format_version": 1,
|
||||
"kind": "process",
|
||||
"name": "wf-plateau-gate-free",
|
||||
"pipeline": [
|
||||
{ "block": "std::sweep", "metric": "sqn_normalized", "select": "argmax" },
|
||||
{ "block": "std::walk_forward", "in_sample_ms": 1209600000, "out_of_sample_ms": 604800000,
|
||||
"step_ms": 604800000, "mode": "rolling", "metric": "net_expectancy_r", "select": "plateau:worst" }
|
||||
]
|
||||
}"#;
|
||||
|
||||
/// #215: the same `walk_forward` plateau select, but a `std::gate` stage now
|
||||
/// sits between the sweep and the walk-forward — the gate can filter
|
||||
/// survivors below the full grid, breaking the lattice the plateau
|
||||
/// neighbourhood assumes, so the executor preflight must still refuse it.
|
||||
const WF_PLATEAU_GATED_PROCESS_DOC: &str = r#"{
|
||||
"format_version": 1,
|
||||
"kind": "process",
|
||||
"name": "wf-plateau-gated",
|
||||
"pipeline": [
|
||||
{ "block": "std::sweep", "metric": "sqn_normalized", "select": "argmax" },
|
||||
{ "block": "std::gate", "all": [ { "metric": "n_trades", "cmp": "ge", "value": 0.0 } ] },
|
||||
{ "block": "std::walk_forward", "in_sample_ms": 1209600000, "out_of_sample_ms": 604800000,
|
||||
"step_ms": 604800000, "mode": "rolling", "metric": "net_expectancy_r", "select": "plateau:worst" }
|
||||
]
|
||||
}"#;
|
||||
|
||||
/// Property (#215): the executor preflight's plateau-in-walk_forward refusal
|
||||
/// is narrowed to gate-preceded stages, not a blanket ban on plateau select
|
||||
/// in walk_forward — pinned through the full document surface (`campaign
|
||||
/// validate`'s third, executor tier — data-free, before any member runs),
|
||||
/// mirroring the pattern `campaign_validate_runs_the_executor_preflight_as_a_
|
||||
/// third_tier` uses for the sibling shape guard. A regression that reverted
|
||||
/// to the blanket refusal would flip the gate-free face to exit 1; a
|
||||
/// regression that dropped the refusal entirely would flip the gated face to
|
||||
/// exit 0 — either direction goes red here.
|
||||
#[test]
|
||||
fn campaign_validate_narrows_plateau_in_wf_refusal_to_gate_preceded_stages() {
|
||||
let _fixture = project_lock();
|
||||
let dir = built_project();
|
||||
let runs_dir = dir.join("runs");
|
||||
std::fs::remove_dir_all(&runs_dir).ok();
|
||||
let _cleanup = ScratchGuard(vec![
|
||||
ScratchPath::Dir(runs_dir.clone()),
|
||||
ScratchPath::File(dir.join("wf-plateau-free.process.json")),
|
||||
ScratchPath::File(dir.join("wf-plateau-free.campaign.json")),
|
||||
ScratchPath::File(dir.join("wf-plateau-gated.process.json")),
|
||||
ScratchPath::File(dir.join("wf-plateau-gated.campaign.json")),
|
||||
]);
|
||||
let bp_id = seed_blueprint(dir, "campaign-validate-wf-plateau-seed");
|
||||
|
||||
// Face 1 — gate-free: sweep -> walk_forward(plateau:worst). The lattice
|
||||
// is intact, so the executor tier accepts it (exit 0).
|
||||
let free_id =
|
||||
register_process_doc(dir, "wf-plateau-free.process.json", WF_PLATEAU_GATE_FREE_PROCESS_DOC);
|
||||
write_doc(
|
||||
dir,
|
||||
"wf-plateau-free.campaign.json",
|
||||
&campaign_doc_json(&bp_id, &free_id, (1, 2), "", ""),
|
||||
);
|
||||
let (free_out, free_code) =
|
||||
run_code_in(dir, &["campaign", "validate", "wf-plateau-free.campaign.json"]);
|
||||
assert_eq!(free_code, Some(0), "gate-free plateau walk_forward validates clean: {free_out}");
|
||||
assert!(
|
||||
free_out.contains(
|
||||
"campaign document valid (executable): pipeline shape and static guards pass"
|
||||
),
|
||||
"stdout/stderr: {free_out}"
|
||||
);
|
||||
|
||||
// Face 2 — gate-preceded: sweep -> gate -> walk_forward(plateau:worst).
|
||||
// The gate can break the lattice, so the executor tier still refuses it
|
||||
// (exit 1), naming the offending stage.
|
||||
let gated_id =
|
||||
register_process_doc(dir, "wf-plateau-gated.process.json", WF_PLATEAU_GATED_PROCESS_DOC);
|
||||
write_doc(
|
||||
dir,
|
||||
"wf-plateau-gated.campaign.json",
|
||||
&campaign_doc_json(&bp_id, &gated_id, (1, 2), "", ""),
|
||||
);
|
||||
let (gated_out, gated_code) =
|
||||
run_code_in(dir, &["campaign", "validate", "wf-plateau-gated.campaign.json"]);
|
||||
assert_eq!(gated_code, Some(1), "gate-preceded plateau walk_forward still refuses: {gated_out}");
|
||||
assert!(
|
||||
gated_out.contains("campaign is not executable:"),
|
||||
"the executor-tier fault header: {gated_out}"
|
||||
);
|
||||
assert!(
|
||||
gated_out.contains(
|
||||
"process stage 2: walk_forward cannot use a plateau select (a gated survivor subset has no parameter lattice)"
|
||||
),
|
||||
"the exec_fault_prose naming the gate-broken lattice: {gated_out}"
|
||||
);
|
||||
assert!(!gated_out.contains("PlateauInWalkForward"), "Debug leak: {gated_out}");
|
||||
}
|
||||
|
||||
/// A v2 process doc with a static `std::monte_carlo resamples: 0` defect.
|
||||
const ZERO_RESAMPLES_MC_PROCESS_DOC: &str = r#"{
|
||||
"format_version": 1,
|
||||
|
||||
Reference in New Issue
Block a user