Aura/docs/specs/walkforward-dissolution.md

Walkforward dissolution — Design Spec

Date: 2026-07-06 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Milestone #210 ("Verb dissolution — the campaign path as canonical orchestration"), third dissolution (after sweep and generalize). Reference issue: Gitea Brummel/Aura #210. Scope reading: decision 2, Reading A (the verb surfaces stay fenced until #159; a real-archive execution of a built-in strategy routes through the campaign path now). Derived forks A/B and the pure-sugar finding are recorded as #210 comments (2026-07-06).

Goal

Dissolve aura walkforward's inline built-in --strategy r-sma --real execution into thin sugar over the one campaign executor (aura_campaign::execute), exactly as sweep and generalize already were. The dispatch_walkforward built-in arm gains an r-sma-real branch that runs through a generated, auto-registered, content-addressed campaign document, reproducing the {"walkforward":{…}} summary line byte-for-byte. The committed anchor walkforward_real_e2e_pins_the_exact_current_grade is the acceptance gate.

run_walkforward is NOT deleted (unlike generalize, which had no synthetic path): the built-in arm splits, and everything the dissolution predicate does not match stays on the inline handler.

Out of scope (stay verb-wired, fenced until #159 — Reading A): the blueprint-file branch (run_blueprint_walkforward, hardcoded DataSource::Synthetic), the synthetic built-in path (any strategy, no --real), and the non-r-sma real path (--strategy sma --real — a different blueprint the r-sma sugar does not synthesize). The --select plateau:* sub-path is carved out to a follow-up (Fork B): the dissolved dispatch refuses it (exit 2) with a forward pointer until the decision-4 wf-stage plateau relaxation (#215) ships.

Architecture

The dissolution reuses the generalize seam verbatim; the only new insight is that the walk-forward verb summary reconstructs from the campaign's recorded outcome as pure sugar — no executor change.

Grounding (verified, recorded on #210):

• The campaign std::walk_forward stage (crates/aura-campaign/src/exec.rs, run_walk_forward_stage) already runs the per-window IS-sweep → optimize_deflated winner → OOS run, and appends the per-window OOS reports as a FamilyKind::WalkForward family. CellOutcome.families surfaces that StageFamily { block: "std::walk_forward", family_id, reports } in-memory to the caller, with each report's metrics.r.trade_rs intact (it is #[serde(skip)] on the wire but present in-memory).
• stitched_total_pips = Σ (per-window metrics.total_pips), summed left-to-right in window (roll) order. The engine stitch offsets each segment by the running sum of prior segments' final cumulative values, so stitched_oos_equity.last() = Σ_i seg_i.last, and seg_i.last == total_pips_i. Verified byte-exact on the anchor: -10398606.666650848. The discarded oos_equity: vec![] (exec.rs:794) is therefore NOT needed.
• pooled oos_r = r_metrics_from_rs(pool of the per-window trade_rs, roll order) — the same reduction the inline pooled_oos_trade_rs performs. No std::monte_carlo stage is appended. (Σ per-window n_trades = 20681 verified == the pooled n_trades.)
• param_stability is the on-demand reduction over the per-window chosen params; reconstructable from the family reports' recorded manifest params (roll order).

The canonical process shape for a pure walk-forward is [std::sweep(argmax), std::walk_forward]: the wf stage's per-window IS-refit needs a survivor population, and the campaign feeds it from a preceding sweep (a selection-free sweep is permitted only as the terminal stage). The preceding sweep's whole family flows on as survivors (select names a recorded selection, never a filter — exec.rs:319), so the wf stage receives the full fast×slow grid and refits it per window. That preceding full-window sweep records an extra Sweep family (a campaign-path audit-trail addition, accepted exactly as generalize's family-set change was — the anchor pins only the summary line).

Fork A (stop mapping): fast/slow are the IS-refit grid (multi-value campaign strategy axes); the stop is a single risk regime RiskRegime::Vol{stop_length, stop_k} (the member runner maps cell.regime → StopRule::Vol). --stop-length / --stop-k are single-value; a multi-value stop is refused.

Load-bearing implementation detail (anchor-gated, not an assumption about current behaviour): the inline roller works in ns (span from probe_window, sizes WF_REAL_*_NS); the campaign wf stage works in ms (span cell.window_ms from the doc's from_ms/to_ms, sizes the StageBlock _ms fields). Since ns = ms × 1e6 linearly, the ms roller sizes (WF_REAL_*_NS / 1_000_000) over the doc window must reproduce the identical 9 calendar windows — the anchor's windows count and exact floats are what verify this end-to-end; any span/roller drift fails there loudly.

Concrete code shapes

The user-facing program (unchanged surface, dissolved execution)

$ aura walkforward --strategy r-sma --real GER40 \
    --from 1735689600000 --to 1767225599000 \
    --fast 3,5 --slow 12,20 --stop-length 14 --stop-k 2.0
{"family_id":"…","report":{…}}          # per-window OOS member line (×9)
…
{"walkforward":{"oos_r":{…},"param_stability":[…],
  "stitched_total_pips":-10398606.666650848,"windows":9}}

The summary line is byte-identical to today's inline output (the acceptance gate). The per-window member lines carry the campaign-generated WalkForward family id rather than the inline walkforward-N handle (a family-handle change, accepted like generalize's — the anchor pins the summary line, not the member handles).

This example IS the acceptance-criterion evidence (aura CLAUDE.md domain invariant 12: the World runs any harness through the one path; invariant 10: closed-vocabulary campaign data, no verb-special logic). The audience — a researcher validating a candidate out-of-sample — reaches for the identical command; nothing about the surface changes, while the execution becomes a durable, diffable, reproducible campaign document (C18). It reintroduces no look-ahead or determinism failure: the campaign wf stage is the same deterministic engine walk_forward (C1/C2), and the dissolution adds no new streamed non-scalar.

--select plateau:mean|plateau:worst is refused on the dissolved path:

$ aura walkforward --strategy r-sma --real GER40 --select plateau:worst …
aura: --select plateau is not yet available on the campaign path; see #215
$ echo $?
2

Sugar translator — mirror of translate_generalize (verb_sugar.rs)

The new translate_walkforward differs from translate_generalize in three ways: the process is [Sweep(argmax), WalkForward] (not [Sweep(argmax), Generalize]); the campaign carries one instrument and multi-value fast/slow axes (the IS-refit grid, not a single candidate); the wf stage's roller sizes are computed from the data kind.

// crates/aura-cli/src/verb_sugar.rs  — new, after translate_generalize
pub(crate) struct GeneratedWalkforward { pub process: ProcessDoc, pub campaign: CampaignDoc }

#[allow(clippy::too_many_arguments)]           // #214: bundle deferred until all verbs dissolve
pub(crate) fn translate_walkforward(
    fast: &[i64], slow: &[i64],                // the IS-refit grid axes (multi-value)
    stop_length: i64, stop_k: f64,             // single-value → one risk regime
    metric: &str,                              // "sqn_normalized" (the RSma wf objective)
    in_sample_ms: u64, out_of_sample_ms: u64, step_ms: u64,   // roller sizes, ms
    name: &str, symbol: &str, from_ms: i64, to_ms: i64,
    blueprint_canonical: &str,
) -> Result<GeneratedWalkforward, String> {
    let process = ProcessDoc {
        format_version: FORMAT_VERSION, kind: DocKind::Process,
        name: "walkforward".to_string(), description: None,
        pipeline: vec![
            // the preceding sweep only enumerates the survivor grid for the wf
            // stage; its recorded argmax selection is not part of the summary
            StageBlock::Sweep { selection: Some(SweepSelection {
                metric: metric.to_string(), select: SelectRule::Argmax, deflate: false }) },
            StageBlock::WalkForward {
                in_sample_ms, out_of_sample_ms, step_ms,
                mode: WfMode::Rolling, metric: metric.to_string(), select: SelectRule::Argmax },
        ],
    };
    let mut doc_axes: BTreeMap<String, Axis> = BTreeMap::new();
    doc_axes.insert("fast.length".to_string(),
        axis_from_values("fast.length", &fast.iter().map(|&v| Scalar::i64(v)).collect::<Vec<_>>())?);
    doc_axes.insert("slow.length".to_string(),
        axis_from_values("slow.length", &slow.iter().map(|&v| Scalar::i64(v)).collect::<Vec<_>>())?);
    let campaign = CampaignDoc {
        format_version: FORMAT_VERSION, kind: DocKind::Campaign,
        name: name.to_string(), description: None,
        data: DataSection { instruments: vec![symbol.to_string()],
                            windows: vec![Window { from_ms, to_ms }] },
        strategies: vec![StrategyEntry {
            r#ref: DocRef::ContentId(content_id_of(blueprint_canonical)), axes: doc_axes }],
        process: ProcessRef { r#ref: DocRef::ContentId(content_id_of(&process_to_json(&process))) },
        risk: vec![RiskRegime::Vol { length: stop_length, k: stop_k }],
        seed: 0,
        presentation: Presentation { persist_taps: vec![], emit: vec!["family_table".to_string()] },
    };
    Ok(GeneratedWalkforward { process, campaign })
}

Sugar runner — reprint the summary from the recorded WalkForward family

// crates/aura-cli/src/verb_sugar.rs  — new, mirror of run_generalize_sugar
#[allow(clippy::too_many_arguments)]
pub(crate) fn run_walkforward_sugar(/* same args as translate_walkforward + env */)
  -> Result<(), String> {
    let generated = translate_walkforward(/* … */)?;
    let probe_params = vec![
        ("fast.length".to_string(), Scalar::i64(fast[0])),
        ("slow.length".to_string(), Scalar::i64(slow[0])),
    ];
    validate_before_register(&generated.process, &generated.campaign, blueprint_canonical, &probe_params, env)?;
    let reg = env.registry();
    let (_p, campaign_id) = register_generated_wf(&reg, &generated)?;   // put_process + put_campaign
    let run = crate::campaign_run::run_campaign_returning(&campaign_id, env)?;

    // The single cell's walk_forward StageFamily (block == "std::walk_forward").
    let wf = run.outcome.cells.iter()
        .flat_map(|c| c.families.iter())
        .find(|f| f.block == "std::walk_forward")
        .ok_or("walkforward produced no walk_forward family")?;

    // Per-window member lines (campaign family id), then the summary line —
    // both reconstructed from wf.reports in roll order.
    for report in &wf.reports { println!("{}", crate::family_member_line(&wf.family_id, report)); }
    println!("{}", crate::walkforward_summary_json_from_reports(&wf.reports));   // new pure-sugar reducer
    Ok(())
}

The new walkforward_summary_json_from_reports is the pure-sugar reducer over the recorded per-window reports (verified byte-exact against the inline walkforward_summary_json): windows = reports.len(); stitched_total_pips = reports.iter().map(|r| r.metrics.total_pips).sum() (left-to-right, roll order — IEEE-754 order preserved); the oos_r block from r_metrics_from_rs(pooled trade_rs); param_stability from the per-window recorded params. It reuses the existing pooled_oos_trade_rs/param_stability reductions where their inputs are reconstructable from the reports.

Dispatch rewrite — split the built-in arm; run_walkforward STAYS for the fenced paths

Unlike generalize (which requires --real and has no synthetic path), dispatch_walkforward's None (built-in) arm today routes BOTH synthetic (aura walkforward without --real) and real through run_walkforward. Under decision 2 / Reading A only the real-archive r-sma execution dissolves; the synthetic path (any strategy) and the non-r-sma real path (--strategy sma --real — a different blueprint the r-sma sugar does not synthesize) stay verb-wired, fenced until #159. So the arm splits on the dissolution predicate and run_walkforward is retained for the fenced branch — this cycle deletes NO inline code; it adds an r-sma-real branch alongside the existing body.

// crates/aura-cli/src/main.rs  — dispatch_walkforward, the None (built-in) arm
None => {
    // …existing usage / blueprint-flag guards, unchanged…
    let dissolves = a.strategy.as_deref() == Some("r-sma") && a.real.is_some();
    if !dissolves {
        // Fenced (Reading A): synthetic (any strategy) + non-r-sma real stay on
        // the inline handler — the EXISTING arm body, byte-for-byte unchanged:
        //   strategy_from / RGrid parse / parse_select / name_persist /
        //   data_choice_from → run_walkforward(strategy, &name, persist,
        //   DataSource::from_choice(data, env), &grid, select, env);
        return; /* existing body, verbatim */
    }
    // r-sma --real: the dissolved sugar path.
    if is_plateau(a.select.as_deref()) {           // Fork B: refuse plateau on the dissolved path
        eprintln!("aura: --select plateau is not yet available on the campaign path; see #215");
        std::process::exit(2);
    }
    let (name, symbol, fast, slow, stop_length, stop_k, from, to) =
        walkforward_args_from(&a) /* unwrap_or_else: eprintln + exit 2, the generalize idiom */;
    // Synthesize the bare open sma_signal blueprint (identical to generalize).
    let signal = sma_signal(None, None);
    let canonical = blueprint_to_json(&signal).expect("a bare sma_signal serializes");
    let reg = env.registry();
    reg.put_blueprint(&topology_hash(&signal), &canonical) /* exit 1 on err */;
    // Window: explicit --from/--to when both present, else the symbol's full
    // archive window (as generalize; no-window grade drifts with the archive).
    let (from_ms, to_ms) = resolve_window(&symbol, from, to, env);
    let (is_ms, oos_ms, step_ms) = wf_ms_sizes();   // WF_REAL_*_NS / 1_000_000
    verb_sugar::run_walkforward_sugar(&fast, &slow, stop_length, stop_k,
        "sqn_normalized", is_ms, oos_ms, step_ms, &name, &symbol, from_ms, to_ms, &canonical, env)
        /* unwrap_or_else: eprintln + exit 1 */;
}

run_walkforward (main.rs:1665) and every helper it uses — walkforward_family, walkforward_summary_json, pooled_oos_trade_rs, param_stability, select_winner, run_oos_r, family_member_line — are retained: recon confirmed each keeps a live non-test caller (the synthetic path, the blueprint walkforward path, and the mc R-bootstrap path). Nothing is deleted; the new sugar functions are additive.

Components

• crates/aura-cli/src/verb_sugar.rs — translate_walkforward, GeneratedWalkforward, register_generated_wf, run_walkforward_sugar, walkforward_summary_json_from_reports (or reuse the inline reducer if it can read &[RunReport]). Reuses validate_before_register, axis_from_values.
• crates/aura-cli/src/main.rs — walkforward_args_from (mirror of generalize_args_from: single instrument, multi-value fast/slow, single-value stop with a multi-value refusal, all-four-knobs-required), a --select plateau refusal, the dispatch_walkforward built-in-arm split (add the r-sma-real sugar branch; retain the existing body + run_walkforward for the fenced else), the ns→ms roller-size helper. No inline handler or helper is removed.
• No executor change (aura-campaign, aura-engine, aura-research untouched) for the argmax path.

Data flow

argv → walkforward_args_from (name, symbol, fast[], slow[], stop, window) → dispatch_walkforward synthesizes + stores the sma_signal blueprint, resolves the window, computes ms roller sizes → run_walkforward_sugar → translate_walkforward builds [Sweep(argmax), WalkForward] + a campaign (fast/slow axes, one instrument, one Vol regime) → validate_before_register → register_generated_wf → run_campaign_returning runs the one executor (sweep enumerates survivors → wf refits per window → appends the WalkForward family) → the sugar reads the WalkForward StageFamily from CellOutcome.families, reprints the member lines + the summary reducer.

Error handling

• Non-r-sma strategy, missing knobs, empty/absent --real: reuse the inline message strings (byte-identical front-end refusals, exit 2), as generalize did.
• Multi-value --stop-length/--stop-k: exit 2 with "the stop is a single risk regime; --stop-length/--stop-k take one value each" (Fork A).
• --select plateau:*: exit 2 with the forward-pointer message (Fork B).
• Referential faults (unbound axis, unresolvable ref): validate_before_register refuses BEFORE any store write (no store litter), reusing the shared seam.
• Data refusal (no local archive): the campaign member runner's existing no_real_data path (exit 1), same as generalize.

Testing strategy

• The committed anchor walkforward_real_e2e_pins_the_exact_current_grade (96dc783) is the equivalence gate: it must stay green through the dissolution (windows=9, stitched_total_pips, oos_r, param_stability means all exact).
• translate_walkforward unit tests (mirror the generalize translator tests): deterministic content ids for identical invocations; the pipeline is [Sweep(argmax), WalkForward] with the right roller sizes + metric; the risk regime carries the stop; diverging stops do not collide content ids; register_generated_wf stores documents the campaign can resolve.
• Dissolution-proof e2e (mirror generalize_dissolves_through_the_campaign_path): a successful real run auto-registers exactly one process + one campaign doc + one campaign-run record, and the walk_forward family is present.
• Front-end refusal tests: multi-value stop (exit 2), --select plateau (exit 2), non-r-sma (exit 2) — each byte-identical or forward-pointing.
• walkforward_summary_json_from_reports unit test: on a small fixture of per-window reports, its output equals the inline walkforward_summary_json on the same windows (the pure-sugar equivalence, in isolation).

Acceptance criteria

1. aura walkforward --strategy r-sma --real <SYM> --from --to --fast <csv> --slow <csv> --stop-length <v> --stop-k <v> runs through the campaign path; the anchor stays green (summary line byte-identical).
2. --strategy r-sma --real no longer flows through run_walkforward; it runs the sugar/campaign path. run_walkforward is retained (not deleted) and still reached for the fenced branches.
3. The fenced branches are behaviour-unchanged (decision 2 / Reading A): the blueprint-file path, the synthetic built-in path (any strategy), and the non-r-sma real path (--strategy sma --real) still run inline through run_walkforward exactly as before; every shared helper (walkforward_family, walkforward_summary_json, select_winner, …) is retained.
4. --select plateau:* is explicitly refused (exit 2) with a forward pointer; the decision-4 wf-stage plateau relaxation is filed as a follow-up issue.
5. Multi-value --stop-length/--stop-k is refused (Fork A).
6. Full workspace suite green; clippy --all-targets -D warnings clean.
7. No executor change (aura-campaign/aura-engine/aura-research diff-free for the argmax path).