Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

In-sample-refit walk-forward over a loaded blueprint (per-window re-optimization) #173

New Issue
Closed
opened 2026-07-01 11:36:30 +02:00 by Brummel · 2 comments
Brummel commented 2026-07-01 11:36:30 +02:00
Owner
Copy Link
Copy Source

Cycle 0095 (#170) shipped aura walkforward <blueprint.json> as a windowed out-of-sample evaluation of the fixed loaded blueprint (rolling OOS windows, no per-window in-sample re-optimization), FamilyKind::WalkForward. The classic IS-refit walk-forward re-optimizes the blueprint's params on each in-sample slice, then runs the winner OOS — which requires the loaded blueprint to declare its sweepable axes so each window can re-fit them.

depends on: #169 (discover a loaded blueprint's sweepable axis names)
context: deferred from #170 cycle 0095 — the windowed-eval form ships the machinery; the IS-refit flavour needs axis-discovery to re-fit per window.

Cycle 0095 (#170) shipped `aura walkforward <blueprint.json>` as a **windowed out-of-sample evaluation** of the *fixed* loaded blueprint (rolling OOS windows, no per-window in-sample re-optimization), FamilyKind::WalkForward. The classic IS-refit walk-forward re-optimizes the blueprint's params on each in-sample slice, then runs the winner OOS — which requires the loaded blueprint to declare its sweepable axes so each window can re-fit them. depends on: #169 (discover a loaded blueprint's sweepable axis names) context: deferred from #170 cycle 0095 — the windowed-eval form ships the machinery; the IS-refit flavour needs axis-discovery to re-fit per window.
Brummel added the idea label 2026-07-01 11:36:30 +02:00
Brummel commented 2026-07-01 13:43:18 +02:00
Author
Owner
Copy Link
Copy Source

Direction chosen: Arm A — real IS-refit walk-forward (user decision)

The World/C21 milestone's remaining walk-forward half is settled to Arm A:
a true in-sample-refit walk-forward over a loaded blueprint (re-optimize params
per IS window, run the winner OOS), not a fixed-blueprint windowed evaluation
under a new verb name (Arm B) and not dropping WF (Arm C).

Basis (user decision): "Natürlich A!" — @Brummel, 2026-07-01.

Sequencing: this issue (IS-refit WF) depends on #169 (discover a loaded
blueprint's sweepable axis names) — the harness must know which axes to re-fit
per window. So the next cycle is #169 (axis discovery), then this issue (#173)
builds the IS-refit walk-forward on top of it. Arm A is the only variant that
honestly carries the "walk-forward" name (matching the hard-wired verb's
IS-optimizing guarantee), and #169 is independently useful (a user wants a
blueprint's --axis names before sweeping).

## Direction chosen: Arm A — real IS-refit walk-forward (user decision) The World/C21 milestone's remaining walk-forward half is settled to **Arm A**: a true in-sample-refit walk-forward over a loaded blueprint (re-optimize params per IS window, run the winner OOS), *not* a fixed-blueprint windowed evaluation under a new verb name (Arm B) and *not* dropping WF (Arm C). Basis (user decision): "Natürlich A!" — @Brummel, 2026-07-01. Sequencing: this issue (IS-refit WF) depends on **#169** (discover a loaded blueprint's sweepable axis names) — the harness must know which axes to re-fit per window. So the next cycle is #169 (axis discovery), then this issue (#173) builds the IS-refit walk-forward on top of it. Arm A is the only variant that honestly carries the "walk-forward" name (matching the hard-wired verb's IS-optimizing guarantee), and #169 is independently useful (a user wants a blueprint's `--axis` names before sweeping).
Brummel commented 2026-07-01 15:06:23 +02:00
Author
Owner
Copy Link
Copy Source

Design reconciliation + derived decisions (specify, cycle 0097)

Spec: 0097-is-refit-walk-forward-over-a-loaded-blueprint. Direction is Arm A
(user decision already on this issue: "Natürlich A!" — @Brummel, 2026-07-01).
This records a premise correction + the derived sub-forks (rationale, not
user-provenance) before the spec is written.

Premise correction (this issue's body is stale)

The body states "Cycle 0095 (#170) shipped aura walkforward <blueprint.json>
as a windowed out-of-sample evaluation ... FamilyKind::WalkForward." That did
not ship. Verified against the current tree: cycle 0095 shipped only
Monte-Carlo (aura mc <bp.json> --seeds N); the ["walkforward", ..] dispatch
(crates/aura-cli/src/main.rs) routes ONLY to the hard-wired parse_walkforward_args
(built-in strategies) — there is no .json-discriminator arm, so no
loaded-blueprint walk-forward of any kind exists. So this cycle builds the
loaded-blueprint WF fresh, in the Arm-A (IS-refit) form — not "adds IS-refit
to an existing windowed-eval." (The earlier Arm-B windowed-eval reading was a
0095 draft decision that was withdrawn when the MC/WF cycle narrowed to MC-only.)

Derived sub-forks (all decided by analogy to the hard-wired Stage1R

walk-forward + the loaded-blueprint sweep; each Basis is a groundable rationale)

  • Fork: what re-fits per window → the loaded blueprint's params, over the
    user's --axis <name>=<csv> grid (the #169 prefixed names, e.g.
    stage1_signal.fast.length). Basis (derived): the hard-wired Stage1R WF arm
    re-fits a grid per IS window (stage1_r_sweep_over → select_winner → run_oos_r);
    the loaded-blueprint form is the same, sourcing its axis namespace from the
    cycle-0096 blueprint_axis_probe (single-sourced with the sweep). An OPEN
    blueprint + ≥1 --axis is required (something to re-fit), mirroring the
    loaded-blueprint sweep's contract; a closed/axis-less invocation is a usage
    error (exit 2), not a degenerate run.
  • Fork: the roller driver → reuse the generic engine walk_forward(roller, space, run_window) verbatim; only the per-window closure body changes (loaded
    blueprint). Basis (derived): the driver is strategy-agnostic — the hard-wired
    SmaCross and Stage1R arms already differ only in their closure body.
  • Fork: window schedule → mirror the hard-wired synthetic default (24-bar IS
    / 12-bar OOS / step 12 over the 60-bar span) via the same wf_full_span() /
    wf_window_sizes() data-source seam. Basis (derived): the loaded-blueprint
    sweep/mc already mirror the hard-wired data regime; the WF should not invent a
    second schedule.
  • Fork: selection metric + objective → sqn_normalized (the Stage1R arm's
    metric — the loaded signal is wrapped in the SAME wrap_stage1r scaffolding,
    so its members carry it), select_winner reused verbatim, --select argmax|plateau:mean|plateau:worst (default argmax). Plateau needs the grid
    lattice, so the per-window IS-sweep uses sweep_with_lattice (as the
    hard-wired stage1_r_sweep_over does). Basis (derived): full parity with the
    hard-wired WF's selection surface, least surprise.
  • Fork: iteration-1 scope (persist ↔ reproduce) → iteration 1 ships the FULL
    loaded-blueprint IS-refit WF: build → FamilyKind::WalkForward → persist
    (put_blueprint store hook + append_family) → per-window OOS member lines +
    walkforward_summary_json, AND makes aura reproduce WalkForward-aware
    (reconstruct each OOS window's windowed source + the window's winner params
    from its member manifest, mirroring the MonteCarlo reproduce branch added in
    cycle 0095). Basis (derived): persist and reproduce are coupled — a persisted
    family with no store hook reports members as "blueprint missing" (the #170
    hazard), so shipping persist without reproduce-awareness would ship a broken
    reproduce. Cycle 0095 (MC) delivered family+verb+persist+reproduce in one
    cycle at comparable size; this mirrors it. (If the plan judges it too large,
    it may split argmax-core vs plateau/trace across implement iterations WITHIN
    this cycle — but persist+reproduce stay together.)

Verb surface: aura walkforward <blueprint.json> --axis <name>=<csv> [--axis …] [--select <argmax|plateau:mean|plateau:worst>] [--name <n>|--trace <n>] — a
.json-discriminator arm on the existing ["walkforward", ..] dispatch,
mirroring the sweep/mc .json arms. Closes the World/C21 milestone's last piece.

## Design reconciliation + derived decisions (specify, cycle 0097) Spec: 0097-is-refit-walk-forward-over-a-loaded-blueprint. Direction is Arm A (user decision already on this issue: "Natürlich A!" — @Brummel, 2026-07-01). This records a premise correction + the derived sub-forks (rationale, not user-provenance) before the spec is written. ### Premise correction (this issue's body is stale) The body states "Cycle 0095 (#170) shipped `aura walkforward <blueprint.json>` as a windowed out-of-sample evaluation ... FamilyKind::WalkForward." **That did not ship.** Verified against the current tree: cycle 0095 shipped only Monte-Carlo (`aura mc <bp.json> --seeds N`); the `["walkforward", ..]` dispatch (crates/aura-cli/src/main.rs) routes ONLY to the hard-wired `parse_walkforward_args` (built-in strategies) — there is **no** `.json`-discriminator arm, so no loaded-blueprint walk-forward of any kind exists. So this cycle builds the loaded-blueprint WF **fresh**, in the Arm-A (IS-refit) form — not "adds IS-refit to an existing windowed-eval." (The earlier Arm-B windowed-eval reading was a 0095 draft decision that was withdrawn when the MC/WF cycle narrowed to MC-only.) ### Derived sub-forks (all decided by analogy to the hard-wired Stage1R walk-forward + the loaded-blueprint sweep; each Basis is a groundable rationale) - **Fork: what re-fits per window** → the loaded blueprint's params, over the user's `--axis <name>=<csv>` grid (the #169 prefixed names, e.g. `stage1_signal.fast.length`). Basis (derived): the hard-wired Stage1R WF arm re-fits a grid per IS window (`stage1_r_sweep_over → select_winner → run_oos_r`); the loaded-blueprint form is the same, sourcing its axis namespace from the cycle-0096 `blueprint_axis_probe` (single-sourced with the sweep). An OPEN blueprint + ≥1 `--axis` is required (something to re-fit), mirroring the loaded-blueprint sweep's contract; a closed/axis-less invocation is a usage error (exit 2), not a degenerate run. - **Fork: the roller driver** → reuse the generic engine `walk_forward(roller, space, run_window)` verbatim; only the per-window closure body changes (loaded blueprint). Basis (derived): the driver is strategy-agnostic — the hard-wired SmaCross and Stage1R arms already differ only in their closure body. - **Fork: window schedule** → mirror the hard-wired synthetic default (24-bar IS / 12-bar OOS / step 12 over the 60-bar span) via the same `wf_full_span()` / `wf_window_sizes()` data-source seam. Basis (derived): the loaded-blueprint sweep/mc already mirror the hard-wired data regime; the WF should not invent a second schedule. - **Fork: selection metric + objective** → `sqn_normalized` (the Stage1R arm's metric — the loaded signal is wrapped in the SAME `wrap_stage1r` scaffolding, so its members carry it), `select_winner` reused verbatim, `--select argmax|plateau:mean|plateau:worst` (default argmax). Plateau needs the grid lattice, so the per-window IS-sweep uses `sweep_with_lattice` (as the hard-wired `stage1_r_sweep_over` does). Basis (derived): full parity with the hard-wired WF's selection surface, least surprise. - **Fork: iteration-1 scope (persist ↔ reproduce)** → iteration 1 ships the FULL loaded-blueprint IS-refit WF: build → `FamilyKind::WalkForward` → **persist** (`put_blueprint` store hook + `append_family`) → per-window OOS member lines + `walkforward_summary_json`, AND makes `aura reproduce` **WalkForward-aware** (reconstruct each OOS window's windowed source + the window's winner params from its member manifest, mirroring the MonteCarlo reproduce branch added in cycle 0095). Basis (derived): persist and reproduce are coupled — a persisted family with no store hook reports members as "blueprint missing" (the #170 hazard), so shipping persist without reproduce-awareness would ship a broken `reproduce`. Cycle 0095 (MC) delivered family+verb+persist+reproduce in one cycle at comparable size; this mirrors it. (If the plan judges it too large, it may split argmax-core vs plateau/trace across implement iterations WITHIN this cycle — but persist+reproduce stay together.) Verb surface: `aura walkforward <blueprint.json> --axis <name>=<csv> [--axis …] [--select <argmax|plateau:mean|plateau:worst>] [--name <n>|--trace <n>]` — a `.json`-discriminator arm on the existing `["walkforward", ..]` dispatch, mirroring the sweep/mc `.json` arms. Closes the World/C21 milestone's last piece.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label idea
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#173
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?