Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Monte-Carlo and walk-forward over a loaded blueprint #170

New Issue
Closed
opened 2026-07-01 00:35:03 +02:00 by Brummel · 5 comments
Brummel commented 2026-07-01 00:35:03 +02:00
Owner
Copy Link
Copy Source

Cycle 0093 (#166) shipped aura sweep <blueprint.json> — families from a loaded
blueprint — but scoped Monte-Carlo and walk-forward over a loaded blueprint OUT
(the charter's "sweep first; MC/WF are the same machinery per verb, a fast
follow-on").

They are the same shape as the sweep arm: a .json-discriminator arm mirroring
aura mc / aura walkforward, building each member via cycle-1's wrap_stage1r
seam (reload-per-member, Composite is !Clone) over the loaded signal, with the
seed-range (MC) / window-config (WF) axis, aggregating to FamilyKind::MonteCarlo /
FamilyKind::WalkForward via append_family. Each member carries the shared
topology_hash. Part of the World/C21 milestone (families from blueprint-data);
the next family-verb cycle after sweep.

Cycle 0093 (#166) shipped `aura sweep <blueprint.json>` — families from a loaded blueprint — but scoped Monte-Carlo and walk-forward over a loaded blueprint OUT (the charter's "sweep first; MC/WF are the same machinery per verb, a fast follow-on"). They are the same shape as the sweep arm: a `.json`-discriminator arm mirroring `aura mc` / `aura walkforward`, building each member via cycle-1's `wrap_stage1r` seam (reload-per-member, `Composite` is !Clone) over the loaded signal, with the seed-range (MC) / window-config (WF) axis, aggregating to `FamilyKind::MonteCarlo` / `FamilyKind::WalkForward` via `append_family`. Each member carries the shared `topology_hash`. Part of the World/C21 milestone (families from blueprint-data); the next family-verb cycle after sweep.
Brummel added the feature label 2026-07-01 00:35:03 +02:00
Brummel added this to the World/C21: run + orchestrate harness families from blueprint-data milestone 2026-07-01 00:35:23 +02:00
Brummel commented 2026-07-01 03:16:23 +02:00
Author
Owner
Copy Link
Copy Source

Note from cycle 0094 (#158): reproduce needs the store-write hook per family verb

Cycle 0094 shipped content-addressed reproduction: aura reproduce <family-id> re-derives a
persisted sweep family from a content-addressed blueprint store (runs/blueprints/<hash>.json),
written once at the sweep persist seam (run_blueprint_sweep -> Registry::put_blueprint).

For this issue, MC / walk-forward over a loaded blueprint must write the blueprint to the same
store at ITS persist seam (the same put_blueprint(topology_hash, canonical) call the sweep
makes), or aura reproduce will report those families' members as "blueprint missing
from store" (exit 2, refuse-don't-guess). The reproduce path itself is verb-agnostic — it only
needs the stored blueprint keyed by the member's topology_hash, which every generated member
already carries.

## Note from cycle 0094 (#158): reproduce needs the store-write hook per family verb Cycle 0094 shipped content-addressed reproduction: `aura reproduce <family-id>` re-derives a persisted sweep family from a content-addressed blueprint store (`runs/blueprints/<hash>.json`), written once at the sweep persist seam (`run_blueprint_sweep` -> `Registry::put_blueprint`). For this issue, MC / walk-forward over a loaded blueprint must write the blueprint to the same store at ITS persist seam (the same `put_blueprint(topology_hash, canonical)` call the sweep makes), or `aura reproduce` will report those families' members as "blueprint <hash> missing from store" (exit 2, refuse-don't-guess). The reproduce path itself is verb-agnostic — it only needs the stored blueprint keyed by the member's `topology_hash`, which every generated member already carries.
Brummel commented 2026-07-01 11:35:55 +02:00
Author
Owner
Copy Link
Copy Source

Design record (specify, cycle 0095) — derived decisions + a grounding correction

Spec: 0095-mc-and-walkforward-over-a-loaded-blueprint. The forks below are
load-bearing and were not fully settled by this issue's body; a grounding
pass against the current code corrected the issue's premise and the resolution
is derived and recorded here (rationale, not user-provenance).

Grounding correction (the issue's premise, refined)

The body assumed run_blueprint_member(signal, point, space, sources, window, seed, pip, topo) being "already parameterized on both seed and window" makes
MC (vary seed) / WF (vary window) fall out for free. Reading the code: seed
and window currently reach only the manifest record (sim_optimal_manifest),
NOT the data — the member runs over the sources argument as given
(data.run_sources() = the fixed showcase series), and the sim graph is
deterministic (C1). So varying seed/window alone would produce N bit-identical
members with different labels — a degenerate family. The only source of
Monte-Carlo / walk-forward variation is re-drawing the input, exactly as the
existing hard-wired mc_family does (SyntheticSpec::source(seed) → a distinct
price walk per seed) and run_mc_r_bootstrap does (moving-block resample). This
correction is the spec's foundation; it is forced by C1, not a preference.

Fork: what the per-member realization facet drives

  • Decision → the facet drives the member's data source, not just the
    manifest: MC's seed selects a synthetic price walk; WF's window selects a
    rolling out-of-sample slice. run_blueprint_member gains its sources from the
    facet at each new persist seam.
    Basis (derived): C1 makes the graph deterministic, so family variation must
    live in the input; the in-repo precedents (mc_family, run_mc_r_bootstrap)
    already draw variation from re-sampled input.

Fork: MC data-generating process

  • Decision → per-seed synthetic price walk (SyntheticSpec::source(seed),
    the mc_family pattern), surfaced as aura mc <blueprint.json> --seeds N
    (seeds 1..=N), fixed default param point, FamilyKind::MonteCarlo. The walk is
    sized to warm the loaded graph (showcase-length regime), so members differ yet
    all produce trades.
    Basis (derived): it is the sole in-repo MC precedent (least surprise), and the
    deliverable here is the FamilyKind + content-addressed-store + reproduce
    machinery over blueprint-data — not trader-grade MC statistics. A block-bootstrap
    of a real recorded series is the statistically stronger form but rides the
    DataServer recorded-data seam (#124); it is filed as a follow-on, not this cycle.

Fork: WF semantics (IS-optimizing vs windowed evaluation)

  • Decision → windowed out-of-sample evaluation of the fixed loaded
    blueprint: rolling windows over the wf span, each member the loaded blueprint
    run over that window's OOS slice (windowed_sources), FamilyKind::WalkForward.
    No per-window in-sample re-optimization.
    Basis (derived): the issue frames it as "the window-config axis, the same
    machinery per verb", which is the windowed-resweep reading; the IS-optimizing
    form needs the loaded blueprint to declare sweepable axes to re-fit per window,
    which depends on axis-discovery (#169, unbuilt). Running a fixed strategy over
    rolling OOS windows is itself a recognized (simpler) walk-forward stability
    check; the IS-refit flavour is filed as a follow-on pending #169.
    Documented limitation: this is a windowed evaluation, not an in-sample-refit
    walk-forward.

Fork: reproduce compatibility (0094 path)

  • Decision → aura reproduce becomes realization-aware: it reconstructs
    a member's data from the facet recorded in its manifest, branching on the
    family kind — MonteCarlo → SyntheticSpec::source(manifest.seed);
    WalkForward → windowed_sources(manifest.window); Sweep → the showcase
    source at the full window (unchanged, keeps 0094 green). A persisted
    aura mc / aura walkforward family must reproduce bit-identically — the
    concrete proof each verb's store-write hook (put_blueprint) fired.
    Basis (derived): the manifest already stores window and seed, so the facet
    is recoverable; C1 makes the reconstruction bit-exact, and bit-identical
    reproduction is the acceptance criterion that proves the store hook fired.

Scope note

One spec covers both verbs plus the shared reproduce-realization-awareness;
the planner may split into iterations. Follow-on tracks filed forward: a
real-data block-bootstrap MC (rides #124) and an in-sample-refit walk-forward
(rides #169). Design settled — ready for spec production.

## Design record (specify, cycle 0095) — derived decisions + a grounding correction Spec: 0095-mc-and-walkforward-over-a-loaded-blueprint. The forks below are load-bearing and were **not** fully settled by this issue's body; a grounding pass against the current code corrected the issue's premise and the resolution is derived and recorded here (rationale, not user-provenance). ### Grounding correction (the issue's premise, refined) The body assumed `run_blueprint_member(signal, point, space, sources, window, seed, pip, topo)` being "already parameterized on both seed and window" makes MC (vary seed) / WF (vary window) fall out for free. Reading the code: `seed` and `window` currently reach only the **manifest record** (`sim_optimal_manifest`), NOT the data — the member runs over the `sources` argument as given (`data.run_sources()` = the fixed showcase series), and the sim graph is deterministic (C1). So varying seed/window alone would produce **N bit-identical members** with different labels — a degenerate family. The only source of Monte-Carlo / walk-forward variation is re-drawing the **input**, exactly as the existing hard-wired `mc_family` does (`SyntheticSpec::source(seed)` → a distinct price walk per seed) and `run_mc_r_bootstrap` does (moving-block resample). This correction is the spec's foundation; it is forced by C1, not a preference. ### Fork: what the per-member realization facet drives - **Decision** → the facet drives the member's **data source**, not just the manifest: MC's seed selects a synthetic price walk; WF's window selects a rolling out-of-sample slice. `run_blueprint_member` gains its sources from the facet at each new persist seam. Basis (derived): C1 makes the graph deterministic, so family variation must live in the input; the in-repo precedents (`mc_family`, `run_mc_r_bootstrap`) already draw variation from re-sampled input. ### Fork: MC data-generating process - **Decision** → per-seed **synthetic price walk** (`SyntheticSpec::source(seed)`, the `mc_family` pattern), surfaced as `aura mc <blueprint.json> --seeds N` (seeds 1..=N), fixed default param point, `FamilyKind::MonteCarlo`. The walk is sized to warm the loaded graph (showcase-length regime), so members differ yet all produce trades. Basis (derived): it is the sole in-repo MC precedent (least surprise), and the deliverable here is the FamilyKind + content-addressed-store + reproduce machinery over blueprint-data — not trader-grade MC statistics. A block-bootstrap of a **real** recorded series is the statistically stronger form but rides the DataServer recorded-data seam (#124); it is filed as a follow-on, not this cycle. ### Fork: WF semantics (IS-optimizing vs windowed evaluation) - **Decision** → **windowed out-of-sample evaluation** of the fixed loaded blueprint: rolling windows over the wf span, each member the loaded blueprint run over that window's OOS slice (`windowed_sources`), `FamilyKind::WalkForward`. **No per-window in-sample re-optimization.** Basis (derived): the issue frames it as "the window-config axis, the same machinery per verb", which is the windowed-resweep reading; the IS-optimizing form needs the loaded blueprint to declare sweepable axes to re-fit per window, which depends on axis-discovery (#169, unbuilt). Running a fixed strategy over rolling OOS windows is itself a recognized (simpler) walk-forward stability check; the IS-refit flavour is filed as a follow-on pending #169. Documented limitation: this is a windowed evaluation, not an in-sample-refit walk-forward. ### Fork: reproduce compatibility (0094 path) - **Decision** → `aura reproduce` becomes **realization-aware**: it reconstructs a member's data from the facet recorded in its manifest, branching on the family kind — `MonteCarlo` → `SyntheticSpec::source(manifest.seed)`; `WalkForward` → `windowed_sources(manifest.window)`; `Sweep` → the showcase source at the full window (unchanged, keeps 0094 green). A persisted `aura mc` / `aura walkforward` family must reproduce bit-identically — the concrete proof each verb's store-write hook (`put_blueprint`) fired. Basis (derived): the manifest already stores `window` and `seed`, so the facet is recoverable; C1 makes the reconstruction bit-exact, and bit-identical reproduction is the acceptance criterion that proves the store hook fired. ### Scope note One spec covers both verbs plus the shared reproduce-realization-awareness; the planner may split into iterations. Follow-on tracks filed forward: a real-data block-bootstrap MC (rides #124) and an in-sample-refit walk-forward (rides #169). Design settled — ready for spec production.
Brummel commented 2026-07-01 11:53:05 +02:00
Author
Owner
Copy Link
Copy Source

Decision update (specify, cycle 0095) — two review gates narrowed the scope

The 0095 draft (both mc and walkforward over a loaded blueprint, with a
synthesized default param point) went through two independent read-only gates.
Both blocked; the resolution narrows this cycle to Monte-Carlo over a loaded
CLOSED blueprint and revises the earlier decision record above. This is the
honest, grounded scope.

Gate 1 — scope-skeptic BLOCK: walk-forward verb-name collision

The hard-wired aura walkforward performs a true per-window in-sample
re-optimization (sweep_over(is) → select_winner → run_oos). A loaded blueprint
declares no sweepable axes yet (that is #169), so a loaded-blueprint walkforward
could only be a fixed-blueprint windowed out-of-sample evaluation — no IS
refit. Shipping that under the identical verb name (split only by a .json
suffix) lets one trading term name two categorically different guarantees.

  • Decision → defer walk-forward. The honest walk-forward over a loaded
    blueprint is the IS-refit form, which needs axis-discovery first; it is tracked
    by #173 (deps #169), not forced into this cycle under a colliding name. The
    earlier "WF = windowed evaluation under the walkforward verb" call is withdrawn.

Gate 2 — grounding BLOCK: no default_point mechanism; unbound-knob path panics

The earlier record assumed a per-knob "declared default" and that aura run
emits a named error on an unbound knob. Grounding falsified both: ParamSpec
carries no default, compile_with_params requires params.len() == space.len(),
and aura run <bp> only bootstraps a closed blueprint (empty params) — an
unbound knob panics (compile_with_params(..).expect(..)), it does not emit a
clean error.

  • Decision → MC runs a closed blueprint over the empty point (&[]),
    exactly as aura run <bp> does; an open blueprint (non-empty wrapped
    param_space) is rejected with a named aura: error + exit 2 before any
    run, pre-empting the panic. No synthesized default point.

Net for #170

This cycle closes the Monte-Carlo half of #170 (aura mc <closed blueprint.json> --seeds N → a FamilyKind::MonteCarlo family, seed → synthetic
walk, shared topology_hash, content-addressed store hook, aura reproduce
bit-identical). The walk-forward half remains open, tracked forward by #173
(deps #169). The milestone therefore does not fully close this cycle.
Spec re-grounded to PASS after the revision.

## Decision update (specify, cycle 0095) — two review gates narrowed the scope The 0095 draft (both `mc` and `walkforward` over a loaded blueprint, with a synthesized default param point) went through two independent read-only gates. Both blocked; the resolution **narrows this cycle to Monte-Carlo over a loaded CLOSED blueprint** and revises the earlier decision record above. This is the honest, grounded scope. ### Gate 1 — scope-skeptic BLOCK: walk-forward verb-name collision The hard-wired `aura walkforward` performs a true per-window **in-sample re-optimization** (`sweep_over(is) → select_winner → run_oos`). A loaded blueprint declares no sweepable axes yet (that is #169), so a loaded-blueprint `walkforward` could only be a *fixed-blueprint windowed out-of-sample evaluation* — **no IS refit**. Shipping that under the identical verb name (split only by a `.json` suffix) lets one trading term name two categorically different guarantees. - **Decision** → **defer walk-forward**. The honest walk-forward over a loaded blueprint is the IS-refit form, which needs axis-discovery first; it is tracked by #173 (deps #169), not forced into this cycle under a colliding name. The earlier "WF = windowed evaluation under the walkforward verb" call is withdrawn. ### Gate 2 — grounding BLOCK: no `default_point` mechanism; unbound-knob path panics The earlier record assumed a per-knob "declared default" and that `aura run` emits a named error on an unbound knob. Grounding falsified both: `ParamSpec` carries no default, `compile_with_params` requires `params.len() == space.len()`, and `aura run <bp>` only bootstraps a **closed** blueprint (empty params) — an unbound knob **panics** (`compile_with_params(..).expect(..)`), it does not emit a clean error. - **Decision** → MC runs a **closed** blueprint over the **empty point** (`&[]`), exactly as `aura run <bp>` does; an **open** blueprint (non-empty wrapped `param_space`) is **rejected with a named `aura:` error + exit 2** before any run, pre-empting the panic. No synthesized default point. ### Net for #170 This cycle closes the **Monte-Carlo half** of #170 (`aura mc <closed blueprint.json> --seeds N` → a `FamilyKind::MonteCarlo` family, seed → synthetic walk, shared `topology_hash`, content-addressed store hook, `aura reproduce` bit-identical). The **walk-forward half remains open**, tracked forward by #173 (deps #169). The milestone therefore does not fully close this cycle. Spec re-grounded to PASS after the revision.
Brummel commented 2026-07-01 13:27:24 +02:00
Author
Owner
Copy Link
Copy Source

Status: the Monte-Carlo half shipped (cycle 0095); walk-forward remains

aura mc <closed blueprint.json> --seeds N shipped in cycle 0095 (feat commit
14e27a9, audit-close ebebc24; unpushed): a FamilyKind::MonteCarlo family
from a loaded closed blueprint, each seed a distinct synthetic price walk
(disjoint-parallel via the engine monte_carlo seam), one stored blueprint per
family keyed by topology_hash, and aura reproduces bit-identically. Verified
green (build + clippy + full suite, 5 new tests). refs #170, not closes.

This issue stays OPEN for the walk-forward half, which is a design decision,
not mechanical execution — tracked forward by #173 (dep #169). The fork: a loaded
blueprint declares no sweepable axes, so a loaded-blueprint walkforward would be
a fixed-blueprint windowed out-of-sample evaluation with no in-sample refit,
whereas the hard-wired aura walkforward re-optimizes params per window
(sweep_over(is) → select_winner → run_oos). Shipping the former under the same
verb name is a guarantee collision. Resolving it needs a direction call —
IS-refit walk-forward (which needs axis-discovery, #169, to know what to re-fit)
vs. a distinctly-named windowed-evaluation verb — so the World/C21 milestone's
last piece is paused here for that decision rather than built toward one arm
autonomously.

## Status: the Monte-Carlo half shipped (cycle 0095); walk-forward remains `aura mc <closed blueprint.json> --seeds N` shipped in cycle 0095 (feat commit `14e27a9`, audit-close `ebebc24`; unpushed): a `FamilyKind::MonteCarlo` family from a loaded closed blueprint, each seed a distinct synthetic price walk (disjoint-parallel via the engine `monte_carlo` seam), one stored blueprint per family keyed by `topology_hash`, and `aura reproduce`s bit-identically. Verified green (build + clippy + full suite, 5 new tests). `refs #170`, not `closes`. **This issue stays OPEN for the walk-forward half**, which is a design decision, not mechanical execution — tracked forward by #173 (dep #169). The fork: a loaded blueprint declares no sweepable axes, so a loaded-blueprint `walkforward` would be a fixed-blueprint windowed out-of-sample evaluation with **no in-sample refit**, whereas the hard-wired `aura walkforward` re-optimizes params per window (`sweep_over(is) → select_winner → run_oos`). Shipping the former under the same verb name is a guarantee collision. Resolving it needs a direction call — IS-refit walk-forward (which needs axis-discovery, #169, to know what to re-fit) vs. a distinctly-named windowed-evaluation verb — so the World/C21 milestone's last piece is paused here for that decision rather than built toward one arm autonomously.
Brummel commented 2026-07-01 17:51:04 +02:00
Author
Owner
Copy Link
Copy Source

Closing as delivered — the MC + walk-forward-over-a-loaded-blueprint deliverable has fully shipped:

  • Monte-Carlo over a loaded blueprint — aura mc <blueprint.json> --seeds N — shipped in cycle 0095 (14e27a9).
  • IS-refit walk-forward over a loaded blueprint — aura walkforward <blueprint.json> --axis <name>=<csv> [--select …] — shipped in cycle 0097 (ff20f74), the milestone capstone: re-fit the --axis grid per in-sample window, run the winner out-of-sample, aggregate to FamilyKind::WalkForward, persisted content-addressed and reproducible bit-identically.

Both were validated end-to-end by the World/C21 milestone field test — all three family kinds (Sweep / MonteCarlo / WalkForward) reproduce bit-identically from consumer-authored blueprints. The two bugs it surfaced are fixed (#176, #177). The fix commits linked here with refs #170 (non-closing), so this stayed open; closing it now completes the milestone.

Closing as delivered — the MC + walk-forward-over-a-loaded-blueprint deliverable has fully shipped: - **Monte-Carlo** over a loaded blueprint — `aura mc <blueprint.json> --seeds N` — shipped in cycle 0095 (`14e27a9`). - **IS-refit walk-forward** over a loaded blueprint — `aura walkforward <blueprint.json> --axis <name>=<csv> [--select …]` — shipped in cycle 0097 (`ff20f74`), the milestone capstone: re-fit the `--axis` grid per in-sample window, run the winner out-of-sample, aggregate to `FamilyKind::WalkForward`, persisted content-addressed and reproducible bit-identically. Both were validated end-to-end by the World/C21 milestone field test — all three family kinds (Sweep / MonteCarlo / WalkForward) reproduce bit-identically from consumer-authored blueprints. The two bugs it surfaced are fixed (#176, #177). The fix commits linked here with `refs #170` (non-closing), so this stayed open; closing it now completes the milestone.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone World/C21: run + orchestrate harness families from blueprint-data
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#170
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?