Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Orchestrate harness families from a loaded blueprint (sweep / MC / walk-forward) #166

New Issue
Closed
opened 2026-06-30 18:02:08 +02:00 by Brummel · 3 comments
Brummel commented 2026-06-30 18:02:08 +02:00
Owner
Copy Link
Copy Source

Cycle 2 of the World/C21 milestone: the World orchestrates families of harnesses built from a loaded blueprint — the meta-level differentiator (C21: families, not the single backtest).

Deliverable

sweep / Monte-Carlo / walk-forward orchestration that accepts a loaded blueprint (the cycle-1 aura run load path) plus the family's axes, builds one harness instance per family member from the blueprint-data, runs them (C1: members are disjoint → parallel), and aggregates to the same FamilyRunRecord shape the hard-wired *_sweep_family paths produce today.

Acceptance (to be sharpened at spec time)

  • A sweep over a loaded blueprint produces a FamilyRunRecord byte-identical to the equivalent hard-wired-path family run (same members, same aggregation).
  • Members are built via the cycle-1 blueprint load path (one source of construction), not a parallel re-implementation.
  • Persists to the family store (families.jsonl) unchanged.

Notes

  • This cycle makes the orchestration data-driven (a blueprint + axes), but does NOT yet retire the hard-wired CLI verbs (#159) — that retirement is gated on the project-as-crate authoring layer (a follow-on milestone), so a local authoring path always exists (invariant 12).
  • Turning the axes themselves into composable builder tools (nesting / chaining) is follow-on surface, out of scope here.

Depends on the cycle-1 run-from-data load path. Part of the World/C21 milestone.

Cycle 2 of the World/C21 milestone: the World orchestrates **families** of harnesses built from a loaded blueprint — the meta-level differentiator (C21: families, not the single backtest). ## Deliverable sweep / Monte-Carlo / walk-forward orchestration that accepts a **loaded blueprint** (the cycle-1 `aura run` load path) plus the family's axes, builds one harness instance per family member from the blueprint-data, runs them (C1: members are disjoint → parallel), and aggregates to the **same `FamilyRunRecord` shape** the hard-wired `*_sweep_family` paths produce today. ## Acceptance (to be sharpened at spec time) - [ ] A sweep over a loaded blueprint produces a `FamilyRunRecord` byte-identical to the equivalent hard-wired-path family run (same members, same aggregation). - [ ] Members are built via the cycle-1 blueprint load path (one source of construction), not a parallel re-implementation. - [ ] Persists to the family store (`families.jsonl`) unchanged. ## Notes - This cycle makes the orchestration **data-driven** (a blueprint + axes), but does NOT yet retire the hard-wired CLI verbs (#159) — that retirement is gated on the project-as-crate authoring layer (a follow-on milestone), so a local authoring path always exists (invariant 12). - Turning the axes themselves into composable builder tools (nesting / chaining) is follow-on surface, out of scope here. Depends on the cycle-1 run-from-data load path. Part of the World/C21 milestone.
Brummel added this to the World/C21: run + orchestrate harness families from blueprint-data milestone 2026-06-30 18:02:08 +02:00
Brummel added the feature label 2026-06-30 18:02:08 +02:00
Brummel commented 2026-06-30 22:50:59 +02:00
Author
Owner
Copy Link
Copy Source

Derived design decisions (cycle-2 grounding, specify Step 1.5)

A read-only recon confirmed cycle 2 (families from a loaded blueprint) is a
DERIVABLE extension of the existing sweep machinery — no design fork. The
member-construction (wrap_stage1r(loaded_signal) → param_space() →
SweepBinder.axis(name, values).sweep(|point| bootstrap_with_cells(point))) and
the family store (append_family) are blueprint-agnostic; only the signal source
and the axis-naming surface change.

Decisions:

  • Scope — sweep first (keystone family). This cycle ships aura sweep <blueprint.json>; Monte-Carlo and walk-forward over a loaded blueprint are the
    same machinery repeated per verb and follow as a fast follow-on (their own
    small cycle), so cycle 2 stays tractable.
  • CLI — by-name --axis <name>=<csv>. A loaded signal is arbitrary (not only
    stage1-r's fast/slow), so axes are user-supplied by name, resolved
    against the loaded blueprint's param_space() (the existing resolve_axes
    by-name binding). The hard-wired --strategy stage1-r --fast/--slow path stays
    untouched (#159). The .json-file discriminator mirrors cycle 1's
    aura run <blueprint.json> arm.
  • Member topology_hash. Each member manifest carries
    topology_hash(&loaded_signal) — all members of a sweep over one signal share
    it (same topology, only params vary; member_key distinguishes members). This
    keeps a generated member reproducible (C18/C24), the natural consequence of
    topology-as-data. (Hard-wired members keep None — their topology is baked,
    identified by commit.)
  • Aggregation unchanged. FamilyKind::Sweep + append_family → families.jsonl,
    byte-identical shape to the hard-wired sweep.

Ready for spec production.

## Derived design decisions (cycle-2 grounding, specify Step 1.5) A read-only recon confirmed cycle 2 (families from a loaded blueprint) is a DERIVABLE extension of the existing sweep machinery — no design fork. The member-construction (`wrap_stage1r(loaded_signal)` → `param_space()` → `SweepBinder.axis(name, values).sweep(|point| bootstrap_with_cells(point))`) and the family store (`append_family`) are blueprint-agnostic; only the signal source and the axis-naming surface change. **Decisions:** - **Scope — sweep first (keystone family).** This cycle ships `aura sweep <blueprint.json>`; Monte-Carlo and walk-forward over a loaded blueprint are the same machinery repeated per verb and follow as a fast follow-on (their own small cycle), so cycle 2 stays tractable. - **CLI — by-name `--axis <name>=<csv>`.** A loaded signal is arbitrary (not only stage1-r's `fast`/`slow`), so axes are user-supplied **by name**, resolved against the loaded blueprint's `param_space()` (the existing `resolve_axes` by-name binding). The hard-wired `--strategy stage1-r --fast/--slow` path stays untouched (#159). The `.json`-file discriminator mirrors cycle 1's `aura run <blueprint.json>` arm. - **Member topology_hash.** Each member manifest carries `topology_hash(&loaded_signal)` — all members of a sweep over one signal share it (same topology, only params vary; `member_key` distinguishes members). This keeps a generated member reproducible (C18/C24), the natural consequence of topology-as-data. (Hard-wired members keep `None` — their topology is baked, identified by `commit`.) - **Aggregation unchanged.** `FamilyKind::Sweep` + `append_family` → `families.jsonl`, byte-identical shape to the hard-wired sweep. Ready for spec production.
Brummel commented 2026-06-30 23:04:33 +02:00
Author
Owner
Copy Link
Copy Source

Grounding refinements from plan-recon (derived decisions)

The recon confirmed the mirror is a clean reuse but surfaced five design points;
all derivable, resolved here.

Q1 (load-bearing) — Composite is !Clone, so the signal is re-loaded per
member. Composite cannot derive Clone (its PrimitiveBuilders hold a
build: Box<dyn Fn> closure — the same reason it carries no PartialEq/Debug,
#164), and bootstrap_with_cells(self, point) consumes the graph. The hard-wired
stage1_r_sweep_family gets a fresh per-member graph for free by re-calling
stage1_r_graph(...) inside its .sweep closure. Decision:
run_blueprint_sweep takes the serialized doc (&str, Send+Sync) and
re-loads a fresh signal via blueprint_from_json(doc, &|t| std_vocabulary(t))
inside the closure, per member — cheap relative to a backtest. topology_hash is
computed once from the doc.

Q2 — stop_open=false, reduce=true, cost=None. The loaded signal is a
price→bias leg with NO stop knobs; the vol-stop lives in wrap_stage1r's
scaffolding, bound to the R-defining STAGE1_R_STOP_* constants (stop_open=false).
The user sweeps the signal's own open knobs (--axis fast.length=…). reduce=true
(folded, mirroring reduce = trace.is_none()); cost=None. (Sweeping the stop would
need stop_open=true; the spec shows only signal axes, so it stays bound.)

Q3 — the family-equivalence test compares each member to run_signal_stage1r
per point, NOT to the hard-wired sweep. The hard-wired stage1_r_sweep_family
runs stop_open=true (the open-stop scaffolding); a blueprint sweep runs
stop_open=false. The clean, exact equivalence is: each run_blueprint_sweep
member == run_signal_stage1r(signal, &point_params, …) (cycle 1's single run —
the SAME stop_open=false scaffolding), trivially identical. This is "the loaded
path agrees with the Rust path", the honest acceptance.

Q4 — parser stays pure → DataChoice; the arm converts.
parse_sweep_blueprint_args returns a DataChoice (via the existing
RealWindowGrammar accumulator), and the dispatch arm converts with
DataSource::from_choice before calling run_blueprint_sweep(… data: DataSource)
— mirroring the existing sweep arm.

Q5 — --axis <name>=<csv> lexes each token to a Scalar by shape. A by-name
axis has no statically-known kind at parse time. Lex: integer-shaped token →
Scalar::i64, else Scalar::f64 (try-i64-then-f64). resolve_axes then
kind-checks each value against the slot's declared ParamSpec.kind; a mismatch is
a clean named error (the KindMismatch arm), an unknown axis name a clean
UnknownKnob — neither panics.

## Grounding refinements from plan-recon (derived decisions) The recon confirmed the mirror is a clean reuse but surfaced five design points; all derivable, resolved here. **Q1 (load-bearing) — `Composite` is `!Clone`, so the signal is re-loaded per member.** `Composite` cannot derive `Clone` (its `PrimitiveBuilder`s hold a `build: Box<dyn Fn>` closure — the same reason it carries no `PartialEq`/`Debug`, #164), and `bootstrap_with_cells(self, point)` consumes the graph. The hard-wired `stage1_r_sweep_family` gets a fresh per-member graph for free by re-calling `stage1_r_graph(...)` inside its `.sweep` closure. **Decision:** `run_blueprint_sweep` takes the serialized **doc** (`&str`, `Send+Sync`) and re-loads a fresh signal via `blueprint_from_json(doc, &|t| std_vocabulary(t))` inside the closure, per member — cheap relative to a backtest. `topology_hash` is computed once from the doc. **Q2 — `stop_open=false`, `reduce=true`, `cost=None`.** The loaded signal is a `price→bias` leg with NO stop knobs; the vol-stop lives in `wrap_stage1r`'s scaffolding, bound to the R-defining `STAGE1_R_STOP_*` constants (`stop_open=false`). The user sweeps the signal's own open knobs (`--axis fast.length=…`). `reduce=true` (folded, mirroring `reduce = trace.is_none()`); `cost=None`. (Sweeping the stop would need `stop_open=true`; the spec shows only signal axes, so it stays bound.) **Q3 — the family-equivalence test compares each member to `run_signal_stage1r` per point, NOT to the hard-wired sweep.** The hard-wired `stage1_r_sweep_family` runs `stop_open=true` (the open-stop scaffolding); a blueprint sweep runs `stop_open=false`. The clean, exact equivalence is: each `run_blueprint_sweep` member == `run_signal_stage1r(signal, &point_params, …)` (cycle 1's single run — the SAME `stop_open=false` scaffolding), trivially identical. This is "the loaded path agrees with the Rust path", the honest acceptance. **Q4 — parser stays pure → `DataChoice`; the arm converts.** `parse_sweep_blueprint_args` returns a `DataChoice` (via the existing `RealWindowGrammar` accumulator), and the dispatch arm converts with `DataSource::from_choice` before calling `run_blueprint_sweep(… data: DataSource)` — mirroring the existing sweep arm. **Q5 — `--axis <name>=<csv>` lexes each token to a `Scalar` by shape.** A by-name axis has no statically-known kind at parse time. Lex: integer-shaped token → `Scalar::i64`, else `Scalar::f64` (try-i64-then-f64). `resolve_axes` then kind-checks each value against the slot's declared `ParamSpec.kind`; a mismatch is a clean named error (the `KindMismatch` arm), an unknown axis name a clean `UnknownKnob` — neither panics.
Brummel commented 2026-07-01 01:03:42 +02:00
Author
Owner
Copy Link
Copy Source

Investigated: reduce vs full R-equivalence at window-end — no divergence

A follow-up flagged that the reduce-mode sweep path (GatedRecorder) might report a
different R from a full single run (Recorder) for a position left open at the window's
end (n_open_at_end). Investigated — there is no divergence; the equivalence holds for
every window-end shape, and it is pinned by three currently-green tests:

  • gated_recorder_then_summarize_r_equals_raw_summarize_r
    (crates/aura-engine/tests/streaming_reduction_equivalence.rs) — unit, drives a record
    that ends in a non-gated open-at-end row; asserts summarize_r(gated) == summarize_r(full),
    n_open_at_end == 1 included.
  • stage1_r_single_run_output_golden (crates/aura-cli/tests/cli_run.rs) — pins
    n_open_at_end:1 for the default single run at fast=2/slow=4 (so that param is
    open-at-end, not a closed-flat ending).
  • blueprint_sweep_member_equals_single_run_and_shares_topology_hash
    (crates/aura-cli/src/main.rs) — integration: a reduce-mode sweep member's full metrics
    equals a full-recorder single run at fast=2/slow=4 (the open-at-end param above).

Mechanism: GatedRecorder.finalize (crates/aura-std/src/gated_recorder.rs) emits the true
final row exactly once when it was not already emitted as a gated (closed) row, and
summarize_r decides n_open_at_end solely from the last row's OPEN flag — so both
paths present the same final row. The harness calls finalize on every node after the
stream drains (crates/aura-engine/src/harness.rs), and both run paths share that
machinery.

The earlier hedge ("chosen params avoid that") was inaccurate: fast=2/slow=4 is itself an
open-at-end param and the equivalence holds there. The one shape not yet pinned by a
direct integration test — a full aura run vs aura sweep member where the final bar is
a non-gated open hold (CLOSED=false, OPEN=true) through the real run loop — is covered by
composition (finalize-is-called + the unit equivalence) and will be pinned directly by the
#158 reproduce coverage (a sweep member must re-derive a standalone run bit-identically),
scoped to include an open-at-end param.

## Investigated: reduce vs full R-equivalence at window-end — no divergence A follow-up flagged that the reduce-mode sweep path (`GatedRecorder`) might report a different R from a full single run (`Recorder`) for a position left open at the window's end (`n_open_at_end`). Investigated — there is no divergence; the equivalence holds for every window-end shape, and it is pinned by three currently-green tests: - `gated_recorder_then_summarize_r_equals_raw_summarize_r` (crates/aura-engine/tests/streaming_reduction_equivalence.rs) — unit, drives a record that ends in a non-gated open-at-end row; asserts `summarize_r(gated) == summarize_r(full)`, `n_open_at_end == 1` included. - `stage1_r_single_run_output_golden` (crates/aura-cli/tests/cli_run.rs) — pins `n_open_at_end:1` for the default single run at fast=2/slow=4 (so that param *is* open-at-end, not a closed-flat ending). - `blueprint_sweep_member_equals_single_run_and_shares_topology_hash` (crates/aura-cli/src/main.rs) — integration: a reduce-mode sweep member's full `metrics` equals a full-recorder single run at fast=2/slow=4 (the open-at-end param above). Mechanism: `GatedRecorder.finalize` (crates/aura-std/src/gated_recorder.rs) emits the true final row exactly once when it was not already emitted as a gated (closed) row, and `summarize_r` decides `n_open_at_end` solely from the last row's `OPEN` flag — so both paths present the same final row. The harness calls `finalize` on every node after the stream drains (crates/aura-engine/src/harness.rs), and both run paths share that machinery. The earlier hedge ("chosen params avoid that") was inaccurate: fast=2/slow=4 is itself an open-at-end param and the equivalence holds there. The one shape not yet pinned by a *direct* integration test — a full `aura run` vs `aura sweep` member where the final bar is a non-gated open hold (CLOSED=false, OPEN=true) through the real run loop — is covered by composition (finalize-is-called + the unit equivalence) and will be pinned directly by the #158 reproduce coverage (a sweep member must re-derive a standalone run bit-identically), scoped to include an open-at-end param.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone World/C21: run + orchestrate harness families from blueprint-data
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#166
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?