Persist per-member trace streams for family runs (sweep/MC/walk-forward), gated by --trace #104

New Issue

Closed

opened 2026-06-20 18:30:46 +02:00 by Brummel · 2 comments

Brummel commented 2026-06-20 18:30:46 +02:00

Owner

Copy Link

Copy Source

Gap (validated)

Single backtest runs persist their drained streams to disk and are chartable:
aura run --trace <name> calls persist_traces (crates/aura-cli/src/main.rs:161),
gated on the --trace opt-in (:249, :320, :921), writing
runs/traces/<name>/{index.json,equity.json,exposure.json} via TraceStore
(crates/aura-registry/src/trace_store.rs:40).

Family runs do not. aura sweep|mc|walkforward call append_family with
metric-only RunReports and persist nothing to the trace store
(run_sweep:500, run_walkforward:522, run_mc:721). Each family member does
run a full harness with equity+exposure recorders and is drained inside its
per-member closure — but the raw rows are immediately folded to
metrics: summarize(...) and discarded (sweep_family:468-469, mc_family:686-687,
run_oos:609-610). SweepFamily/McFamily/WalkForwardResult carry only
RunReports (manifest + metrics), never the streams.

Consequence: the family-comparison view — the C21 differentiator (families of
harnesses: sweep / MC / walk-forward) — has no per-member trace data to chart.
The streams that would feed it exist transiently and are thrown away.

Goal

Each family member's equity + exposure streams land on disk, mirroring
single-run persistence, so (a) any single member is chartable today with the
existing viewer, and (b) the later family-comparison view has data to render.

Scope — the write side only

• Opt-in --trace <name> on aura sweep|mc|walkforward, mirroring
  aura run --trace. Without it, behaviour is byte-unchanged (no member trace
  dirs written — the current cardinality cost stays user-chosen, never imposed).
• Persist each member as a standalone run-dir at
  runs/traces/<name>/<member_key>/ via the existing persist_traces /
  TraceStore. The engine stays untouched: the per-member closures already hold
  the drained rows (rx_eq.try_iter().collect()); they bind those rows once and,
  when --trace is active, call persist_traces(&member_name, ...) before
  folding to metrics.
• member_key per family kind (deterministic, derivable from what each
  family already keys its output lines by):
  • sweep — odometer/point index (m00, m01, …),
  • Monte-Carlo — the draw seed (seed1, seed2, …),
  • walk-forward — the OOS window index (w00, w01, …).
• No view-side change needed for single members. Because TraceStore
  resolves <name>/<member_key> as a subpath, the existing
  aura chart <name>/<member_key> charts any one member as-is.

Out of scope (explicit — deferred to their own cycles)

• The family-comparison view itself (overlay / small-multiples / rank-and-top-N
  of N members) — a genuine UI design fork, not settled here.
• Decimation / LOD (full resolution persisted, as single runs do today).
• Binary / columnar-on-disk trace format (the JSON-text container is a known,
  latent constant-factor concern; out of scope until cardinality is real).

Acceptance shape

• aura sweep --trace s writes runs/traces/s/m00/, …/m01/, … each a complete
  run-dir (index.json + equity.json + exposure.json) whose SoA content
  round-trips through TraceStore::read.
• aura mc --trace m writes one dir per seed; aura walkforward --trace w writes
  one dir per OOS window.
• No --trace → no trace dirs created; existing family stdout/registry output
  byte-unchanged.
• Determinism (C1) preserved: the persisted member streams are a side-effect of
  the same disjoint per-member sims, not a change to what they compute.

Invariants touched

C6 (the recorded stream is the replay source of truth), C7 (SoA scalar taps),
C8 (sinks/recorders), C18 (run registry) and C21 (families) — all upheld; this
adds a persistence side-effect at the family member boundary, mirroring the
single-run path, and changes no computation.

## Gap (validated) Single backtest runs persist their drained streams to disk and are chartable: `aura run --trace <name>` calls `persist_traces` (`crates/aura-cli/src/main.rs:161`), gated on the `--trace` opt-in (`:249`, `:320`, `:921`), writing `runs/traces/<name>/{index.json,equity.json,exposure.json}` via `TraceStore` (`crates/aura-registry/src/trace_store.rs:40`). Family runs do **not**. `aura sweep|mc|walkforward` call `append_family` with metric-only `RunReport`s and persist nothing to the trace store (`run_sweep`:500, `run_walkforward`:522, `run_mc`:721). Each family member *does* run a full harness with equity+exposure recorders and *is* drained inside its per-member closure — but the raw rows are immediately folded to `metrics: summarize(...)` and discarded (`sweep_family`:468-469, `mc_family`:686-687, `run_oos`:609-610). `SweepFamily`/`McFamily`/`WalkForwardResult` carry only `RunReport`s (manifest + metrics), never the streams. Consequence: the family-comparison view — the C21 differentiator (families of harnesses: sweep / MC / walk-forward) — has no per-member trace data to chart. The streams that would feed it exist transiently and are thrown away. ## Goal Each family member's equity + exposure streams land on disk, mirroring single-run persistence, so (a) any single member is chartable *today* with the existing viewer, and (b) the later family-comparison view has data to render. ## Scope — the write side only - **Opt-in `--trace <name>`** on `aura sweep|mc|walkforward`, mirroring `aura run --trace`. Without it, behaviour is byte-unchanged (no member trace dirs written — the current cardinality cost stays user-chosen, never imposed). - **Persist each member as a standalone run-dir** at `runs/traces/<name>/<member_key>/` via the existing `persist_traces` / `TraceStore`. The engine stays untouched: the per-member closures already hold the drained rows (`rx_eq.try_iter().collect()`); they bind those rows once and, when `--trace` is active, call `persist_traces(&member_name, ...)` before folding to metrics. - **`member_key` per family kind** (deterministic, derivable from what each family already keys its output lines by): - sweep — odometer/point index (`m00`, `m01`, …), - Monte-Carlo — the draw seed (`seed1`, `seed2`, …), - walk-forward — the OOS window index (`w00`, `w01`, …). - **No view-side change needed for single members.** Because `TraceStore` resolves `<name>/<member_key>` as a subpath, the existing `aura chart <name>/<member_key>` charts any one member as-is. ## Out of scope (explicit — deferred to their own cycles) - The **family-comparison view** itself (overlay / small-multiples / rank-and-top-N of N members) — a genuine UI design fork, not settled here. - Decimation / LOD (full resolution persisted, as single runs do today). - Binary / columnar-on-disk trace format (the JSON-text container is a known, latent constant-factor concern; out of scope until cardinality is real). ## Acceptance shape - `aura sweep --trace s` writes `runs/traces/s/m00/`, `…/m01/`, … each a complete run-dir (`index.json` + `equity.json` + `exposure.json`) whose SoA content round-trips through `TraceStore::read`. - `aura mc --trace m` writes one dir per seed; `aura walkforward --trace w` writes one dir per OOS window. - No `--trace` → no trace dirs created; existing family stdout/registry output byte-unchanged. - Determinism (C1) preserved: the persisted member streams are a side-effect of the same disjoint per-member sims, not a change to what they compute. ## Invariants touched C6 (the recorded stream is the replay source of truth), C7 (SoA scalar taps), C8 (sinks/recorders), C18 (run registry) and C21 (families) — all upheld; this adds a persistence side-effect at the family member boundary, mirroring the single-run path, and changes no computation.

Brummel added the feature label 2026-06-20 18:30:46 +02:00

Brummel commented 2026-06-20 18:32:57 +02:00

Author

Owner

Copy Link

Copy Source

Design record (specify, Step 1.5 — derived fork decisions)

Spec: 0058-family-trace-persistence. Boss-mode autonomous run; the forks
below were derivable (sources / code / existing-design consistency / risk),
so they are decided here, not bounced. Each is labelled derived with its
rationale, per the bold-decide stance.

• Fork A — how member traces get retained: persist-in-closure vs. engine
  carries the traces → persist inside the per-member closure (aura-cli),
  gated by the active --trace name; the engine stays untouched.
  Basis (derived): the three closures already live in aura-cli and already
  drain the rows (rx_eq.try_iter().collect() at sweep_family:468,
  mc_family:686, run_oos:609). persist_traces is, by its own doc,
  engine-untouched TraceStore wiring. Threading traces through
  SweepFamily/McFamily/WalkForwardResult + the engine's
  sweep/monte_carlo/walk_forward higher-order fns would enlarge the blast
  radius across aura-engine for no gain. C1 computation stays pure; persistence
  is a sink side-effect, exactly as in single runs.

• Fork B — opt-in vs. always-on → opt-in via a new --trace <name> on
  aura sweep|mc|walkforward, mirroring aura run --trace. Without it,
  behaviour is byte-unchanged.
  Basis (derived): consistency with the single-run gate (:249/:320/:921)

  • risk — it bounds the cardinality cost (N members × full resolution) to a
    user-chosen action, never imposed.

• Fork C — member-key scheme + on-disk layout → each member persisted as a
  standalone run-dir at runs/traces/<name>/<member_key>/, keyed
  sweep = m{NN} (odometer index), Monte-Carlo = seed{N} (draw seed),
  walk-forward = w{NN} (OOS window index).
  Basis (derived): each family already identifies its members by exactly these
  in stdout (mc prints "seed": draw.seed; sweep / walk-forward print per
  point / per window in order). TraceStore resolves <name>/<member_key> as
  a subpath, so the existing single-run viewer charts a member with no
  view-side code.

• Fork D — view scope → the family-comparison VIEW (overlay /
  small-multiples / rank-top-N of N members) is out of scope, its own later
  cycle.
  Basis (derived): it is a genuine UI design fork (how to render N members) with
  real trade-offs the write side does not settle; deferring keeps this cycle
  bounded and single-plan. A scope boundary, not a guess at UI taste.

## Design record (specify, Step 1.5 — derived fork decisions) Spec: `0058-family-trace-persistence`. Boss-mode autonomous run; the forks below were derivable (sources / code / existing-design consistency / risk), so they are decided here, not bounced. Each is labelled **derived** with its rationale, per the bold-decide stance. - **Fork A — how member traces get retained: persist-in-closure vs. engine carries the traces** → **persist inside the per-member closure** (aura-cli), gated by the active `--trace` name; the engine stays untouched. Basis (derived): the three closures already live in aura-cli and already drain the rows (`rx_eq.try_iter().collect()` at `sweep_family`:468, `mc_family`:686, `run_oos`:609). `persist_traces` is, by its own doc, engine-untouched `TraceStore` wiring. Threading traces through `SweepFamily`/`McFamily`/`WalkForwardResult` + the engine's `sweep`/`monte_carlo`/`walk_forward` higher-order fns would enlarge the blast radius across aura-engine for no gain. C1 computation stays pure; persistence is a sink side-effect, exactly as in single runs. - **Fork B — opt-in vs. always-on** → **opt-in via a new `--trace <name>`** on `aura sweep|mc|walkforward`, mirroring `aura run --trace`. Without it, behaviour is byte-unchanged. Basis (derived): consistency with the single-run gate (`:249`/`:320`/`:921`) + risk — it bounds the cardinality cost (N members × full resolution) to a user-chosen action, never imposed. - **Fork C — member-key scheme + on-disk layout** → each member persisted as a **standalone run-dir at `runs/traces/<name>/<member_key>/`**, keyed sweep = `m{NN}` (odometer index), Monte-Carlo = `seed{N}` (draw seed), walk-forward = `w{NN}` (OOS window index). Basis (derived): each family already identifies its members by exactly these in stdout (`mc` prints `"seed": draw.seed`; sweep / walk-forward print per point / per window in order). `TraceStore` resolves `<name>/<member_key>` as a subpath, so the existing single-run viewer charts a member with no view-side code. - **Fork D — view scope** → the **family-comparison VIEW** (overlay / small-multiples / rank-top-N of N members) is **out of scope**, its own later cycle. Basis (derived): it is a genuine UI design fork (how to render N members) with real trade-offs the write side does not settle; deferring keeps this cycle bounded and single-plan. A scope boundary, not a guess at UI taste.

Brummel commented 2026-06-20 18:35:19 +02:00

Author

Owner

Copy Link

Copy Source

Correction to Fork C (member-key scheme) — forced by C1 parallelism

Grounding the closure signatures revealed the engine higher-order fns are
F: Fn(...) -> RunReport + Sync (sweep.rs:314, mc.rs:124,
walkforward.rs:187): family members run in parallel across threads (C1
parallelism across sims). A runtime ordinal counter (m00, m01, … assigned
as the closure fires) would therefore be non-deterministic — the
member→index mapping would depend on thread scheduling, a C1 violation. The
final family.points vector is in deterministic odometer order, but the closure
does not receive its odometer position, and persisting after the sweep would
require the engine to carry the traces (the rejected alternative of Fork A).

Revised decision (derived): member keys are content-derived and
deterministic, computed from what each closure already receives:

• Monte-Carlo → seed{N} (the draw seed, passed to the closure).
• sweep → derived from the swept param values the closure already names via
  zip_params(&space, point) (a stable, collision-free encoding of the grid
  point — distinct points differ in ≥1 swept axis).
• walk-forward → derived deterministically from the OOS window (its bounds /
  index, which uniquely identify the window).

Basis (derived): preserves Fork A (engine untouched, persist-in-closure) and
C1-determinism under the parallel Fn + Sync model, at the cost of pretty
ordinal keys. Content-derived keys are in fact more informative (self-describing)
and are guaranteed deterministic + collision-free, which a runtime counter is
not. Concurrent TraceStore::write to disjoint member dirs needs no lock
(disjoint paths; create_dir_all is idempotent) — consistent with C1's
"concurrently executable without locking".

## Correction to Fork C (member-key scheme) — forced by C1 parallelism Grounding the closure signatures revealed the engine higher-order fns are `F: Fn(...) -> RunReport + Sync` (`sweep.rs:314`, `mc.rs:124`, `walkforward.rs:187`): family members run **in parallel across threads** (C1 parallelism *across* sims). A runtime ordinal counter (`m00`, `m01`, … assigned as the closure fires) would therefore be **non-deterministic** — the member→index mapping would depend on thread scheduling, a C1 violation. The final `family.points` vector is in deterministic odometer order, but the closure does not receive its odometer position, and persisting *after* the sweep would require the engine to carry the traces (the rejected alternative of Fork A). **Revised decision (derived):** member keys are **content-derived and deterministic**, computed from what each closure already receives: - Monte-Carlo → `seed{N}` (the draw seed, passed to the closure). - sweep → derived from the swept param values the closure already names via `zip_params(&space, point)` (a stable, collision-free encoding of the grid point — distinct points differ in ≥1 swept axis). - walk-forward → derived deterministically from the OOS window (its bounds / index, which uniquely identify the window). Basis (derived): preserves Fork A (engine untouched, persist-in-closure) and C1-determinism under the parallel `Fn + Sync` model, at the cost of pretty ordinal keys. Content-derived keys are in fact more informative (self-describing) and are guaranteed deterministic + collision-free, which a runtime counter is not. Concurrent `TraceStore::write` to disjoint member dirs needs no lock (disjoint paths; `create_dir_all` is idempotent) — consistent with C1's "concurrently executable without locking".

Brummel referenced this issue 2026-06-20 20:00:51 +02:00

Generalise sweep member-key derivation beyond the two hardcoded grid axes #105

Brummel referenced this issue from a commit 2026-06-21 13:09:56 +02:00

spec: 0058 family-trace-persistence (boss-signed)

Brummel referenced this issue from a commit 2026-06-21 13:09:56 +02:00

plan: 0058 family-trace-persistence

Brummel referenced this issue from a commit 2026-06-21 13:09:56 +02:00

feat(family-traces): persist per-member streams for sweep/MC/walk-forward

Brummel closed this issue 2026-06-21 13:09:56 +02:00

Brummel referenced this issue from a commit 2026-06-21 13:09:56 +02:00

audit(family-traces): cycle close 0058 — drift-clean, retire spec/plan

Brummel referenced this issue from a commit 2026-06-21 13:09:57 +02:00

audit(sweep-key): cycle close 0059 — drift-clean, retire spec/plan

Brummel referenced this issue 2026-06-21 13:58:52 +02:00

Real-data source path for family runs (aura sweep|walkforward --real) #106

Brummel referenced this issue from a commit 2026-06-22 11:20:23 +02:00

audit(families-view): cycle close 0061 — drift-clean, ledger records the comparison view

Brummel referenced this issue 2026-07-10 00:05:08 +02:00

Deliver per-member trace-writing on sweep/walkforward (--trace writes traces) #224

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

pre-sugar-retirement

Labels

Clear labels

BLOCKER

Core functionality blocked; preempts other selection

bug

Observable misbehaviour

feature

Substantial extension; no full spec required

idea

Droppable; no commitment

No Label feature

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

claude (Claude) Brummel

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#104