Real-data source path for family runs (aura sweep|walkforward --real) #106

New Issue

2026-06-21T13:58:52+02:00

Brummel commented

2026-06-21 13:58:52 +02:00

Design record (specify, in-context entry)

This issue seeds the real-data source path for family runs. The design was
settled in an in-context discussion (the family-trace demo for #104 over
synthetic data, then "interessant wäre das ganze auch noch mit echten Daten über
einen relevanten Zeitraum"); this records it. No prior tracker issue existed, so
this is the reference issue for the /boss run that implements it.

Gap

aura run --real <SYMBOL> streams real M1 close bars from the local data-server
archive through the #71 M1FieldSource seam. The family builders do not:
sweep_family, momentum_sweep_family, and walkforward_family /
sweep_over / run_oos are hardcoded to a synthetic VecSource
(showcase_prices / walkforward_prices, crates/aura-cli/src/main.rs:524-525, 600-601, 730-731, 768-769, 795-796). There is no --real path for families, so
the C21 family surface can only ever show synthetic toy streams.

Goal

Opt-in --real <SYMBOL> [--from <ms>] [--to <ms>] on aura sweep and
aura walkforward, mirroring aura run --real: each family member runs over real
M1 close bars instead of the synthetic stream. Without --real, behaviour is
byte-unchanged.

Load-bearing forks resolved (derived; bold-decide stance)

Fork A — which family kinds get --real → sweep + walkforward; MC
excluded. Basis (derived): mc_family (main.rs:868-871) varies, per seed,
the realization of a synthetic price walk (SyntheticSpec.source(seed)). Real
data has exactly one realization, so every seed-member would stream identical
bars → the MC axis is undefined over real data. A real MC needs a different
stochastic axis (block-bootstrap of the real returns) — its own design cycle.
sweep (param axis) and walk-forward (time axis) are both well-defined over real
data.
Fork B — how the real source is injected → a source-provider per member
closure, mirroring run_sample_real's open closure (main.rs:342). Each
member / window builds a fresh M1FieldSource (Sources are single-pass;
parallel members each need their own); the DataServer is Arc-shared
(thread-safe cache). The family builders take a source-provider parameter
instead of the hardcoded VecSource::new(...). Engine untouched
(M1FieldSource already impls Source).
Fork C — pip size → from aura_ingest::instrument_spec(symbol), refused
(stderr + exit 2) before any data access for an un-vetted symbol, mirroring
run_sample_real:323. The sim-optimal broker + manifest use the real pip, not
SYNTHETIC_PIP_SIZE.
Fork D — walk-forward IS/OOS sizing over real time → the WindowRoller
already works in epoch-unit timestamps (walkforward.rs:18-21), which are
nanoseconds for real M1; M1FieldSource::open_window takes Timestamp, so the
windowed source binds directly to WindowBounds. The real path uses fixed
sensible calendar-time defaults for IS / OOS / step (the spec fixes the exact
values); no new CLI flags this iteration (flags are a later refinement).
Engine untouched — only the CLI-side default sizes change from bar-index
(synthetic) to calendar time.
Fork E — parser grammar → aura sweep [--strategy <sma|momentum>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--trace <n>] and aura walkforward [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name | --trace <n>], mirroring
parse_real_args. --real is exclusive with the synthetic default; without it,
byte-unchanged.

Out of scope (deferred)

MC over real data — needs a bootstrap-resampling axis; its own cycle.
CLI flags for IS/OOS/step sizing — fixed defaults this iteration.
The family-comparison view (overlay / small-multiples) — already deferred by #104.
Decimation / LOD, binary trace container — latent, deferred by #104.

Invariants

C1 (determinism — same real window streamed twice is bit-identical, already true
for run --real), C2 (no look-ahead — WindowRoller enforces oos.0 > is.1),
C6 (record-then-replay — real bars are a pre-recorded archive), C7 (SoA), C9
(engine/project — real data is the data-server's job, engine untouched), C18/C21
(families). All upheld; this adds a source-injection seam at the CLI family
boundary and changes no engine computation.

## Design record (specify, in-context entry) This issue seeds the **real-data source path for family runs**. The design was settled in an in-context discussion (the family-trace demo for #104 over synthetic data, then *"interessant wäre das ganze auch noch mit echten Daten über einen relevanten Zeitraum"*); this records it. No prior tracker issue existed, so this is the reference issue for the `/boss` run that implements it. ## Gap `aura run --real <SYMBOL>` streams real M1 close bars from the local data-server archive through the #71 `M1FieldSource` seam. The **family builders do not**: `sweep_family`, `momentum_sweep_family`, and `walkforward_family` / `sweep_over` / `run_oos` are hardcoded to a synthetic `VecSource` (`showcase_prices` / `walkforward_prices`, `crates/aura-cli/src/main.rs:524-525, 600-601, 730-731, 768-769, 795-796`). There is no `--real` path for families, so the C21 family surface can only ever show synthetic toy streams. ## Goal Opt-in `--real <SYMBOL> [--from <ms>] [--to <ms>]` on `aura sweep` and `aura walkforward`, mirroring `aura run --real`: each family member runs over real M1 close bars instead of the synthetic stream. Without `--real`, behaviour is byte-unchanged. ## Load-bearing forks resolved (derived; bold-decide stance) - **Fork A — which family kinds get `--real`** → **sweep + walkforward; MC excluded.** Basis (derived): `mc_family` (`main.rs:868-871`) varies, per seed, the *realization of a synthetic price walk* (`SyntheticSpec.source(seed)`). Real data has exactly one realization, so every seed-member would stream identical bars → the MC axis is undefined over real data. A real MC needs a different stochastic axis (block-bootstrap of the real returns) — its own design cycle. sweep (param axis) and walk-forward (time axis) are both well-defined over real data. - **Fork B — how the real source is injected** → **a source-provider per member closure**, mirroring `run_sample_real`'s `open` closure (`main.rs:342`). Each member / window builds a fresh `M1FieldSource` (Sources are single-pass; parallel members each need their own); the `DataServer` is `Arc`-shared (thread-safe cache). The family builders take a source-provider parameter instead of the hardcoded `VecSource::new(...)`. Engine untouched (`M1FieldSource` already impls `Source`). - **Fork C — pip size** → from `aura_ingest::instrument_spec(symbol)`, refused (stderr + exit 2) before any data access for an un-vetted symbol, mirroring `run_sample_real:323`. The sim-optimal broker + manifest use the real pip, not `SYNTHETIC_PIP_SIZE`. - **Fork D — walk-forward IS/OOS sizing over real time** → the `WindowRoller` already works in epoch-unit timestamps (`walkforward.rs:18-21`), which are nanoseconds for real M1; `M1FieldSource::open_window` takes `Timestamp`, so the windowed source binds directly to `WindowBounds`. The real path uses fixed sensible calendar-time defaults for IS / OOS / step (the spec fixes the exact values); **no new CLI flags** this iteration (flags are a later refinement). Engine untouched — only the CLI-side default sizes change from bar-index (synthetic) to calendar time. - **Fork E — parser grammar** → `aura sweep [--strategy <sma|momentum>] [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--trace <n>]` and `aura walkforward [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name | --trace <n>]`, mirroring `parse_real_args`. `--real` is exclusive with the synthetic default; without it, byte-unchanged. ## Out of scope (deferred) - **MC over real data** — needs a bootstrap-resampling axis; its own cycle. - **CLI flags for IS/OOS/step sizing** — fixed defaults this iteration. - The **family-comparison view** (overlay / small-multiples) — already deferred by #104. - **Decimation / LOD**, binary trace container — latent, deferred by #104. ## Invariants C1 (determinism — same real window streamed twice is bit-identical, already true for `run --real`), C2 (no look-ahead — `WindowRoller` enforces `oos.0 > is.1`), C6 (record-then-replay — real bars are a pre-recorded archive), C7 (SoA), C9 (engine/project — real data is the data-server's job, engine untouched), C18/C21 (families). All upheld; this adds a source-injection seam at the CLI family boundary and changes no engine computation.

Brummel added the feature label 2026-06-21 13:58:52 +02:00

Brummel commented

2026-06-21 14:12:07 +02:00

Plan recon — two derived sub-forks (specify→planner)

Plan-recon (read-only) surfaced two mechanics the spec left to taste; both
derivable, decided here under the bold stance.

Fork F — where the walk-forward IS/OOS/step sizes live → a 5th provider
method DataSource::wf_window_sizes(&self) -> (i64, i64, i64). Synthetic returns
(24, 12, 12) (bar-index, over the 60-bar synthetic span, byte-unchanged); Real
returns (WF_REAL_IS_NS, WF_REAL_OOS_NS, WF_REAL_STEP_NS) = (90d, 30d, 30d) in
ns. Basis (derived): the WindowRoller already takes sizes in the stream's
epoch-unit; the two sizing regimes (bar-index synthetic vs calendar-ns real)
differ only in the numbers, so one provider method keeps both at a single site,
exactly beside full_window() which the same provider already supplies.
Fork G — where build-or-refuse lives → the parser stays pure and yields a
DataChoice (unit-testable: --real SYM, window pairs, rejects). The
build-or-refuse (un-vetted symbol → refuse; symbol absent from archive → refuse)
mirrors run_sample_real's inline stderr + exit(2) pattern (main.rs:323-341)
in a DataSource constructor, E2E-tested via an integration test like the
existing run_real_unspecced_symbol_refuses_with_exit_2. Basis (derived):
consistency with the one real-data path that already ships; a process exit(2)
is not unit-testable without a process spawn, which the integration tests do
anyway, so the refuse path is covered E2E rather than left untested.

Authoritative call-site set for the signature changes is
cargo build -p aura-cli --all-targets (the compiler enumerates the 6
#[cfg(test)] callers a prod-only grep misses), not a hand grep. Engine / ingest
/ registry remain untouched. Spec 0060 committed boss-signed (grounding-check
PASS, 8/8 assumptions ratified); plan 0060 next.

## Plan recon — two derived sub-forks (specify→planner) Plan-recon (read-only) surfaced two mechanics the spec left to taste; both derivable, decided here under the bold stance. - **Fork F — where the walk-forward IS/OOS/step sizes live** → a 5th provider method `DataSource::wf_window_sizes(&self) -> (i64, i64, i64)`. Synthetic returns `(24, 12, 12)` (bar-index, over the 60-bar synthetic span, byte-unchanged); Real returns `(WF_REAL_IS_NS, WF_REAL_OOS_NS, WF_REAL_STEP_NS)` = `(90d, 30d, 30d)` in ns. Basis (derived): the `WindowRoller` already takes sizes in the stream's epoch-unit; the two sizing regimes (bar-index synthetic vs calendar-ns real) differ only in the numbers, so one provider method keeps both at a single site, exactly beside `full_window()` which the same provider already supplies. - **Fork G — where build-or-refuse lives** → the parser stays pure and yields a `DataChoice` (unit-testable: `--real SYM`, window pairs, rejects). The build-or-refuse (un-vetted symbol → refuse; symbol absent from archive → refuse) mirrors `run_sample_real`'s inline `stderr + exit(2)` pattern (main.rs:323-341) in a `DataSource` constructor, E2E-tested via an integration test like the existing `run_real_unspecced_symbol_refuses_with_exit_2`. Basis (derived): consistency with the one real-data path that already ships; a process `exit(2)` is not unit-testable without a process spawn, which the integration tests do anyway, so the refuse path is covered E2E rather than left untested. Authoritative call-site set for the signature changes is `cargo build -p aura-cli --all-targets` (the compiler enumerates the 6 `#[cfg(test)]` callers a prod-only grep misses), not a hand grep. Engine / ingest / registry remain untouched. Spec 0060 committed boss-signed (grounding-check PASS, 8/8 assumptions ratified); plan 0060 next.

Brummel commented

2026-06-21 15:33:39 +02:00

Shipped (cycle 0060) — closing

aura sweep|walkforward --real <SYMBOL> [--from <ms>] [--to <ms>] now streams real
M1 close bars per family member; opt-in, synthetic path byte-unchanged. Verified
end-to-end over real EURUSD: sweep --real → 4 members; walkforward --real over
2024 → 9 rolling OOS windows; un-vetted symbol / mc --real / window-without-real
each exit 2.

Commits: 9637730 (spec, boss-signed — grounding-check PASS 8/8) · bd5a08d
(plan) · cf10217 (DataSource provider + DRY-extracted real-data helpers) ·
2a6ca47 (pip-param + &DataSource wiring) · 8e5d14b (--real parser + gated
real tests) · 607e548 (audit close, ledger amendment).

Two corrections surfaced during implementation, both fixed forward:

wf_full_span (a design gap I missed in the plan): walk-forward is a
windowed consumer whose span comes from the 60-bar walkforward_prices, not
full_window()'s 18-bar showcase span. Added DataSource::wf_full_span(); the
implementer correctly bounced rather than silently substitute.
RealWindowGrammar (an improvement): the --real/--from/--to grammar is
shared between the two parsers (strict flag-repeat / empty-symbol rejection
mirroring parse_real_args) instead of duplicated.

Architect: cycle contract-clean (C1/C2/C6/C9/C10/C12 upheld, engine/ingest/registry
untouched). The one drift was a ledger gap, closed in 607e548 (C22 real-data
family amendment). Carry-on [low]: the real oos{ns} key has no explicit
uniqueness pin beyond starts_with("oos") — acceptable, covered in practice.

Deferred to their own cycles (recorded here as the forward queue): CLI flags for
walk-forward IS/OOS/step sizing (fixed 90d/30d/30d defaults this cycle); MC over
real data (needs a bootstrap-resampling axis); the family-comparison view.

## Shipped (cycle 0060) — closing `aura sweep|walkforward --real <SYMBOL> [--from <ms>] [--to <ms>]` now streams real M1 close bars per family member; opt-in, synthetic path byte-unchanged. Verified end-to-end over real EURUSD: `sweep --real` → 4 members; `walkforward --real` over 2024 → 9 rolling OOS windows; un-vetted symbol / `mc --real` / window-without-real each exit 2. Commits: `9637730` (spec, boss-signed — grounding-check PASS 8/8) · `bd5a08d` (plan) · `cf10217` (DataSource provider + DRY-extracted real-data helpers) · `2a6ca47` (pip-param + `&DataSource` wiring) · `8e5d14b` (`--real` parser + gated real tests) · `607e548` (audit close, ledger amendment). Two corrections surfaced during implementation, both fixed forward: - **`wf_full_span` (a design gap I missed in the plan):** walk-forward is a *windowed* consumer whose span comes from the 60-bar `walkforward_prices`, not `full_window()`'s 18-bar showcase span. Added `DataSource::wf_full_span()`; the implementer correctly bounced rather than silently substitute. - **`RealWindowGrammar` (an improvement):** the `--real`/`--from`/`--to` grammar is shared between the two parsers (strict flag-repeat / empty-symbol rejection mirroring `parse_real_args`) instead of duplicated. Architect: cycle contract-clean (C1/C2/C6/C9/C10/C12 upheld, engine/ingest/registry untouched). The one drift was a ledger gap, closed in `607e548` (C22 real-data family amendment). Carry-on [low]: the real `oos{ns}` key has no explicit uniqueness pin beyond `starts_with("oos")` — acceptable, covered in practice. Deferred to their own cycles (recorded here as the forward queue): CLI flags for walk-forward IS/OOS/step sizing (fixed 90d/30d/30d defaults this cycle); MC over real data (needs a bootstrap-resampling axis); the family-comparison view.

Brummel closed this issue

2026-06-21 15:33:39 +02:00

Brummel referenced this issue from a commit

2026-06-21 16:12:33 +02:00

spec: 0060 real-data source path for family runs (boss-signed)

Brummel referenced this issue from a commit

2026-06-21 16:12:33 +02:00

plan: 0060 real-data source path for family runs

Brummel referenced this issue from a commit

2026-06-21 16:12:33 +02:00

feat(real-family): DataSource provider + shared real-data helpers

Brummel referenced this issue from a commit