spec: 0060 real-data source path for family runs (boss-signed)

Opt-in `--real <SYMBOL> [--from <ms>] [--to <ms>]` for `aura sweep` and `aura walkforward`, mirroring `aura run --real`: each family member streams real M1 close bars from the local data-server archive instead of the synthetic stream. MC is excluded — its seed varies a synthetic price-walk realization, undefined over real data (one realization -> identical members; a real MC needs a bootstrap-resampling axis, its own cycle). Engine, ingest, and registry untouched: a CLI-side `DataSource` provider replaces the hardcoded `VecSource`. `M1FieldSource` already impls `Source`; `WindowRoller` already rolls over epoch-unit timestamps. Boss-signed: Step-5 grounding-check PASS — all 8 load-bearing current-behaviour assumptions ratified by named, currently-green tests. refs #106
2026-06-21 14:05:34 +02:00
parent b92aab713c
commit 9637730520
1 changed files with 273 additions and 0 deletions
@@ -0,0 +1,273 @@
+# Real-data source path for family runs — Design Spec
+
+**Date:** 2026-06-21
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+> **Reference issue:** Brummel/Aura #106 (carries the load-bearing fork
+> decisions, derived under the bold-decide stance).
+
+## Goal
+
+Add an opt-in real-data source path to the **family** commands `aura sweep` and
+`aura walkforward`, mirroring the existing `aura run --real`. With
+`--real <SYMBOL> [--from <ms>] [--to <ms>]`, each family member runs over real M1
+close bars streamed from the local data-server archive (the #71 `M1FieldSource`
+seam) instead of the built-in synthetic stream. Without `--real`, behaviour is
+**byte-unchanged**.
+
+`aura mc` is **excluded** (Fork A): its seed varies the realization of a
+*synthetic* price walk (`SyntheticSpec.source(seed)`), which is undefined over
+real data (one realization → identical members). Real MC needs a
+bootstrap-resampling axis — a separate cycle.
+
+## Architecture
+
+The synthetic-vs-real choice is captured by one small CLI-side provider type,
+`DataSource`, threaded into the family builders in place of the hardcoded
+`VecSource::new(showcase_prices())`. The engine, the blueprint topology, and the
+family types (`SweepFamily` / `WalkForwardResult`) are untouched — `M1FieldSource`
+already impls `aura_engine::Source`, and `WindowRoller` already rolls over
+epoch-unit timestamps.
+
+```text
+DataSource (CLI provider)
+├─ Synthetic                              → VecSource::new(showcase_prices()/walkforward_prices())
+└─ Real { server: Arc<DataServer>,        → M1FieldSource::open[_window](&server, symbol, from, to, Close)
+          symbol, from_ms, to_ms, pip }
+
+Responsibilities:
+  pip_size()                 → f64        (Synthetic: SYNTHETIC_PIP_SIZE; Real: instrument_spec(symbol).pip_size)
+  full_window()              → (Timestamp,Timestamp)   probed once (Real: drain a probe source, as run_sample_real does)
+  run_source()               → Vec<Box<dyn Source>>    fresh per member (sweep: the whole --from..--to stream)
+  windowed_source(from,to)   → Vec<Box<dyn Source>>    fresh per IS/OOS window (walk-forward)
+```
+
+A `DataSource::Real` is built only after `instrument_spec(symbol)` succeeds and
+`server.has_symbol(symbol)` is true — both checked **before** any member runs, so
+an un-vetted symbol or absent data refuses with stderr + `exit(2)` (no panic, no
+partial family), exactly as `run_sample_real` does today.
+
+## Concrete code shapes
+
+### Worked user-facing examples (the acceptance evidence)
+
+```console
+# Param sweep over real EURUSD — every grid member streams the same real window;
+# member dirs are the generic portable keys from #105 (#104 persistence).
+$ aura sweep --real EURUSD --from 1704067200000 --to 1735689599999 --trace swp24
+{"family_id":"swp24-0","report":{"manifest":{"broker":"sim-optimal(pip_size=0.0001)",
+  "params":[["signals.trend.fast.length",{"I64":2}], ...],"window":[1704153600000000000, ...]}, ...}}
+... (4 members)
+$ ls runs/traces/swp24/
+signals.trend.fast.length-2_signals.trend.slow.length-4/   ...   # 4 member dirs, each index/equity/exposure.json
+
+# Rolling walk-forward over real EURUSD 2024 — IS/OOS roll over real calendar time;
+# one OOS member dir per window.
+$ aura walkforward --real EURUSD --from 1704067200000 --to 1735689599999 --trace wf24
+{"family_id":"wf24-0","report":{ ... "window":[<oos0_from_ns>, <oos0_to_ns>] }}
+... (one line per OOS window) + the stitched summary line
+$ ls runs/traces/wf24/
+oos<ns>/  oos<ns>/  ...                                          # one OOS member dir per window
+
+# Un-vetted symbol refuses before any data access (Fork C), unchanged from run --real:
+$ aura sweep --real AAPL.US
+aura: no vetted pip/instrument spec for symbol 'AAPL.US' — refusing ...   # exit 2
+
+# No --real → byte-identical to today (synthetic):
+$ aura sweep --trace swp        # 4 synthetic members, exactly as before
+```
+
+### Provider type (new)
+
+```rust
+// crates/aura-cli/src/main.rs
+enum DataSource {
+    Synthetic,                                  // showcase_prices / walkforward_prices, SYNTHETIC_PIP_SIZE
+    Real { server: Arc<DataServer>, symbol: String, from_ms: Option<i64>, to_ms: Option<i64>, pip: f64 },
+}
+
+impl DataSource {
+    fn pip_size(&self) -> f64 { /* SYNTHETIC_PIP_SIZE | self.pip */ }
+
+    // Build a Real provider or refuse (stderr + exit 2) on un-vetted symbol / absent data.
+    // Mirrors run_sample_real:323-341 (spec lookup BEFORE data access).
+    fn real(symbol: &str, from_ms: Option<i64>, to_ms: Option<i64>) -> DataSource { ... }
+
+    // Sweep: the whole --from..--to stream (Synthetic: the built-in showcase stream).
+    fn run_sources(&self) -> Vec<Box<dyn Source>> { ... }     // fresh each call (single-pass)
+
+    // Walk-forward: a sub-window [from,to] (Synthetic: walkforward_window_source).
+    fn windowed_sources(&self, from: Timestamp, to: Timestamp) -> Vec<Box<dyn Source>> { ... }
+}
+```
+
+### sweep_family / momentum_sweep_family — before → after
+
+```rust
+// BEFORE (main.rs:519-538): hardcoded synthetic source + SYNTHETIC_PIP_SIZE
+fn sweep_family(trace: Option<&str>) -> SweepFamily {
+    ...
+    binder.sweep(|point| {
+        let sources: Vec<Box<dyn Source>> = vec![Box::new(VecSource::new(showcase_prices()))];
+        let window = window_of(&sources).expect("non-empty showcase stream");
+        ...
+        let manifest = sim_optimal_manifest(named, window, 0, SYNTHETIC_PIP_SIZE);
+        ...
+    })
+}
+
+// AFTER: source + pip + window come from the provider; the member closure is otherwise unchanged.
+fn sweep_family(trace: Option<&str>, data: &DataSource) -> SweepFamily {
+    let pip = data.pip_size();
+    let window = data.full_window();                 // probed once (Real: a probe source)
+    ...
+    binder.sweep(|point| {
+        let sources = data.run_sources();            // fresh per member
+        ...
+        h.run(sources);
+        let manifest = sim_optimal_manifest(named, window, 0, pip);
+        ...
+    })
+}
+```
+
+`sample_blueprint_with_sinks()` gains a `pip_size: f64` parameter (its
+`SimBroker::builder(SYNTHETIC_PIP_SIZE)` at main.rs:475 is the hardcode); the
+graph-render callers (`build_sample` at :490) pass `SYNTHETIC_PIP_SIZE`. The
+`momentum_blueprint_with_sinks()` broker (main.rs:566) likewise threads the pip.
+(Momentum sweep over real data: `momentum_sweep_family(trace, data)` mirrors
+`sweep_family`.)
+
+### walkforward — before → after
+
+```rust
+// BEFORE (main.rs:729-748): synthetic span + bar-index roller + synthetic window source
+fn walkforward_family(trace: Option<&str>) -> WalkForwardResult {
+    let sources = vec![Box::new(VecSource::new(walkforward_prices()))];
+    let span = window_of(&sources).expect("non-empty synthetic stream");
+    let roller = WindowRoller::new(span, 24, 12, 12, RollMode::Rolling).expect(...);
+    ...
+    walk_forward(roller, space, |w| {
+        let is_family = sweep_over(w.is.0, w.is.1);                  // synthetic window source inside
+        let best = optimize(&is_family, "total_pips")...;
+        let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace);
+        ...
+    })
+}
+
+// AFTER: span = the real --from..--to (Synthetic: the built-in span); roller sizes
+// come from the provider — bar-index for Synthetic (24/12/12), calendar-ns for Real.
+fn walkforward_family(trace: Option<&str>, data: &DataSource) -> WalkForwardResult {
+    let span = data.full_window();
+    let (is_len, oos_len, step) = data.wf_window_sizes();   // Synthetic: (24,12,12); Real: (90d,30d,30d) in ns
+    let roller = WindowRoller::new(span, is_len, oos_len, step, RollMode::Rolling)
+        .unwrap_or_else(|e| { eprintln!("aura: walk-forward window config: {e}"); std::process::exit(2) });
+    ...
+    walk_forward(roller, space, |w| {
+        let is_family = sweep_over(w.is.0, w.is.1, data);     // provider's windowed_sources inside
+        ...
+        let (oos_equity, oos_report) = run_oos(&best.params, w.oos.0, w.oos.1, trace, data);
+        ...
+    })
+}
+```
+
+`sweep_over` and `run_oos` take `data: &DataSource` and call
+`data.windowed_sources(from, to)` in place of `walkforward_window_source(from,
+to)`; their `sim_optimal_manifest(..., data.pip_size())` threads the pip. The
+real IS/OOS/step defaults (Fork D), as named ns constants:
+
+```rust
+const WF_DAY_NS: i64 = 86_400_000_000_000;          // 1 day in ns (M1 stream timestamp unit)
+const WF_REAL_IS_NS: i64  = 90 * WF_DAY_NS;          // 90-day in-sample
+const WF_REAL_OOS_NS: i64 = 30 * WF_DAY_NS;          // 30-day out-of-sample
+const WF_REAL_STEP_NS: i64 = 30 * WF_DAY_NS;         // 30-day step (contiguous OOS tiling)
+```
+
+A real window shorter than one full IS+OOS span yields `WalkForwardError` from
+`WindowRoller::new` → stderr + `exit(2)` (the typed gate, not a panic).
+
+### Parser + dispatch — before → after
+
+```rust
+// sweep: add an optional --real tail to parse_sweep_args; --real is exclusive with the
+// synthetic default. Returns the data choice alongside (strategy, name, persist).
+fn parse_sweep_args(rest) -> Result<(Strategy, String, bool, DataChoice), String>
+//   DataChoice = Synthetic | Real { symbol, from_ms, to_ms }   (parsed, not yet opened)
+
+// walkforward: a parse_walkforward_args mirroring it (no --strategy axis):
+//   [--real <SYMBOL> [--from <ms>] [--to <ms>]] [--name <n> | --trace <n>]
+
+// dispatch arms build the DataSource (which may refuse) then call run_sweep / run_walkforward.
+```
+
+`run_sweep` / `run_walkforward` gain a `DataSource` parameter and pass it to the
+family builders.
+
+## Components
+
+| Component | Change |
+|-----------|--------|
+| `DataSource` enum + impl | **new** — synthetic/real provider (pip, window, sources) |
+| `parse_sweep_args` | add `--real` tail → `DataChoice`; keep `--strategy`/`--name`/`--trace` |
+| `parse_walkforward_args` | **new** — `--real` + `--name`/`--trace` grammar |
+| `sample_blueprint_with_sinks` / `momentum_blueprint_with_sinks` | add `pip_size: f64` param (un-hardcode the broker) |
+| `sweep_family` / `momentum_sweep_family` | take `&DataSource`; source/pip/window from it |
+| `walkforward_family` / `sweep_over` / `run_oos` | take `&DataSource`; provider window source; real roller sizes |
+| `run_sweep` / `run_walkforward` + dispatch arms | thread `DataSource`; build-or-refuse before running |
+| `USAGE` | document the `--real` tails |
+
+The engine (`aura-engine`), `aura-ingest`, and the registry are **not** modified.
+
+## Data flow
+
+1. Parse the tail → `DataChoice` (pure, unit-testable: symbol + optional window).
+2. Build `DataSource` — `Real` looks up the pip (refuse if un-vetted) and checks
+   `has_symbol` (refuse if absent), both before any member runs.
+3. Family builder asks the provider for the pip, the full window (probed once),
+   and a fresh source per member / per IS/OOS window.
+4. Each member runs the same disjoint harness (C1) over real bars; `--trace`
+   persists it under `runs/traces/<name>/<member_key>/` exactly as #104.
+
+## Error handling
+
+- Un-vetted symbol → `instrument_spec` `None` → stderr + `exit(2)`, before data.
+- Symbol absent from the archive → `has_symbol` false → stderr + `exit(2)`.
+- Real window too short for window 0 → `WindowRoller::new` `Err` → stderr +
+  `exit(2)`.
+- `--real` without a symbol, unknown flag, non-`i64` ms, `--name` with `--trace`,
+  `--real` with no synthetic fallback conflict → `Err(usage)` from the parser →
+  stderr + `exit(2)`.
+- No `--real` → the synthetic path runs unchanged.
+
+## Testing strategy
+
+- **Parser unit tests** (`--bin aura`): `parse_sweep_args` accepts `--real SYM`,
+  `--real SYM --from .. --to ..`, rejects `--real` (no symbol) and `--real` +
+  conflicting flags; `parse_walkforward_args` mirror.
+- **`DataSource` unit tests**: `pip_size()` Synthetic vs Real; `Synthetic`
+  `run_sources()` non-empty.
+- **Byte-unchanged guard**: existing `sweep_report` / walk-forward report tests
+  stay green (the synthetic path is the default), proving no `--real` is
+  byte-identical.
+- **Integration** (`--test cli_run`, gated on local data like the existing real
+  test): `aura sweep --real EURUSD --from <m> --to <m> --trace t` persists member
+  dirs whose keys match `^[A-Za-z0-9._-]+$`; `aura walkforward --real EURUSD …
+  --trace t` persists one `oos<ns>` dir per window; both members chart
+  (`aura chart t/<key>` contains `uPlot`). C1: the same real sweep twice is
+  bit-identical. Un-vetted symbol refuses with exit 2.
+
+## Acceptance criteria
+
+- `aura sweep --real EURUSD --from <m> --to <m> --trace s` runs every grid member
+  over real M1 close bars and persists one member dir per point.
+- `aura walkforward --real EURUSD --from <m> --to <m> --trace w` rolls IS/OOS over
+  real calendar time and persists one OOS member dir per window.
+- `aura mc` is unchanged (real explicitly excluded).
+- No `--real` → family stdout + registry + trace output byte-unchanged.
+- Un-vetted symbol / absent data / too-short window → stderr + `exit(2)`, no
+  panic, no partial family.
+- C1 preserved: the same real window streamed twice is bit-identical.
+- `cargo build --workspace`, `cargo test --workspace`, `cargo clippy --workspace
+  --all-targets -- -D warnings` all green.