From fa9dac35aa9b7603192ff31f3e914550810718fb Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 18 Jun 2026 12:26:11 +0200 Subject: [PATCH] spec: real-data source-open seam (Runway-1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidate aura-ingest's OHLC source-open surface onto the engine-native epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a real-data source builds from aura-ingest alone, with the one ms<->ns crossing owned by the seam. Behaviour-preserving (byte-identical RunReports). Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions confirmed; compare.rs compiler-invisible site + AC1 scope fix applied). refs #80, #81, #92 --- docs/specs/0052-real-data-source-open-seam.md | 249 ++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 docs/specs/0052-real-data-source-open-seam.md diff --git a/docs/specs/0052-real-data-source-open-seam.md b/docs/specs/0052-real-data-source-open-seam.md new file mode 100644 index 0000000..1b3ee4c --- /dev/null +++ b/docs/specs/0052-real-data-source-open-seam.md @@ -0,0 +1,249 @@ +# Real-data source-open seam — Design Spec + +**Date:** 2026-06-18 +**Status:** Grounded (workflow `ground-spec-0052`: 25/28 assertions confirmed; 1 blocker + 1 scope fix applied below); proceeding to planner per user instruction +**Authors:** orchestrator + Claude +**Cycle:** Runway-1 — milestone "Runway — real-data ergonomics & honesty hardening". Closes #80, #81, #92. +**Note:** Per-cycle specs are git-tracked but ephemeral — committed while their cycle is live and removed (`git rm`) at cycle close (the git-ignore of 28958f2 was reverted in aedaa5d). This file is a working artifact for the planner this cycle, never a durable API reference. + +## Goal + +Consolidate the `aura-ingest` public surface so a real-data source — a single +base column **and** the OHLC bundle — is buildable from `aura-ingest` alone, +with the epoch-ns ↔ Unix-ms window-currency owned by the seam. This removes two +field-tested frictions: the per-consumer `ns_to_ms` hand-divide (a transposable +silent-wrong-window class, #80) and the duplicated, order-sensitive OHLC opener +(#92), plus the undocumented direct reach into the external `data_server` crate +(#81). Behaviour-preserving: existing runs reach byte-identical `RunReport`s. + +## Architecture + +Four additions to `crates/aura-ingest/src/lib.rs`, all additive; the existing +ms-based `M1FieldSource::open` and `unix_ms_to_epoch_ns` are untouched (the +behaviour-preserving guarantee for current call sites). + +1. **Seam-owned inverse (private).** `epoch_ns_to_unix_ms(Timestamp) -> i64`, + the inverse of the existing `unix_ms_to_epoch_ns`. **Not** `pub`: the seam + owns the ms↔ns convention (C3); a consumer never converts. (This is the #80 + decision — see the reconciliation comment on #80.) + +2. **Timestamp-window opener.** `M1FieldSource::open_window(server, symbol, + from: Option, to: Option, field) -> Option`, + mirroring the existing `open` but in the engine's native `Timestamp` + (epoch-ns) currency; it maps each bound through `epoch_ns_to_unix_ms` and + delegates to `open`. `Option` preserves open-ended windows, as `open` does. + +3. **Canonical OHLC opener.** `open_ohlc(server, symbol, from: Timestamp, to: + Timestamp) -> Option>>` builds the four `M1FieldSource`s + for `Open, High, Low, Close` **in that fixed order** (the C4 merge tie-break + the resampler's `Barrier(0)` group depends on, #92), each via `open_window`, + all sharing the one `Arc`. Closed `Timestamp` window (a backtest / + walk-forward fold is always bounded). This is the single vetted home of the + O/H/L/C order; the trap of hand-spelling it four times is gone. + +4. **Default-archive convenience + re-exports.** `aura-ingest` re-exports + `DataServer` and `DEFAULT_DATA_PATH` from `data_server`, and adds + `default_data_server() -> Arc` = `Arc::new(DataServer::new( + DEFAULT_DATA_PATH))`. One shared `Arc` (one cache) flows into + `open_ohlc` across the four fields and across disjoint sims — the C12 + sharing the streaming model rests on. (This is the #81 decision; the + issue-sketched per-field `open_default_archive` was rejected because it would + build one cache per field — see the reconciliation comment on #81.) + +**Return type — `Vec` not `[_; 4]`.** `Harness::run` consumes +`Vec>` (harness.rs:371), so a `Vec` feeds straight in with no +conversion; the order-safety #92 asks for is delivered by the single vetted +helper regardless of container. (Issue #92 sketched `[_; 4]` as an "e.g."; this +is the non-load-bearing container choice.) + +## Concrete code shapes + +### User-facing worked example — the walk-forward fold closure (the #80/#81/#92 evidence) + +**Before** (today; `crates/aura-ingest/examples/ger40_breakout_walkforward.rs` +and `tests/ger40_breakout_world.rs`), each consumer re-deriving the seam's job: + +```rust +use data_server::{DataServer, DEFAULT_DATA_PATH}; // #81: direct reach into the external crate + +// #80: local hand-divide, re-written per consumer; transposable from/to +fn ns_to_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 } + +// #92: local open_ohlc_sources, hardcoded to SYMBOL, O/H/L/C order spelled by hand +let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH)); +let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun { + let (is_from, is_to) = (ns_to_ms(w.is.0), ns_to_ms(w.is.1)); // <- the divide + let sources = open_ohlc_sources(&server, is_from, is_to).expect("…"); + /* bootstrap + run + summarize */ +}); +``` + +**After** — the seam owns the currency, the opener, and the server construction: + +```rust +use aura_ingest::{default_data_server, open_ohlc}; // builds from aura-ingest alone + +let server = default_data_server(); // one shared Arc (C12) +let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun { + let sources = open_ohlc(&server, SYMBOL, w.is.0, w.is.1) // Timestamp in — no divide + .expect("window overlaps GER40 data"); + /* bootstrap + run + summarize */ +}); +``` + +No `ns_to_ms`, no local OHLC helper, no `data_server::` import. + +### Before → after API surface (secondary) + +```rust +// lib.rs — ADD (private): +fn epoch_ns_to_unix_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 } + +// lib.rs — ADD on impl M1FieldSource: +pub fn open_window( + server: &Arc, symbol: &str, + from: Option, to: Option, field: M1Field, +) -> Option { + Self::open(server, symbol, from.map(epoch_ns_to_unix_ms), to.map(epoch_ns_to_unix_ms), field) +} + +// lib.rs — ADD (free fns): +pub fn open_ohlc( + server: &Arc, symbol: &str, from: Timestamp, to: Timestamp, +) -> Option>>; // O,H,L,C order, shares `server` + +pub fn default_data_server() -> Arc; // Arc::new(DataServer::new(DEFAULT_DATA_PATH)) + +// lib.rs — ADD (re-exports): +pub use data_server::{DataServer, DEFAULT_DATA_PATH}; + +// UNCHANGED (behaviour-preserving): pub fn open(.. Option ..), pub fn unix_ms_to_epoch_ns +``` + +## Components + +- `crates/aura-ingest/src/lib.rs` — the four additions + re-exports above. +- `crates/aura-ingest/examples/shared/breakout_real.rs`: + - remove the local `open_ohlc_sources` (Unix-ms, hardcoded `SYMBOL`); the + library `open_ohlc(server, symbol, from: Timestamp, to: Timestamp)` replaces + every use of it (symbol now a parameter). + - retype the shared window-builder `utc_month_window_ms(year, month) -> + (i64, i64)` (breakout_real.rs:334) to `utc_month_window(year, month) -> + (Timestamp, Timestamp)` — the *same* UTC instants, typed as the + engine-native `Timestamp` (it wraps each bound through the existing + `unix_ms_to_epoch_ns`). This is what makes the whole consumer layer + Timestamp-native, so no caller round-trips ms→Timestamp→ms. + - retype `report_from_trace` (breakout_real.rs:407, `from_ms/to_ms: i64`, + window wrap at :422-425) to take `from/to: Timestamp` and drop the + `unix_ms_to_epoch_ns` wrap — same shape as the other shared helpers. (The + `utc_month_window` retype turns this into a compile error, so the type + checker reaches it; named here for completeness.) +- `crates/aura-ingest/examples/ger40_breakout_compare.rs` — the **one + compiler-invisible** migration site. It opens OHLC via its own hand-spelled + O/H/L/C loop (compare.rs:74-79), imports `data_server` directly (:29), and + builds its window inline via chrono (:114-121); it calls **neither** + `open_ohlc_sources` **nor** `utc_month_window_ms`, so deleting/retyping those + does **not** flag it. It must be migrated by inspection: replace the loop with + `open_ohlc`, the inline chrono window with `utc_month_window`, and the + `data_server` import with the `aura-ingest` re-exports — else it silently + survives with exactly the #92 order-trap and #81 direct-import this spec kills. + +### Migration policy (all callers of the removed helper) + +`open_ohlc_sources` has call sites well beyond the two walk-forward consumers — +**8 call expressions across 6 files** (today: `examples/ger40_breakout_walkforward.rs`, +`examples/ger40_breakout_sweep.rs`, `examples/ger40_breakout_real.rs`, +`tests/ger40_breakout_world.rs` (×2, :54/:228), `tests/ger40_breakout_real.rs`, +`tests/ger40_breakout_blueprint.rs` (×2, :83/:94)). The planner migrates every +*expression*, not one-per-file. Separately, `ger40_breakout_compare.rs` opens +OHLC without this helper (its own loop) and so is **not** in this census — it is +the compiler-invisible site handled below. **All** migrate to the *same* +Timestamp-native surface — there is **no** ms-based shim retained, and the only +epoch-ns↔Unix-ms conversion left anywhere in the consumer layer is the one inside +the seam: + +- **Walk-forward consumers**: delete the local `ns_to_ms`; pass the + `WindowBounds` `Timestamp` bounds (`w.is.0`, `w.is.1`, `w.oos.0`, `w.oos.1`) + straight to `open_ohlc`. +- **Calendar-window consumers**: their bounds come from `utc_month_window` (now + `Timestamp`); thread that `Timestamp` window through any local helper + (`run_point` / `run_blueprint` / `run_flatgraph`, whose `from_ms/to_ms: i64` + params become `from/to: Timestamp`) into `open_ohlc`. Internal `RunManifest` + construction that previously wrapped `unix_ms_to_epoch_ns(from_ms)` now + receives the `Timestamp` directly (drop the wrap). +- **Direct `data_server::` importers**: switch `use data_server::{DataServer, + DEFAULT_DATA_PATH}` to the `aura-ingest` re-exports, and `Arc::new( + DataServer::new(DEFAULT_DATA_PATH))` to `default_data_server()`. +- **The compiler-invisible site — `ger40_breakout_compare.rs`**: migrated by + inspection, not by a compile error (it calls neither removed/retyped helper). + Replace its hand-spelled O/H/L/C loop (:74-79) with `open_ohlc`, its inline + chrono window (:114-121) with `utc_month_window`, and its direct `data_server` + import (:29) with the `aura-ingest` re-exports + `default_data_server()`. + +Deleting `open_ohlc_sources` and retyping `utc_month_window` / `report_from_trace` +turns every site *except `ger40_breakout_compare.rs`* into a compile error until +migrated, so the type checker enumerates that set; the single shape above is +applied to each, and `compare.rs` is migrated by inspection. Every run is +byte-identical (same instants, same windows) — behaviour-preserving. + +## Data flow + +`WindowRoller` → `WindowBounds { is, oos: (Timestamp, Timestamp) }` → `open_ohlc` +(Timestamp) → `epoch_ns_to_unix_ms` (the one ms↔ns crossing, internal) → +`DataServer::stream_m1_windowed` (Unix-ms) → four `M1FieldSource` (O/H/L/C) → +`Vec>` → `Harness::run`. The currency boundary is crossed +exactly once, inside the seam. + +## Error handling + +`open_ohlc` returns `None` if any of the four field opens returns `None` (no +archived file overlaps the window) — the same file-level `?` short-circuit and +the same `Some`-with-empty vs `None` semantics the current `open` / +`open_ohlc_sources` already document (lib.rs:126-135). A window overlapping a +file but holding zero bars yields sources whose first `peek` is `None`, not a +`None` from `open_ohlc`. + +## Testing strategy + +- **Hermetic (no DataServer):** `epoch_ns_to_unix_ms(unix_ms_to_epoch_ns(x)) == + x` round-trip, beside the existing `unix_ms_to_epoch_ns_scales_ms_to_ns` + (lib.rs:279). This pins the seam-owned inverse without an archive. +- **Gated (real data, skip-with-note when the archive is absent, mirroring + `real_bars.rs` / `streaming_seam.rs` / `ger40_*`):** + - `open_ohlc(&server, SYMBOL, from_ts, to_ts)` yields four sources and a + Timestamp-window backtest reaches the **same** `RunReport` as the existing + ms-path open over the equivalent window (behaviour-preserving proof). + - the migrated walk-forward example + world test run green with zero + `ns_to_ms` (the #80 evidence). +- **Full workspace suite green, unchanged** (`cargo test --workspace`); clippy + clean. No existing ms-based call site changes behaviour. + +## Acceptance criteria + +1. A single-field source (`M1FieldSource::open_window`) and the OHLC bundle + (`open_ohlc`) both build using only `aura-ingest`'s public exports. In the + migrated OHLC / walk-forward consumer layer (the examples + tests this spec + names, including `ger40_breakout_compare.rs`) **no `data_server::` import + remains**. Out of scope: the gated single-field ms-path tests (`real_bars.rs`, + `unbounded_window.rs`, `ger40_archive_utc_dst.rs`, `streaming_seam.rs`) and + `aura-cli`'s single-field open legitimately keep their `data_server` import — + they are not part of this OHLC-opener migration. +2. A real-data walk-forward feeds `WindowRoller` `Timestamp` bounds straight + into `open_ohlc` with zero consumer-side `ns_to_ms` / `/ 1_000_000`. +3. The four OHLC sources are produced in `Open, High, Low, Close` order by the + single library helper (the C4 merge tie-break lives in one vetted place). +4. `epoch_ns_to_unix_ms` is private (the seam owns the convention; not part of + the public surface). +5. Behaviour-preserving: the full suite is green and every run result is + byte-identical to before (same `RunReport`s); the existing ms-based `open` + and `unix_ms_to_epoch_ns` *signatures* are untouched (consumer files do + change — that is the migration, not a behaviour change). +6. The migration is complete: `open_ohlc_sources` and every consumer-side + `ns_to_ms` are gone, `utc_month_window` is `Timestamp`-typed, and every + former caller opens via `open_ohlc`. The type checker enumerates most of the + set (deleting `open_ohlc_sources` / retyping `utc_month_window` / + `report_from_trace` turns each into a compile error), **except** + `ger40_breakout_compare.rs`, which opens OHLC via its own hand-spelled + O/H/L/C loop and imports `data_server` directly — it compiles unchanged and + is migrated by inspection, not by a compile error. No ms-based shim remains + in the consumer layer.