Aura/docs/specs/0052-real-data-source-open-seam.md

Real-data source-open seam — Design Spec

Date: 2026-06-18 Status: Grounded (workflow ground-spec-0052: 25/28 assertions confirmed; 1 blocker + 1 scope fix applied below); proceeding to planner per user instruction Authors: orchestrator + Claude Cycle: Runway-1 — milestone "Runway — real-data ergonomics & honesty hardening". Closes #80, #81, #92. Note: Per-cycle specs are git-tracked but ephemeral — committed while their cycle is live and removed (git rm) at cycle close (the git-ignore of 28958f2 was reverted in aedaa5d). This file is a working artifact for the planner this cycle, never a durable API reference.

Goal

Consolidate the aura-ingest public surface so a real-data source — a single base column and the OHLC bundle — is buildable from aura-ingest alone, with the epoch-ns ↔ Unix-ms window-currency owned by the seam. This removes two field-tested frictions: the per-consumer ns_to_ms hand-divide (a transposable silent-wrong-window class, #80) and the duplicated, order-sensitive OHLC opener (#92), plus the undocumented direct reach into the external data_server crate (#81). Behaviour-preserving: existing runs reach byte-identical RunReports.

Architecture

Four additions to crates/aura-ingest/src/lib.rs, all additive; the existing ms-based M1FieldSource::open and unix_ms_to_epoch_ns are untouched (the behaviour-preserving guarantee for current call sites).

1. Seam-owned inverse (private). epoch_ns_to_unix_ms(Timestamp) -> i64, the inverse of the existing unix_ms_to_epoch_ns. Not pub: the seam owns the ms↔ns convention (C3); a consumer never converts. (This is the #80 decision — see the reconciliation comment on #80.)

2. Timestamp-window opener. M1FieldSource::open_window(server, symbol, from: Option<Timestamp>, to: Option<Timestamp>, field) -> Option<Self>, mirroring the existing open but in the engine's native Timestamp (epoch-ns) currency; it maps each bound through epoch_ns_to_unix_ms and delegates to open. Option preserves open-ended windows, as open does.

3. Canonical OHLC opener. open_ohlc(server, symbol, from: Timestamp, to: Timestamp) -> Option<Vec<Box<dyn Source>>> builds the four M1FieldSources for Open, High, Low, Close in that fixed order (the C4 merge tie-break the resampler's Barrier(0) group depends on, #92), each via open_window, all sharing the one Arc<DataServer>. Closed Timestamp window (a backtest / walk-forward fold is always bounded). This is the single vetted home of the O/H/L/C order; the trap of hand-spelling it four times is gone.

4. Default-archive convenience + re-exports. aura-ingest re-exports DataServer and DEFAULT_DATA_PATH from data_server, and adds default_data_server() -> Arc<DataServer> = Arc::new(DataServer::new( DEFAULT_DATA_PATH)). One shared Arc<DataServer> (one cache) flows into open_ohlc across the four fields and across disjoint sims — the C12 sharing the streaming model rests on. (This is the #81 decision; the issue-sketched per-field open_default_archive was rejected because it would build one cache per field — see the reconciliation comment on #81.)

Return type — Vec not [_; 4]. Harness::run consumes Vec<Box<dyn Source>> (harness.rs:371), so a Vec feeds straight in with no conversion; the order-safety #92 asks for is delivered by the single vetted helper regardless of container. (Issue #92 sketched [_; 4] as an "e.g."; this is the non-load-bearing container choice.)

Concrete code shapes

User-facing worked example — the walk-forward fold closure (the #80/#81/#92 evidence)

Before (today; crates/aura-ingest/examples/ger40_breakout_walkforward.rs and tests/ger40_breakout_world.rs), each consumer re-deriving the seam's job:

use data_server::{DataServer, DEFAULT_DATA_PATH};   // #81: direct reach into the external crate

// #80: local hand-divide, re-written per consumer; transposable from/to
fn ns_to_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }

// #92: local open_ohlc_sources, hardcoded to SYMBOL, O/H/L/C order spelled by hand
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
    let (is_from, is_to) = (ns_to_ms(w.is.0), ns_to_ms(w.is.1));        // <- the divide
    let sources = open_ohlc_sources(&server, is_from, is_to).expect("…");
    /* bootstrap + run + summarize */
});

After — the seam owns the currency, the opener, and the server construction:

use aura_ingest::{default_data_server, open_ohlc};   // builds from aura-ingest alone

let server = default_data_server();                   // one shared Arc<DataServer> (C12)
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
    let sources = open_ohlc(&server, SYMBOL, w.is.0, w.is.1)   // Timestamp in — no divide
        .expect("window overlaps GER40 data");
    /* bootstrap + run + summarize */
});

No ns_to_ms, no local OHLC helper, no data_server:: import.

Before → after API surface (secondary)

// lib.rs — ADD (private):
fn epoch_ns_to_unix_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }

// lib.rs — ADD on impl M1FieldSource:
pub fn open_window(
    server: &Arc<DataServer>, symbol: &str,
    from: Option<Timestamp>, to: Option<Timestamp>, field: M1Field,
) -> Option<Self> {
    Self::open(server, symbol, from.map(epoch_ns_to_unix_ms), to.map(epoch_ns_to_unix_ms), field)
}

// lib.rs — ADD (free fns):
pub fn open_ohlc(
    server: &Arc<DataServer>, symbol: &str, from: Timestamp, to: Timestamp,
) -> Option<Vec<Box<dyn Source>>>;   // O,H,L,C order, shares `server`

pub fn default_data_server() -> Arc<DataServer>;   // Arc::new(DataServer::new(DEFAULT_DATA_PATH))

// lib.rs — ADD (re-exports):
pub use data_server::{DataServer, DEFAULT_DATA_PATH};

// UNCHANGED (behaviour-preserving): pub fn open(.. Option<i64> ..), pub fn unix_ms_to_epoch_ns

Components

• crates/aura-ingest/src/lib.rs — the four additions + re-exports above.
• crates/aura-ingest/examples/shared/breakout_real.rs:
  • remove the local open_ohlc_sources (Unix-ms, hardcoded SYMBOL); the library open_ohlc(server, symbol, from: Timestamp, to: Timestamp) replaces every use of it (symbol now a parameter).
  • retype the shared window-builder utc_month_window_ms(year, month) -> (i64, i64) (breakout_real.rs:334) to utc_month_window(year, month) -> (Timestamp, Timestamp) — the same UTC instants, typed as the engine-native Timestamp (it wraps each bound through the existing unix_ms_to_epoch_ns). This is what makes the whole consumer layer Timestamp-native, so no caller round-trips ms→Timestamp→ms.
  • retype report_from_trace (breakout_real.rs:407, from_ms/to_ms: i64, window wrap at :422-425) to take from/to: Timestamp and drop the unix_ms_to_epoch_ns wrap — same shape as the other shared helpers. (The utc_month_window retype turns this into a compile error, so the type checker reaches it; named here for completeness.)
• crates/aura-ingest/examples/ger40_breakout_compare.rs — the one compiler-invisible migration site. It opens OHLC via its own hand-spelled O/H/L/C loop (compare.rs:74-79), imports data_server directly (:29), and builds its window inline via chrono (:114-121); it calls neither open_ohlc_sources nor utc_month_window_ms, so deleting/retyping those does not flag it. It must be migrated by inspection: replace the loop with open_ohlc, the inline chrono window with utc_month_window, and the data_server import with the aura-ingest re-exports — else it silently survives with exactly the #92 order-trap and #81 direct-import this spec kills.

Migration policy (all callers of the removed helper)

open_ohlc_sources has call sites well beyond the two walk-forward consumers — 8 call expressions across 6 files (today: examples/ger40_breakout_walkforward.rs, examples/ger40_breakout_sweep.rs, examples/ger40_breakout_real.rs, tests/ger40_breakout_world.rs (×2, :54/:228), tests/ger40_breakout_real.rs, tests/ger40_breakout_blueprint.rs (×2, :83/:94)). The planner migrates every expression, not one-per-file. Separately, ger40_breakout_compare.rs opens OHLC without this helper (its own loop) and so is not in this census — it is the compiler-invisible site handled below. All migrate to the same Timestamp-native surface — there is no ms-based shim retained, and the only epoch-ns↔Unix-ms conversion left anywhere in the consumer layer is the one inside the seam:

• Walk-forward consumers: delete the local ns_to_ms; pass the WindowBounds Timestamp bounds (w.is.0, w.is.1, w.oos.0, w.oos.1) straight to open_ohlc.
• Calendar-window consumers: their bounds come from utc_month_window (now Timestamp); thread that Timestamp window through any local helper (run_point / run_blueprint / run_flatgraph, whose from_ms/to_ms: i64 params become from/to: Timestamp) into open_ohlc. Internal RunManifest construction that previously wrapped unix_ms_to_epoch_ns(from_ms) now receives the Timestamp directly (drop the wrap).
• Direct data_server:: importers: switch use data_server::{DataServer, DEFAULT_DATA_PATH} to the aura-ingest re-exports, and Arc::new( DataServer::new(DEFAULT_DATA_PATH)) to default_data_server().
• The compiler-invisible site — ger40_breakout_compare.rs: migrated by inspection, not by a compile error (it calls neither removed/retyped helper). Replace its hand-spelled O/H/L/C loop (:74-79) with open_ohlc, its inline chrono window (:114-121) with utc_month_window, and its direct data_server import (:29) with the aura-ingest re-exports + default_data_server().

Deleting open_ohlc_sources and retyping utc_month_window / report_from_trace turns every site except ger40_breakout_compare.rs into a compile error until migrated, so the type checker enumerates that set; the single shape above is applied to each, and compare.rs is migrated by inspection. Every run is byte-identical (same instants, same windows) — behaviour-preserving.

Data flow

WindowRoller → WindowBounds { is, oos: (Timestamp, Timestamp) } → open_ohlc (Timestamp) → epoch_ns_to_unix_ms (the one ms↔ns crossing, internal) → DataServer::stream_m1_windowed (Unix-ms) → four M1FieldSource (O/H/L/C) → Vec<Box<dyn Source>> → Harness::run. The currency boundary is crossed exactly once, inside the seam.

Error handling

open_ohlc returns None if any of the four field opens returns None (no archived file overlaps the window) — the same file-level ? short-circuit and the same Some-with-empty vs None semantics the current open / open_ohlc_sources already document (lib.rs:126-135). A window overlapping a file but holding zero bars yields sources whose first peek is None, not a None from open_ohlc.

Testing strategy

• Hermetic (no DataServer): epoch_ns_to_unix_ms(unix_ms_to_epoch_ns(x)) == x round-trip, beside the existing unix_ms_to_epoch_ns_scales_ms_to_ns (lib.rs:279). This pins the seam-owned inverse without an archive.
• Gated (real data, skip-with-note when the archive is absent, mirroring real_bars.rs / streaming_seam.rs / ger40_*):
  • open_ohlc(&server, SYMBOL, from_ts, to_ts) yields four sources and a Timestamp-window backtest reaches the same RunReport as the existing ms-path open over the equivalent window (behaviour-preserving proof).
  • the migrated walk-forward example + world test run green with zero ns_to_ms (the #80 evidence).
• Full workspace suite green, unchanged (cargo test --workspace); clippy clean. No existing ms-based call site changes behaviour.

Acceptance criteria

1. A single-field source (M1FieldSource::open_window) and the OHLC bundle (open_ohlc) both build using only aura-ingest's public exports. In the migrated OHLC / walk-forward consumer layer (the examples + tests this spec names, including ger40_breakout_compare.rs) no data_server:: import remains. Out of scope: the gated single-field ms-path tests (real_bars.rs, unbounded_window.rs, ger40_archive_utc_dst.rs, streaming_seam.rs) and aura-cli's single-field open legitimately keep their data_server import — they are not part of this OHLC-opener migration.
2. A real-data walk-forward feeds WindowRoller Timestamp bounds straight into open_ohlc with zero consumer-side ns_to_ms / / 1_000_000.
3. The four OHLC sources are produced in Open, High, Low, Close order by the single library helper (the C4 merge tie-break lives in one vetted place).
4. epoch_ns_to_unix_ms is private (the seam owns the convention; not part of the public surface).
5. Behaviour-preserving: the full suite is green and every run result is byte-identical to before (same RunReports); the existing ms-based open and unix_ms_to_epoch_ns signatures are untouched (consumer files do change — that is the migration, not a behaviour change).
6. The migration is complete: open_ohlc_sources and every consumer-side ns_to_ms are gone, utc_month_window is Timestamp-typed, and every former caller opens via open_ohlc. The type checker enumerates most of the set (deleting open_ohlc_sources / retyping utc_month_window / report_from_trace turns each into a compile error), except ger40_breakout_compare.rs, which opens OHLC via its own hand-spelled O/H/L/C loop and imports data_server directly — it compiles unchanged and is migrated by inspection, not by a compile error. No ms-based shim remains in the consumer layer.