Aura/docs/specs/0052-real-data-source-open-seam.md

# Real-data source-open seam — Design Spec

**Date:** 2026-06-18
**Status:** Grounded (workflow `ground-spec-0052`: 25/28 assertions confirmed; 1 blocker + 1 scope fix applied below); proceeding to planner per user instruction
**Authors:** orchestrator + Claude
**Cycle:** Runway-1 — milestone "Runway — real-data ergonomics & honesty hardening". Closes #80, #81, #92.
**Note:** Per-cycle specs are git-tracked but ephemeral — committed while their cycle is live and removed (`git rm`) at cycle close (the git-ignore of 28958f2 was reverted in aedaa5d). This file is a working artifact for the planner this cycle, never a durable API reference.

## Goal

Consolidate the `aura-ingest` public surface so a real-data source — a single
base column **and** the OHLC bundle — is buildable from `aura-ingest` alone,
with the epoch-ns ↔ Unix-ms window-currency owned by the seam. This removes two
field-tested frictions: the per-consumer `ns_to_ms` hand-divide (a transposable
silent-wrong-window class, #80) and the duplicated, order-sensitive OHLC opener
(#92), plus the undocumented direct reach into the external `data_server` crate
(#81). Behaviour-preserving: existing runs reach byte-identical `RunReport`s.

## Architecture

Four additions to `crates/aura-ingest/src/lib.rs`, all additive; the existing
ms-based `M1FieldSource::open` and `unix_ms_to_epoch_ns` are untouched (the
behaviour-preserving guarantee for current call sites).

1. **Seam-owned inverse (private).** `epoch_ns_to_unix_ms(Timestamp) -> i64`,
   the inverse of the existing `unix_ms_to_epoch_ns`. **Not** `pub`: the seam
   owns the ms↔ns convention (C3); a consumer never converts. (This is the #80
   decision — see the reconciliation comment on #80.)

2. **Timestamp-window opener.** `M1FieldSource::open_window(server, symbol,
   from: Option<Timestamp>, to: Option<Timestamp>, field) -> Option<Self>`,
   mirroring the existing `open` but in the engine's native `Timestamp`
   (epoch-ns) currency; it maps each bound through `epoch_ns_to_unix_ms` and
   delegates to `open`. `Option` preserves open-ended windows, as `open` does.

3. **Canonical OHLC opener.** `open_ohlc(server, symbol, from: Timestamp, to:
   Timestamp) -> Option<Vec<Box<dyn Source>>>` builds the four `M1FieldSource`s
   for `Open, High, Low, Close` **in that fixed order** (the C4 merge tie-break
   the resampler's `Barrier(0)` group depends on, #92), each via `open_window`,
   all sharing the one `Arc<DataServer>`. Closed `Timestamp` window (a backtest /
   walk-forward fold is always bounded). This is the single vetted home of the
   O/H/L/C order; the trap of hand-spelling it four times is gone.

4. **Default-archive convenience + re-exports.** `aura-ingest` re-exports
   `DataServer` and `DEFAULT_DATA_PATH` from `data_server`, and adds
   `default_data_server() -> Arc<DataServer>` = `Arc::new(DataServer::new(
   DEFAULT_DATA_PATH))`. One shared `Arc<DataServer>` (one cache) flows into
   `open_ohlc` across the four fields and across disjoint sims — the C12
   sharing the streaming model rests on. (This is the #81 decision; the
   issue-sketched per-field `open_default_archive` was rejected because it would
   build one cache per field — see the reconciliation comment on #81.)

**Return type — `Vec` not `[_; 4]`.** `Harness::run` consumes
`Vec<Box<dyn Source>>` (harness.rs:371), so a `Vec` feeds straight in with no
conversion; the order-safety #92 asks for is delivered by the single vetted
helper regardless of container. (Issue #92 sketched `[_; 4]` as an "e.g."; this
is the non-load-bearing container choice.)

## Concrete code shapes

### User-facing worked example — the walk-forward fold closure (the #80/#81/#92 evidence)

**Before** (today; `crates/aura-ingest/examples/ger40_breakout_walkforward.rs`
and `tests/ger40_breakout_world.rs`), each consumer re-deriving the seam's job:

```rust
use data_server::{DataServer, DEFAULT_DATA_PATH};   // #81: direct reach into the external crate

// #80: local hand-divide, re-written per consumer; transposable from/to
fn ns_to_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }

// #92: local open_ohlc_sources, hardcoded to SYMBOL, O/H/L/C order spelled by hand
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
    let (is_from, is_to) = (ns_to_ms(w.is.0), ns_to_ms(w.is.1));        // <- the divide
    let sources = open_ohlc_sources(&server, is_from, is_to).expect("…");
    /* bootstrap + run + summarize */
});
```

**After** — the seam owns the currency, the opener, and the server construction:

```rust
use aura_ingest::{default_data_server, open_ohlc};   // builds from aura-ingest alone

let server = default_data_server();                   // one shared Arc<DataServer> (C12)
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
    let sources = open_ohlc(&server, SYMBOL, w.is.0, w.is.1)   // Timestamp in — no divide
        .expect("window overlaps GER40 data");
    /* bootstrap + run + summarize */
});
```

No `ns_to_ms`, no local OHLC helper, no `data_server::` import.

### Before → after API surface (secondary)

```rust
// lib.rs — ADD (private):
fn epoch_ns_to_unix_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }

// lib.rs — ADD on impl M1FieldSource:
pub fn open_window(
    server: &Arc<DataServer>, symbol: &str,
    from: Option<Timestamp>, to: Option<Timestamp>, field: M1Field,
) -> Option<Self> {
    Self::open(server, symbol, from.map(epoch_ns_to_unix_ms), to.map(epoch_ns_to_unix_ms), field)
}

// lib.rs — ADD (free fns):
pub fn open_ohlc(
    server: &Arc<DataServer>, symbol: &str, from: Timestamp, to: Timestamp,
) -> Option<Vec<Box<dyn Source>>>;   // O,H,L,C order, shares `server`

pub fn default_data_server() -> Arc<DataServer>;   // Arc::new(DataServer::new(DEFAULT_DATA_PATH))

// lib.rs — ADD (re-exports):
pub use data_server::{DataServer, DEFAULT_DATA_PATH};

// UNCHANGED (behaviour-preserving): pub fn open(.. Option<i64> ..), pub fn unix_ms_to_epoch_ns
```

## Components

- `crates/aura-ingest/src/lib.rs` — the four additions + re-exports above.
- `crates/aura-ingest/examples/shared/breakout_real.rs`:
  - remove the local `open_ohlc_sources` (Unix-ms, hardcoded `SYMBOL`); the
    library `open_ohlc(server, symbol, from: Timestamp, to: Timestamp)` replaces
    every use of it (symbol now a parameter).
  - retype the shared window-builder `utc_month_window_ms(year, month) ->
    (i64, i64)` (breakout_real.rs:334) to `utc_month_window(year, month) ->
    (Timestamp, Timestamp)` — the *same* UTC instants, typed as the
    engine-native `Timestamp` (it wraps each bound through the existing
    `unix_ms_to_epoch_ns`). This is what makes the whole consumer layer
    Timestamp-native, so no caller round-trips ms→Timestamp→ms.
  - retype `report_from_trace` (breakout_real.rs:407, `from_ms/to_ms: i64`,
    window wrap at :422-425) to take `from/to: Timestamp` and drop the
    `unix_ms_to_epoch_ns` wrap — same shape as the other shared helpers. (The
    `utc_month_window` retype turns this into a compile error, so the type
    checker reaches it; named here for completeness.)
- `crates/aura-ingest/examples/ger40_breakout_compare.rs` — the **one
  compiler-invisible** migration site. It opens OHLC via its own hand-spelled
  O/H/L/C loop (compare.rs:74-79), imports `data_server` directly (:29), and
  builds its window inline via chrono (:114-121); it calls **neither**
  `open_ohlc_sources` **nor** `utc_month_window_ms`, so deleting/retyping those
  does **not** flag it. It must be migrated by inspection: replace the loop with
  `open_ohlc`, the inline chrono window with `utc_month_window`, and the
  `data_server` import with the `aura-ingest` re-exports — else it silently
  survives with exactly the #92 order-trap and #81 direct-import this spec kills.

### Migration policy (all callers of the removed helper)

`open_ohlc_sources` has call sites well beyond the two walk-forward consumers —
**8 call expressions across 6 files** (today: `examples/ger40_breakout_walkforward.rs`,
`examples/ger40_breakout_sweep.rs`, `examples/ger40_breakout_real.rs`,
`tests/ger40_breakout_world.rs` (×2, :54/:228), `tests/ger40_breakout_real.rs`,
`tests/ger40_breakout_blueprint.rs` (×2, :83/:94)). The planner migrates every
*expression*, not one-per-file. Separately, `ger40_breakout_compare.rs` opens
OHLC without this helper (its own loop) and so is **not** in this census — it is
the compiler-invisible site handled below. **All** migrate to the *same*
Timestamp-native surface — there is **no** ms-based shim retained, and the only
epoch-ns↔Unix-ms conversion left anywhere in the consumer layer is the one inside
the seam:

- **Walk-forward consumers**: delete the local `ns_to_ms`; pass the
  `WindowBounds` `Timestamp` bounds (`w.is.0`, `w.is.1`, `w.oos.0`, `w.oos.1`)
  straight to `open_ohlc`.
- **Calendar-window consumers**: their bounds come from `utc_month_window` (now
  `Timestamp`); thread that `Timestamp` window through any local helper
  (`run_point` / `run_blueprint` / `run_flatgraph`, whose `from_ms/to_ms: i64`
  params become `from/to: Timestamp`) into `open_ohlc`. Internal `RunManifest`
  construction that previously wrapped `unix_ms_to_epoch_ns(from_ms)` now
  receives the `Timestamp` directly (drop the wrap).
- **Direct `data_server::` importers**: switch `use data_server::{DataServer,
  DEFAULT_DATA_PATH}` to the `aura-ingest` re-exports, and `Arc::new(
  DataServer::new(DEFAULT_DATA_PATH))` to `default_data_server()`.
- **The compiler-invisible site — `ger40_breakout_compare.rs`**: migrated by
  inspection, not by a compile error (it calls neither removed/retyped helper).
  Replace its hand-spelled O/H/L/C loop (:74-79) with `open_ohlc`, its inline
  chrono window (:114-121) with `utc_month_window`, and its direct `data_server`
  import (:29) with the `aura-ingest` re-exports + `default_data_server()`.

Deleting `open_ohlc_sources` and retyping `utc_month_window` / `report_from_trace`
turns every site *except `ger40_breakout_compare.rs`* into a compile error until
migrated, so the type checker enumerates that set; the single shape above is
applied to each, and `compare.rs` is migrated by inspection. Every run is
byte-identical (same instants, same windows) — behaviour-preserving.

## Data flow

`WindowRoller` → `WindowBounds { is, oos: (Timestamp, Timestamp) }` → `open_ohlc`
(Timestamp) → `epoch_ns_to_unix_ms` (the one ms↔ns crossing, internal) →
`DataServer::stream_m1_windowed` (Unix-ms) → four `M1FieldSource` (O/H/L/C) →
`Vec<Box<dyn Source>>` → `Harness::run`. The currency boundary is crossed
exactly once, inside the seam.

## Error handling

`open_ohlc` returns `None` if any of the four field opens returns `None` (no
archived file overlaps the window) — the same file-level `?` short-circuit and
the same `Some`-with-empty vs `None` semantics the current `open` /
`open_ohlc_sources` already document (lib.rs:126-135). A window overlapping a
file but holding zero bars yields sources whose first `peek` is `None`, not a
`None` from `open_ohlc`.

## Testing strategy

- **Hermetic (no DataServer):** `epoch_ns_to_unix_ms(unix_ms_to_epoch_ns(x)) ==
  x` round-trip, beside the existing `unix_ms_to_epoch_ns_scales_ms_to_ns`
  (lib.rs:279). This pins the seam-owned inverse without an archive.
- **Gated (real data, skip-with-note when the archive is absent, mirroring
  `real_bars.rs` / `streaming_seam.rs` / `ger40_*`):**
  - `open_ohlc(&server, SYMBOL, from_ts, to_ts)` yields four sources and a
    Timestamp-window backtest reaches the **same** `RunReport` as the existing
    ms-path open over the equivalent window (behaviour-preserving proof).
  - the migrated walk-forward example + world test run green with zero
    `ns_to_ms` (the #80 evidence).
- **Full workspace suite green, unchanged** (`cargo test --workspace`); clippy
  clean. No existing ms-based call site changes behaviour.

## Acceptance criteria

1. A single-field source (`M1FieldSource::open_window`) and the OHLC bundle
   (`open_ohlc`) both build using only `aura-ingest`'s public exports. In the
   migrated OHLC / walk-forward consumer layer (the examples + tests this spec
   names, including `ger40_breakout_compare.rs`) **no `data_server::` import
   remains**. Out of scope: the gated single-field ms-path tests (`real_bars.rs`,
   `unbounded_window.rs`, `ger40_archive_utc_dst.rs`, `streaming_seam.rs`) and
   `aura-cli`'s single-field open legitimately keep their `data_server` import —
   they are not part of this OHLC-opener migration.
2. A real-data walk-forward feeds `WindowRoller` `Timestamp` bounds straight
   into `open_ohlc` with zero consumer-side `ns_to_ms` / `/ 1_000_000`.
3. The four OHLC sources are produced in `Open, High, Low, Close` order by the
   single library helper (the C4 merge tie-break lives in one vetted place).
4. `epoch_ns_to_unix_ms` is private (the seam owns the convention; not part of
   the public surface).
5. Behaviour-preserving: the full suite is green and every run result is
   byte-identical to before (same `RunReport`s); the existing ms-based `open`
   and `unix_ms_to_epoch_ns` *signatures* are untouched (consumer files do
   change — that is the migration, not a behaviour change).
6. The migration is complete: `open_ohlc_sources` and every consumer-side
   `ns_to_ms` are gone, `utc_month_window` is `Timestamp`-typed, and every
   former caller opens via `open_ohlc`. The type checker enumerates most of the
   set (deleting `open_ohlc_sources` / retyping `utc_month_window` /
   `report_from_trace` turns each into a compile error), **except**
   `ger40_breakout_compare.rs`, which opens OHLC via its own hand-spelled
   O/H/L/C loop and imports `data_server` directly — it compiles unchanged and
   is migrated by inspection, not by a compile error. No ms-based shim remains
   in the consumer layer.