Brummel/Aura
1
0
Fork 0
Code Issues 33 Pull Requests Actions Packages Projects Releases Wiki Activity

spec: real-data source-open seam (Runway-1)

Browse Source 
Consolidate aura-ingest's OHLC source-open surface onto the engine-native
epoch-ns Timestamp currency: a private seam-owned inverse, a Timestamp-window
opener, a canonical OHLC bundle opener (fixed C4 O/H/L/C order), and a
DataServer/DEFAULT_DATA_PATH re-export + default_data_server convenience — so a
real-data source builds from aura-ingest alone, with the one ms<->ns crossing
owned by the seam. Behaviour-preserving (byte-identical RunReports).

Grounded against the tree via the ground-spec-0052 workflow (25/28 assertions
confirmed; compare.rs compiler-invisible site + AC1 scope fix applied).

refs #80, #81, #92
This commit is contained in:
Brummel
2026-06-18 12:26:11 +02:00
parent aedaa5d11b
commit fa9dac35aa
1 changed files with 249 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/0052-real-data-source-open-seam.md
+249
View File
@@ -0,0 +1,249 @@
# Real-data source-open seam — Design Spec
**Date:** 2026-06-18
**Status:** Grounded (workflow `ground-spec-0052`: 25/28 assertions confirmed; 1 blocker + 1 scope fix applied below); proceeding to planner per user instruction
**Authors:** orchestrator + Claude
**Cycle:** Runway-1 — milestone "Runway — real-data ergonomics & honesty hardening". Closes #80, #81, #92.
**Note:** Per-cycle specs are git-tracked but ephemeral — committed while their cycle is live and removed (`git rm`) at cycle close (the git-ignore of 28958f2 was reverted in aedaa5d). This file is a working artifact for the planner this cycle, never a durable API reference.
## Goal
Consolidate the `aura-ingest` public surface so a real-data source — a single
base column **and** the OHLC bundle — is buildable from `aura-ingest` alone,
with the epoch-ns ↔ Unix-ms window-currency owned by the seam. This removes two
field-tested frictions: the per-consumer `ns_to_ms` hand-divide (a transposable
silent-wrong-window class, #80) and the duplicated, order-sensitive OHLC opener
(#92), plus the undocumented direct reach into the external `data_server` crate
(#81). Behaviour-preserving: existing runs reach byte-identical `RunReport`s.
## Architecture
Four additions to `crates/aura-ingest/src/lib.rs`, all additive; the existing
ms-based `M1FieldSource::open` and `unix_ms_to_epoch_ns` are untouched (the
behaviour-preserving guarantee for current call sites).
1. **Seam-owned inverse (private).** `epoch_ns_to_unix_ms(Timestamp) -> i64`,
the inverse of the existing `unix_ms_to_epoch_ns`. **Not** `pub`: the seam
owns the ms↔ns convention (C3); a consumer never converts. (This is the #80
decision — see the reconciliation comment on #80.)
2. **Timestamp-window opener.** `M1FieldSource::open_window(server, symbol,
from: Option<Timestamp>, to: Option<Timestamp>, field) -> Option<Self>`,
mirroring the existing `open` but in the engine's native `Timestamp`
(epoch-ns) currency; it maps each bound through `epoch_ns_to_unix_ms` and
delegates to `open`. `Option` preserves open-ended windows, as `open` does.
3. **Canonical OHLC opener.** `open_ohlc(server, symbol, from: Timestamp, to:
Timestamp) -> Option<Vec<Box<dyn Source>>>` builds the four `M1FieldSource`s
for `Open, High, Low, Close` **in that fixed order** (the C4 merge tie-break
the resampler's `Barrier(0)` group depends on, #92), each via `open_window`,
all sharing the one `Arc<DataServer>`. Closed `Timestamp` window (a backtest /
walk-forward fold is always bounded). This is the single vetted home of the
O/H/L/C order; the trap of hand-spelling it four times is gone.
4. **Default-archive convenience + re-exports.** `aura-ingest` re-exports
`DataServer` and `DEFAULT_DATA_PATH` from `data_server`, and adds
`default_data_server() -> Arc<DataServer>` = `Arc::new(DataServer::new(
DEFAULT_DATA_PATH))`. One shared `Arc<DataServer>` (one cache) flows into
`open_ohlc` across the four fields and across disjoint sims — the C12
sharing the streaming model rests on. (This is the #81 decision; the
issue-sketched per-field `open_default_archive` was rejected because it would
build one cache per field — see the reconciliation comment on #81.)
**Return type — `Vec` not `[_; 4]`.** `Harness::run` consumes
`Vec<Box<dyn Source>>` (harness.rs:371), so a `Vec` feeds straight in with no
conversion; the order-safety #92 asks for is delivered by the single vetted
helper regardless of container. (Issue #92 sketched `[_; 4]` as an "e.g."; this
is the non-load-bearing container choice.)
## Concrete code shapes
### User-facing worked example — the walk-forward fold closure (the #80/#81/#92 evidence)
**Before** (today; `crates/aura-ingest/examples/ger40_breakout_walkforward.rs`
and `tests/ger40_breakout_world.rs`), each consumer re-deriving the seam's job:
```rust
use data_server::{DataServer, DEFAULT_DATA_PATH}; // #81: direct reach into the external crate
// #80: local hand-divide, re-written per consumer; transposable from/to
fn ns_to_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }
// #92: local open_ohlc_sources, hardcoded to SYMBOL, O/H/L/C order spelled by hand
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
let (is_from, is_to) = (ns_to_ms(w.is.0), ns_to_ms(w.is.1)); // <- the divide
let sources = open_ohlc_sources(&server, is_from, is_to).expect("…");
/* bootstrap + run + summarize */
});
```
**After** — the seam owns the currency, the opener, and the server construction:
```rust
use aura_ingest::{default_data_server, open_ohlc}; // builds from aura-ingest alone
let server = default_data_server(); // one shared Arc<DataServer> (C12)
let result = walk_forward(roller, space, move |w: WindowBounds| -> WindowRun {
let sources = open_ohlc(&server, SYMBOL, w.is.0, w.is.1) // Timestamp in — no divide
.expect("window overlaps GER40 data");
/* bootstrap + run + summarize */
});
```
No `ns_to_ms`, no local OHLC helper, no `data_server::` import.
### Before → after API surface (secondary)
```rust
// lib.rs — ADD (private):
fn epoch_ns_to_unix_ms(ts: Timestamp) -> i64 { ts.0 / 1_000_000 }
// lib.rs — ADD on impl M1FieldSource:
pub fn open_window(
server: &Arc<DataServer>, symbol: &str,
from: Option<Timestamp>, to: Option<Timestamp>, field: M1Field,
) -> Option<Self> {
Self::open(server, symbol, from.map(epoch_ns_to_unix_ms), to.map(epoch_ns_to_unix_ms), field)
}
// lib.rs — ADD (free fns):
pub fn open_ohlc(
server: &Arc<DataServer>, symbol: &str, from: Timestamp, to: Timestamp,
) -> Option<Vec<Box<dyn Source>>>; // O,H,L,C order, shares `server`
pub fn default_data_server() -> Arc<DataServer>; // Arc::new(DataServer::new(DEFAULT_DATA_PATH))
// lib.rs — ADD (re-exports):
pub use data_server::{DataServer, DEFAULT_DATA_PATH};
// UNCHANGED (behaviour-preserving): pub fn open(.. Option<i64> ..), pub fn unix_ms_to_epoch_ns
```
## Components
- `crates/aura-ingest/src/lib.rs` — the four additions + re-exports above.
- `crates/aura-ingest/examples/shared/breakout_real.rs`:
- remove the local `open_ohlc_sources` (Unix-ms, hardcoded `SYMBOL`); the
library `open_ohlc(server, symbol, from: Timestamp, to: Timestamp)` replaces
every use of it (symbol now a parameter).
- retype the shared window-builder `utc_month_window_ms(year, month) ->
(i64, i64)` (breakout_real.rs:334) to `utc_month_window(year, month) ->
(Timestamp, Timestamp)` — the *same* UTC instants, typed as the
engine-native `Timestamp` (it wraps each bound through the existing
`unix_ms_to_epoch_ns`). This is what makes the whole consumer layer
Timestamp-native, so no caller round-trips ms→Timestamp→ms.
- retype `report_from_trace` (breakout_real.rs:407, `from_ms/to_ms: i64`,
window wrap at :422-425) to take `from/to: Timestamp` and drop the
`unix_ms_to_epoch_ns` wrap — same shape as the other shared helpers. (The
`utc_month_window` retype turns this into a compile error, so the type
checker reaches it; named here for completeness.)
- `crates/aura-ingest/examples/ger40_breakout_compare.rs` — the **one
compiler-invisible** migration site. It opens OHLC via its own hand-spelled
O/H/L/C loop (compare.rs:74-79), imports `data_server` directly (:29), and
builds its window inline via chrono (:114-121); it calls **neither**
`open_ohlc_sources` **nor** `utc_month_window_ms`, so deleting/retyping those
does **not** flag it. It must be migrated by inspection: replace the loop with
`open_ohlc`, the inline chrono window with `utc_month_window`, and the
`data_server` import with the `aura-ingest` re-exports — else it silently
survives with exactly the #92 order-trap and #81 direct-import this spec kills.
### Migration policy (all callers of the removed helper)
`open_ohlc_sources` has call sites well beyond the two walk-forward consumers —
**8 call expressions across 6 files** (today: `examples/ger40_breakout_walkforward.rs`,
`examples/ger40_breakout_sweep.rs`, `examples/ger40_breakout_real.rs`,
`tests/ger40_breakout_world.rs` (×2, :54/:228), `tests/ger40_breakout_real.rs`,
`tests/ger40_breakout_blueprint.rs` (×2, :83/:94)). The planner migrates every
*expression*, not one-per-file. Separately, `ger40_breakout_compare.rs` opens
OHLC without this helper (its own loop) and so is **not** in this census — it is
the compiler-invisible site handled below. **All** migrate to the *same*
Timestamp-native surface — there is **no** ms-based shim retained, and the only
epoch-ns↔Unix-ms conversion left anywhere in the consumer layer is the one inside
the seam:
- **Walk-forward consumers**: delete the local `ns_to_ms`; pass the
`WindowBounds` `Timestamp` bounds (`w.is.0`, `w.is.1`, `w.oos.0`, `w.oos.1`)
straight to `open_ohlc`.
- **Calendar-window consumers**: their bounds come from `utc_month_window` (now
`Timestamp`); thread that `Timestamp` window through any local helper
(`run_point` / `run_blueprint` / `run_flatgraph`, whose `from_ms/to_ms: i64`
params become `from/to: Timestamp`) into `open_ohlc`. Internal `RunManifest`
construction that previously wrapped `unix_ms_to_epoch_ns(from_ms)` now
receives the `Timestamp` directly (drop the wrap).
- **Direct `data_server::` importers**: switch `use data_server::{DataServer,
DEFAULT_DATA_PATH}` to the `aura-ingest` re-exports, and `Arc::new(
DataServer::new(DEFAULT_DATA_PATH))` to `default_data_server()`.
- **The compiler-invisible site — `ger40_breakout_compare.rs`**: migrated by
inspection, not by a compile error (it calls neither removed/retyped helper).
Replace its hand-spelled O/H/L/C loop (:74-79) with `open_ohlc`, its inline
chrono window (:114-121) with `utc_month_window`, and its direct `data_server`
import (:29) with the `aura-ingest` re-exports + `default_data_server()`.
Deleting `open_ohlc_sources` and retyping `utc_month_window` / `report_from_trace`
turns every site *except `ger40_breakout_compare.rs`* into a compile error until
migrated, so the type checker enumerates that set; the single shape above is
applied to each, and `compare.rs` is migrated by inspection. Every run is
byte-identical (same instants, same windows) — behaviour-preserving.
## Data flow
`WindowRoller` → `WindowBounds { is, oos: (Timestamp, Timestamp) }` → `open_ohlc`
(Timestamp) → `epoch_ns_to_unix_ms` (the one ms↔ns crossing, internal) →
`DataServer::stream_m1_windowed` (Unix-ms) → four `M1FieldSource` (O/H/L/C) →
`Vec<Box<dyn Source>>` → `Harness::run`. The currency boundary is crossed
exactly once, inside the seam.
## Error handling
`open_ohlc` returns `None` if any of the four field opens returns `None` (no
archived file overlaps the window) — the same file-level `?` short-circuit and
the same `Some`-with-empty vs `None` semantics the current `open` /
`open_ohlc_sources` already document (lib.rs:126-135). A window overlapping a
file but holding zero bars yields sources whose first `peek` is `None`, not a
`None` from `open_ohlc`.
## Testing strategy
- **Hermetic (no DataServer):** `epoch_ns_to_unix_ms(unix_ms_to_epoch_ns(x)) ==
x` round-trip, beside the existing `unix_ms_to_epoch_ns_scales_ms_to_ns`
(lib.rs:279). This pins the seam-owned inverse without an archive.
- **Gated (real data, skip-with-note when the archive is absent, mirroring
`real_bars.rs` / `streaming_seam.rs` / `ger40_*`):**
- `open_ohlc(&server, SYMBOL, from_ts, to_ts)` yields four sources and a
Timestamp-window backtest reaches the **same** `RunReport` as the existing
ms-path open over the equivalent window (behaviour-preserving proof).
- the migrated walk-forward example + world test run green with zero
`ns_to_ms` (the #80 evidence).
- **Full workspace suite green, unchanged** (`cargo test --workspace`); clippy
clean. No existing ms-based call site changes behaviour.
## Acceptance criteria
1. A single-field source (`M1FieldSource::open_window`) and the OHLC bundle
(`open_ohlc`) both build using only `aura-ingest`'s public exports. In the
migrated OHLC / walk-forward consumer layer (the examples + tests this spec
names, including `ger40_breakout_compare.rs`) **no `data_server::` import
remains**. Out of scope: the gated single-field ms-path tests (`real_bars.rs`,
`unbounded_window.rs`, `ger40_archive_utc_dst.rs`, `streaming_seam.rs`) and
`aura-cli`'s single-field open legitimately keep their `data_server` import —
they are not part of this OHLC-opener migration.
2. A real-data walk-forward feeds `WindowRoller` `Timestamp` bounds straight
into `open_ohlc` with zero consumer-side `ns_to_ms` / `/ 1_000_000`.
3. The four OHLC sources are produced in `Open, High, Low, Close` order by the
single library helper (the C4 merge tie-break lives in one vetted place).
4. `epoch_ns_to_unix_ms` is private (the seam owns the convention; not part of
the public surface).
5. Behaviour-preserving: the full suite is green and every run result is
byte-identical to before (same `RunReport`s); the existing ms-based `open`
and `unix_ms_to_epoch_ns` *signatures* are untouched (consumer files do
change — that is the migration, not a behaviour change).
6. The migration is complete: `open_ohlc_sources` and every consumer-side
`ns_to_ms` are gone, `utc_month_window` is `Timestamp`-typed, and every
former caller opens via `open_ohlc`. The type checker enumerates most of the
set (deleting `open_ohlc_sources` / retyping `utc_month_window` /
`report_from_trace` turns each into a compile error), **except**
`ger40_breakout_compare.rs`, which opens OHLC via its own hand-spelled
O/H/L/C loop and imports `data_server` directly — it compiles unchanged and
is migrated by inspection, not by a compile error. No ms-based shim remains
in the consumer layer.