Aura/docs/specs/0011-ingest-data-server-m1.md

# data-server M1 ingestion at the one merge boundary — Design Spec

**Date:** 2026-06-04
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

## Goal

Replace hand-built synthetic price streams with aura's first **real** data source.
This cycle ships the **ingestion boundary** (C3): a new `aura-ingest` crate that
pulls `data-server` as a cargo git dependency, transposes its Array-of-Structs
`M1Parsed` records into aura's Structure-of-Arrays base columns (C7), normalizes
`data-server`'s Unix-millisecond time to aura's canonical epoch-ns `Timestamp` at
that one boundary (C3), and feeds the existing k-way merge (`Harness::run`) a real
close-price stream — so a backtest runs over real bars, deterministically (C1).

Scope is the **boundary plus its tests**, not a new CLI surface. The `aura run`
CLI keeps its argument-free built-in sample harness (cycle 0010); wiring a
`--symbol/--from/--to` invocation surface is a deliberate follow-up cycle, kept
out of this one because it would force the still-unsettled `aura run` arg-parse
strictness (issue #16) and a date-format surface before the boundary itself is
proven. M1 is the format shipped here; tick (ask/bid) is a natural extension of
the same transpose, deferred to a follow-up.

## Architecture

`data-server` (`~/dev/libs/data-server`, Gitea `Brummel/data-server`) is a
standalone loader/cache for Pepperstone binary M1/tick files. It yields parsed
records lock-free as `Arc<[T]>` chunks in chronological order, with `time_ms: i64`
(Unix-ms). aura's engine consumes **materialized** ascending source streams
`Vec<Vec<(Timestamp, Scalar)>>` and performs the one k-way merge inside
`Harness::run` (cycle 0004). The ingestion boundary bridges the two.

**Eager materialization** (decided this cycle): the boundary drains a
`data-server` window into owned SoA columns and hands the engine the stream shape
it already consumes. No engine API change. C12's *cross-sim* zero-copy `Arc<[T]>`
sharing is **not** realized here — it has no consumer yet (the multi-sim
orchestration axes are unbuilt), so building a lazy/shared `Source` abstraction
now would be speculative infra for an absent consumer. It is deferred to the
orchestration cycle that actually shares one dataset across many harnesses; the
transpose logic written here carries over to that layer unchanged.

**`aura-ingest` is the external-dependency firewall.** The aura workspace is
zero-external-dependency by deep design (the hand-rolled JSON in
`aura-engine::report` exists precisely to keep it so). `data-server` transitively
pulls `chrono`, `regex`, and `zip` — the one sanctioned external dependency
(INDEX.md, External components; this issue). Isolating it in its **own** crate
keeps `aura-core` / `aura-std` / `aura-engine` — the hot path and the frozen
deploy artifact (C8/C6: deploy replays *recorded* streams, never re-ingests) —
external-dependency-free. This is the substantive reason the boundary is a
dedicated crate rather than a feature-gated dep on `aura-engine`: the crate edge
*is* the firewall.

**Offline-build consequence.** With `aura-ingest` a workspace member,
`cargo build/test --workspace` now requires `data-server` to be fetchable or
already in the cargo cache (one Gitea fetch over LAN, then cached). The previous
"workspace compiles with no network" invariant becomes "compiles from a populated
cargo cache"; this is the accepted cost of the first real data source, and the
firewall confines the transitive external tree to this one crate's build.

## Concrete code shapes

### Delivered code — the boundary the cycle ships

```rust
// crates/aura-ingest/src/lib.rs
use aura_core::{Scalar, Timestamp};
use data_server::records::M1Parsed;

/// Normalize data-server's Unix-millisecond time to aura's canonical epoch-ns
/// `Timestamp` — the single unit normalization of C3, performed at the one
/// ingestion boundary and nowhere else. `ms * 1_000_000`; the i64 range covers
/// all market data through year ~2262 (well beyond any Pepperstone file).
pub fn unix_ms_to_epoch_ns(time_ms: i64) -> Timestamp {
    Timestamp(time_ms * 1_000_000)
}

/// One M1 window transposed Array-of-Structs -> Structure-of-Arrays (C7): the
/// OHLCV bar as a bundle of base columns, with time already normalized to
/// epoch-ns. `volume` is the one i64 column; the price/spread columns are f64.
/// All columns share an index: `ts[i]` is the timestamp of `close[i]`, etc.
pub struct M1Columns {
    pub ts: Vec<Timestamp>, // normalized epoch-ns, ascending (data-server is chronological)
    pub open: Vec<f64>,
    pub high: Vec<f64>,
    pub low: Vec<f64>,
    pub close: Vec<f64>,
    pub spread: Vec<f64>,
    pub volume: Vec<i64>,
}

/// Transpose data-server's AoS M1 records into aura's SoA columns (C7),
/// normalizing time at the boundary (C3). Pure: identical bars yield identical
/// columns (C1) — it reads no clock and no external state.
pub fn transpose_m1(bars: &[M1Parsed]) -> M1Columns {
    let mut c = M1Columns::with_capacity(bars.len());
    for b in bars {
        c.ts.push(unix_ms_to_epoch_ns(b.time_ms));
        c.open.push(b.open);
        c.high.push(b.high);
        c.low.push(b.low);
        c.close.push(b.close);
        c.spread.push(b.spread);
        c.volume.push(b.volume);
    }
    c
}

impl M1Columns {
    /// The close column as an engine source stream: each normalized timestamp
    /// zipped with its close as a `Scalar::F64`. Ascending in timestamp iff the
    /// bars were (the C3 ingestion precondition `Harness::run` relies on). This
    /// is the price input the sample SMA-cross strategy consumes; a project that
    /// wants other fields reads the public columns directly and zips its own.
    pub fn close_stream(&self) -> Vec<(Timestamp, Scalar)> {
        self.ts.iter().zip(&self.close).map(|(&t, &v)| (t, Scalar::F64(v))).collect()
    }
}
```

```rust
// crates/aura-ingest/src/lib.rs (data-server adapter)
use std::sync::Arc;
use data_server::DataServer;

/// Drain a data-server M1 window into transposed SoA columns. Reads the
/// chronological chunk iterator to exhaustion into one AoS buffer, then
/// transposes once at the boundary (C3 — one merge point, not per chunk).
/// `None` if the symbol has no data in `[from_ms, to_ms]` (data-server's own
/// `None`); the inclusive Unix-ms window bounds are data-server's contract.
pub fn load_m1_window(
    server: &Arc<DataServer>,
    symbol: &str,
    from_ms: i64,
    to_ms: i64,
) -> Option<M1Columns> {
    let mut it = server.stream_m1_windowed(symbol, Some(from_ms), Some(to_ms))?;
    let mut bars: Vec<M1Parsed> = Vec::new();
    while let Some(chunk) = it.next_chunk() {
        bars.extend_from_slice(&chunk); // M1Parsed: Copy; Arc<[_]> derefs to a slice
    }
    Some(transpose_m1(&bars))
}
```

### North-star slice — a real backtest over real bars (the gated integration test)

```rust
// crates/aura-ingest/tests/real_bars.rs   (skips cleanly where local data is absent)
use std::sync::Arc;
use data_server::{DataServer, DEFAULT_DATA_PATH};
use aura_ingest::load_m1_window;

#[test]
fn sample_strategy_runs_over_real_m1_bars_deterministically() {
    let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
    if !server.has_symbol("AAPL.US") {
        eprintln!("skip: no local data at {DEFAULT_DATA_PATH}");
        return; // hermetic elsewhere; exercises the real path where files exist
    }
    // 2006-08 in Unix-ms (inclusive bounds; data-server skips files outside).
    let (from_ms, to_ms) = (1_154_390_400_000, 1_157_068_799_999);

    let run = || {
        let cols = load_m1_window(&server, "AAPL.US", from_ms, to_ms)
            .expect("symbol has data in window");
        let prices = cols.close_stream();          // real close bars as the source
        let report = run_sample_over(prices);      // SMA-cross -> Exposure -> SimBroker -> report
        report
    };

    let r1 = run();
    assert!(r1.metrics.total_pips.is_finite());     // a backtest actually ran over real bars
    assert!(!r1.manifest_window_is_empty());        // the window carried real bars
    assert_eq!(run().to_json(), r1.to_json());      // same window -> bit-identical (C1)
}
```

`run_sample_over(prices)` is a test-local helper that bootstraps the cycle-0007
signal-quality wiring — `Sma::new(2)` / `Sma::new(4)` → `Sub` → `Exposure::new(0.5)`
→ `SimBroker::new(0.0001)` with two shipped `aura_std::Recorder` sinks (equity on
the broker, exposure on the `Exposure` node, price tapped into the broker) — runs
it on the single real close stream, drains the receivers, and folds
`aura_engine::{f64_field, summarize}` into a `RunReport` whose `window` is the
first/last real bar timestamp. It mirrors `aura-cli`'s `sample_harness`, fed real
prices instead of synthetic.

### Supporting shape — workspace + crate wiring (secondary)

```toml
# Cargo.toml (workspace) — add the member
members = [
    "crates/aura-core",
    "crates/aura-std",
    "crates/aura-engine",
    "crates/aura-cli",
    "crates/aura-ingest", # NEW — the ingestion boundary / external-dep firewall
]
```

```toml
# crates/aura-ingest/Cargo.toml
[package]
name = "aura-ingest"
edition.workspace = true
version.workspace = true
license.workspace = true
publish.workspace = true

[dependencies]
aura-core = { path = "../aura-core" }
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }

[dev-dependencies]
aura-engine = { path = "../aura-engine" } # the integration test bootstraps a harness + report
aura-std = { path = "../aura-std" }       # Sma / Sub / Exposure / SimBroker / Recorder
```

## Components

- **`aura-ingest` crate.** The new workspace member. Production surface:
  `unix_ms_to_epoch_ns`, `M1Columns` (public SoA fields), `transpose_m1`,
  `M1Columns::close_stream`, `load_m1_window`. Production deps: `aura-core`
  (Scalar/Timestamp) + `data-server` (records + loader). No production dependency
  on `aura-engine`/`aura-std` — the boundary produces the engine's *input shape*,
  it does not wire harnesses.
- **`transpose_m1` + `unix_ms_to_epoch_ns`.** The pure C3/C7 core: AoS→SoA and
  Unix-ms→epoch-ns. No I/O, no clock — hermetically testable on hand-built
  records.
- **`load_m1_window`.** The thin `data-server` adapter: drains the chunk iterator
  and calls `transpose_m1` once. This is the only function that touches files /
  the cache; it is exercised only by the gated integration test.
- **`M1Columns::close_stream`.** The adapter from a transposed column to the
  engine's `Vec<(Timestamp, Scalar)>` source-stream shape.

## Data flow

```
data-server file (ZIP, Unix-ms, AoS)
   │  stream_m1_windowed(symbol, from_ms, to_ms)  -> SymbolChunkIter
   ▼
Arc<[M1Parsed]> chunks  (chronological)
   │  load_m1_window: drain to one AoS buffer
   ▼
&[M1Parsed]
   │  transpose_m1  ── unix_ms_to_epoch_ns (C3 normalize) ──▶ SoA columns (C7)
   ▼
M1Columns { ts: epoch-ns, open/high/low/close/spread: f64, volume: i64 }
   │  close_stream()
   ▼
Vec<(Timestamp, Scalar::F64)>   ── the one real source stream ──┐
   │                                                            │
   ▼                                                            ▼
Harness::run(vec![stream])  ──(k-way merge, cycle 0004; one record = one cycle, C4)──▶
   SMA(2)/SMA(4) → Sub → Exposure → SimBroker → Recorder sinks
   │  drain → f64_field → summarize
   ▼
RunReport { manifest{window = first/last real bar ts}, metrics }
```

The merge is unchanged: the boundary delivers one ascending stream, so the k-way
merge is a 1-way pass and each bar is one cycle in global timestamp order (C3/C4).
Multi-stream merge (e.g. several symbols, or OHLCV-as-barrier) is already proven
by the engine's own tests; this cycle exercises the single-real-source path.

## Error handling

- **Symbol/window miss.** `load_m1_window` returns `None` when `data-server`
  returns `None` (unknown symbol, or no file in the window). The integration test
  branches on `has_symbol` first and skips cleanly; a real consumer decides
  per-call.
- **No local data files.** The integration test prints a skip note and returns
  (not a failure), so `cargo test --workspace` stays green on a machine without
  `/mnt/tickdata` — correctness lives in the hermetic unit tests, the real path is
  exercised where data exists.
- **Timestamp range.** `time_ms * 1_000_000` cannot overflow i64 for any real
  Pepperstone timestamp (year ≤ 2262); no checked arithmetic is warranted, and a
  pathological future date is out of scope (it would be a data-server-side
  concern).
- **Empty window.** A window that resolves to zero bars yields empty columns;
  `close_stream` yields an empty stream; `Harness::run` over an empty stream does
  nothing and `summarize` returns the documented zeros. No panic.

## Testing strategy

Hermetic unit tests (`crates/aura-ingest/src/lib.rs`, no files, no network):

- `unix_ms_to_epoch_ns` maps a known Unix-ms to the right epoch-ns (e.g. the
  data-server epoch test value: `1_488_326_400_000` ms → `…000_000` ns).
- `transpose_m1` on a hand-built `&[M1Parsed]` produces field-correct, equal-length
  SoA columns; `volume` lands as i64, prices/spread as f64; `ts` is normalized.
- `transpose_m1` is pure: two transposes of the same bars are equal (C1 at the
  boundary).
- `close_stream` zips ts↔close in order and preserves ascending timestamps.
- empty input → empty columns → empty stream (the no-panic edge).

Gated integration test (`crates/aura-ingest/tests/real_bars.rs`): the north-star
slice above — real close bars → sample strategy → sim-optimal broker → `RunReport`,
finite metrics, and two runs over the same window bit-identical (C1). Skips with a
note where `/mnt/tickdata` is absent.

Gates (profile `commands`): `cargo test --workspace`,
`cargo clippy --workspace --all-targets -- -D warnings`,
`RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps`. All must be clean,
which now includes building `aura-ingest` and its `data-server` dependency tree.

## Acceptance criteria

Maps the issue's acceptance boxes to this spec:

- [x] **data-server pulled as a cargo git dependency** — `aura-ingest/Cargo.toml`,
  git dep on the Gitea URL, `branch = "main"`.
- [x] **M1 AoS records transposed into SoA columns at the boundary** —
  `transpose_m1` → `M1Columns`, hermetically tested.
- [x] **time_ms (Unix-ms) normalized to epoch-ns at the boundary** —
  `unix_ms_to_epoch_ns`, tested against a known conversion.
- [x] **The k-way merge consumes the real source; a backtest runs over real bars**
  — `close_stream` → `Harness::run` → `RunReport` in the gated integration test.
- [x] **Determinism preserved (C1): same window → same cycle stream** — purity
  unit test + the two-run bit-identical assertion in the integration test.

### Non-goals (this cycle)

- No `aura run --symbol/--from/--to` CLI surface (deferred; would force issue
  #16's arg-parse decision and a date-format surface).
- No tick (ask/bid) ingestion (deferred; same transpose shape, a follow-up).
- No lazy/shared `Source` abstraction / `Arc<[T]>` cross-sim sharing (deferred to
  the multi-sim orchestration cycle that has a consumer for it; C12).
- No OHLCV-barrier bundler node in the shipped path — the sample strategy consumes
  the close column; the other columns are transposed and publicly available but
  not wired into a bar-record node here.