Ingest data-server M1/tick files at the ingestion boundary #7

New Issue

2026-06-04T15:25:19+02:00

Brummel commented

2026-06-04 15:25:19 +02:00

The harness is currently fed hand-built synthetic streams. C3 (docs/design/
INDEX.md) defines the one merge boundary; this issue wires aura's first real
data source there.

Scope

Pull data-server (~/dev/libs/data-server, Brummel/data-server) as a cargo git
dependency. Transpose its AoS records (M1Parsed / TickParsed, with
time_ms: i64 Unix-ms) into aura's SoA columns (C7) and normalize time_ms ->
canonical epoch-ns timestamp at the single ingestion boundary (C3). History
is shared read-only as Arc<[T]> chunks (data-server is built for this; C12).
Feed the existing k-way merge (cycle 0004) from this real source instead of
hand-fed streams.

C3/C4 keep cycle semantics identical between backtest and live: one input record
= one cycle, advanced in global timestamp order, ties broken by source order.

Note: this is the one network/external dependency. It is deliberately absent
from the bare skeleton so the workspace compiles offline (INDEX, External
components); it is pulled in only when this task starts.

Anchors

C3 — one merge at ingestion only; normalize source time units there.
C4 — data-driven cycle clock; one record = one cycle.
C7 — transpose AoS -> SoA; canonical epoch-ns timestamp.
C12 — raw data shared read-only as Arc<[T]> across sims.
External: data-server at http://192.168.178.103:3000/Brummel/data-server.git

Acceptance

data-server pulled as a cargo git dependency.
M1/tick AoS records transposed into SoA columns at the boundary.
time_ms (Unix-ms) normalized to epoch-ns timestamp at the boundary.
The k-way merge consumes the real source; a backtest runs over real bars.
Determinism preserved (C1): same window -> same cycle stream.

The harness is currently fed hand-built synthetic streams. C3 (docs/design/ INDEX.md) defines the one merge boundary; this issue wires aura's first real data source there. ## Scope Pull `data-server` (~/dev/libs/data-server, Brummel/data-server) as a cargo git dependency. Transpose its AoS records (`M1Parsed` / `TickParsed`, with `time_ms: i64` Unix-ms) into aura's SoA columns (C7) and normalize `time_ms` -> canonical epoch-ns `timestamp` at the single ingestion boundary (C3). History is shared read-only as `Arc<[T]>` chunks (data-server is built for this; C12). Feed the existing k-way merge (cycle 0004) from this real source instead of hand-fed streams. C3/C4 keep cycle semantics identical between backtest and live: one input record = one cycle, advanced in global timestamp order, ties broken by source order. Note: this is the one network/external dependency. It is deliberately absent from the bare skeleton so the workspace compiles offline (INDEX, External components); it is pulled in only when this task starts. ## Anchors - C3 — one merge at ingestion only; normalize source time units there. - C4 — data-driven cycle clock; one record = one cycle. - C7 — transpose AoS -> SoA; canonical epoch-ns timestamp. - C12 — raw data shared read-only as `Arc<[T]>` across sims. - External: data-server at http://192.168.178.103:3000/Brummel/data-server.git ## Acceptance - [ ] data-server pulled as a cargo git dependency. - [ ] M1/tick AoS records transposed into SoA columns at the boundary. - [ ] `time_ms` (Unix-ms) normalized to epoch-ns `timestamp` at the boundary. - [ ] The k-way merge consumes the real source; a backtest runs over real bars. - [ ] Determinism preserved (C1): same window -> same cycle stream.

Brummel added this to the Walking skeleton milestone 2026-06-04 15:25:19 +02:00

Brummel added the feature label 2026-06-04 15:25:19 +02:00

Brummel referenced this issue

2026-06-04 15:25:33 +02:00

Wire an aura run CLI that bootstraps and runs a sample harness end-to-end #8

Brummel referenced this issue from a commit

2026-06-04 17:46:30 +02:00

design(C10): reframe strategy output to intent/exposure stream; position table becomes a derived layer

Brummel referenced this issue from a commit

2026-06-04 17:46:30 +02:00

audit: cycle 0007 tidy — add C10 Realization (cycle 0007) note

Brummel referenced this issue from a commit