Generalize the trace store into a recorded-stream store for all engine inputs #320

New Issue

2026-07-23T15:37:07+02:00

claude commented

2026-07-23 15:37:07 +02:00

Observation

aura cannot assume where input data comes from or how fast it arrives. The only market-data path today reads Pepperstone .m1/.tick archives through the external data-server crate (a cargo git dependency, crates/aura-ingest/Cargo.toml) — a filesystem loader with an in-process RAM cache (FileCache). Verified 2026-07-23 on the development host: the default archive directory resolves to an NFS4 automount (findmnt: fstype nfs4, hard mount option), so every cold read crosses the network, the cache amortizes only within one process lifetime, and a NAS outage makes reads hang rather than fail. There is no HTTP/network client anywhere in the workspace (grep for reqwest|ureq|hyper|TcpStream over crates/ and the data-server repo returns nothing) — the network dependency is the mount itself.

The record-then-replay contract (C11, docs/design/contracts/c11-sources-record-replay.md) already states the law this calls for: anything nondeterministic, external, or slow is materialized into a recorded, timestamped stream before it enters the engine. Its current-state note marks the recording direction — a live source materialized for later replay — as design law that has not yet landed. C26 (docs/design/contracts/c26-input-binding.md) names the deferred extension point: the input-binding value-space grows "additively from archive columns to recorded-stream references".

Proposal

Generalize what data-server does specifically for Pepperstone data into an aura-owned recorded-stream store, layered so that no performance or availability assumption survives above the recording edge:

Layer	Role	Runs
1 — Adapters	Per-source fetch/decode: the Pepperstone archive decoder, a future REST/API client, an LLM/news recorder. May be slow, flaky, network-bound.	import time only
2 — Store	Canonical, immutable recordings: chunked binary columns + manifest/index per recording. The determinism-boundary artifact.	—
3 — Serving	Chunk cache with `Arc<[T]>` sharing across sim-family members (the C12 residency guarantee, `docs/design/contracts/c12-atomic-sim-unit.md`, today provided by data-server's `FileCache`), lazy windowed `Source` impls.	replay

The engine is untouched: layer 3 feeds bind_sources and the single k-way ingestion merge (C3, docs/design/contracts/c03-single-merge.md) exactly as M1FieldSource does today.

Import is itself a run. A recording is produced by a trivial harness (source → recording sink) whose tap trace is the recorded stream. This keeps C22's identity intact for inputs — "displayable = exactly what a sink recorded" (docs/design/contracts/c22-playground-traces.md) — and produces the recording's manifest through the existing run-manifest machinery instead of a parallel metadata scheme.

Directions settled at filing (2026-07-23)

Record semantics, not cache semantics. Recordings are immutable and identity-carrying (source, window, fetch time); a re-fetch is a new recording, never an in-place refresh. Vendors revise history, and determinism (C1, docs/design/contracts/c01-determinism.md) requires the input's identity to be pinned — a transparent cache cannot provide that. The performance benefit of a cache is retained anyway, since replay reads only the recording. Closed monthly archives are the easy case: identity (archive file, hash) never invalidates; only a growing source needs the new-recording rule.
Subsumption is the end state. The data-server splits along its two roles: the Pepperstone decoder becomes a layer-1 import adapter (runs once per archive file, never per backtest); its chunking/cache/serving technique moves into layer 3, where the C12 Arc<[T]> guarantee already lives. "Pepperstone import" reduces to reading the archive into the store; the replay path never touches the Pepperstone format again.
Homogeneous replay. Backtest logic reads inputs only from the store; the API boundary (layer 1) has no contact with replay.

Open decisions

Crate home. The current TraceStore lives in aura-registry — the process column, which per the stratification contract (C28, docs/design/contracts/c28-stratification.md) consumes run artifacts, not market streams; the market layer must not import the process column. Candidate resolution: the container mechanics (format, index, chunk cache, serving) become a base crate, with run traces and recorded inputs as two namespaces/clients of one container; aura-registry's trace store becomes a thin run-scoped view over it.
The growing edge. A growing source (the current archive month; any future live feed) needs extensible recordings — appendable chunk segments with per-segment identity — rather than whole re-imports. A chunked format permits this, but it must be designed in from the start.

Relations

Milestone "Trace store — completing the tap-subscriber cut" proceeds unchanged; this issue is the follow-on generalization.
#307 (binary on-disk trace format) becomes the store's single encoding. The requirements this proposal adds — seekable window-addressable reads, multi-column streamed writes, native column types — are recorded in a comment on #307.
#308 (one delivery shape) and #311 (trace-name collision contract) gain weight here: delivery and naming identity become input-path concerns, not only presentation concerns.
#71 (closed; the Source seam) is the migration template: its acceptance discipline — suite byte-identity across the seam change — carries over as "store-served replay is byte-identical to data-server-served replay" before the data-server is retired.

## Observation aura cannot assume where input data comes from or how fast it arrives. The only market-data path today reads Pepperstone `.m1`/`.tick` archives through the external `data-server` crate (a cargo git dependency, `crates/aura-ingest/Cargo.toml`) — a filesystem loader with an in-process RAM cache (`FileCache`). Verified 2026-07-23 on the development host: the default archive directory resolves to an NFS4 automount (`findmnt`: fstype `nfs4`, `hard` mount option), so every cold read crosses the network, the cache amortizes only within one process lifetime, and a NAS outage makes reads hang rather than fail. There is no HTTP/network client anywhere in the workspace (grep for `reqwest|ureq|hyper|TcpStream` over `crates/` and the data-server repo returns nothing) — the network dependency is the mount itself. The record-then-replay contract (C11, `docs/design/contracts/c11-sources-record-replay.md`) already states the law this calls for: anything nondeterministic, external, or slow is materialized into a recorded, timestamped stream before it enters the engine. Its current-state note marks the recording direction — a live source materialized for later replay — as design law that has not yet landed. C26 (`docs/design/contracts/c26-input-binding.md`) names the deferred extension point: the input-binding value-space grows "additively from archive columns to recorded-stream references". ## Proposal Generalize what `data-server` does specifically for Pepperstone data into an aura-owned **recorded-stream store**, layered so that no performance or availability assumption survives above the recording edge: | Layer | Role | Runs | |---|---|---| | 1 — Adapters | Per-source fetch/decode: the Pepperstone archive decoder, a future REST/API client, an LLM/news recorder. May be slow, flaky, network-bound. | import time only | | 2 — Store | Canonical, immutable recordings: chunked binary columns + manifest/index per recording. The determinism-boundary artifact. | — | | 3 — Serving | Chunk cache with `Arc<[T]>` sharing across sim-family members (the C12 residency guarantee, `docs/design/contracts/c12-atomic-sim-unit.md`, today provided by data-server's `FileCache`), lazy windowed `Source` impls. | replay | The engine is untouched: layer 3 feeds `bind_sources` and the single k-way ingestion merge (C3, `docs/design/contracts/c03-single-merge.md`) exactly as `M1FieldSource` does today. **Import is itself a run.** A recording is produced by a trivial harness (source → recording sink) whose tap trace is the recorded stream. This keeps C22's identity intact for inputs — "displayable = exactly what a sink recorded" (`docs/design/contracts/c22-playground-traces.md`) — and produces the recording's manifest through the existing run-manifest machinery instead of a parallel metadata scheme. ## Directions settled at filing (2026-07-23) - **Record semantics, not cache semantics.** Recordings are immutable and identity-carrying (source, window, fetch time); a re-fetch is a new recording, never an in-place refresh. Vendors revise history, and determinism (C1, `docs/design/contracts/c01-determinism.md`) requires the input's identity to be pinned — a transparent cache cannot provide that. The performance benefit of a cache is retained anyway, since replay reads only the recording. Closed monthly archives are the easy case: identity `(archive file, hash)` never invalidates; only a growing source needs the new-recording rule. - **Subsumption is the end state.** The data-server splits along its two roles: the Pepperstone decoder becomes a layer-1 import adapter (runs once per archive file, never per backtest); its chunking/cache/serving technique moves into layer 3, where the C12 `Arc<[T]>` guarantee already lives. "Pepperstone import" reduces to reading the archive into the store; the replay path never touches the Pepperstone format again. - **Homogeneous replay.** Backtest logic reads inputs only from the store; the API boundary (layer 1) has no contact with replay. ## Open decisions - **Crate home.** The current `TraceStore` lives in `aura-registry` — the process column, which per the stratification contract (C28, `docs/design/contracts/c28-stratification.md`) consumes run artifacts, not market streams; the market layer must not import the process column. Candidate resolution: the container mechanics (format, index, chunk cache, serving) become a base crate, with run traces and recorded inputs as two namespaces/clients of one container; `aura-registry`'s trace store becomes a thin run-scoped view over it. - **The growing edge.** A growing source (the current archive month; any future live feed) needs extensible recordings — appendable chunk segments with per-segment identity — rather than whole re-imports. A chunked format permits this, but it must be designed in from the start. ## Relations - Milestone "Trace store — completing the tap-subscriber cut" proceeds unchanged; this issue is the follow-on generalization. - #307 (binary on-disk trace format) becomes the store's single encoding. The requirements this proposal adds — seekable window-addressable reads, multi-column streamed writes, native column types — are recorded in a comment on #307. - #308 (one delivery shape) and #311 (trace-name collision contract) gain weight here: delivery and naming identity become input-path concerns, not only presentation concerns. - #71 (closed; the `Source` seam) is the migration template: its acceptance discipline — suite byte-identity across the seam change — carries over as "store-served replay is byte-identical to data-server-served replay" before the data-server is retired.

claude added the idea label 2026-07-23 15:37:07 +02:00

claude referenced this issue

2026-07-23 15:37:26 +02:00

Binary/compact on-disk trace format — JSON does not scale to multi-million-point series #307

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#320