Binary/compact on-disk trace format — JSON does not scale to multi-million-point series #307

New Issue

2026-07-21T18:49:29+02:00

claude commented

2026-07-21 18:49:29 +02:00

Surfaced during spec review of the tap-subscribers cycle (refs #283, #77).

Problem

The trace store persists every tap as a ColumnarTrace JSON file (crates/aura-registry/src/trace_store.rs; shape {tap, kinds, ts: Vec<i64>, columns: Vec<Vec<f64>>}). For the full-history study this cycle is built for (~4.5 M M1 cycles), one tap file becomes ~60–120 MB of number text:

Read side is the real cost: read/read_family do read_to_string + serde_json::from_str — the whole file as a String plus the parsed vectors in memory, per tap, per chart request.
Write side pays number formatting (itoa/ryu) per value — tolerable, and after #283 it streams at constant memory, but a binary column dump would make it near-free.

Direction (not prescribing)

A compact columnar format for the two hot arrays (ts, columns) — e.g. length-prefixed little-endian i64/f64 blocks, JSON retained only for the small index.json/manifest metadata. Decision surface: the read side (read, read_family, the chart serve/decimation path), forward/backward compatibility or a one-shot migration, and whether kinds stays a JSON sidecar.

Context

The #283 tap-subscribers cycle deliberately made the opposite call for its own scope: no format migration, streamed writer pinned byte-equal to the legacy JSON writer, every reader untouched. That keeps the cycle small; this issue is the tracked follow-up lever for when trace size actually hurts. The streamed-writer seam (TapTraceWriter) landing in that cycle is the natural place a binary encoder would plug into.

context: filed on user decision during spec review, 2026-07-21.

Surfaced during spec review of the tap-subscribers cycle (refs #283, #77). ## Problem The trace store persists every tap as a `ColumnarTrace` JSON file (`crates/aura-registry/src/trace_store.rs`; shape `{tap, kinds, ts: Vec<i64>, columns: Vec<Vec<f64>>}`). For the full-history study this cycle is built for (~4.5 M M1 cycles), one tap file becomes ~60–120 MB of number text: - **Read side is the real cost:** `read`/`read_family` do `read_to_string` + `serde_json::from_str` — the whole file as a `String` plus the parsed vectors in memory, per tap, per chart request. - Write side pays number formatting (itoa/ryu) per value — tolerable, and after #283 it streams at constant memory, but a binary column dump would make it near-free. ## Direction (not prescribing) A compact columnar format for the two hot arrays (`ts`, `columns`) — e.g. length-prefixed little-endian i64/f64 blocks, JSON retained only for the small `index.json`/manifest metadata. Decision surface: the read side (`read`, `read_family`, the chart serve/decimation path), forward/backward compatibility or a one-shot migration, and whether `kinds` stays a JSON sidecar. ## Context The #283 tap-subscribers cycle deliberately made the opposite call for its own scope: no format migration, streamed writer pinned byte-equal to the legacy JSON writer, every reader untouched. That keeps the cycle small; this issue is the tracked follow-up lever for when trace size actually hurts. The streamed-writer seam (`TapTraceWriter`) landing in that cycle is the natural place a binary encoder would plug into. context: filed on user decision during spec review, 2026-07-21.

claude added the idea label 2026-07-21 18:49:29 +02:00

Brummel referenced this issue from a commit

2026-07-22 00:42:33 +02:00

docs(ledger, glossary): tap-subscribers cycle tidy — C8/C27/C28 lifts, tap subscription vocabulary

claude referenced this issue

2026-07-23 06:36:23 +02:00

Persist a price/signal tap pair so `measure ic` is reachable from the release binary #312

claude added this to the Trace store — completing the tap-subscriber cut milestone 2026-07-23 13:42:58 +02:00

claude referenced this issue

2026-07-23 15:37:07 +02:00

Generalize the trace store into a recorded-stream store for all engine inputs #320

claude commented

2026-07-23 15:37:26 +02:00

Additional requirements from #320 (generalizing the trace store into a recorded-stream store that holds engine inputs as well as taps), recorded here so the encoding chosen for this issue does not force a second migration if #320 lands:

Seekable, window-addressable reads. TraceStore::read/read_family currently read_to_string + parse each whole tap file (crates/aura-registry/src/trace_store.rs:120-143); at input scale (multi-year M1/tick series) that reproduces the eager-materialization wall the Source seam (#71) exists to avoid. The format needs chunked storage with random access by row index and timestamp window (bounds + windowed pull), so a lazy Source impl can serve one chunk at a time.
Multi-column streamed writes. ColumnarTrace is already N-column (kinds/columns are parallel arrays over one ts axis, crates/aura-engine/src/report.rs:201-206) and the chart reader handles width > 1, but the incremental TapTraceWriter is hard-wired to exactly one column (kind: ScalarKind singular, crates/aura-registry/src/trace_store.rs:377-386; finish() emits a single nested "columns":[[…]] list). An OHLCV recording needs N columns sharing one ts axis through the streamed path.
Native column types. The current write path coerces all four scalar kinds to f64 (cell_to_f64, crates/aura-registry/src/trace_store.rs:356-369), the base type surviving only as a string tag in kinds. A binary format can store i64/bool/timestamp columns natively — f64 holds integers exactly only up to 2^53, a bound the format should not have to lean on.

These requirements originate in #320; an encoding shipped tap-scoped without them forces a second on-disk migration if #320 lands. Whether they bind immediately or #307 ships tap-scoped first is part of #320's crate/namespace decision.

Additional requirements from #320 (generalizing the trace store into a recorded-stream store that holds engine inputs as well as taps), recorded here so the encoding chosen for this issue does not force a second migration if #320 lands: - **Seekable, window-addressable reads.** `TraceStore::read`/`read_family` currently `read_to_string` + parse each whole tap file (`crates/aura-registry/src/trace_store.rs:120-143`); at input scale (multi-year M1/tick series) that reproduces the eager-materialization wall the `Source` seam (#71) exists to avoid. The format needs chunked storage with random access by row index and timestamp window (`bounds` + windowed pull), so a lazy `Source` impl can serve one chunk at a time. - **Multi-column streamed writes.** `ColumnarTrace` is already N-column (`kinds`/`columns` are parallel arrays over one `ts` axis, `crates/aura-engine/src/report.rs:201-206`) and the chart reader handles width > 1, but the incremental `TapTraceWriter` is hard-wired to exactly one column (`kind: ScalarKind` singular, `crates/aura-registry/src/trace_store.rs:377-386`; `finish()` emits a single nested `"columns":[[…]]` list). An OHLCV recording needs N columns sharing one `ts` axis through the streamed path. - **Native column types.** The current write path coerces all four scalar kinds to `f64` (`cell_to_f64`, `crates/aura-registry/src/trace_store.rs:356-369`), the base type surviving only as a string tag in `kinds`. A binary format can store `i64`/`bool`/`timestamp` columns natively — `f64` holds integers exactly only up to 2^53, a bound the format should not have to lean on. These requirements originate in #320; an encoding shipped tap-scoped without them forces a second on-disk migration if #320 lands. Whether they bind immediately or #307 ships tap-scoped first is part of #320's crate/namespace decision.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#307