aura-ingest/aura-engine: join multi-tap recorder traces on timestamp, not by index #93

New Issue

2026-06-17T18:16:56+02:00

Brummel commented

2026-06-17 18:16:56 +02:00

Surfaced building the GER40 real-data example (commit 5b5f034,
crates/aura-ingest/examples/ger40_breakout_real.rs).

When a graph taps several Recorder sinks to build a per-bar trace, the
sinks do not all fire on the same cardinality:

a Gt tap downstream of Delay(1) is one row shorter — Delay is cold on
the first bar and emits nothing, so everything downstream skips that cycle;
a Session tap filters bars that precede a session open (emits None
out of session).

So the recorded channels have different lengths and different firing
instants. Fusing them positionally (zip by index) misaligns the columns or
panics outright — the example's first cut did exactly that and panicked.

A correct multi-sink trace must join on the recorded timestamp, not by
position. The example fixed it with a ts-keyed join (drain_trace /
BarTrace in examples/shared/breakout_real.rs), defaulting side-taps where
they did not fire that bar.

Proposed: either a documented note on the sink/Recorder API ("taps fire at
their own node's cadence; join multi-tap traces on ts, never by index") or a
small shipped join_on_ts helper, so the next consumer does not rediscover
this via a panic. Non-blocking. Relates to C8/C18 (a sink is the only
observable surface).

**Surfaced building the GER40 real-data example** (commit `5b5f034`, `crates/aura-ingest/examples/ger40_breakout_real.rs`). When a graph taps several `Recorder` sinks to build a per-bar trace, the sinks do **not** all fire on the same cardinality: - a `Gt` tap downstream of `Delay(1)` is one row shorter — `Delay` is cold on the first bar and emits nothing, so everything downstream skips that cycle; - a `Session` tap filters bars that precede a session open (emits `None` out of session). So the recorded channels have **different lengths and different firing instants**. Fusing them positionally (zip by index) misaligns the columns or panics outright — the example's first cut did exactly that and panicked. A correct multi-sink trace must **join on the recorded timestamp**, not by position. The example fixed it with a ts-keyed join (`drain_trace` / `BarTrace` in `examples/shared/breakout_real.rs`), defaulting side-taps where they did not fire that bar. **Proposed:** either a documented note on the sink/Recorder API ("taps fire at their own node's cadence; join multi-tap traces on ts, never by index") or a small shipped `join_on_ts` helper, so the next consumer does not rediscover this via a panic. Non-blocking. Relates to C8/C18 (a sink is the only observable surface).

Brummel added the idea label 2026-06-17 18:16:56 +02:00

Brummel added this to the Runway — real-data ergonomics & honesty hardening milestone 2026-06-18 11:02:34 +02:00

Brummel commented

2026-06-18 14:09:26 +02:00

Design reconciliation (specify, in-context entry)

Spec: 0053-multi-tap-ts-join. The fork below is still listed open on this
issue; it was resolved in the in-context design discussion (brainstorm).

Fork: default-on-miss semantics for the generic ts-join → Option-per-side.
join_on_ts returns Vec<Option<Vec<Scalar>>> for the side taps — the engine
reports presence (None where a tap did not fire at a ts), and the consumer
interprets absence (the 0.0/-1/false defaults stay caller-side, since they
are consumer interpretations of "Gt didn't fire" / "before session open", never
engine truths). Rejected: per-side caller-supplied default-row (bakes "absence
has a default" into the engine API), and doc-note-only (delivers no reuse).
Provenance: user selected "Option je Seite (empfohlen)" in the brainstorm
approach-choice question, 2026-06-18.

## Design reconciliation (specify, in-context entry) Spec: `0053-multi-tap-ts-join`. The fork below is still listed open on this issue; it was resolved in the in-context design discussion (brainstorm). - **Fork: default-on-miss semantics for the generic ts-join** → **Option-per-side**. `join_on_ts` returns `Vec<Option<Vec<Scalar>>>` for the side taps — the engine reports *presence* (`None` where a tap did not fire at a ts), and the consumer interprets absence (the `0.0`/`-1`/`false` defaults stay caller-side, since they are consumer interpretations of "Gt didn't fire" / "before session open", never engine truths). Rejected: per-side caller-supplied default-row (bakes "absence has a default" into the engine API), and doc-note-only (delivers no reuse). Provenance: user selected "Option je Seite (empfohlen)" in the brainstorm approach-choice question, 2026-06-18.

Brummel commented

2026-06-18 15:05:10 +02:00

Resolved in cycle 0053. A reusable, spine-anchored join_on_ts(spine, sides) -> Vec<JoinedRow> helper now lives in aura-engine's report layer (sibling of summarize/f64_field), fusing N recorded (Timestamp, Vec<Scalar>) tap streams on the recorded timestamp — Option-per-side, so the engine reports presence and the consumer interprets absence. drain_trace is refactored onto it (the hand-rolled _by_ts HashMaps are gone), and the Recorder doc now states the cadence/join rule ("join multi-tap traces on ts, never by index").

Commits:

74324d1 feat(aura-engine): join_on_ts helper + JoinedRow + two RED unit tests + lib.rs re-export
35c5adc refactor(aura-ingest): drain_trace fused via join_on_ts + Recorder prose cadence note
7deb38e chore: cycle 0053 close (audit clean)

Verified: workspace build/test/clippy clean; the gated GER40 byte-identity tests stay green against the real archive (drain_trace's trace is byte-preserved across the refactor).

Resolved in cycle 0053. A reusable, spine-anchored `join_on_ts(spine, sides) -> Vec<JoinedRow>` helper now lives in `aura-engine`'s report layer (sibling of `summarize`/`f64_field`), fusing N recorded `(Timestamp, Vec<Scalar>)` tap streams on the recorded timestamp — Option-per-side, so the engine reports presence and the consumer interprets absence. `drain_trace` is refactored onto it (the hand-rolled `_by_ts` HashMaps are gone), and the `Recorder` doc now states the cadence/join rule ("join multi-tap traces on ts, never by index"). Commits: - `74324d1` feat(aura-engine): join_on_ts helper + JoinedRow + two RED unit tests + lib.rs re-export - `35c5adc` refactor(aura-ingest): drain_trace fused via join_on_ts + Recorder prose cadence note - `7deb38e` chore: cycle 0053 close (audit clean) Verified: workspace build/test/clippy clean; the gated GER40 byte-identity tests stay green against the real archive (drain_trace's trace is byte-preserved across the refactor).

Brummel closed this issue

2026-06-18 15:05:11 +02:00

Brummel referenced this issue from a commit

2026-06-18 15:14:31 +02:00

spec: 0053 multi-tap ts-join

Brummel referenced this issue from a commit

2026-06-18 15:14:31 +02:00

plan: 0053 multi-tap ts-join

Brummel referenced this issue from a commit

2026-06-18 15:14:31 +02:00

feat(aura-engine): join_on_ts — timestamp join for multi-tap sink traces

Brummel referenced this issue from a commit