Aura/docs/specs/0006-sink-recording.md

Sink recording — recording is a node role, not a type — Design Spec

Date: 2026-06-04 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Goal

Replace the engine's single observe: usize recording affordance with real recording-by-node, so that one run can record many streams instead of exactly one node's row. This is the BLOCKER (#2) that the milestone's multi-producer × multi-consumer × multi-sink stress matrix (#3) structurally depends on.

The cycle settles a conceptual point and makes the substrate reflect it: a "sink" is a role, not a type. Recording is a node, in its eval, pushing a value to a destination it holds as a field (a channel, a chart handle, a buffer) — an out-of-graph side effect. There is no Sink type, no engine sink concept, no structural marker. A node that only records is the pure-consumer case (C8); a node may also return an output the engine forwards downstream, recording and producing in the same eval (C8 "producer, consumer, or both").

The load-bearing principle, recorded in the ledger by this cycle: in-graph routing stays engine-owned data (the edge table — introspectable, topologically ordered, kind-checked at wiring); the escape out of the graph is a node-side side effect. The boundary between them is exactly the determinism / graph-as-data boundary (C1/C7/C9, and the recording boundary of C18/C22).

Closes #2. Unblocks #3. No trading domain (no position table, no broker, no equity) — pure compute substrate.

Architecture

Three moving parts, all small, because the substrate already carries a sink as an ordinary node:

1. Ctx::now() (aura-core). A recording node stamps each record with the cycle's timestamp. The cycle timestamp is the engine's clock, not any node's output, so it must reach eval through Ctx. Ctx gains a now: Timestamp field and a now() accessor. Reading "now" is causal — it is the present cycle's timestamp, never the future (C2) — so it introduces no look-ahead. Ctx stays Copy (Timestamp is Copy).

2. observe is removed (aura-engine). The observe: usize field on Harness, its observe >= n bootstrap BadIndex check, and the per-cycle "collect the observed node's row" logic all go. Harness::bootstrap drops its fourth parameter. Harness::run no longer yields a stream — it returns (). Recording is now a node-side concern; retention is owned by whoever constructed the recording node and holds its destination's read end.

3. The run loop is otherwise unchanged as a router. It still evaluates nodes in topological order, gates by firing (C5/C6), and scatters each fired producer's returned row field-wise along out_edges into consumer slots (C7/0005). A recording node's eval reads its typed input windows + now(), pushes a record to its held destination, and returns None (pure sink) or Some(row) (producer-and-sink). The engine forwards the Some arm exactly as today and is wholly oblivious to the side-effect push. The one change inside the loop is that Ctx is now constructed with the cycle ts.

What the engine does not gain: a Sink type, a SinkHandler trait, any dyn FnMut, any engine registry of recording points, any "is this a sink" flag. A recording node is identified by nothing structural — the constructing World knows its recording points because it built them.

Concrete code shapes

User-facing program (the worked example, = the acceptance evidence)

Recording two distinct interior streams of one run — the SMA(2) and SMA(4) outputs of a fan-out DAG — into two independent recording nodes. The destination shown is a channel, the faithful miniature of a real recording sink (a chart on another thread, the run registry): the node holds the Sender, the caller holds the Receiver and owns retention.

Recorder here is this cycle's test-local fixture, not a shipped type — it stands in for the recording node a downstream author writes (their ChartSink, their RegistrySink), each a node with output: vec![] of the same shape. Nothing in the engine surface is a Sink; the example shows the shape every recording node has.

use std::sync::mpsc;
use aura_core::{Scalar, ScalarKind, Timestamp};
use aura_engine::{Edge, Harness, SourceSpec, Target};
use aura_std::Sma;

// Two recording nodes, each tapping one indicator. `Recorder` is a node with
// `output: vec![]` that, on every fired cycle, reads its one f64 input and sends
// `(now, value)` to a channel the caller drains.
let (tx_fast, rx_fast) = mpsc::channel();
let (tx_slow, rx_slow) = mpsc::channel();

let mut h = Harness::bootstrap(
    vec![
        Box::new(Sma::new(2)),                       // 0
        Box::new(Sma::new(4)),                       // 1
        Box::new(Recorder::new(&[ScalarKind::F64], tx_fast)), // 2: taps node 0
        Box::new(Recorder::new(&[ScalarKind::F64], tx_slow)), // 3: taps node 1
    ],
    vec![SourceSpec {
        kind: ScalarKind::F64,
        targets: vec![Target { node: 0, slot: 0 }, Target { node: 1, slot: 0 }],
    }],
    vec![
        Edge { from: 0, to: 2, slot: 0, from_field: 0 }, // SMA(2) -> recorder fast
        Edge { from: 1, to: 3, slot: 0, from_field: 0 }, // SMA(4) -> recorder slow
    ],
    // no observe argument
)
.expect("valid DAG");

h.run(vec![f64_stream(&[(1, 10.0), (2, 12.0), (3, 14.0), (4, 16.0), (5, 18.0)])]);

let fast: Vec<(Timestamp, Vec<Scalar>)> = rx_fast.try_iter().collect();
let slow: Vec<(Timestamp, Vec<Scalar>)> = rx_slow.try_iter().collect();
// `fast` holds SMA(2)'s stream, `slow` holds SMA(4)'s — two streams, one run.

Producer-and-sink in one node (the C8 "both" case)

A node may record and return an output the engine forwards. This is not a special node kind — just a normal eval with a side effect before its return:

fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
    let v = ctx.f64_in(0)[0];
    let _ = self.tx.send((ctx.now(), vec![Scalar::F64(v)])); // sink side effect: out of graph
    self.out[0] = Scalar::F64(v);
    Some(&self.out)                                          // producer output: engine forwards it
}

Must-fail fixture (correct behaviour is rejection)

A recording node declares typed input slots like any consumer, so an edge whose producer field kind does not match the slot kind is rejected at bootstrap — the existing per-field kind check (0005) already covers it; recording adds no new hole:

// Recorder declares an f64 input; an i64 producer field bound into it is rejected.
let err = Harness::bootstrap(
    vec![Box::new(I64Source { /* one i64 output field */ }),
         Box::new(Recorder::new(&[ScalarKind::F64], tx))],
    vec![/* ... */],
    vec![Edge { from: 0, to: 1, slot: 0, from_field: 0 }], // i64 field -> f64 slot
    // no observe
)
.unwrap_err();
assert_eq!(err, BootstrapError::KindMismatch {
    producer: ScalarKind::I64, consumer: ScalarKind::F64,
});

Implementation shape (before → after — supporting, not the headline)

Ctx (aura-core/src/ctx.rs):

// before
pub struct Ctx<'a> { inputs: &'a [AnyColumn] }
impl<'a> Ctx<'a> {
    pub fn new(inputs: &'a [AnyColumn]) -> Self { Self { inputs } }
}

// after
pub struct Ctx<'a> { inputs: &'a [AnyColumn], now: Timestamp }
impl<'a> Ctx<'a> {
    pub fn new(inputs: &'a [AnyColumn], now: Timestamp) -> Self { Self { inputs, now } }
    /// The current cycle's timestamp (C4). Causal — the present, never the
    /// future (C2).
    pub fn now(&self) -> Timestamp { self.now }
}

Harness (aura-engine/src/harness.rs):

// before
pub struct Harness { nodes, topo, out_edges, sources, observe: usize }
pub fn bootstrap(nodes, sources, edges, observe: usize) -> Result<Harness, BootstrapError>
pub fn run(&mut self, streams) -> Vec<Option<Vec<Scalar>>>

// after
pub struct Harness { nodes, topo, out_edges, sources }   // no observe
pub fn bootstrap(nodes, sources, edges) -> Result<Harness, BootstrapError>  // no observe param
pub fn run(&mut self, streams)                            // returns ()

Inside run the only change is the Ctx construction and the deletion of the observe bookkeeping:

// before: Ctx::new(&nb.inputs);  ... if nidx == observe { observed = result.map(...) } ... out.push(observed)
// after:  Ctx::new(&nb.inputs, ts);  (no observe branch, no out vector, no return)

Components

• aura-core — Ctx gains now: Timestamp + now(); Ctx::new takes the timestamp. The three in-crate Ctx::new call sites (its unit tests) pass a timestamp. Node/NodeSchema/FieldSpec/Scalar are untouched — a recording node uses the existing eval -> Option<&[Scalar]> contract verbatim.
• aura-engine — Harness loses observe (field, bootstrap check, per-cycle collection); bootstrap loses its fourth parameter; run returns () and constructs Ctx with ts. A test-local Recorder fixture (see Testing strategy) plus the migrated and new tests. No new public type ships — the engine surface shrinks.
• design ledger — Realization (cycle 0006) notes on C8 (the pure-consumer / sink half is now realized: recording = an eval side effect to a held destination; "sink" is a role, not a type; a node may produce and record at once) and C22 (sinks-as-recording-mechanism is realized at the substrate level; the recorded trace is what a recording node pushed out; the in-graph routing vs out-of-graph escape boundary = the determinism boundary). No new contract; the Harness API change (observe removed) is recorded.

Data flow

Per source tick: the engine advances one cycle with timestamp ts (C4), forwards the source value into its target slots, then evaluates nodes in topological order. A fired producer returns a row; the engine copies it into the reused scratch buffer and scatters scratch[from_field] into each out-edge's consumer slot (unchanged, C7/0005). A fired recording node reads its typed input windows (newest of each, ctx.<kind>_in(i)[0]) plus ctx.now(), assembles a record, and pushes it to the destination it holds; it returns None (pure sink) or Some(row) (also a producer). Nothing the recording node pushes re-enters the graph — a recording node has no out-edges for its recorded values (a "both" node's forwarded output is a separate, declared port).

Recorded streams are sparse and timestamped: a recording node emits a record only on the cycles it fires (its firing policy — Any or Barrier — sets the rate), each tagged with now(). This supersedes the old dense Vec<Option<row>> (one entry per cycle) and matches a trace of timestamped events (C18/C22); streams of different-rate recorders no longer have to share a cycle index.

Error handling

• A recording node's input slots are kind-checked at bootstrap exactly like any consumer's (the per-field KindMismatch and range BadIndex from 0005). A mis-wired recording edge fails at bootstrap, before any data flows.
• The observe >= n BadIndex check is removed with observe. No BootstrapError variant is added or removed otherwise.
• ctx.now() is total — every cycle has a timestamp; there is no cold/None case.
• The test destination is an mpsc::Sender; a send whose receiver was dropped returns Err, which the recording fixture ignores (let _ = self.tx.send(..)) — the receiver outlives the run in every test.

Testing strategy

A test-local Recorder fixture (in harness.rs's #[cfg(test)] mod tests, alongside the existing Ohlcv / AsOfSum / BarrierSum fixtures — C9: no speculative library surface). It holds its declared input kinds and an mpsc::Sender<(Timestamp, Vec<Scalar>)>. Read-back is via the channel, not via Rc/RefCell — this keeps aura-engine/src free of the interior-mutability constructs the purity invariant (C7) forbids, regardless of how the audit's purity grep is scoped, and is a faithful miniature of a real out-of-graph recording destination. Its eval reads each input slot's newest value into a Vec<Scalar>, sends (ctx.now(), row), returns None.

Migration (existing tests, behaviour preserved): every firing/DAG test currently asserting on run's returned Vec<Option<Vec<Scalar>>> via observe is rewritten to tap the previously-observed node with a Recorder and assert on the drained channel. Because recording is sparse (fired cycles only), the expected vectors drop their None hold-rows and carry timestamps: chain_source_sma_runs, fan_out_join_dag_runs_deterministically, mode_a_as_of_fires_on_any_fresh_and_holds, mode_b_barrier_fires_only_on_timestamp_coincidence, within_source_diamond_rejoin_barrier_fires, mixed_a_and_b_or_combine_on_one_node, ohlcv_bundles_five_field_record, edge_binds_single_field_high_minus_low, distinct_edges_read_distinct_fields. The bootstrap_rejects_* tests drop the observe argument; bootstrap_rejects_a_cycle, bootstrap_rejects_a_kind_mismatch, bootstrap_rejects_from_field_out_of_range, and bootstrap_rejects_per_field_kind_mismatch are otherwise unchanged. bootstrap_rejects_a_bad_index currently triggers BadIndex via observe 5 (an out-of-range observe index) — with observe removed, its trigger no longer exists, so it is repurposed to raise BadIndex via an out-of-range edge target (e.g. Edge { to: 9, .. } into a two-node graph): BadIndex itself is unchanged, only the path that reaches it.

New proof tests (the #2 deliverable; the exhaustive M×N×K matrix stays #3):

• multi_sink_records_distinct_interior_streams — two recorders tap SMA(2) and SMA(4) in one run; both drained streams are individually correct. The headline "one run records many streams".
• recorder_taps_all_fields_of_a_record — one recorder with five inputs taps all five OHLCV fields via five edges; its recorded row is the whole bar (ties 0005 in: recording a multi-field producer is N field-wise edges, no whole-record bind).
• recorder_records_mixed_scalar_kinds — a recorder with i64 + f64 + bool + timestamp inputs records a four-field mixed-kind row, proving recording is not f64-only.
• node_is_producer_and_sink_at_once — a node returns an output forwarded to a downstream consumer and records to a channel in the same eval; both the forwarded computation and the recorded stream are correct. Proves the C8 "both" role.
• recording_is_deterministic — two fresh harnesses, two channels, identical input; the two drained streams are bit-identical (C1).
• Recorder firing modes: a Firing::Barrier recorder records only on barrier-complete cycles; a Firing::Any recorder records on any-fresh — asserted by the timestamps present in the drained stream.
• bootstrap_rejects_kind_mismatched_recorder_edge — the must-fail above.

aura-core: a ctx_now_returns_cycle_timestamp unit test.

Gates: cargo build/test --workspace, cargo clippy --workspace --all-targets -D warnings, and the audit's surface-purity grep (no dyn Any / Rc / RefCell in engine source — the channel read-back keeps it clean).

Acceptance criteria

The project's feature-acceptance criterion (CLAUDE.md): the audience naturally reaches for it; it measurably improves correctness/removes redundancy; it reintroduces no failure class a core constraint exists to prevent.

• Naturally reached for. The worked example above is the program a downstream author writes to record more than one stream from a run — impossible before (one observe index). #3's matrix consumes exactly this surface.
• Improves correctness / removes redundancy. One mechanism (a node recording via a held destination) replaces the special-case observe index; the engine surface shrinks (a field, a parameter, a return type, and a per-cycle branch all removed). Multi-stream recording becomes structurally reachable.
• Reintroduces no forbidden failure class. Determinism (C1) holds — a recorded side effect cannot feed back into the graph (no out-edge for recorded values), and the determinism test re-runs bit-identical. Purity (C7) holds — no dyn Any/Rc/RefCell in engine source; the engine gains no dyn FnMut. Causality (C2) holds — now() is the present. Field-wise binding (C8/0005) is untouched — recording a K-field producer is K edges.
• Ship gate: observe gone; bootstrap and run signatures as specified; Ctx::now() present; the full test matrix green; the determinism re-run bit-identical; the C8/C22 realization notes added to the ledger; all three gates (build/test, clippy, purity grep) clean.