spec: 0005 node output record

Cycle 0005 (BLOCKER, milestone walking-skeleton): generalize a node's output from a single scalar to a record of base-scalar columns (K >= 1), with a scalar being the degenerate K = 1 case. A node keeps exactly one output port; that port's payload becomes an ordered list of named, typed base columns (a composite stream, C7). This unblocks every multi-column producer the engine needs next -- OHLCV bars and, later, the C10 position-event table -- none of which can exist while a node emits at most one column. This revises C8 and sharpens C7. The load-bearing design call, made with the user: the eval output representation under C7's "no per-event heap allocation". eval becomes `-> Option<&[Scalar]>` -- the node fills a buffer it owns (sized once at construction) and returns a borrowed K-field row; the engine copies it into one reused scratch row (resolving the borrow, zero per-cycle alloc once warm) and scatters each field to its out-edges. `None` still means filter/not-warmed. Rejected alternatives, on substance: returning `Option<Vec<Scalar>>` allocates per fire (violates C7's forbids); an inline fixed `[Scalar; N]` record bakes a magic width cap; engine-owned output columns invert the eval model (write-into-out + a fired flag) and split "fired" from "produced" for an output-lookback benefit we do not need (history is bound as an input, not read from a node's own past). The borrowed row is the minimal faithful generalization of today's `Some(Scalar)` push route, K-wide. Binding stays field-wise only: `Edge` gains `from_field`; consuming a whole record is N edges; no "bind whole record" mechanism. The K fields of one record are co-fresh by construction (one eval, one timestamp), so C6 is untouched. The record is structural bundling over the four base types (C7 intact) -- no fifth scalar type, no dyn Any, no heterogeneous buffer. Scope is the substrate only: the contract change, the Sma/Sub degenerate migration, the engine from_field wiring, and a neutral OHLCV proof (a 5-field bundler fed by five timestamp-aligned barrier sources + a downstream Sub binding high - low field-wise, plus must-fail cases: from_field out of range -> BadIndex, per-field kind mismatch -> KindMismatch). The C10 position table and broker nodes are out of scope -- later cycles that consume this capability. Self-review clean; grounding-check PASS (14 load-bearing "before"-state assumptions ratified by named green tests; C6/C7/C8 match the ledger as written; parse-every-block a documented no-op -- profile declares no spec_validation). refs #1 refs walking-skeleton Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:35:31 +02:00
parent 9ff90b5d80
commit e9b4d51c30
1 changed files with 324 additions and 0 deletions
@@ -0,0 +1,324 @@
+# Node output is a record (composite stream) — Design Spec
+
+**Date:** 2026-06-03
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+> **Parent issue:** Gitea #1 (BLOCKER, milestone *Walking skeleton*).
+> Revises contract **C8**, sharpens **C7**.
+
+## Goal
+
+Generalize a node's output from a single scalar to a **record of base-scalar
+columns** (K ≥ 1), with a scalar being the degenerate K = 1 case. A node still has
+exactly **one output port**; that port's payload is an ordered list of named,
+typed base columns (a composite stream, C7). This unblocks every multi-column
+producer the engine needs next — OHLCV bars and, later, the position-event table
+(C10) — none of which can exist while a node emits at most one column.
+
+This cycle delivers the *substrate* only: the contract change, the field-wise
+binding mechanism, the Sma/Sub migration, and a neutral OHLCV proof. The position
+table (C10) and broker nodes are explicitly out of scope — they are later cycles
+that *consume* this capability.
+
+## Architecture
+
+Three coordinated changes, in compile-gate order:
+
+1. **`aura-core` — the contract.** `NodeSchema.output` changes from one
+   `ScalarKind` to `Vec<FieldSpec>` (an ordered list of `{ name, kind }` base
+   columns). `Node::eval` changes from `-> Option<Scalar>` to
+   `-> Option<&[Scalar]>`: a node fills a buffer it owns (sized once, at
+   construction) and returns a borrowed row of K scalars; `None` still means
+   *filter / not-yet-warmed*. Scalar output is the degenerate **1-field** record —
+   there is no separate scalar path.
+
+2. **`aura-std` — the degenerate migration.** `Sma` and `Sub` declare a 1-field
+   output (`name: "value"`), hold a `[Scalar; 1]` buffer, and return `Some(&out)`.
+   Behaviour is unchanged: a 1-field record is a scalar.
+
+3. **`aura-engine` — field-wise binding.** `Edge` gains `from_field: usize` —
+   which field of the producer's record this edge forwards. Bootstrap kind-checks
+   per field (`producer.output[from_field].kind == consumer.inputs[slot].kind`)
+   and range-checks `from_field`. The run loop copies a producer's returned row
+   into one reused scratch buffer (resolving the borrow, zero per-cycle heap
+   allocation once its capacity stabilizes), then scatters `scratch[from_field]`
+   into each out-edge's input slot. Freshness stamping is per slot, unchanged, so
+   the K fields of one record are **co-fresh by construction** (one `eval`, one
+   timestamp).
+
+The record is **structural bundling over the four base types (C7 intact)**, not a
+fifth scalar type: no `dyn Any`, no heterogeneous buffer. Every field lands in a
+base column at the edge exactly as today's single scalar does; the streamed /
+stored form stays SoA.
+
+## Concrete code shapes
+
+### What the cycle delivers — the neutral OHLCV proof (test-local fixtures)
+
+The cycle has no end-user surface; its deliverable is the executable proof that a
+node can emit K > 1 fields and a consumer can bind them individually. The
+north-star slice: OHLCV as a 5-field record (the position-event table will be the
+same shape), with a downstream node binding *single* fields by index.
+
+```rust
+// A neutral multi-field producer: five f64 inputs bundled into one 5-field
+// record. No trading-domain logic — it proves the output mechanism in isolation.
+struct Ohlcv {
+    out: [Scalar; 5],
+}
+impl Node for Ohlcv {
+    fn schema(&self) -> NodeSchema {
+        NodeSchema {
+            inputs: vec![
+                InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Barrier(0) },
+                InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Barrier(0) },
+                InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Barrier(0) },
+                InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Barrier(0) },
+                InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Barrier(0) },
+            ],
+            output: vec![
+                FieldSpec { name: "open", kind: ScalarKind::F64 },
+                FieldSpec { name: "high", kind: ScalarKind::F64 },
+                FieldSpec { name: "low", kind: ScalarKind::F64 },
+                FieldSpec { name: "close", kind: ScalarKind::F64 },
+                FieldSpec { name: "volume", kind: ScalarKind::F64 },
+            ],
+        }
+    }
+    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
+        for i in 0..5 {
+            let w = ctx.f64_in(i);
+            if w.is_empty() {
+                return None; // not yet warmed
+            }
+            self.out[i] = Scalar::F64(w[0]);
+        }
+        Some(&self.out) // one 5-field record, all fields co-fresh
+    }
+}
+
+// The proof wiring: five timestamp-aligned sources feed O/H/L/C/V into Ohlcv's
+// barrier group (the canonical "OHLC from separate sources" case, C6), so Ohlcv
+// emits one complete bar per timestamp. A downstream Sub binds field 1 (high)
+// minus field 2 (low) == the bar range. Observing Sub proves field-wise binding
+// routes the right columns. Nodes are [Ohlcv (0), Sub (1)]; the five sources are
+// SourceSpec entries targeting Ohlcv's five input slots.
+//   Edge { from: 0 /*Ohlcv*/, to: 1 /*Sub*/, slot: 0, from_field: 1 /*high*/ }
+//   Edge { from: 0 /*Ohlcv*/, to: 1 /*Sub*/, slot: 1, from_field: 2 /*low*/  }
+```
+
+The **must-fail** half of the proof (wrong code must be rejected at bootstrap):
+
+```rust
+// from_field past the producer's output width -> BadIndex
+Edge { from: 0, to: 1, slot: 0, from_field: 9 }   // Ohlcv has 5 fields
+
+// a field whose kind mismatches the consumer slot -> KindMismatch
+// (e.g. binding an i64 field into an f64 input slot)
+```
+
+### Before → after implementation shapes (secondary)
+
+**`aura-core/src/node.rs`** — the contract:
+
+```rust
+// added
+pub struct FieldSpec {
+    pub name: &'static str,
+    pub kind: ScalarKind,
+}
+
+// before
+pub struct NodeSchema {
+    pub inputs: Vec<InputSpec>,
+    pub output: ScalarKind,
+}
+pub trait Node {
+    fn schema(&self) -> NodeSchema;
+    fn eval(&mut self, ctx: Ctx<'_>) -> Option<Scalar>;
+}
+
+// after
+pub struct NodeSchema {
+    pub inputs: Vec<InputSpec>,
+    pub output: Vec<FieldSpec>, // ordered base columns; len 1 = scalar (degenerate)
+}
+pub trait Node {
+    fn schema(&self) -> NodeSchema;
+    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]>; // borrowed K-field row
+}
+```
+
+`FieldSpec` derives `Clone, Copy, Debug, PartialEq, Eq` (like `InputSpec`);
+`NodeSchema` keeps its derives.
+
+**`aura-std/src/sma.rs`** — the degenerate migration (Sub is analogous):
+
+```rust
+// before
+pub struct Sma { length: usize }
+// schema output: ScalarKind::F64
+fn eval(&mut self, ctx: Ctx<'_>) -> Option<Scalar> {
+    let w = ctx.f64_in(0);
+    if w.len() < self.length { return None; }
+    let mut sum = 0.0;
+    for k in 0..self.length { sum += w[k]; }
+    Some(Scalar::F64(sum / self.length as f64))
+}
+
+// after
+pub struct Sma { length: usize, out: [Scalar; 1] }
+// new() initializes out: [Scalar::F64(0.0)]
+// schema output: vec![FieldSpec { name: "value", kind: ScalarKind::F64 }]
+fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
+    let w = ctx.f64_in(0);
+    if w.len() < self.length { return None; }
+    let mut sum = 0.0;
+    for k in 0..self.length { sum += w[k]; }
+    self.out[0] = Scalar::F64(sum / self.length as f64);
+    Some(&self.out)
+}
+```
+
+**`aura-engine/src/harness.rs`** — the edge and the run loop:
+
+```rust
+// before
+pub struct Edge { pub from: usize, pub to: usize, pub slot: usize }
+
+// after
+pub struct Edge { pub from: usize, pub to: usize, pub slot: usize, pub from_field: usize }
+```
+
+Bootstrap edge check — from a single `from.output` compare to a per-field one:
+
+```rust
+// after
+for &e in &edges {
+    let from = schemas.get(e.from).ok_or(BootstrapError::BadIndex)?;
+    let to = schemas.get(e.to).ok_or(BootstrapError::BadIndex)?;
+    let field = from.output.get(e.from_field).ok_or(BootstrapError::BadIndex)?;
+    let slot = to.inputs.get(e.slot).ok_or(BootstrapError::BadIndex)?;
+    if field.kind != slot.kind {
+        return Err(BootstrapError::KindMismatch { producer: field.kind, consumer: slot.kind });
+    }
+    out_edges[e.from].push(e);
+}
+```
+
+Run loop — the eval-and-forward, with `result: Option<&[Scalar]>` (which is
+`Copy`, since `&[Scalar]` is a shared ref) and one reused `scratch` row declared
+once before the loop:
+
+```rust
+// after (per evaluated node nidx)
+let result: Option<&[Scalar]> = {
+    let nb = &mut nodes[nidx];
+    nb.node.eval(Ctx::new(&nb.inputs))
+};
+if nidx == observe {
+    observed = result.map(|row| row.to_vec());
+}
+if let Some(row) = result {
+    debug_assert_eq!(row.len(), schemas_out_len[nidx]); // node honours its schema width
+    scratch.clear();
+    scratch.extend_from_slice(row); // copy ends the producer borrow; no per-cycle alloc once warm
+    for e in out_edges[nidx].iter() {
+        let nb = &mut nodes[e.to];
+        nb.inputs[e.slot].push(scratch[e.from_field]).expect("edge kind checked at wiring");
+        nb.slots[e.slot] = SlotState { fresh_at: cycle_id, last_ts: ts };
+    }
+}
+```
+
+`run`'s return type changes from `Vec<Option<Scalar>>` to
+`Vec<Option<Vec<Scalar>>>` — the observed node's full row per cycle. The
+per-fire `to_vec()` is in the materialization surface (the returned collection,
+already a `Vec` today), not the inter-node hot path C7's no-alloc rule governs.
+
+## Components
+
+| Component | Change |
+|-----------|--------|
+| `aura-core::FieldSpec` | new — `{ name: &'static str, kind: ScalarKind }`, an output column descriptor |
+| `aura-core::NodeSchema` | `output: ScalarKind` → `output: Vec<FieldSpec>` |
+| `aura-core::Node::eval` | `-> Option<Scalar>` → `-> Option<&[Scalar]>` |
+| `aura-core` lib doc | update the "single output kind" line |
+| `aura-std::Sma`, `::Sub` | hold an `out` buffer; 1-field schema; return `Some(&out)`; in-crate tests adjusted to the slice return |
+| `aura-engine::Edge` | gains `from_field: usize` |
+| `aura-engine::Harness::bootstrap` | per-field edge kind/range check |
+| `aura-engine::Harness::run` | reused scratch row; field-indexed forward; `-> Vec<Option<Vec<Scalar>>>` |
+| `aura-engine` harness doc | the output-forward framing (one scalar → one record field) |
+| `aura-engine` tests | `Ohlcv` fixture; field-binding proof; must-fail (`from_field` OOB, field kind-mismatch); existing 0003/0004 tests adapted to `from_field` + the `Vec` return |
+
+## Data flow
+
+1. The k-way merge picks the next `(timestamp, source-index)` source head and
+   pushes its value into target slots, stamping freshness — unchanged (C3/C4).
+2. In topological order, a fired node's `eval` fills its owned buffer and returns
+   `Some(&row)` (or `None` to hold/filter).
+3. The engine copies `row` into the reused scratch buffer (ending the producer
+   borrow), then for each out-edge pushes `scratch[from_field]` into the consumer
+   slot, stamping that slot fresh at the current `cycle_id`/`ts`.
+4. Because all K fields are written in one `eval` and forwarded in one cycle, every
+   consumer of any field of the record sees the same timestamp — co-freshness is
+   structural, needing no barrier among fields (C6 stays for cross-source sync).
+5. The observed node's row is cloned into the run's output vector per cycle.
+
+## Error handling
+
+- **`from_field` out of range** (`from_field >= producer.output.len()`) →
+  `BootstrapError::BadIndex`, caught once at wiring (C7: type check paid at
+  bootstrap).
+- **Per-field kind mismatch** (`producer.output[from_field].kind !=
+  consumer.inputs[slot].kind`) → `BootstrapError::KindMismatch`.
+- **Node violates its declared width** (returned row length ≠ `schema.output.len()`)
+  → `debug_assert_eq!` in the run loop; a node-author bug, not a wiring fault, so it
+  is a debug assertion, not a typed `BootstrapError`.
+- **Runtime per-field kind** is enforced by `AnyColumn::push` as today
+  (`.expect("edge kind checked at wiring")`) — the bootstrap check makes the expect
+  unreachable for well-typed wiring.
+
+## Testing strategy
+
+- **Degenerate compat** — the existing 0003 fan-out/join DAG and the 0004 firing
+  rails (AsOfSum/BarrierSum/MixedSum, diamond rejoin) keep their expected vectors;
+  every edge gains `from_field: 0`, every fixture returns a 1-field slice. Proves
+  the scalar = 1-field-record identity preserves all prior behaviour.
+- **K > 1 output** — `Ohlcv` (5 f64 inputs → 5-field record); observing it yields
+  the full 5-field row per cycle.
+- **Field-wise binding** — source → `Ohlcv` → `Sub`(high − low); observing `Sub`
+  proves `from_field` routes field 1 and field 2 (not field 0). A second consumer
+  binding different fields (e.g. close − open) on the same `Ohlcv` confirms several
+  edges read distinct fields of one record.
+- **Co-freshness** — the field-binding consumer fires on the bar's single
+  timestamp (no barrier needed among the bound fields), confirming structural
+  co-freshness.
+- **Determinism** — a bit-identical re-run of the K > 1 proof (C1).
+- **Must-fail (rejection)** — `from_field` past the output width → `BadIndex`; a
+  field kind-mismatch → `KindMismatch`.
+- **Purity** — `grep -rnE 'RefCell|Rc<|dyn Any' crates/*/src` stays empty (no
+  fifth type, no heterogeneous payload).
+
+Gates: `cargo build/test/clippy --workspace --all-targets -- -D warnings`, plus
+the purity grep.
+
+## Acceptance criteria
+
+1. A node emits a record of K ≥ 1 named base columns through one output port;
+   `Sma`/`Sub` are the K = 1 degenerate case with unchanged behaviour.
+2. An `Edge` binds exactly one producer field (`from_field`) into one consumer
+   slot; consuming a whole record is N such edges; there is no "bind whole record"
+   mechanism.
+3. The K fields of one record are co-fresh by construction; C6 is untouched.
+4. C7 holds: no fifth scalar type, no `dyn Any`, no heterogeneous buffer, no
+   per-cycle heap allocation on the inter-node forward path; the purity grep is
+   clean.
+5. Bootstrap rejects an out-of-range `from_field` (`BadIndex`) and a per-field
+   kind mismatch (`KindMismatch`).
+6. The neutral OHLCV proof (producer + field-binding consumer + must-fail cases)
+   is green; all prior 0003/0004 tests pass adapted to `from_field`.
+7. The design ledger records the C8 revision and the C7 sharpening (ledger edit is
+   part of this cycle; the glossary `composite`/`node` record-reality pass is an
+   audit-time follow-up).