spec: 0047 cell hot-path carrier (boss-signed)

Auto-signed under /boss spec auto-sign: all objective gates green (Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS) and a unanimous five-lens spec-skeptic panel SOUND, after one editorial round on the ambiguity lens (the test-migration rule was sharpened to a three-role classification: eval-output -> Cell, column-write -> Scalar, channel-send -> Scalar). Spec for the deferred C7/C8 carrier swap flagged by the C7 realization note: Cell becomes the hot-path value carrier (Node::eval -> Option<&[Cell]>, node out-buffers, inter-node forward via a new AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic boundaries (the param path, AnyColumn::get, source ingestion). Behaviour-preserving (C1): only the carrier type narrows. refs #74
2026-06-16 12:46:18 +02:00
parent 8eff6398ca
commit 73e1558383
1 changed files with 388 additions and 0 deletions
@@ -0,0 +1,388 @@
+# Cell as the hot-path value carrier — Design Spec
+
+**Date:** 2026-06-16
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+**Tracker:** closes #74
+
+## Goal
+
+Make the tag-free `Cell` the value carrier on aura's hot path, and narrow
+`Scalar` (= `{ kind: ScalarKind, cell: Cell }`) to the *self-describing dynamic
+boundaries* only. This is the deferred carrier swap the C7 realization note in
+`docs/design/INDEX.md` (commits `cd3d1ca` / `049f22a`) explicitly flagged:
+
+> `Cell` is **not yet** the hot-path carrier — `eval -> Option<&[Scalar]>` and
+> the edges still flow `Scalar`; replacing the carrier with `Cell` is a larger
+> step that will get its own note (and touch C8's `eval` contract).
+
+The change is **behaviour-preserving** (C1): same input → same run,
+byte-identical metrics and manifests. Only the carrier *type* narrows; the
+streamed values are unchanged.
+
+## Architecture
+
+### The settled principle (the decision procedure for every value site)
+
+C7 already holds that the scalar kind lives at the schema / column / port, not
+in the value. The carrier swap makes that literal on the single-value carrier:
+
+- **Kind structurally co-present at the site → bare `Cell`.** If the kind is
+  already known from the surrounding context (a column's own kind, a declared
+  output `FieldSpec.kind`, a bootstrap-verified edge), the value must not
+  re-carry a redundant per-value tag.
+- **Value travels detached from its kind → `Scalar`.** If a value must
+  self-describe because it is serialized, rendered, bound by name, or merged
+  heterogeneously, it stays the fatter `{ kind, cell }` form.
+
+Applying that procedure partitions every current `Scalar` site into a
+**convert-set** and a **keep-set**.
+
+### Convert-set (→ `Cell`)
+
+| Site | Why the kind is co-present |
+|------|----------------------------|
+| `Node::eval -> Option<&[Cell]>` (`crates/aura-core/src/node.rs`) | Each output field's kind is declared in the node's own `NodeSchema.output[i].kind`; the harness holds the schema. |
+| Each node's output buffer `out: [Scalar; N]` → `[Cell; N]` | The node declares its own output kinds and constructs the values; it never needs to read its own buffer back through a tag. |
+| Harness forward path (`crates/aura-engine/src/harness.rs`) — `scratch: Vec<Cell>` + the inter-node edge push | The consumer column's kind is fixed at bootstrap and the edge `from_field`→slot kind match is bootstrap-verified. |
+| `AnyColumn::push_cell(Cell)` — **new**, branch-free, infallible | Bootstrap guarantees the kind match, so the runtime push needs no kind check and no `Result`. |
+
+### Keep-set (stays `Scalar`)
+
+| Site | Why it must self-describe |
+|------|---------------------------|
+| `AnyColumn::get -> Option<Scalar>` (`crates/aura-core/src/any.rs`) | The type-erased read for sinks / recorder / serialization / playground — the caller has no static kind, so the value must carry its own. |
+| The whole param path: `PrimitiveBuilder::build(&[Scalar])` + build closure, `bind(slot, value: Scalar)`, `BoundParam.value`, `zip_params`, `Blueprint::compile_with_params(&[Scalar])`, the sweep / mc / walkforward param-point args, aura-cli `run_oos` / `scalar_as_param_f64` | A param point routinely travels **detached** from its co-indexed param-space — serialized into the `RunManifest`, passed to `run_one(seed, point)`. `bind` kind-checks the value against the slot. Self-description is load-bearing exactly where the manifest needs it. |
+| Source ingestion: `Source::next -> Option<(Timestamp, Scalar)>` + the source-target push (`crates/aura-engine/src/harness.rs`) | The k-way-merged ingestion boundary (C3) is heterogeneous and self-describing by nature; the source *produces* a `Scalar`, so stripping it to a cell at the push would be churn at exactly the boundary whose job is to carry heterogeneous values. |
+| The recorder's out-of-graph channel row `Vec<Scalar>` (`crates/aura-std/src/recorder.rs`) | An out-of-graph destination (C8 sink side effect) — the consumer at the other end of the `mpsc` has no schema, so the row self-describes. |
+
+### The boundary, in one picture
+
+```
+  Source::next ──Scalar──▶ [push] ──▶ AnyColumn (typed) ──get──Scalar──▶ sinks/serde
+   (ingestion, C3)                         │
+                                           │ Ctx windows (native f64/i64/…)
+                                           ▼
+                                     Node::eval ──&[Cell]──▶ [push_cell] ──▶ AnyColumn (next node)
+                                           ▲                                   (inter-node forward)
+                                   out: [Cell; N]
+
+  params:  &[Scalar] ─────────────────────────────────────▶ build / bind / manifest (self-describing)
+```
+
+The hot inter-node forward (`eval` → `push_cell`) carries bare cells; the two
+*dynamic* edges (ingestion in, `get` / serde out) and the param plane carry
+`Scalar`.
+
+## Concrete code shapes
+
+### Worked example — the node-author surface (the empirical evidence)
+
+The authoring surface is the consumer here: every node author writes `eval` and
+an output buffer. `Add` is representative.
+
+**Before** (`crates/aura-std/src/add.rs`):
+
+```rust
+pub struct Add {
+    out: [Scalar; 1],
+}
+impl Add {
+    pub fn new() -> Self {
+        Self { out: [Scalar::f64(0.0)] }
+    }
+}
+impl Node for Add {
+    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
+        let a = ctx.f64_in(0);
+        let b = ctx.f64_in(1);
+        if a.is_empty() || b.is_empty() {
+            return None;
+        }
+        self.out[0] = Scalar::f64(a[0] + b[0]);
+        Some(&self.out)
+    }
+}
+```
+
+**After**:
+
+```rust
+pub struct Add {
+    out: [Cell; 1],
+}
+impl Add {
+    pub fn new() -> Self {
+        Self { out: [Cell::from_f64(0.0)] }
+    }
+}
+impl Node for Add {
+    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Cell]> {
+        let a = ctx.f64_in(0);
+        let b = ctx.f64_in(1);
+        if a.is_empty() || b.is_empty() {
+            return None;
+        }
+        self.out[0] = Cell::from_f64(a[0] + b[0]);
+        Some(&self.out)
+    }
+}
+```
+
+The change is strictly a *simplification* for the author: the kind is named
+once on the constructor (`Cell::from_f64`) and never re-asserted, because the
+node already declared `FieldSpec { kind: F64 }` in its schema. There is no
+authoring concept added — the read side (`ctx.f64_in`) is already native and is
+untouched.
+
+A node-author eval test migrates its output assertion from the `Scalar` carrier
+to the `Cell` one (the read side and the param side stay `Scalar`):
+
+```rust
+// before
+assert_eq!(add.eval(ctx), Some([Scalar::f64(14.0)].as_slice()));
+// after
+assert_eq!(add.eval(ctx), Some([Cell::from_f64(14.0)].as_slice()));
+```
+
+`Cell` derives `PartialEq`/`Eq` (bit-exact), so the assertion compares the raw
+words — which for a deterministic f64 sum is exactly the intended check.
+
+#### Test-migration rule (precise — what migrates vs. what stays)
+
+A unit test touches `Scalar` in three distinct roles; only one migrates. The
+rule keys on the role, not on "it is a test":
+
+1. **Eval-output assertions → migrate to `Cell`.** The value a test reads back
+   *out of `eval`* is now a cell: `Some([Scalar::f64(x)].as_slice())` →
+   `Some([Cell::from_f64(x)].as_slice())`, and a value extractor on an eval row
+   migrates too — the `sim_broker` helper `Some([s]) => s.as_f64()` →
+   `Some([c]) => c.f64()`. This is the **only** role that changes.
+2. **Test-fixture column *writes* → stay `Scalar` (`push`).** Every node test
+   seeds inputs with `inputs[i].push(Scalar::f64(x)).unwrap()` (e.g.
+   `exposure.rs`, `sma.rs`, all of aura-std + the harness fixtures, ~40 sites).
+   A column write models a **source feed** into that column — it is the
+   ingestion boundary in miniature, squarely in the keep-set. These lines are
+   **left untouched**; they keep the fallible `push(Scalar)` (which is retained
+   precisely for ingestion + tests). They are **not** rewritten to `push_cell`.
+3. **Out-of-graph channel sends → stay `Scalar`.** A fixture that records to a
+   channel sends a self-describing `Vec<Scalar>` row — the recorder
+   (`recorder.rs`) and any dual-role test fixture that both forwards an `eval`
+   output (convert) **and** sends a channel row, e.g. `TapForward`
+   (`harness.rs`, which sends `vec![Scalar::f64(v)]`). For such a fixture the
+   `eval` signature + out-buffer migrate to `Cell`, but its channel send stays
+   `Scalar`. The two roles in one node migrate independently, by role.
+
+In one line: **only what comes *out of* `eval` becomes a `Cell`; what goes
+*into* a column or *out to* a channel stays a `Scalar`.**
+
+### Engine-internal shape — `AnyColumn::push_cell` (secondary)
+
+**Before** — the forward path pushes a tagged `Scalar` and unwraps a kind check
+the code already trusts (`crates/aura-engine/src/harness.rs`):
+
+```rust
+// AnyColumn::push (kept, for the source-ingestion boundary + tests)
+pub fn push(&mut self, v: Scalar) -> Result<(), KindMismatch> {
+    match (self, v.kind()) {
+        (AnyColumn::F64(c), ScalarKind::F64) => { c.push(v.as_f64()); Ok(()) }
+        // … one arm per kind …
+        (this, got) => Err(KindMismatch { expected: this.kind(), got }),
+    }
+}
+
+// harness forward loop
+let result: Option<&[Scalar]> = nb.node.eval(Ctx::new(&nb.inputs, ts));
+if let Some(row) = result {
+    scratch.clear();
+    scratch.extend_from_slice(row);                 // scratch: Vec<Scalar>
+    for e in out_edges[nidx].iter() {
+        nb.inputs[e.slot]
+            .push(scratch[e.from_field])
+            .expect("edge kind checked at wiring");  // ← already treated infallible
+    }
+}
+```
+
+**After** — a branch-free, infallible cell push for the inter-node forward; the
+fallible `Scalar` `push` stays for ingestion:
+
+```rust
+// AnyColumn — new method; the kind is the column's own, the cell is reinterpreted
+pub fn push_cell(&mut self, c: Cell) {
+    match self {
+        AnyColumn::I64(col) => col.push(c.i64()),
+        AnyColumn::F64(col) => col.push(c.f64()),
+        AnyColumn::Bool(col) => col.push(c.bool()),
+        AnyColumn::Ts(col) => col.push(c.ts()),
+    }
+}
+
+// harness forward loop
+let result: Option<&[Cell]> = nb.node.eval(Ctx::new(&nb.inputs, ts));
+if let Some(row) = result {
+    scratch.clear();
+    scratch.extend_from_slice(row);                 // scratch: Vec<Cell>
+    for e in out_edges[nidx].iter() {
+        nb.inputs[e.slot].push_cell(scratch[e.from_field]); // infallible
+    }
+}
+```
+
+The kind check is not lost — it is paid once at bootstrap (`harness.rs`
+verifies `from.output[from_field].kind == to.inputs[slot].kind` for every edge,
+green tests `bootstrap_rejects_a_kind_mismatch`,
+`bootstrap_rejects_per_field_kind_mismatch`,
+`bootstrap_rejects_kind_mismatched_recorder_edge`). `push_cell` realizes C7's
+"type check paid once at wiring, then direct dispatch".
+
+The source-target forward keeps the `Scalar` path (ingestion is in the
+keep-set):
+
+```rust
+// unchanged — source value is a self-describing Scalar
+nb.inputs[t.slot].push(value).expect("source kind checked at wiring");
+```
+
+### `AnyColumn::get` stays `Scalar` (secondary)
+
+`get` is the type-erased read for the boundary; it reconstructs the tag from
+the column kind on the way out, exactly as today:
+
+```rust
+pub fn get(&self, k: usize) -> Option<Scalar> {
+    match self {
+        AnyColumn::I64(c) => c.get(k).map(Scalar::i64),
+        // … unchanged …
+    }
+}
+```
+
+## Components
+
+- **`crates/aura-core/src/node.rs`** — `Node::eval` return type
+  `Option<&[Scalar]>` → `Option<&[Cell]>`. The param path in the same file
+  (`PrimitiveBuilder::build`/`new`/`bind`, `BoundParam.value`, `zip_params`)
+  stays `Scalar`. Test/fixture nodes (`Bare`, `Probe`) migrate their `eval`
+  signature; `Probe`'s param-vector field stays `Vec<Scalar>` (it stores a
+  param point).
+- **`crates/aura-core/src/any.rs`** — add `push_cell(Cell)`; keep `push(Scalar)`
+  and `get -> Option<Scalar>`. The `aura_core::Cell` re-export already exists.
+- **`crates/aura-std/`** — `add`, `sub`, `sma`, `ema`, `exposure`, `lincomb`,
+  `sim_broker`: `out: [Scalar; N]` → `[Cell; N]`, constructors and `eval` bodies
+  use `Cell::from_*`; `eval` signature changes. `recorder`: `eval` signature
+  changes (returns `None`); the channel row stays `Vec<Scalar>` and the
+  per-kind `Scalar::*` construction from `get(0)?` stays. Each node's eval tests
+  migrate output assertions to `Cell`; the `sim_broker` test helper
+  `Some([s]) => s.as_f64()` becomes `Some([c]) => c.f64()`.
+- **`crates/aura-engine/src/harness.rs`** — `scratch: Vec<Cell>`; the inter-node
+  forward uses `push_cell`; the source forward keeps `push(Scalar)`. The
+  in-file test/fixture nodes migrate their `eval` signatures and out-buffers per
+  the test-migration rule above: a dual-role fixture such as `TapForward`
+  migrates its `eval` output to `Cell` but keeps its out-of-graph channel send
+  `Scalar`, and test column writes (`inputs[i].push(Scalar::f64(x))`) stay on
+  the fallible `push`.
+- **`crates/aura-engine/src/blueprint.rs`, `graph_model.rs`** — the
+  test/fixture nodes that `impl Node` migrate their `eval` signatures + buffers.
+  `graph_model.rs::scalar_str` (renders a `BoundParam.value`) stays `Scalar`.
+  `blueprint.rs::compile_with_params(&[Scalar])` and the sweep entry points stay
+  `Scalar` (param plane).
+- **`crates/aura-engine/src/sweep.rs`, `mc.rs`, `walkforward.rs`** — the
+  `&[Scalar]` param-point args and `scalar_as_f64` stay `Scalar` (param plane).
+- **`crates/aura-cli/src/main.rs`** — `run_oos(&[Scalar])`,
+  `scalar_as_param_f64(&Scalar)` stay `Scalar` (param plane).
+- **`crates/aura-engine/src/report.rs`** — `f64_field(&[(Timestamp,
+  Vec<Scalar>)])` stays `Scalar` (it reads recorder channel rows).
+- **No change to** `crates/aura-core/src/ctx.rs` (the read side is already
+  native — `f64_in`/`i64_in`/… return typed `Window`s), `column.rs`, `error.rs`,
+  `scalar.rs`, `cell.rs`.
+
+## Data flow
+
+1. A source yields `(Timestamp, Scalar)`; the harness pushes the self-describing
+   `Scalar` into its target column via the fallible `push` (kind verified at
+   wiring). **[keep]**
+2. The engine evaluates each fired node in topo order. `eval` reads native
+   windows from `Ctx` and returns `Option<&[Cell]>` — a borrowed row of bare
+   cells into the node's own buffer. **[convert]**
+3. The harness copies the row into `scratch: Vec<Cell>` and, per out-edge,
+   `push_cell`s one cell into the consumer column. No per-value kind check —
+   bootstrap guaranteed it. **[convert]**
+4. A sink/recorder reads `AnyColumn::get -> Scalar` (or native windows via
+   `Ctx`) and emits a self-describing row out of the graph. **[keep]**
+5. Params flow as `&[Scalar]` from the builder/sweep into `compile_with_params`
+   and the `RunManifest`. **[keep]**
+
+The values at every step are bit-identical to before; only steps 2–3 change the
+*static type* of the carrier from `Scalar` to `Cell`.
+
+## Error handling
+
+- **`push_cell` is infallible by construction.** It cannot return
+  `KindMismatch` because there is no second kind to compare — it pushes the cell
+  into *this* column's concrete `Column<T>`, and the only way the wrong cell
+  reaches the wrong column is a bootstrap-verified edge being wrong, which the
+  bootstrap kind-check (green tests above) already forbids. `KindMismatch` and
+  the fallible `push` remain for the ingestion boundary and the unit tests that
+  exercise the guard directly.
+- **The per-value runtime kind check on *node output* is removed.** Today, a
+  node whose `eval` built a `Scalar` of the wrong kind for its declared output
+  slot would be caught by the forward-path `push(...).expect(...)` (a panic).
+  After the swap, a node that builds a cell of the wrong *kind* (e.g.
+  `Cell::from_i64` in an `f64`-output node) pushes raw bits with no panic. This
+  is an accepted, bounded trade:
+  - It is the *same class* of authoring bug as a node returning the wrong row
+    *width*, which C8 already guards only with a `debug_assert`
+    (`row.len() == out_len`), not a hard error.
+  - The node's own `eval` test asserts the exact output value; a wrong-kind
+    cell yields a garbage value that fails that assertion at the node's own test
+    layer.
+  - It is consistent with C7: the output kind is the node's *declared* contract
+    (`FieldSpec.kind`), honored by the node, not re-checked per value on the hot
+    path. The check that *does* matter cross-node — that producer and consumer
+    agree — stays at bootstrap.
+- **No new panics or `Result`s on the hot path.** The change only *removes* a
+  `Result` the code already `.expect`ed.
+
+## Testing strategy
+
+- **Behaviour preservation is the headline.** The existing suite is the
+  regression net: every node `eval` test, the harness end-to-end forward tests,
+  the sweep/mc/walk-forward/registry tests, and the manifest/metric tests must
+  stay green after the carrier migration. Their *values* do not change; only the
+  output-side assertions re-spell `Scalar::f64(x)` as `Cell::from_f64(x)`.
+- **`push_cell` round-trip.** Add a focused `AnyColumn` test: `push_cell` of
+  each kind round-trips through `get` to the same `Scalar` (mirrors the existing
+  `same_kind_push_round_trips`), proving the cell is interpreted by the column's
+  kind.
+- **`push` guard retained.** The existing `wrong_kind_push_is_rejected` stays
+  green (the fallible `Scalar` `push` is unchanged), proving the ingestion guard
+  is intact.
+- **Determinism spot-check.** A harness run's recorded equity/exposure rows and
+  `RunManifest` are unchanged vs. before the swap — covered by the existing
+  end-to-end report tests (`f64_field` projections + `summarize`); no new
+  golden file needed because those tests already pin the exact numeric output.
+- All four gates clean: `cargo build / clippy --all-targets -D warnings /
+  test / doc --workspace`.
+
+## Acceptance criteria
+
+Judged against aura's domain invariants (the project's feature-acceptance
+criterion):
+
+1. **C7 made literal on the carrier.** `Node::eval` and every node out-buffer
+   carry bare `Cell` (8 bytes, no tag, no per-value branch); the kind lives only
+   at the schema/column. `Scalar` survives exactly on the param / `get` / source
+   boundaries enumerated above — nowhere on the inter-node hot path.
+2. **C1 preserved.** Same input → same run: every existing test green with only
+   carrier-type spelling changes on `eval`-output assertions; recorded rows and
+   manifests byte-identical.
+3. **No reintroduced failure class.** The cross-node kind agreement stays
+   enforced at bootstrap (the green bootstrap-reject tests); the only removed
+   check is the redundant runtime one the code already `.expect`ed. The
+   node-output-kind responsibility is the node author's, caught by the node's own
+   test — the same posture C8 already takes for output width.
+4. **Authoring surface no harder.** The worked `Add` before→after shows the node
+   author writes *less* ceremony (one `Cell::from_f64` per output, no tag named
+   twice), with no new concept on the read side.
+5. All four workspace gates clean.