From 73e15583834fe3d9bf20db2364a696a5d79f3673 Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 16 Jun 2026 12:46:18 +0200 Subject: [PATCH] spec: 0047 cell hot-path carrier (boss-signed) Auto-signed under /boss spec auto-sign: all objective gates green (Step-1.5 precondition, Step-4 self-review, Step-5 grounding-check PASS) and a unanimous five-lens spec-skeptic panel SOUND, after one editorial round on the ambiguity lens (the test-migration rule was sharpened to a three-role classification: eval-output -> Cell, column-write -> Scalar, channel-send -> Scalar). Spec for the deferred C7/C8 carrier swap flagged by the C7 realization note: Cell becomes the hot-path value carrier (Node::eval -> Option<&[Cell]>, node out-buffers, inter-node forward via a new AnyColumn::push_cell), and Scalar narrows to the self-describing dynamic boundaries (the param path, AnyColumn::get, source ingestion). Behaviour-preserving (C1): only the carrier type narrows. refs #74 --- docs/specs/0047-cell-hot-path-carrier.md | 388 +++++++++++++++++++++++ 1 file changed, 388 insertions(+) create mode 100644 docs/specs/0047-cell-hot-path-carrier.md diff --git a/docs/specs/0047-cell-hot-path-carrier.md b/docs/specs/0047-cell-hot-path-carrier.md new file mode 100644 index 0000000..4e1c8a4 --- /dev/null +++ b/docs/specs/0047-cell-hot-path-carrier.md @@ -0,0 +1,388 @@ +# Cell as the hot-path value carrier — Design Spec + +**Date:** 2026-06-16 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude +**Tracker:** closes #74 + +## Goal + +Make the tag-free `Cell` the value carrier on aura's hot path, and narrow +`Scalar` (= `{ kind: ScalarKind, cell: Cell }`) to the *self-describing dynamic +boundaries* only. This is the deferred carrier swap the C7 realization note in +`docs/design/INDEX.md` (commits `cd3d1ca` / `049f22a`) explicitly flagged: + +> `Cell` is **not yet** the hot-path carrier — `eval -> Option<&[Scalar]>` and +> the edges still flow `Scalar`; replacing the carrier with `Cell` is a larger +> step that will get its own note (and touch C8's `eval` contract). + +The change is **behaviour-preserving** (C1): same input → same run, +byte-identical metrics and manifests. Only the carrier *type* narrows; the +streamed values are unchanged. + +## Architecture + +### The settled principle (the decision procedure for every value site) + +C7 already holds that the scalar kind lives at the schema / column / port, not +in the value. The carrier swap makes that literal on the single-value carrier: + +- **Kind structurally co-present at the site → bare `Cell`.** If the kind is + already known from the surrounding context (a column's own kind, a declared + output `FieldSpec.kind`, a bootstrap-verified edge), the value must not + re-carry a redundant per-value tag. +- **Value travels detached from its kind → `Scalar`.** If a value must + self-describe because it is serialized, rendered, bound by name, or merged + heterogeneously, it stays the fatter `{ kind, cell }` form. + +Applying that procedure partitions every current `Scalar` site into a +**convert-set** and a **keep-set**. + +### Convert-set (→ `Cell`) + +| Site | Why the kind is co-present | +|------|----------------------------| +| `Node::eval -> Option<&[Cell]>` (`crates/aura-core/src/node.rs`) | Each output field's kind is declared in the node's own `NodeSchema.output[i].kind`; the harness holds the schema. | +| Each node's output buffer `out: [Scalar; N]` → `[Cell; N]` | The node declares its own output kinds and constructs the values; it never needs to read its own buffer back through a tag. | +| Harness forward path (`crates/aura-engine/src/harness.rs`) — `scratch: Vec` + the inter-node edge push | The consumer column's kind is fixed at bootstrap and the edge `from_field`→slot kind match is bootstrap-verified. | +| `AnyColumn::push_cell(Cell)` — **new**, branch-free, infallible | Bootstrap guarantees the kind match, so the runtime push needs no kind check and no `Result`. | + +### Keep-set (stays `Scalar`) + +| Site | Why it must self-describe | +|------|---------------------------| +| `AnyColumn::get -> Option` (`crates/aura-core/src/any.rs`) | The type-erased read for sinks / recorder / serialization / playground — the caller has no static kind, so the value must carry its own. | +| The whole param path: `PrimitiveBuilder::build(&[Scalar])` + build closure, `bind(slot, value: Scalar)`, `BoundParam.value`, `zip_params`, `Blueprint::compile_with_params(&[Scalar])`, the sweep / mc / walkforward param-point args, aura-cli `run_oos` / `scalar_as_param_f64` | A param point routinely travels **detached** from its co-indexed param-space — serialized into the `RunManifest`, passed to `run_one(seed, point)`. `bind` kind-checks the value against the slot. Self-description is load-bearing exactly where the manifest needs it. | +| Source ingestion: `Source::next -> Option<(Timestamp, Scalar)>` + the source-target push (`crates/aura-engine/src/harness.rs`) | The k-way-merged ingestion boundary (C3) is heterogeneous and self-describing by nature; the source *produces* a `Scalar`, so stripping it to a cell at the push would be churn at exactly the boundary whose job is to carry heterogeneous values. | +| The recorder's out-of-graph channel row `Vec` (`crates/aura-std/src/recorder.rs`) | An out-of-graph destination (C8 sink side effect) — the consumer at the other end of the `mpsc` has no schema, so the row self-describes. | + +### The boundary, in one picture + +``` + Source::next ──Scalar──▶ [push] ──▶ AnyColumn (typed) ──get──Scalar──▶ sinks/serde + (ingestion, C3) │ + │ Ctx windows (native f64/i64/…) + ▼ + Node::eval ──&[Cell]──▶ [push_cell] ──▶ AnyColumn (next node) + ▲ (inter-node forward) + out: [Cell; N] + + params: &[Scalar] ─────────────────────────────────────▶ build / bind / manifest (self-describing) +``` + +The hot inter-node forward (`eval` → `push_cell`) carries bare cells; the two +*dynamic* edges (ingestion in, `get` / serde out) and the param plane carry +`Scalar`. + +## Concrete code shapes + +### Worked example — the node-author surface (the empirical evidence) + +The authoring surface is the consumer here: every node author writes `eval` and +an output buffer. `Add` is representative. + +**Before** (`crates/aura-std/src/add.rs`): + +```rust +pub struct Add { + out: [Scalar; 1], +} +impl Add { + pub fn new() -> Self { + Self { out: [Scalar::f64(0.0)] } + } +} +impl Node for Add { + fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> { + let a = ctx.f64_in(0); + let b = ctx.f64_in(1); + if a.is_empty() || b.is_empty() { + return None; + } + self.out[0] = Scalar::f64(a[0] + b[0]); + Some(&self.out) + } +} +``` + +**After**: + +```rust +pub struct Add { + out: [Cell; 1], +} +impl Add { + pub fn new() -> Self { + Self { out: [Cell::from_f64(0.0)] } + } +} +impl Node for Add { + fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Cell]> { + let a = ctx.f64_in(0); + let b = ctx.f64_in(1); + if a.is_empty() || b.is_empty() { + return None; + } + self.out[0] = Cell::from_f64(a[0] + b[0]); + Some(&self.out) + } +} +``` + +The change is strictly a *simplification* for the author: the kind is named +once on the constructor (`Cell::from_f64`) and never re-asserted, because the +node already declared `FieldSpec { kind: F64 }` in its schema. There is no +authoring concept added — the read side (`ctx.f64_in`) is already native and is +untouched. + +A node-author eval test migrates its output assertion from the `Scalar` carrier +to the `Cell` one (the read side and the param side stay `Scalar`): + +```rust +// before +assert_eq!(add.eval(ctx), Some([Scalar::f64(14.0)].as_slice())); +// after +assert_eq!(add.eval(ctx), Some([Cell::from_f64(14.0)].as_slice())); +``` + +`Cell` derives `PartialEq`/`Eq` (bit-exact), so the assertion compares the raw +words — which for a deterministic f64 sum is exactly the intended check. + +#### Test-migration rule (precise — what migrates vs. what stays) + +A unit test touches `Scalar` in three distinct roles; only one migrates. The +rule keys on the role, not on "it is a test": + +1. **Eval-output assertions → migrate to `Cell`.** The value a test reads back + *out of `eval`* is now a cell: `Some([Scalar::f64(x)].as_slice())` → + `Some([Cell::from_f64(x)].as_slice())`, and a value extractor on an eval row + migrates too — the `sim_broker` helper `Some([s]) => s.as_f64()` → + `Some([c]) => c.f64()`. This is the **only** role that changes. +2. **Test-fixture column *writes* → stay `Scalar` (`push`).** Every node test + seeds inputs with `inputs[i].push(Scalar::f64(x)).unwrap()` (e.g. + `exposure.rs`, `sma.rs`, all of aura-std + the harness fixtures, ~40 sites). + A column write models a **source feed** into that column — it is the + ingestion boundary in miniature, squarely in the keep-set. These lines are + **left untouched**; they keep the fallible `push(Scalar)` (which is retained + precisely for ingestion + tests). They are **not** rewritten to `push_cell`. +3. **Out-of-graph channel sends → stay `Scalar`.** A fixture that records to a + channel sends a self-describing `Vec` row — the recorder + (`recorder.rs`) and any dual-role test fixture that both forwards an `eval` + output (convert) **and** sends a channel row, e.g. `TapForward` + (`harness.rs`, which sends `vec![Scalar::f64(v)]`). For such a fixture the + `eval` signature + out-buffer migrate to `Cell`, but its channel send stays + `Scalar`. The two roles in one node migrate independently, by role. + +In one line: **only what comes *out of* `eval` becomes a `Cell`; what goes +*into* a column or *out to* a channel stays a `Scalar`.** + +### Engine-internal shape — `AnyColumn::push_cell` (secondary) + +**Before** — the forward path pushes a tagged `Scalar` and unwraps a kind check +the code already trusts (`crates/aura-engine/src/harness.rs`): + +```rust +// AnyColumn::push (kept, for the source-ingestion boundary + tests) +pub fn push(&mut self, v: Scalar) -> Result<(), KindMismatch> { + match (self, v.kind()) { + (AnyColumn::F64(c), ScalarKind::F64) => { c.push(v.as_f64()); Ok(()) } + // … one arm per kind … + (this, got) => Err(KindMismatch { expected: this.kind(), got }), + } +} + +// harness forward loop +let result: Option<&[Scalar]> = nb.node.eval(Ctx::new(&nb.inputs, ts)); +if let Some(row) = result { + scratch.clear(); + scratch.extend_from_slice(row); // scratch: Vec + for e in out_edges[nidx].iter() { + nb.inputs[e.slot] + .push(scratch[e.from_field]) + .expect("edge kind checked at wiring"); // ← already treated infallible + } +} +``` + +**After** — a branch-free, infallible cell push for the inter-node forward; the +fallible `Scalar` `push` stays for ingestion: + +```rust +// AnyColumn — new method; the kind is the column's own, the cell is reinterpreted +pub fn push_cell(&mut self, c: Cell) { + match self { + AnyColumn::I64(col) => col.push(c.i64()), + AnyColumn::F64(col) => col.push(c.f64()), + AnyColumn::Bool(col) => col.push(c.bool()), + AnyColumn::Ts(col) => col.push(c.ts()), + } +} + +// harness forward loop +let result: Option<&[Cell]> = nb.node.eval(Ctx::new(&nb.inputs, ts)); +if let Some(row) = result { + scratch.clear(); + scratch.extend_from_slice(row); // scratch: Vec + for e in out_edges[nidx].iter() { + nb.inputs[e.slot].push_cell(scratch[e.from_field]); // infallible + } +} +``` + +The kind check is not lost — it is paid once at bootstrap (`harness.rs` +verifies `from.output[from_field].kind == to.inputs[slot].kind` for every edge, +green tests `bootstrap_rejects_a_kind_mismatch`, +`bootstrap_rejects_per_field_kind_mismatch`, +`bootstrap_rejects_kind_mismatched_recorder_edge`). `push_cell` realizes C7's +"type check paid once at wiring, then direct dispatch". + +The source-target forward keeps the `Scalar` path (ingestion is in the +keep-set): + +```rust +// unchanged — source value is a self-describing Scalar +nb.inputs[t.slot].push(value).expect("source kind checked at wiring"); +``` + +### `AnyColumn::get` stays `Scalar` (secondary) + +`get` is the type-erased read for the boundary; it reconstructs the tag from +the column kind on the way out, exactly as today: + +```rust +pub fn get(&self, k: usize) -> Option { + match self { + AnyColumn::I64(c) => c.get(k).map(Scalar::i64), + // … unchanged … + } +} +``` + +## Components + +- **`crates/aura-core/src/node.rs`** — `Node::eval` return type + `Option<&[Scalar]>` → `Option<&[Cell]>`. The param path in the same file + (`PrimitiveBuilder::build`/`new`/`bind`, `BoundParam.value`, `zip_params`) + stays `Scalar`. Test/fixture nodes (`Bare`, `Probe`) migrate their `eval` + signature; `Probe`'s param-vector field stays `Vec` (it stores a + param point). +- **`crates/aura-core/src/any.rs`** — add `push_cell(Cell)`; keep `push(Scalar)` + and `get -> Option`. The `aura_core::Cell` re-export already exists. +- **`crates/aura-std/`** — `add`, `sub`, `sma`, `ema`, `exposure`, `lincomb`, + `sim_broker`: `out: [Scalar; N]` → `[Cell; N]`, constructors and `eval` bodies + use `Cell::from_*`; `eval` signature changes. `recorder`: `eval` signature + changes (returns `None`); the channel row stays `Vec` and the + per-kind `Scalar::*` construction from `get(0)?` stays. Each node's eval tests + migrate output assertions to `Cell`; the `sim_broker` test helper + `Some([s]) => s.as_f64()` becomes `Some([c]) => c.f64()`. +- **`crates/aura-engine/src/harness.rs`** — `scratch: Vec`; the inter-node + forward uses `push_cell`; the source forward keeps `push(Scalar)`. The + in-file test/fixture nodes migrate their `eval` signatures and out-buffers per + the test-migration rule above: a dual-role fixture such as `TapForward` + migrates its `eval` output to `Cell` but keeps its out-of-graph channel send + `Scalar`, and test column writes (`inputs[i].push(Scalar::f64(x))`) stay on + the fallible `push`. +- **`crates/aura-engine/src/blueprint.rs`, `graph_model.rs`** — the + test/fixture nodes that `impl Node` migrate their `eval` signatures + buffers. + `graph_model.rs::scalar_str` (renders a `BoundParam.value`) stays `Scalar`. + `blueprint.rs::compile_with_params(&[Scalar])` and the sweep entry points stay + `Scalar` (param plane). +- **`crates/aura-engine/src/sweep.rs`, `mc.rs`, `walkforward.rs`** — the + `&[Scalar]` param-point args and `scalar_as_f64` stay `Scalar` (param plane). +- **`crates/aura-cli/src/main.rs`** — `run_oos(&[Scalar])`, + `scalar_as_param_f64(&Scalar)` stay `Scalar` (param plane). +- **`crates/aura-engine/src/report.rs`** — `f64_field(&[(Timestamp, + Vec)])` stays `Scalar` (it reads recorder channel rows). +- **No change to** `crates/aura-core/src/ctx.rs` (the read side is already + native — `f64_in`/`i64_in`/… return typed `Window`s), `column.rs`, `error.rs`, + `scalar.rs`, `cell.rs`. + +## Data flow + +1. A source yields `(Timestamp, Scalar)`; the harness pushes the self-describing + `Scalar` into its target column via the fallible `push` (kind verified at + wiring). **[keep]** +2. The engine evaluates each fired node in topo order. `eval` reads native + windows from `Ctx` and returns `Option<&[Cell]>` — a borrowed row of bare + cells into the node's own buffer. **[convert]** +3. The harness copies the row into `scratch: Vec` and, per out-edge, + `push_cell`s one cell into the consumer column. No per-value kind check — + bootstrap guaranteed it. **[convert]** +4. A sink/recorder reads `AnyColumn::get -> Scalar` (or native windows via + `Ctx`) and emits a self-describing row out of the graph. **[keep]** +5. Params flow as `&[Scalar]` from the builder/sweep into `compile_with_params` + and the `RunManifest`. **[keep]** + +The values at every step are bit-identical to before; only steps 2–3 change the +*static type* of the carrier from `Scalar` to `Cell`. + +## Error handling + +- **`push_cell` is infallible by construction.** It cannot return + `KindMismatch` because there is no second kind to compare — it pushes the cell + into *this* column's concrete `Column`, and the only way the wrong cell + reaches the wrong column is a bootstrap-verified edge being wrong, which the + bootstrap kind-check (green tests above) already forbids. `KindMismatch` and + the fallible `push` remain for the ingestion boundary and the unit tests that + exercise the guard directly. +- **The per-value runtime kind check on *node output* is removed.** Today, a + node whose `eval` built a `Scalar` of the wrong kind for its declared output + slot would be caught by the forward-path `push(...).expect(...)` (a panic). + After the swap, a node that builds a cell of the wrong *kind* (e.g. + `Cell::from_i64` in an `f64`-output node) pushes raw bits with no panic. This + is an accepted, bounded trade: + - It is the *same class* of authoring bug as a node returning the wrong row + *width*, which C8 already guards only with a `debug_assert` + (`row.len() == out_len`), not a hard error. + - The node's own `eval` test asserts the exact output value; a wrong-kind + cell yields a garbage value that fails that assertion at the node's own test + layer. + - It is consistent with C7: the output kind is the node's *declared* contract + (`FieldSpec.kind`), honored by the node, not re-checked per value on the hot + path. The check that *does* matter cross-node — that producer and consumer + agree — stays at bootstrap. +- **No new panics or `Result`s on the hot path.** The change only *removes* a + `Result` the code already `.expect`ed. + +## Testing strategy + +- **Behaviour preservation is the headline.** The existing suite is the + regression net: every node `eval` test, the harness end-to-end forward tests, + the sweep/mc/walk-forward/registry tests, and the manifest/metric tests must + stay green after the carrier migration. Their *values* do not change; only the + output-side assertions re-spell `Scalar::f64(x)` as `Cell::from_f64(x)`. +- **`push_cell` round-trip.** Add a focused `AnyColumn` test: `push_cell` of + each kind round-trips through `get` to the same `Scalar` (mirrors the existing + `same_kind_push_round_trips`), proving the cell is interpreted by the column's + kind. +- **`push` guard retained.** The existing `wrong_kind_push_is_rejected` stays + green (the fallible `Scalar` `push` is unchanged), proving the ingestion guard + is intact. +- **Determinism spot-check.** A harness run's recorded equity/exposure rows and + `RunManifest` are unchanged vs. before the swap — covered by the existing + end-to-end report tests (`f64_field` projections + `summarize`); no new + golden file needed because those tests already pin the exact numeric output. +- All four gates clean: `cargo build / clippy --all-targets -D warnings / + test / doc --workspace`. + +## Acceptance criteria + +Judged against aura's domain invariants (the project's feature-acceptance +criterion): + +1. **C7 made literal on the carrier.** `Node::eval` and every node out-buffer + carry bare `Cell` (8 bytes, no tag, no per-value branch); the kind lives only + at the schema/column. `Scalar` survives exactly on the param / `get` / source + boundaries enumerated above — nowhere on the inter-node hot path. +2. **C1 preserved.** Same input → same run: every existing test green with only + carrier-type spelling changes on `eval`-output assertions; recorded rows and + manifests byte-identical. +3. **No reintroduced failure class.** The cross-node kind agreement stays + enforced at bootstrap (the green bootstrap-reject tests); the only removed + check is the redundant runtime one the code already `.expect`ed. The + node-output-kind responsibility is the node author's, caught by the node's own + test — the same posture C8 already takes for output width. +4. **Authoring surface no harder.** The worked `Add` before→after shows the node + author writes *less* ceremony (one `Cell::from_f64` per output, no tag named + twice), with no new concept on the read side. +5. All four workspace gates clean.