Aura/docs/specs/0047-cell-hot-path-carrier.md

Cell as the hot-path value carrier — Design Spec

Date: 2026-06-16 Status: Draft — awaiting user spec review Authors: orchestrator + Claude Tracker: closes #74

Goal

Make the tag-free Cell the value carrier on aura's hot path, and narrow Scalar (= { kind: ScalarKind, cell: Cell }) to the self-describing dynamic boundaries only. This is the deferred carrier swap the C7 realization note in docs/design/INDEX.md (commits cd3d1ca / 049f22a) explicitly flagged:

Cell is not yet the hot-path carrier — eval -> Option<&[Scalar]> and the edges still flow Scalar; replacing the carrier with Cell is a larger step that will get its own note (and touch C8's eval contract).

The change is behaviour-preserving (C1): same input → same run, byte-identical metrics and manifests. Only the carrier type narrows; the streamed values are unchanged.

Architecture

The settled principle (the decision procedure for every value site)

C7 already holds that the scalar kind lives at the schema / column / port, not in the value. The carrier swap makes that literal on the single-value carrier:

• Kind structurally co-present at the site → bare Cell. If the kind is already known from the surrounding context (a column's own kind, a declared output FieldSpec.kind, a bootstrap-verified edge), the value must not re-carry a redundant per-value tag.
• Value travels detached from its kind → Scalar. If a value must self-describe because it is serialized, rendered, bound by name, or merged heterogeneously, it stays the fatter { kind, cell } form.

Applying that procedure partitions every current Scalar site into a convert-set and a keep-set.

Convert-set (→ Cell)

Site	Why the kind is co-present
Node::eval -> Option<&[Cell]> (crates/aura-core/src/node.rs)	Each output field's kind is declared in the node's own NodeSchema.output[i].kind; the harness holds the schema.
Each node's output buffer out: [Scalar; N] → [Cell; N]	The node declares its own output kinds and constructs the values; it never needs to read its own buffer back through a tag.
Harness forward path (crates/aura-engine/src/harness.rs) — scratch: Vec<Cell> + the inter-node edge push	The consumer column's kind is fixed at bootstrap and the edge from_field→slot kind match is bootstrap-verified.
AnyColumn::push_cell(Cell) — new, branch-free, infallible	Bootstrap guarantees the kind match, so the runtime push needs no kind check and no Result.

Keep-set (stays Scalar)

Site	Why it must self-describe
AnyColumn::get -> Option<Scalar> (crates/aura-core/src/any.rs)	The type-erased read for sinks / recorder / serialization / playground — the caller has no static kind, so the value must carry its own.
The whole param path: PrimitiveBuilder::build(&[Scalar]) + build closure, bind(slot, value: Scalar), BoundParam.value, zip_params, Blueprint::compile_with_params(&[Scalar]), the sweep / mc / walkforward param-point args, aura-cli run_oos / scalar_as_param_f64	A param point routinely travels detached from its co-indexed param-space — serialized into the RunManifest, passed to run_one(seed, point). bind kind-checks the value against the slot. Self-description is load-bearing exactly where the manifest needs it.
Source ingestion: Source::next -> Option<(Timestamp, Scalar)> + the source-target push (crates/aura-engine/src/harness.rs)	The k-way-merged ingestion boundary (C3) is heterogeneous and self-describing by nature; the source produces a Scalar, so stripping it to a cell at the push would be churn at exactly the boundary whose job is to carry heterogeneous values.
The recorder's out-of-graph channel row Vec<Scalar> (crates/aura-std/src/recorder.rs)	An out-of-graph destination (C8 sink side effect) — the consumer at the other end of the mpsc has no schema, so the row self-describes.

The boundary, in one picture

  Source::next ──Scalar──▶ [push] ──▶ AnyColumn (typed) ──get──Scalar──▶ sinks/serde
   (ingestion, C3)                         │
                                           │ Ctx windows (native f64/i64/…)
                                           ▼
                                     Node::eval ──&[Cell]──▶ [push_cell] ──▶ AnyColumn (next node)
                                           ▲                                   (inter-node forward)
                                   out: [Cell; N]

  params:  &[Scalar] ─────────────────────────────────────▶ build / bind / manifest (self-describing)

The hot inter-node forward (eval → push_cell) carries bare cells; the two dynamic edges (ingestion in, get / serde out) and the param plane carry Scalar.

Concrete code shapes

Worked example — the node-author surface (the empirical evidence)

The authoring surface is the consumer here: every node author writes eval and an output buffer. Add is representative.

Before (crates/aura-std/src/add.rs):

pub struct Add {
    out: [Scalar; 1],
}
impl Add {
    pub fn new() -> Self {
        Self { out: [Scalar::f64(0.0)] }
    }
}
impl Node for Add {
    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
        let a = ctx.f64_in(0);
        let b = ctx.f64_in(1);
        if a.is_empty() || b.is_empty() {
            return None;
        }
        self.out[0] = Scalar::f64(a[0] + b[0]);
        Some(&self.out)
    }
}

After:

pub struct Add {
    out: [Cell; 1],
}
impl Add {
    pub fn new() -> Self {
        Self { out: [Cell::from_f64(0.0)] }
    }
}
impl Node for Add {
    fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Cell]> {
        let a = ctx.f64_in(0);
        let b = ctx.f64_in(1);
        if a.is_empty() || b.is_empty() {
            return None;
        }
        self.out[0] = Cell::from_f64(a[0] + b[0]);
        Some(&self.out)
    }
}

The change is strictly a simplification for the author: the kind is named once on the constructor (Cell::from_f64) and never re-asserted, because the node already declared FieldSpec { kind: F64 } in its schema. There is no authoring concept added — the read side (ctx.f64_in) is already native and is untouched.

A node-author eval test migrates its output assertion from the Scalar carrier to the Cell one (the read side and the param side stay Scalar):

// before
assert_eq!(add.eval(ctx), Some([Scalar::f64(14.0)].as_slice()));
// after
assert_eq!(add.eval(ctx), Some([Cell::from_f64(14.0)].as_slice()));

Cell derives PartialEq/Eq (bit-exact), so the assertion compares the raw words — which for a deterministic f64 sum is exactly the intended check.

Test-migration rule (precise — what migrates vs. what stays)

A unit test touches Scalar in three distinct roles; only one migrates. The rule keys on the role, not on "it is a test":

1. Eval-output assertions → migrate to Cell. The value a test reads back out of eval is now a cell: Some([Scalar::f64(x)].as_slice()) → Some([Cell::from_f64(x)].as_slice()), and a value extractor on an eval row migrates too — the sim_broker helper Some([s]) => s.as_f64() → Some([c]) => c.f64(). This is the only role that changes.
2. Test-fixture column writes → stay Scalar (push). Every node test seeds inputs with inputs[i].push(Scalar::f64(x)).unwrap() (e.g. exposure.rs, sma.rs, all of aura-std + the harness fixtures, ~40 sites). A column write models a source feed into that column — it is the ingestion boundary in miniature, squarely in the keep-set. These lines are left untouched; they keep the fallible push(Scalar) (which is retained precisely for ingestion + tests). They are not rewritten to push_cell.
3. Out-of-graph channel sends → stay Scalar. A fixture that records to a channel sends a self-describing Vec<Scalar> row — the recorder (recorder.rs) and any dual-role test fixture that both forwards an eval output (convert) and sends a channel row, e.g. TapForward (harness.rs, which sends vec![Scalar::f64(v)]). For such a fixture the eval signature + out-buffer migrate to Cell, but its channel send stays Scalar. The two roles in one node migrate independently, by role.

In one line: only what comes out of eval becomes a Cell; what goes into a column or out to a channel stays a Scalar.

Engine-internal shape — AnyColumn::push_cell (secondary)

Before — the forward path pushes a tagged Scalar and unwraps a kind check the code already trusts (crates/aura-engine/src/harness.rs):

// AnyColumn::push (kept, for the source-ingestion boundary + tests)
pub fn push(&mut self, v: Scalar) -> Result<(), KindMismatch> {
    match (self, v.kind()) {
        (AnyColumn::F64(c), ScalarKind::F64) => { c.push(v.as_f64()); Ok(()) }
        // … one arm per kind …
        (this, got) => Err(KindMismatch { expected: this.kind(), got }),
    }
}

// harness forward loop
let result: Option<&[Scalar]> = nb.node.eval(Ctx::new(&nb.inputs, ts));
if let Some(row) = result {
    scratch.clear();
    scratch.extend_from_slice(row);                 // scratch: Vec<Scalar>
    for e in out_edges[nidx].iter() {
        nb.inputs[e.slot]
            .push(scratch[e.from_field])
            .expect("edge kind checked at wiring");  // ← already treated infallible
    }
}

After — a branch-free, infallible cell push for the inter-node forward; the fallible Scalar push stays for ingestion:

// AnyColumn — new method; the kind is the column's own, the cell is reinterpreted
pub fn push_cell(&mut self, c: Cell) {
    match self {
        AnyColumn::I64(col) => col.push(c.i64()),
        AnyColumn::F64(col) => col.push(c.f64()),
        AnyColumn::Bool(col) => col.push(c.bool()),
        AnyColumn::Ts(col) => col.push(c.ts()),
    }
}

// harness forward loop
let result: Option<&[Cell]> = nb.node.eval(Ctx::new(&nb.inputs, ts));
if let Some(row) = result {
    scratch.clear();
    scratch.extend_from_slice(row);                 // scratch: Vec<Cell>
    for e in out_edges[nidx].iter() {
        nb.inputs[e.slot].push_cell(scratch[e.from_field]); // infallible
    }
}

The kind check is not lost — it is paid once at bootstrap (harness.rs verifies from.output[from_field].kind == to.inputs[slot].kind for every edge, green tests bootstrap_rejects_a_kind_mismatch, bootstrap_rejects_per_field_kind_mismatch, bootstrap_rejects_kind_mismatched_recorder_edge). push_cell realizes C7's "type check paid once at wiring, then direct dispatch".

The source-target forward keeps the Scalar path (ingestion is in the keep-set):

// unchanged — source value is a self-describing Scalar
nb.inputs[t.slot].push(value).expect("source kind checked at wiring");

AnyColumn::get stays Scalar (secondary)

get is the type-erased read for the boundary; it reconstructs the tag from the column kind on the way out, exactly as today:

pub fn get(&self, k: usize) -> Option<Scalar> {
    match self {
        AnyColumn::I64(c) => c.get(k).map(Scalar::i64),
        // … unchanged …
    }
}

Components

• crates/aura-core/src/node.rs — Node::eval return type Option<&[Scalar]> → Option<&[Cell]>. The param path in the same file (PrimitiveBuilder::build/new/bind, BoundParam.value, zip_params) stays Scalar. Test/fixture nodes (Bare, Probe) migrate their eval signature; Probe's param-vector field stays Vec<Scalar> (it stores a param point).
• crates/aura-core/src/any.rs — add push_cell(Cell); keep push(Scalar) and get -> Option<Scalar>. The aura_core::Cell re-export already exists.
• crates/aura-std/ — add, sub, sma, ema, exposure, lincomb, sim_broker: out: [Scalar; N] → [Cell; N], constructors and eval bodies use Cell::from_*; eval signature changes. recorder: eval signature changes (returns None); the channel row stays Vec<Scalar> and the per-kind Scalar::* construction from get(0)? stays. Each node's eval tests migrate output assertions to Cell; the sim_broker test helper Some([s]) => s.as_f64() becomes Some([c]) => c.f64().
• crates/aura-engine/src/harness.rs — scratch: Vec<Cell>; the inter-node forward uses push_cell; the source forward keeps push(Scalar). The in-file test/fixture nodes migrate their eval signatures and out-buffers per the test-migration rule above: a dual-role fixture such as TapForward migrates its eval output to Cell but keeps its out-of-graph channel send Scalar, and test column writes (inputs[i].push(Scalar::f64(x))) stay on the fallible push.
• crates/aura-engine/src/blueprint.rs, graph_model.rs — the test/fixture nodes that impl Node migrate their eval signatures + buffers. graph_model.rs::scalar_str (renders a BoundParam.value) stays Scalar. blueprint.rs::compile_with_params(&[Scalar]) and the sweep entry points stay Scalar (param plane).
• crates/aura-engine/src/sweep.rs, mc.rs, walkforward.rs — the &[Scalar] param-point args and scalar_as_f64 stay Scalar (param plane).
• crates/aura-cli/src/main.rs — run_oos(&[Scalar]), scalar_as_param_f64(&Scalar) stay Scalar (param plane).
• crates/aura-engine/src/report.rs — f64_field(&[(Timestamp, Vec<Scalar>)]) stays Scalar (it reads recorder channel rows).
• No change to crates/aura-core/src/ctx.rs (the read side is already native — f64_in/i64_in/… return typed Windows), column.rs, error.rs, scalar.rs, cell.rs.

Data flow

1. A source yields (Timestamp, Scalar); the harness pushes the self-describing Scalar into its target column via the fallible push (kind verified at wiring). [keep]
2. The engine evaluates each fired node in topo order. eval reads native windows from Ctx and returns Option<&[Cell]> — a borrowed row of bare cells into the node's own buffer. [convert]
3. The harness copies the row into scratch: Vec<Cell> and, per out-edge, push_cells one cell into the consumer column. No per-value kind check — bootstrap guaranteed it. [convert]
4. A sink/recorder reads AnyColumn::get -> Scalar (or native windows via Ctx) and emits a self-describing row out of the graph. [keep]
5. Params flow as &[Scalar] from the builder/sweep into compile_with_params and the RunManifest. [keep]

The values at every step are bit-identical to before; only steps 2–3 change the static type of the carrier from Scalar to Cell.

Error handling

• push_cell is infallible by construction. It cannot return KindMismatch because there is no second kind to compare — it pushes the cell into this column's concrete Column<T>, and the only way the wrong cell reaches the wrong column is a bootstrap-verified edge being wrong, which the bootstrap kind-check (green tests above) already forbids. KindMismatch and the fallible push remain for the ingestion boundary and the unit tests that exercise the guard directly.
• The per-value runtime kind check on node output is removed. Today, a node whose eval built a Scalar of the wrong kind for its declared output slot would be caught by the forward-path push(...).expect(...) (a panic). After the swap, a node that builds a cell of the wrong kind (e.g. Cell::from_i64 in an f64-output node) pushes raw bits with no panic. This is an accepted, bounded trade:
  • It is the same class of authoring bug as a node returning the wrong row width, which C8 already guards only with a debug_assert (row.len() == out_len), not a hard error.
  • The node's own eval test asserts the exact output value; a wrong-kind cell yields a garbage value that fails that assertion at the node's own test layer.
  • It is consistent with C7: the output kind is the node's declared contract (FieldSpec.kind), honored by the node, not re-checked per value on the hot path. The check that does matter cross-node — that producer and consumer agree — stays at bootstrap.
• No new panics or Results on the hot path. The change only removes a Result the code already .expected.

Testing strategy

• Behaviour preservation is the headline. The existing suite is the regression net: every node eval test, the harness end-to-end forward tests, the sweep/mc/walk-forward/registry tests, and the manifest/metric tests must stay green after the carrier migration. Their values do not change; only the output-side assertions re-spell Scalar::f64(x) as Cell::from_f64(x).
• push_cell round-trip. Add a focused AnyColumn test: push_cell of each kind round-trips through get to the same Scalar (mirrors the existing same_kind_push_round_trips), proving the cell is interpreted by the column's kind.
• push guard retained. The existing wrong_kind_push_is_rejected stays green (the fallible Scalar push is unchanged), proving the ingestion guard is intact.
• Determinism spot-check. A harness run's recorded equity/exposure rows and RunManifest are unchanged vs. before the swap — covered by the existing end-to-end report tests (f64_field projections + summarize); no new golden file needed because those tests already pin the exact numeric output.
• All four gates clean: cargo build / clippy --all-targets -D warnings / test / doc --workspace.

Acceptance criteria

Judged against aura's domain invariants (the project's feature-acceptance criterion):

1. C7 made literal on the carrier. Node::eval and every node out-buffer carry bare Cell (8 bytes, no tag, no per-value branch); the kind lives only at the schema/column. Scalar survives exactly on the param / get / source boundaries enumerated above — nowhere on the inter-node hot path.
2. C1 preserved. Same input → same run: every existing test green with only carrier-type spelling changes on eval-output assertions; recorded rows and manifests byte-identical.
3. No reintroduced failure class. The cross-node kind agreement stays enforced at bootstrap (the green bootstrap-reject tests); the only removed check is the redundant runtime one the code already .expected. The node-output-kind responsibility is the node author's, caught by the node's own test — the same posture C8 already takes for output width.
4. Authoring surface no harder. The worked Add before→after shows the node author writes less ceremony (one Cell::from_f64 per output, no tag named twice), with no new concept on the read side.
5. All four workspace gates clean.