diff --git a/docs/specs/0001-core-streaming-substrate.md b/docs/specs/0001-core-streaming-substrate.md new file mode 100644 index 0000000..3f68ac4 --- /dev/null +++ b/docs/specs/0001-core-streaming-substrate.md @@ -0,0 +1,331 @@ +# Core Streaming Substrate — Design Spec + +**Date:** 2026-06-03 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +## Goal + +Deliver the foundation everything in aura streams through: the four scalar base +types, the bounded-lookback Structure-of-Arrays column with financial-style +indexing, the zero-copy read window a node sees, the type-erased edge carrier, +and the per-column run-count that the freshness model is built on. This is the +first cycle of the walking-skeleton milestone (ingest → one signal → +deterministic backtest → position table → sim-optimal broker → pip-equity); it +is the substrate the subsequent `Node`/`Ctx`/engine-loop cycles build on. + +The reference is RustAst's `src/ast/rtl/series` and `streams` — a working +implementation of this substrate. aura keeps its *shape* (newest-at-0 indexing, +a per-series push counter for freshness, SoA composites) and **sharpens** it: +no boxed `Value`, no `VecDeque`, no `RefCell`/`Rc`, no per-event allocation. + +This cycle implements only the data substrate. **Out of scope** (subsequent +cycles): the `Node` trait, the evaluation `Ctx`, firing policies A/B, the +deterministic sim loop / freshness-gated recompute, the ingestion boundary / +k-way merge, and the named OHLCV bundle. Those are named here only to fix this +cycle's boundary; this cycle delivers the primitives they will consume. + +## Architecture + +`aura-core` gains four small modules, no dependencies added: + +- `scalar` — the closed four-type scalar set: the `Timestamp` newtype, the + `Scalar` Copy-POD carrier, and the `ScalarKind` tag (C7). +- `column` — `Column`: a fixed-capacity ring buffer, pre-sized to a lookback + limit at construction, financial-indexed (index 0 = newest), carrying a + monotonic `run_count` (C5/C8). `Window<'_, T>` is its zero-copy read view. +- `any` — `AnyColumn`: the type-erased edge carrier, an enum over the four + `Column` kinds. The kind check is paid at the edge (wiring time); the hot + loop pushes into the concrete `Column` monomorphically (C7). +- `error` — `KindMismatch`, the wiring-time guard error returned when a `Scalar` + of the wrong kind is pushed into an `AnyColumn`. + +The whole substrate is **interior-mutability-free**. A sim is single-threaded +and non-concurrent (C1), so the engine owns columns by `&mut` for the push side +and hands out `&Column` / `Window` (shared borrow) on the read side — RustAst's +`RefCell`-on-every-series disappears. + +## Concrete code shapes + +### North-star slice (the consumer this substrate exists to serve) + +A future `aura-std` SMA node — *not built this cycle* — reads its input as an +indexed, newest-at-0 window. This is the minimal honest slice proving the +substrate's read API fits the node use case it exists for: + +```rust +// Illustrative future consumer (NOT delivered this cycle): the read shape an +// indicator node wants. `length` is the node's pre-sized lookback (C8). +fn sma(window: &Window<'_, f64>, length: usize) -> Option { + if window.len() < length { + return None; // not yet warmed up (C8: None = not-yet-warmed-up) + } + let sum: f64 = (0..length).map(|k| window[k]).sum(); // [0] newest, [length-1] oldest + Some(sum / length as f64) +} +``` + +The point of the cycle is that `Window` makes this code possible with no +lookback bookkeeping in the node: the engine pre-sized the buffer, the node only +reads by index. + +### Delivered code — the scalar set (`scalar.rs`) + +```rust +/// Canonical engine time: epoch-nanoseconds UTC. Newtype over i64 (C7). +/// `Default` (epoch 0) is required so `Column` can pre-size its ring. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Timestamp(pub i64); + +/// The four scalar base types, type-erased for an edge. Copy POD — no `dyn Any`, +/// no heap, no Rc (C7 forbids all three on the hot path). +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Scalar { + I64(i64), + F64(f64), + Bool(bool), + Ts(Timestamp), +} + +/// The kind tag of a scalar / column, used for the edge-time type check. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum ScalarKind { + I64, + F64, + Bool, + Timestamp, +} + +impl Scalar { + pub fn kind(self) -> ScalarKind { /* match */ } +} + +// Total, infallible widenings into the carrier: +impl From for Scalar { /* ... */ } +impl From for Scalar { /* ... */ } +impl From for Scalar { /* ... */ } +impl From for Scalar { /* ... */ } +``` + +### Delivered code — the column and its window (`column.rs`) + +```rust +/// A fixed-capacity, bounded-lookback SoA column. Pre-sized at construction +/// (C8: no realloc in the hot loop); financial-indexed (index 0 = newest); +/// carries a monotonic push counter for the freshness model (C5). +pub struct Column { + buf: Box<[T]>, // capacity == lookback; never reallocated + head: usize, // ring write cursor (next slot) + len: usize, // valid elements, saturates at capacity + run_count: u64, // monotonic total pushes — RustAst's total_count (C5) +} + +impl Column { + /// `lookback` is the window depth the engine sized for this edge (C8). + pub fn with_capacity(lookback: usize) -> Self { /* Box<[T]> of len lookback */ } + + /// Append the newest value. O(1), no allocation. Bumps `run_count`. + /// On overflow the oldest value is overwritten (true ring). + pub fn push(&mut self, v: T) { /* buf[head]=v; head=(head+1)%cap; len=min(len+1,cap); run_count+=1 */ } + + /// Financial index: 0 = newest, `len()-1` = oldest. None if `k >= len` + /// (C8: None == not-yet-warmed-up / out of range). Zero-copy. + pub fn get(&self, k: usize) -> Option { /* buf[(head - 1 - k) mod cap] */ } + + pub fn len(&self) -> usize { self.len } + pub fn capacity(&self) -> usize { self.buf.len() } + pub fn run_count(&self) -> u64 { self.run_count } + + /// A zero-copy read view handed to a reading node. + pub fn window(&self) -> Window<'_, T> { Window { col: self } } +} + +/// Read-only, zero-copy view into a `Column` — what a node sees (C8). +pub struct Window<'a, T> { + col: &'a Column, +} + +impl<'a, T: Copy + Default> Window<'a, T> { + pub fn get(&self, k: usize) -> Option { self.col.get(k) } + pub fn len(&self) -> usize { self.col.len() } + pub fn run_count(&self) -> u64 { self.col.run_count() } +} + +/// `window[k]` — newest-at-0 indexed access; panics on out-of-range, matching +/// slice `Index` semantics. Nodes that may be cold use `get` instead. +impl<'a, T: Copy + Default> core::ops::Index for Window<'a, T> { + type Output = T; + fn index(&self, k: usize) -> &T { /* &buf[(head-1-k) mod cap], bounds-checked */ } +} +``` + +### Delivered code — the type-erased edge (`any.rs`) + +```rust +/// A type-erased edge: the four concrete columns behind one enum (C7). The kind +/// check happens here, at wiring time; the hot loop grabs the concrete column +/// once (`as_f64_mut`) and pushes monomorphically — no per-event dispatch. +pub enum AnyColumn { + I64(Column), + F64(Column), + Bool(Column), + Ts(Column), +} + +impl AnyColumn { + pub fn with_capacity(kind: ScalarKind, lookback: usize) -> Self { /* match kind */ } + + pub fn kind(&self) -> ScalarKind { /* match self */ } + pub fn len(&self) -> usize { /* match self */ } + pub fn run_count(&self) -> u64 { /* match self */ } + + /// Edge-time typed push. Returns `KindMismatch` if `v.kind() != self.kind()` + /// — the C7 type-erasure guard. The hot path avoids this by taking the + /// concrete `Column` once via `as_*_mut`. + pub fn push(&mut self, v: Scalar) -> Result<(), KindMismatch> { /* match (self, v) */ } + + /// Read one type-erased value (0 = newest). None if cold / out of range. + pub fn get(&self, k: usize) -> Option { /* match self -> Scalar::… */ } + + /// Concrete-column accessors for the monomorphic hot path. + pub fn as_f64_mut(&mut self) -> Option<&mut Column> { /* … */ } + pub fn as_i64_mut(&mut self) -> Option<&mut Column> { /* … */ } + pub fn as_bool_mut(&mut self) -> Option<&mut Column> { /* … */ } + pub fn as_ts_mut(&mut self) -> Option<&mut Column> { /* … */ } +} +``` + +### Delivered code — the must-fail guard fixture (`error.rs` + a test) + +The C7 type-erasure contract is load-bearing: a wrong-kind push must be rejected, +never silently coerced. The test whose *correct behaviour is rejection*: + +```rust +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct KindMismatch { + pub expected: ScalarKind, + pub got: ScalarKind, +} + +#[test] +fn wrong_kind_push_is_rejected() { + let mut edge = AnyColumn::with_capacity(ScalarKind::I64, 8); + let err = edge.push(Scalar::F64(1.5)).unwrap_err(); // pushing f64 into an i64 edge + assert_eq!(err, KindMismatch { expected: ScalarKind::I64, got: ScalarKind::F64 }); + assert_eq!(edge.len(), 0); // nothing was stored + assert_eq!(edge.run_count(), 0); // and the counter did not move +} +``` + +### Implementation shape (secondary — before → after) + +`aura-core/src/lib.rs` today is doc-comment-only (no `pub` items). After this +cycle it declares the four modules and re-exports the surface: + +```rust +// before: lib.rs is documentation only, defines no types. + +// after: +mod scalar; +mod column; +mod any; +mod error; + +pub use scalar::{Scalar, ScalarKind, Timestamp}; +pub use column::{Column, Window}; +pub use any::AnyColumn; +pub use error::KindMismatch; +``` + +## Components + +- **`Timestamp`** — `i64` epoch-ns newtype, fully ordered (the ingestion + boundary will normalize source-native units into it in a later cycle; here it + is just the type). +- **`Scalar` / `ScalarKind`** — the closed carrier + tag. `Scalar` is `Copy`; + `From` impls give ergonomic widening; `kind()` projects to the tag. +- **`Column`** — the ring. Owns a pre-sized `Box<[T]>`; `push` is O(1) and + allocation-free; `get`/`Index` implement newest-at-0; `run_count` is the + freshness primitive. +- **`Window<'_, T>`** — the borrow a reader gets; carries no storage, only a + reference; `Index` for the warm path, `get` for the possibly-cold path. +- **`AnyColumn`** — the edge; type check at the boundary, concrete-column + accessors for the hot loop. +- **`KindMismatch`** — the guard error. + +## Data flow + +1. The engine (later cycle) sizes an edge: `AnyColumn::with_capacity(kind, + lookback)` — buffer allocated once, never again (C8). +2. A producer pushes its newest output for the cycle. Wiring-time / dynamic + path: `edge.push(scalar)?` (kind-checked). Hot path: the engine took + `edge.as_f64_mut()` once at sim start and calls `col.push(x)` monomorphically + each cycle — no per-event dispatch, no allocation (C7). +3. `run_count` increments on every push; the freshness model (later cycle) + compares an input's `run_count` against the value it last saw to decide + fire-vs-hold (C5). This cycle delivers the counter, not the comparison loop. +4. A reading node receives `col.window()` and reads by index, newest at 0 (C8). + +There is no merge, no clock, and no recompute logic in this cycle — those are the +engine's, downstream. The substrate is pure data structures + their invariants. + +## Error handling + +- **Wrong-kind push** → `Err(KindMismatch { expected, got })`; the column is + left untouched (len and run_count unchanged). This is the only fallible + operation in the substrate. +- **Cold / out-of-range read** → `get` returns `None` (C8: not-yet-warmed-up); + `Index` panics like a slice (the warm-path ergonomic, used only when the node + has already checked `len`). +- **No panics on the push/normal-read path**; no allocation after construction; + no interior mutability to poison. + +## Testing strategy + +Unit tests live beside each module; all run under `cargo test --workspace`. + +- **`Column` ring semantics:** push then `get(0)` is newest; `get(len-1)` is + oldest; pushing more than capacity keeps exactly the newest `capacity` values + and drops the oldest (wraparound correctness); `len` saturates at capacity; + `get(k>=len) == None`. +- **`run_count` is the freshness primitive:** monotonic across pushes + (including after the ring wraps — it counts *pushes*, not stored len); never + moved by a read. +- **`Window` is a zero-copy view:** values read through the window equal the + column's; taking a window does not change the column's capacity (no realloc); + `Index` and `get` agree on the warm range. +- **`Scalar` / `ScalarKind`:** `kind()` is correct for each variant; `From` + widenings round-trip through `get` on an `AnyColumn`. +- **`AnyColumn` type erasure:** same-kind push stores and is read back as the + right-kind `Scalar`; `run_count`/`len` delegate; the concrete `as_*_mut` + accessor returns `Some` only for the matching kind. +- **The must-fail guard** (`wrong_kind_push_is_rejected`, shown above): a + wrong-kind push is rejected and leaves the column untouched. + +## Acceptance criteria + +Applying aura's feature-acceptance lens (CLAUDE.md): + +1. **The intended audience naturally reaches for it.** The node author (LLM via + builder APIs, later cycles) reads inputs as `window[k]` newest-at-0 — the + north-star slice above is exactly the SMA an author writes, with zero + lookback bookkeeping. The substrate is the thing every node, source, and sink + stands on. +2. **It measurably improves correctness / removes redundancy.** Engine-sized, + pre-allocated windows mean node code *cannot* mis-manage lookback (C8 + rationale); the closed `Scalar` set + `AnyColumn` removes RustAst's boxed + `Value` and per-series `RefCell` entirely. +3. **It reintroduces no failure class the core constraints exist to kill.** No + `dyn Any`, no per-event heap allocation, no interior mutability, fixed + pre-sized buffers (C1/C7/C8 all upheld); the wrong-kind push is structurally + rejected, not coerced. + +Concretely, the cycle ships when: + +- `cargo build --workspace`, `cargo test --workspace`, and + `cargo clippy --workspace --all-targets -- -D warnings` are all green. +- Every test in the Testing-strategy list exists and passes, including the + `wrong_kind_push_is_rejected` must-fail guard. +- `aura-core` exports `Scalar`, `ScalarKind`, `Timestamp`, `Column`, `Window`, + `AnyColumn`, `KindMismatch` and nothing leaks `RefCell`/`Rc`/`dyn Any` on the + substrate surface.