d2d0d0f4fd6e1780d37dbffdd4c8a2f0c8f957de
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
d5602ec5ad |
feat(0087): blueprint serialization & loader — close the graph-as-data loop
C24 first cut (#155): a blueprint — the param-generic, named graph-as-data a Rust builder produces — is now a first-class serializable data value with a canonical, versioned format, and the engine has the missing load path (data -> blueprint -> FlatGraph), not only the Rust-builder construction path. What ships: - `blueprint_to_json` / `blueprint_from_json` in a new `aura-engine` blueprint_serde module: a faithful serde DTO projection (the build closures dropped; re-derived on load), top-level `format_version` envelope, canonical compact JSON (struct field order, defaults omitted via skip_serializing_if). - A resolver seam: the loader is generic over an injected `Fn(&str) -> Option<PrimitiveBuilder>`; the concrete closed `match` over the aura-std vocabulary lives in aura-std (`std_vocabulary`), never in the engine (the engine stays domain-free: lib deps aura-core + aura-analysis only, never the node-vocabulary crate). This is C24's compiled-in closed-set referenced by type identity, NOT a dynamic/marketplace node registry (domain invariant 9). - serde derives on the serialized plain types (Edge, Target, OutField, Role, BoundParam, ScalarKind); BoundParam additionally re-exported from aura-core's root (an additive fix the plan's file list missed). Load-bearing scope decisions (derived, logged on #155): - Sinks are excluded from the serialized blueprint — a recording sink captures an mpsc::Sender (runtime identity, not param/topology, C19 value-empty); sink addressing is the C18-registry / #101 concern. The round-trip test attaches identical sinks post-load to both twins. - Construction-arg builders (LinComb(arity), CostSum(n), SimBroker(pip), Session(..)) are out of the round-trippable set: their structural args are a C20 structural-axis concern the param-generic format does not yet encode; an absent type_id fails the load cleanly (UnknownNodeType), never a silent wrong graph. #155's vocabulary is the 22 zero-arg aura-std builders. Additively extensible later (#156). Orchestrator hardening on review: the serializer now emits bound params in ascending original-slot order (mirroring the loader), so the canonical form is bind-order-independent — a precondition for content-addressing (#158). Identity today (every #155 node has <=1 param); holds by construction for future multi-param nodes. Acceptance (all green): a serialized blueprint runs bit-identical (C1) to its Rust-built twin; serializer/loader are inverse (canonical idempotence, incl. a recursive nested composite); unknown-type and unsupported-version fail named, never panicking. cargo test --workspace 748 passed; clippy --all-targets -D warnings clean. closes #155 |
||
|
Brummel
|
86746e3d5d |
refactor(aura-core): Scalar as a native tagged enum, disjoint from Cell; typed RunManifest.params
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
|
||
|
Brummel
|
43716be10e |
refactor(aura-engine): Cell into the construction path — param-plane base/frontend split
Cell becomes the carrier of the construction path; Scalar narrows to the author/render boundaries. The validated/enumerated param point carries no redundant kind (it lives once, in the declared param-space); at the author edge the kind is a checksum — two independent sources, the typed value vs. the slot — so the self-describing Scalar stays there. The name->slot binding is dynamic (C23), so the check is necessarily a runtime one and the value must self-describe for it. Base/frontend split (the AnyColumn push/push_cell pattern one level up): compile_with_cells / bootstrap_with_cells are the kind-check-free base; compile_with_params / bootstrap_with_params are the frontend that adds only the per-value kind checksum, strips to cells (new Scalar::cell(), the partner of Scalar::from_cell), and delegates. lower_items loses its per-primitive kind-check; PrimitiveBuilder::build and the std node builders (sma/ema/exposure/lincomb) read cells. Boundary: construction -> Cell (PrimitiveBuilder::build, lower_items, GridSpace.axes/points, SweepPoint.params, the sweep closure). Author edges stay Scalar (GridSpace::new, bind, compile_with_params/bootstrap_with_params). walkforward chosen_params stays a self-describing Scalar report record (Option A — WalkForwardResult carries no space); the cell winner is reconstructed once at the WindowRun site via from_cell. injective-check moved ahead of the arity-check in the frontend to preserve the pre-split error order (DuplicateParamPath before ParamArity). The lossy i64->f64 param projection (scalar_as_f64 / scalar_as_param_f64) is deliberately untouched — a separate follow-up. Behaviour-preserving (C1): build --all-targets / test / clippy -D warnings / doc all clean across the workspace. |
||
|
Brummel
|
cd3d1ca9ed |
refactor(aura-core): split Scalar into a tag-free Cell + ScalarKind
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
|
||
|
Brummel
|
e97906a0e1 |
test(aura-core): pin bare-literal Into<Scalar> kind inference
Authoring layers that lower raw literals via `impl Into<Scalar>` (the named-param-binding surface, spec 0030 / #35) rely on a bare integer literal inferring as i64 and a bare float as f64 — because i64/f64/bool are the only `From<…> for Scalar` impls, each literal class resolves to exactly one. This was true but untested: `from_widenings` uses suffixed literals (`7i64`), so a regression in the inference path would survive it. Behaviour-preserving pin of existing aura-core behaviour. Closes the one grounding gap the grounding-check flagged on spec 0030; also guards the property independently — adding a second integer `From` impl would now break loudly here rather than silently at a `.with("knob", 2)` call site. refs #35 |
||
|
Brummel
|
eec876975c |
feat(engine): serde foundation + amend C16 dependency policy (cycle D / iter 1)
Iteration 1 of the run-registry cycle (#33): lay the dependency + serde foundation the registry's typed read-path needs, and amend the contract that forbade it. Contract change (load-bearing — C18/C16). C16's blanket "zero-external- dependency by commitment" + "aura-ingest is the sole external-dependency firewall" framing is struck and replaced by a considered, per-case dependency policy: dependencies are admitted by deliberate review (what a crate pulls in vs. what it buys), with particular scrutiny for anything entering the frozen deploy artifact (C13); standard vetted crates (serde, rayon, ...) pass that review and are used wherever they do the job, including in the bot; hand-rolling what a vetted crate does is the anti-pattern. C16's engine/project split + three-tier node reuse are unchanged. The five source comments that asserted the struck framing (aura-engine/aura-ingest Cargo.toml + aura-ingest/report.rs docs) are rewritten to match; no live source or ledger text still asserts it. Per-case justification for serde (the policy now requires one): it gives the run-report types a typed (de)serialization path — exactly what ranking/compare over stored runs needs, and what the writer-only hand-rolled JSON (C14) could never provide; closes the typed-read-path gap (#17). Its closure is tiny, ubiquitous, heavily audited, and its output deterministic (C1-safe). serde_json is test-only this iteration (dev-dependency); no production serde_json yet. Changes: [workspace.dependencies] serde (derive) + serde_json; serde derives on Timestamp (aura-core) and RunMetrics/RunManifest/RunReport (aura-engine), with serde_json round-trip tests (window renders as a [from,to] integer array via the transparent Timestamp newtype). The hand-rolled to_json writers are untouched (still drive stdout). No aura-registry crate, no SweepPoint change — those are iterations 2 and 3. Verified: cargo test --workspace green (incl. the two new round-trip tests); clippy --workspace --all-targets -D warnings clean; no surviving assertion of the struck zero-dep framing in live source or the ledger. refs #33 |
||
|
Brummel
|
4b64409036 |
feat(aura-core,aura-std,aura-engine,aura-cli): inject a param-set vector at bootstrap (#31)
Cycle B of milestone "The World — parameter-space & sweep". A blueprint is now value-empty: a leaf holds a param-generic recipe, not a built node, and a positional Scalar vector is bound slot-by-slot at bootstrap — so one blueprint bootstraps into many distinct instances under different vectors, with no cdylib rebuild (C12/C19). This is the binding primitive a sweep (#32) drives. Ratified design (brainstorm): value-empty reconstruct-through-new() over mutate-in-place and over a default-bearing variant. The value lives only in the injected vector (no baked default), keeping the blueprint a pure param-generic recipe (C19); every injected value flows through the node's own constructor (the single sizing/validation gate). - aura-core: LeafFactory { name, params, build } — the recipe (params -> sized node through `new`); Scalar::as_i64/as_f64 value accessors. Node trait unchanged. - aura-std: each of the 7 nodes exposes factory() (SMA length:I64, Exposure scale:F64, LinComb arity x weights[i]:F64; Sub/Add/SimBroker/Recorder paramless, capturing their non-param construction args — pip_size, the Recorder channel). - aura-engine: BlueprintNode::Leaf(LeafFactory), From<LeafFactory>; param_space() reads factory.params() pre-build; compile_with_params/bootstrap_with_params build each leaf from its kind-checked slice while lowering (build-then-wire), arity checked up front via param_space().len(); CompileError::{ParamKindMismatch, ParamArity}. compile/inline/edge-rewrite are structurally unchanged, so the compilat stays bit-identical for a given point (the bit-identity and both param_space mirror tests stay green). The vestigial pre-build Composite::schema / BlueprintNode::schema (no live caller — interface resolution is structural on the built flat nodes) are removed. - aura-cli: the blueprint view labels leaves by bare type via LeafFactory::label() (`[SMA]`) — the ascii-dag renderer cannot render wide cluster-sibling labels (see spec); the compiled view still labels valued (`SMA(2)`). The sample supplies its point as a vector; the mis-wiring swap moved to the compiled view. The #34 dual-traversal drift hazard is subsumed: compile_with_params consumes the vector in the same recipe walk param_space() reports, so the two share one traversal. Verified: cargo build/test --workspace green (127 tests, incl. bit-identity, both mirror tests, and 4 new injection tests — different-vector-different-run, kind mismatch, arity, determinism); clippy --workspace --all-targets -D warnings clean; `aura graph` blueprint view renders cleanly. Scope: one vector -> one instance. Deferred: sweep enumeration (#32), domain validation (#32/C20), single-run authoring convenience (#35). closes #31 |
||
|
Brummel
|
a4760497ec |
feat(core): the streaming substrate — scalars, columns, type-erased edges
Cycle 0001 of the walking-skeleton milestone: the aura-core data substrate
everything streams through.
- scalar.rs: the closed four-type set — Scalar (Copy POD carrier), ScalarKind
(the kind tag), Timestamp (i64 epoch-ns newtype). No boxed Value, no dyn Any,
no heap (C7).
- column.rs: Column<T>, a fixed-capacity bounded-lookback ring pre-sized at
construction (no hot-loop realloc, C8), financial-indexed (index 0 = newest),
carrying a monotonic run_count freshness primitive (C5). Window<'_,T> is its
zero-copy read view with newest-at-0 Index. No RefCell/Rc — a sim is
single-threaded (C1), so push is &mut and read is &.
- any.rs: AnyColumn, the type-erased edge over the four Column kinds. push()
kind-checks at the edge (wiring time) and rejects a wrong-kind value with
KindMismatch, leaving the column untouched; the hot path bypasses the check
via the monomorphic as_*_mut accessors (C7).
- error.rs: KindMismatch { expected, got }, the guard error.
Sharpens the RustAst rtl/ reference (VecDeque + RefCell + boxed Value) into a
fixed ring of Copy scalars with direct dispatch. Node trait, Ctx, firing
policies, and the sim loop are deferred to subsequent cycles.
Verified independently: cargo build --workspace (0 warnings), cargo test
--workspace (14 aura-core tests green incl. ring-wrap, run_count monotonicity,
zero-copy window, and the wrong_kind_push_is_rejected guard), cargo clippy
--workspace --all-targets -D warnings (clean), and a surface-purity grep
confirming no RefCell/Rc/dyn Any on the substrate.
refs #0001-core-streaming-substrate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|