diff --git a/docs/specs/0087-blueprint-serialization-loader.md b/docs/specs/0087-blueprint-serialization-loader.md new file mode 100644 index 0000000..7567f0b --- /dev/null +++ b/docs/specs/0087-blueprint-serialization-loader.md @@ -0,0 +1,333 @@ +# Blueprint serialization & loader (close the graph-as-data loop) — Design Spec + +**Date:** 2026-06-29 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude +**Cycle:** 0087 +**Issue:** #155 (milestone "Topology-as-data: blueprint serialization & loader (C24)") + +## Goal + +Close the graph-as-data loop mandated by **C24** (ledger, committed `f80571a`): +make a **blueprint** — the param-generic, named graph-as-data a Rust builder +produces (C9/C19) — a **first-class serializable data value** with a stable, +versioned, canonical format, *and* give the engine the missing **load path** +(`data → blueprint → FlatGraph`). Today only the forward render half exists +(`model_to_json`, `crates/aura-engine/src/graph_model.rs`); there is no faithful +serializer and no loader. + +Scope is **format + serializer + loader only** (#155). Explicitly out of scope, +each its own milestone issue: + +- the forward-compat *must-understand* refusal gate (Tier-2 unknown node type / + too-new version) — #156; +- the introspectable construction service that *emits* blueprints — #157; +- content-addressed topology identity in the run manifest — #158. + +### Acceptance criterion (applied prospectively) + +aura's feature-acceptance default: the feature solves a real design-mandated +problem and contradicts no stated design commitment. This cycle: + +- **Solves a mandated problem.** C24/C21/C18 cannot orchestrate, structurally + search, or *reproduce* topology-varying families while topology stays opaque + Rust source. The load path is the substrate they stand on. +- **Reintroduces no forbidden failure class.** The format carries **no node + logic** (C17 — computation stays compiled Rust; no RustAst regression), is a + **static DAG** (non-Turing by construction), and references nodes by + **compiled-in type identity over a closed vocabulary** — *not* a by-name + marketplace/registry (C24 forbid; domain invariant 9). +- **Empirical evidence** (this is an infrastructure cycle; the evidence is the + delivered code + a must-pass test, per the no-surface-still-show-code rule): + the round-trip test below, whose correct behaviour is a **bit-identical run** + (C1) of a serialized-then-loaded blueprint against its Rust-built twin, plus + the canonical JSON a real example blueprint serializes to (shown under + *Concrete code shapes*). + +## Architecture + +Four pieces, placed to honour the engine-stays-domain-free boundary +(`aura-engine → aura-core` only; `aura-std` is a *test-only* dep of the engine — +see `crates/aura-engine/Cargo.toml`): + +1. **A serde DTO layer** (new module `crates/aura-engine/src/blueprint_serde.rs`). + `Composite` and `PrimitiveBuilder` cannot derive serde — they transitively + hold `build: Box Box>` (`node.rs:115`). So the + format is a **separate, faithful projection** to serde-derivable DTO mirrors, + *not* an extension of the lossy render model `model_to_json` (which carries no + constructor link and no param values — confirmed lossy). This mirrors the + established read-widening idiom in `crates/aura-registry/src/compat.rs`. + +2. **The serializer** `blueprint_to_json(&Composite) -> Result`: + projects a `&Composite` to the DTO (dropping the build closures, which the + loader re-derives) and emits **canonical** JSON — compact, struct-declaration + field order, defaults omitted via `skip_serializing_if`. Canonicality is the + precondition for stable content-addressing (#158) and for aura's existing + byte-identical-golden discipline. + +3. **The loader** `blueprint_from_json(data, resolve) -> Result`: + reads the `format_version` envelope first, deserializes the DTO, and + reconstructs the `Composite`. The build closures are recovered **by type + identity** through an **injected node resolver** (next point); bound params + and instance names are re-applied from the DTO. The terminal step downstream + is the unchanged `Composite::compile_with_params` (`blueprint.rs:261`). + +4. **The node resolver seam.** The loader is **generic over an injected + resolver** `Fn(&str) -> Option`, defined engine-side + (`aura-core`/`aura-engine`, so the engine stays domain-free). The **concrete + closed dispatch over the `aura-std` vocabulary** lives in the crate that owns + those nodes — a new `std_vocabulary(type_id)` in `aura-std`, a plain + compiled-in `match` over its ~29 `builder()` factories. This is **not** the + forbidden registry (no dynamic registration, no marketplace, no engine-side + node table): it is exactly C24's "compiled-in closed-set referenced by type + identity". The injected seam is also where a project `cdylib` later supplies + its own vocabulary (C16/#157) — but loading a cdylib is out of #155's scope; + this cut wires the seam and one `aura-std` implementation. + +### The load-bearing fork and its resolution (recorded on #155) + +**Fork:** the loader fundamentally needs a `type-id → constructor` resolution, +and domain invariant 9 forbids "a node registry inside aura"; the recon flagged +this as a possible ledger-level decision. + +**Decision (derived, not a user preference):** the *closed, compiled-in, +injected* resolver is **not** the forbidden registry — it realises the mechanism +C24 *already* mandates ("compiled-in type identity — the closed, typed node +vocabulary of `aura-std` + the project `cdylib`"; forbids only "a by-name node +marketplace/registry … the format is not a distribution mechanism"). Invariant +9's "no node registry" governs *multi-project management / distribution*, not a +crate's compile-time `match` over its own known nodes. The resolution is further +**forced** by the crate graph: `aura-engine → aura-core` only, so the engine +*cannot* hold an `aura-std` node table — the resolver *must* be injected and the +vocabulary table *must* live outside the engine. This is a realisation of C24, +not a contract change, so it is a `specify` decision, not a `brainstorm` bounce. +(Basis: C24 body `docs/design/INDEX.md:1751-1766`, `f80571a`; invariant 9 in +`crates/aura/CLAUDE.md`; `crates/aura-engine/Cargo.toml` engine-domain-free +comment.) + +### What is and is not in the serialized blueprint + +- **In:** topology (nodes, edges, input roles, exposed outputs), each primitive + node's **type identity**, its instance name, and its **bound params** (the + knobs the author fixed). Nested composites recurse structurally. +- **Out (param point):** the *free* param-space is **not** baked to a single + point — a blueprint is param-generic (C19). The free point is supplied at + `compile_with_params` exactly as today; the run manifest records the point + separately (C18, `report.rs`). The round-trip test binds the **same** point to + both twins. +- **Out (sinks / observation):** a recording sink (`Recorder::builder(.., tx)`, + `test_fixtures.rs:67`) captures an `mpsc::Sender` in its build closure — + runtime identity, neither a param nor topology, so it structurally cannot live + in a param-generic, value-empty blueprint (C19). Sink-destination addressing + is the C18-registry / trace-persistence concern (#101), out of #155. The + round-trip test attaches **identical sinks post-load** to both twins and + compares the recorded traces. (How a serialized harness later re-acquires its + sinks — observation is part of the harness in C12 — is #101/#158's question, + not #155's; this cut serialises the closure-free topology and stops there.) + +## Concrete code shapes + +### Delivered API + the canonical artifact (the north-star slice) + +What a World / CLI eventually does — serialize an authored signal graph, store +the bytes, reload and run them (the capability this cycle delivers the substrate +for): + +```rust +let bp: Composite = sma_cross_signal(); // authored via the Rust builder (C17) +let json = blueprint_to_json(&bp)?; // canonical, versioned bytes +std::fs::write("sma_cross.bp.json", &json)?; // topology is now a stored value (C24) + +// later / elsewhere — no Rust rebuild, just data: +let data = std::fs::read_to_string("sma_cross.bp.json")?; +let bp2 = blueprint_from_json(&data, &|t| aura_std::std_vocabulary(t))?; +let flat = bp2.compile_with_params(&[Scalar::i64(2), Scalar::i64(4)])?; // runs (C1) +``` + +The canonical bytes for a minimal SMA-cross **signal** graph (`sma_fast`, +`sma_slow`, a `Sub`, a `Bias`; fast length bound to 2, slow open): + +```json +{"format_version":1,"blueprint":{"name":"sma_cross","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow"}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias"}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}]}],"output":[{"node":3,"field":0,"name":"bias"}]}} +``` + +Note: `slow` carries no `name`-less-than-needed clutter — `bound` and `name` are +**omitted** when empty/None (`skip_serializing_if`), so an absent optional is +byte-identical to the pre-extension form (the additive-extension precondition +#156 builds on). + +### DTO layer (new — `blueprint_serde.rs`), before → after + +There is no "before"; this is new. Shape: + +```rust +pub const BLUEPRINT_FORMAT_VERSION: u32 = 1; + +#[derive(Serialize, Deserialize)] +pub struct BlueprintDoc { pub format_version: u32, pub blueprint: CompositeData } + +#[derive(Serialize, Deserialize)] +pub struct CompositeData { + pub name: String, + pub nodes: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] pub edges: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] pub input_roles: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] pub output: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum NodeData { Primitive(PrimitiveData), Composite(CompositeData) } + +#[derive(Serialize, Deserialize)] +pub struct PrimitiveData { + #[serde(rename = "type")] pub type_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] pub name: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] pub bound: Vec, +} + +pub fn blueprint_to_json(c: &Composite) -> Result; +pub fn blueprint_from_json(data: &str, resolve: &dyn Fn(&str) -> Option) + -> Result; +``` + +`RoleData` mirrors `Role { name, targets: Vec, source: Option }` +(`source` skipped when `None`). The plain data types `Edge`, `Target`, `OutField` +(`harness.rs`, `blueprint.rs:30`) and `BoundParam`, `ScalarKind`, `Firing` +(`node.rs`) gain `#[derive(Serialize, Deserialize)]` — they hold no closures and +already derive `Clone/Debug/PartialEq`. `Scalar` already derives serde +(`scalar.rs:40`). **No `HashMap` enters any DTO** — every field is a `Vec` or +scalar, so struct-declaration field order is the only ordering and JSON output is +deterministic (canonical). + +### Reconstruction (the loader's crux), shape + +```rust +fn reconstruct(d: &CompositeData, resolve: &dyn Fn(&str) -> Option) + -> Result +{ + let mut nodes = Vec::new(); + for nd in &d.nodes { + nodes.push(match nd { + NodeData::Primitive(p) => { + let mut b = resolve(&p.type_id) + .ok_or_else(|| LoadError::UnknownNodeType(p.type_id.clone()))?; + if let Some(n) = &p.name { b = b.named(n); } + // bound params re-applied by NAME; order-independent in the + // effective param vector (each bind computes pos against the + // shrunk schema) — ascending original pos is the canonical order. + let mut bound: Vec<&BoundParam> = p.bound.iter().collect(); + bound.sort_by_key(|b| b.pos); + for bp in bound { b = b.bind(&bp.name, bp.value); } + BlueprintNode::Primitive(b) + } + NodeData::Composite(c) => BlueprintNode::Composite(reconstruct(c, resolve)?), + }); + } + Ok(Composite::new(d.name.clone(), nodes, d.edges.clone(), + d.input_roles.iter().map(Into::into).collect(), + d.output.clone())) +} +``` + +### Resolver (new — `aura-std/src/vocabulary.rs`), shape + +```rust +/// Closed, compiled-in dispatch over aura-std's OWN node vocabulary. Not a +/// dynamic registry (invariant 9): a `match` the crate that owns the nodes +/// writes, referenced by compiled-in type identity (C24). +pub fn std_vocabulary(type_id: &str) -> Option { + Some(match type_id { + "SMA" => Sma::builder(), + "Sub" => Sub::builder(), + "Bias" => Bias::builder(), + "VolStop" => VolStop::builder(), + // … the remaining aura-std builder() factories … + _ => return None, + }) +} +``` + +The `type_id` is the `PrimitiveBuilder.name` (`&'static str`, e.g. `"SMA"`, +`node.rs:108`) — the type label, *not* the lowercased instance default. + +## Components + +| Component | Crate / file | Responsibility | +|---|---|---| +| DTO mirrors + envelope | `aura-engine/src/blueprint_serde.rs` (new) | serde-derivable projection of `Composite`; `format_version` | +| `blueprint_to_json` | same | `&Composite` → canonical JSON | +| `blueprint_from_json` + `reconstruct` | same | JSON + resolver → `Composite` | +| `SerializeError` / `LoadError` | same | typed, named failures | +| serde derives on plain types | `aura-core/src/node.rs`, `aura-engine/src/harness.rs`, `blueprint.rs` | `Edge`, `Target`, `OutField`, `BoundParam`, `ScalarKind`, `Firing`, `RoleData` source | +| `std_vocabulary` | `aura-std/src/vocabulary.rs` (new) | closed `type_id → builder()` dispatch | +| round-trip test | `aura-engine` tests (test-only `aura-std` dep, mirroring `blueprint.rs:1957`) | bit-identity + canonical-idempotence | + +## Data flow + +**Serialize:** `&Composite` → walk nodes/edges/roles/output → `BlueprintDoc` +(drop build closures, keep `name`/`type_id`/`bound`) → `serde_json::to_string` +(compact, defaults omitted) → canonical bytes. + +**Load:** bytes → `serde_json::from_str::` → check `format_version +== 1` (else `LoadError::UnsupportedVersion`) → `reconstruct`: per primitive, +`resolve(type_id)` → `PrimitiveBuilder`, re-apply `named` + `bind`; per nested +composite, recurse → `Composite::new(...)` → caller runs +`compile_with_params(point)` → `FlatGraph` → run. + +## Error handling + +`LoadError` (typed, each names its cause — never a panic, never a silent wrong +graph): + +- `Json(serde_json::Error)` — malformed/structurally-invalid JSON. +- `UnsupportedVersion { found, supported }` — `format_version` mismatch. This is + the *minimal* version-awareness #155 needs; the full must-understand horizon + (multiple versions, unknown-section policy) is #156. +- `UnknownNodeType(String)` — `resolve` returned `None` for a `type_id`. (The + *richer* Tier-2 refusal semantics are #156; #155 fails cleanly and named.) + +`bind`'s existing preconditions (unknown/ambiguous param name, kind mismatch) +panic by design at authoring (`node.rs:205`); a serialized blueprint that names a +param the resolved builder does not declare is a corrupt/incompatible artifact — +#156 owns turning that into a clean Tier-2 refusal; #155 inherits the existing +`bind` panic as the honest first-cut failure and does not widen scope to catch +it. + +## Testing strategy + +1. **Bit-identical run (the headline acceptance, C1).** Mirror + `composite_sma_cross_runs_bit_identical_to_hand_wired` (`blueprint.rs:1957`): + author a **sink-free** SMA-cross *signal* composite; `blueprint_to_json` it; + `blueprint_from_json` with `std_vocabulary`; attach **identical** recording + sinks post-load to both the Rust twin and the loaded blueprint; bootstrap both + with the **same** param point; assert the recorded traces are byte-for-byte + equal and non-empty. +2. **Canonical round-trip idempotence.** `blueprint_to_json(loaded) == + blueprint_to_json(original)` — serialize→load→serialize is byte-stable. +3. **Omit-defaults canonicality.** A node with no instance name and no bound + params serializes with neither `name` nor `bound` key present (golden + substring), proving an absent optional is byte-identical to the pre-extension + form (#156's additive precondition). +4. **Named failures.** An unknown `type_id` → `LoadError::UnknownNodeType`; a + `format_version` of `2` → `LoadError::UnsupportedVersion` — both asserted by + variant, neither panicking. +5. **A richer composite round-trips** (e.g. a nested `risk_executor`-style + composite from `aura-composites`, sinks excluded) to exercise recursion + + bound params + roles, proving the loop is not SMA-specific. + +## Acceptance criteria + +- [ ] A serialized blueprint loads and runs to a **bit-identical** recorded trace + (C1) against its Rust-built twin — same wiring, same sink output (test 1). +- [ ] Serializer and loader are **inverse** on the in-repo example blueprints — + canonical round-trip identity (tests 2, 5). +- [ ] The format carries a top-level **`format_version`** and a **canonical + omit-defaults** encoding (test 3). +- [ ] Unknown node type and unsupported version fail **cleanly and named**, never + panicking, never silently building a different graph (test 4). +- [ ] The engine stays **domain-free** (`aura-engine → aura-core` only); the + vocabulary table lives in `aura-std`; the resolver is injected (no + engine-side node table — invariant 9, C24). +- [ ] `cargo test --workspace` and `cargo clippy --workspace --all-targets -- -D + warnings` are green.