spec: 0087 blueprint serialization & loader (boss-signed)

The C24 first cut (#155): a stable, versioned, canonical serialization of a param-generic Composite blueprint plus the data -> blueprint -> FlatGraph loader, closing the graph-as-data loop. Format references nodes by compiled-in type identity over a closed vocabulary (no logic, non-Turing); the loader is generic over an injected resolver with the concrete aura-std vocabulary match living outside the engine (engine stays domain-free). Acceptance: a serialized blueprint runs bit-identical (C1) to its Rust-built twin. Signed under /boss on the Step-5 grounding-check PASS (all load-bearing assumptions ratified by currently-green tests). refs #155
2026-06-29 13:25:57 +02:00
parent f80571a46a
commit ce86b4ec30
1 changed files with 333 additions and 0 deletions
@@ -0,0 +1,333 @@
+# Blueprint serialization & loader (close the graph-as-data loop) — Design Spec
+
+**Date:** 2026-06-29
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+**Cycle:** 0087
+**Issue:** #155 (milestone "Topology-as-data: blueprint serialization & loader (C24)")
+
+## Goal
+
+Close the graph-as-data loop mandated by **C24** (ledger, committed `f80571a`):
+make a **blueprint** — the param-generic, named graph-as-data a Rust builder
+produces (C9/C19) — a **first-class serializable data value** with a stable,
+versioned, canonical format, *and* give the engine the missing **load path**
+(`data → blueprint → FlatGraph`). Today only the forward render half exists
+(`model_to_json`, `crates/aura-engine/src/graph_model.rs`); there is no faithful
+serializer and no loader.
+
+Scope is **format + serializer + loader only** (#155). Explicitly out of scope,
+each its own milestone issue:
+
+- the forward-compat *must-understand* refusal gate (Tier-2 unknown node type /
+  too-new version) — #156;
+- the introspectable construction service that *emits* blueprints — #157;
+- content-addressed topology identity in the run manifest — #158.
+
+### Acceptance criterion (applied prospectively)
+
+aura's feature-acceptance default: the feature solves a real design-mandated
+problem and contradicts no stated design commitment. This cycle:
+
+- **Solves a mandated problem.** C24/C21/C18 cannot orchestrate, structurally
+  search, or *reproduce* topology-varying families while topology stays opaque
+  Rust source. The load path is the substrate they stand on.
+- **Reintroduces no forbidden failure class.** The format carries **no node
+  logic** (C17 — computation stays compiled Rust; no RustAst regression), is a
+  **static DAG** (non-Turing by construction), and references nodes by
+  **compiled-in type identity over a closed vocabulary** — *not* a by-name
+  marketplace/registry (C24 forbid; domain invariant 9).
+- **Empirical evidence** (this is an infrastructure cycle; the evidence is the
+  delivered code + a must-pass test, per the no-surface-still-show-code rule):
+  the round-trip test below, whose correct behaviour is a **bit-identical run**
+  (C1) of a serialized-then-loaded blueprint against its Rust-built twin, plus
+  the canonical JSON a real example blueprint serializes to (shown under
+  *Concrete code shapes*).
+
+## Architecture
+
+Four pieces, placed to honour the engine-stays-domain-free boundary
+(`aura-engine → aura-core` only; `aura-std` is a *test-only* dep of the engine —
+see `crates/aura-engine/Cargo.toml`):
+
+1. **A serde DTO layer** (new module `crates/aura-engine/src/blueprint_serde.rs`).
+   `Composite` and `PrimitiveBuilder` cannot derive serde — they transitively
+   hold `build: Box<dyn Fn(&[Cell]) -> Box<dyn Node>>` (`node.rs:115`). So the
+   format is a **separate, faithful projection** to serde-derivable DTO mirrors,
+   *not* an extension of the lossy render model `model_to_json` (which carries no
+   constructor link and no param values — confirmed lossy). This mirrors the
+   established read-widening idiom in `crates/aura-registry/src/compat.rs`.
+
+2. **The serializer** `blueprint_to_json(&Composite) -> Result<String, _>`:
+   projects a `&Composite` to the DTO (dropping the build closures, which the
+   loader re-derives) and emits **canonical** JSON — compact, struct-declaration
+   field order, defaults omitted via `skip_serializing_if`. Canonicality is the
+   precondition for stable content-addressing (#158) and for aura's existing
+   byte-identical-golden discipline.
+
+3. **The loader** `blueprint_from_json(data, resolve) -> Result<Composite, _>`:
+   reads the `format_version` envelope first, deserializes the DTO, and
+   reconstructs the `Composite`. The build closures are recovered **by type
+   identity** through an **injected node resolver** (next point); bound params
+   and instance names are re-applied from the DTO. The terminal step downstream
+   is the unchanged `Composite::compile_with_params` (`blueprint.rs:261`).
+
+4. **The node resolver seam.** The loader is **generic over an injected
+   resolver** `Fn(&str) -> Option<PrimitiveBuilder>`, defined engine-side
+   (`aura-core`/`aura-engine`, so the engine stays domain-free). The **concrete
+   closed dispatch over the `aura-std` vocabulary** lives in the crate that owns
+   those nodes — a new `std_vocabulary(type_id)` in `aura-std`, a plain
+   compiled-in `match` over its ~29 `builder()` factories. This is **not** the
+   forbidden registry (no dynamic registration, no marketplace, no engine-side
+   node table): it is exactly C24's "compiled-in closed-set referenced by type
+   identity". The injected seam is also where a project `cdylib` later supplies
+   its own vocabulary (C16/#157) — but loading a cdylib is out of #155's scope;
+   this cut wires the seam and one `aura-std` implementation.
+
+### The load-bearing fork and its resolution (recorded on #155)
+
+**Fork:** the loader fundamentally needs a `type-id → constructor` resolution,
+and domain invariant 9 forbids "a node registry inside aura"; the recon flagged
+this as a possible ledger-level decision.
+
+**Decision (derived, not a user preference):** the *closed, compiled-in,
+injected* resolver is **not** the forbidden registry — it realises the mechanism
+C24 *already* mandates ("compiled-in type identity — the closed, typed node
+vocabulary of `aura-std` + the project `cdylib`"; forbids only "a by-name node
+marketplace/registry … the format is not a distribution mechanism"). Invariant
+9's "no node registry" governs *multi-project management / distribution*, not a
+crate's compile-time `match` over its own known nodes. The resolution is further
+**forced** by the crate graph: `aura-engine → aura-core` only, so the engine
+*cannot* hold an `aura-std` node table — the resolver *must* be injected and the
+vocabulary table *must* live outside the engine. This is a realisation of C24,
+not a contract change, so it is a `specify` decision, not a `brainstorm` bounce.
+(Basis: C24 body `docs/design/INDEX.md:1751-1766`, `f80571a`; invariant 9 in
+`crates/aura/CLAUDE.md`; `crates/aura-engine/Cargo.toml` engine-domain-free
+comment.)
+
+### What is and is not in the serialized blueprint
+
+- **In:** topology (nodes, edges, input roles, exposed outputs), each primitive
+  node's **type identity**, its instance name, and its **bound params** (the
+  knobs the author fixed). Nested composites recurse structurally.
+- **Out (param point):** the *free* param-space is **not** baked to a single
+  point — a blueprint is param-generic (C19). The free point is supplied at
+  `compile_with_params` exactly as today; the run manifest records the point
+  separately (C18, `report.rs`). The round-trip test binds the **same** point to
+  both twins.
+- **Out (sinks / observation):** a recording sink (`Recorder::builder(.., tx)`,
+  `test_fixtures.rs:67`) captures an `mpsc::Sender` in its build closure —
+  runtime identity, neither a param nor topology, so it structurally cannot live
+  in a param-generic, value-empty blueprint (C19). Sink-destination addressing
+  is the C18-registry / trace-persistence concern (#101), out of #155. The
+  round-trip test attaches **identical sinks post-load** to both twins and
+  compares the recorded traces. (How a serialized harness later re-acquires its
+  sinks — observation is part of the harness in C12 — is #101/#158's question,
+  not #155's; this cut serialises the closure-free topology and stops there.)
+
+## Concrete code shapes
+
+### Delivered API + the canonical artifact (the north-star slice)
+
+What a World / CLI eventually does — serialize an authored signal graph, store
+the bytes, reload and run them (the capability this cycle delivers the substrate
+for):
+
+```rust
+let bp: Composite = sma_cross_signal();           // authored via the Rust builder (C17)
+let json = blueprint_to_json(&bp)?;               // canonical, versioned bytes
+std::fs::write("sma_cross.bp.json", &json)?;      // topology is now a stored value (C24)
+
+// later / elsewhere — no Rust rebuild, just data:
+let data = std::fs::read_to_string("sma_cross.bp.json")?;
+let bp2 = blueprint_from_json(&data, &|t| aura_std::std_vocabulary(t))?;
+let flat = bp2.compile_with_params(&[Scalar::i64(2), Scalar::i64(4)])?;   // runs (C1)
+```
+
+The canonical bytes for a minimal SMA-cross **signal** graph (`sma_fast`,
+`sma_slow`, a `Sub`, a `Bias`; fast length bound to 2, slow open):
+
+```json
+{"format_version":1,"blueprint":{"name":"sma_cross","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow"}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias"}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}]}],"output":[{"node":3,"field":0,"name":"bias"}]}}
+```
+
+Note: `slow` carries no `name`-less-than-needed clutter — `bound` and `name` are
+**omitted** when empty/None (`skip_serializing_if`), so an absent optional is
+byte-identical to the pre-extension form (the additive-extension precondition
+#156 builds on).
+
+### DTO layer (new — `blueprint_serde.rs`), before → after
+
+There is no "before"; this is new. Shape:
+
+```rust
+pub const BLUEPRINT_FORMAT_VERSION: u32 = 1;
+
+#[derive(Serialize, Deserialize)]
+pub struct BlueprintDoc { pub format_version: u32, pub blueprint: CompositeData }
+
+#[derive(Serialize, Deserialize)]
+pub struct CompositeData {
+    pub name: String,
+    pub nodes: Vec<NodeData>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub edges: Vec<Edge>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub input_roles: Vec<RoleData>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub output: Vec<OutField>,
+}
+
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum NodeData { Primitive(PrimitiveData), Composite(CompositeData) }
+
+#[derive(Serialize, Deserialize)]
+pub struct PrimitiveData {
+    #[serde(rename = "type")] pub type_id: String,
+    #[serde(default, skip_serializing_if = "Option::is_none")] pub name: Option<String>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub bound: Vec<BoundParam>,
+}
+
+pub fn blueprint_to_json(c: &Composite) -> Result<String, SerializeError>;
+pub fn blueprint_from_json(data: &str, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
+    -> Result<Composite, LoadError>;
+```
+
+`RoleData` mirrors `Role { name, targets: Vec<Target>, source: Option<ScalarKind> }`
+(`source` skipped when `None`). The plain data types `Edge`, `Target`, `OutField`
+(`harness.rs`, `blueprint.rs:30`) and `BoundParam`, `ScalarKind`, `Firing`
+(`node.rs`) gain `#[derive(Serialize, Deserialize)]` — they hold no closures and
+already derive `Clone/Debug/PartialEq`. `Scalar` already derives serde
+(`scalar.rs:40`). **No `HashMap` enters any DTO** — every field is a `Vec` or
+scalar, so struct-declaration field order is the only ordering and JSON output is
+deterministic (canonical).
+
+### Reconstruction (the loader's crux), shape
+
+```rust
+fn reconstruct(d: &CompositeData, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
+    -> Result<Composite, LoadError>
+{
+    let mut nodes = Vec::new();
+    for nd in &d.nodes {
+        nodes.push(match nd {
+            NodeData::Primitive(p) => {
+                let mut b = resolve(&p.type_id)
+                    .ok_or_else(|| LoadError::UnknownNodeType(p.type_id.clone()))?;
+                if let Some(n) = &p.name { b = b.named(n); }
+                // bound params re-applied by NAME; order-independent in the
+                // effective param vector (each bind computes pos against the
+                // shrunk schema) — ascending original pos is the canonical order.
+                let mut bound: Vec<&BoundParam> = p.bound.iter().collect();
+                bound.sort_by_key(|b| b.pos);
+                for bp in bound { b = b.bind(&bp.name, bp.value); }
+                BlueprintNode::Primitive(b)
+            }
+            NodeData::Composite(c) => BlueprintNode::Composite(reconstruct(c, resolve)?),
+        });
+    }
+    Ok(Composite::new(d.name.clone(), nodes, d.edges.clone(),
+                      d.input_roles.iter().map(Into::into).collect(),
+                      d.output.clone()))
+}
+```
+
+### Resolver (new — `aura-std/src/vocabulary.rs`), shape
+
+```rust
+/// Closed, compiled-in dispatch over aura-std's OWN node vocabulary. Not a
+/// dynamic registry (invariant 9): a `match` the crate that owns the nodes
+/// writes, referenced by compiled-in type identity (C24).
+pub fn std_vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
+    Some(match type_id {
+        "SMA"       => Sma::builder(),
+        "Sub"       => Sub::builder(),
+        "Bias"      => Bias::builder(),
+        "VolStop"   => VolStop::builder(),
+        // … the remaining aura-std builder() factories …
+        _ => return None,
+    })
+}
+```
+
+The `type_id` is the `PrimitiveBuilder.name` (`&'static str`, e.g. `"SMA"`,
+`node.rs:108`) — the type label, *not* the lowercased instance default.
+
+## Components
+
+| Component | Crate / file | Responsibility |
+|---|---|---|
+| DTO mirrors + envelope | `aura-engine/src/blueprint_serde.rs` (new) | serde-derivable projection of `Composite`; `format_version` |
+| `blueprint_to_json` | same | `&Composite` → canonical JSON |
+| `blueprint_from_json` + `reconstruct` | same | JSON + resolver → `Composite` |
+| `SerializeError` / `LoadError` | same | typed, named failures |
+| serde derives on plain types | `aura-core/src/node.rs`, `aura-engine/src/harness.rs`, `blueprint.rs` | `Edge`, `Target`, `OutField`, `BoundParam`, `ScalarKind`, `Firing`, `RoleData` source |
+| `std_vocabulary` | `aura-std/src/vocabulary.rs` (new) | closed `type_id → builder()` dispatch |
+| round-trip test | `aura-engine` tests (test-only `aura-std` dep, mirroring `blueprint.rs:1957`) | bit-identity + canonical-idempotence |
+
+## Data flow
+
+**Serialize:** `&Composite` → walk nodes/edges/roles/output → `BlueprintDoc`
+(drop build closures, keep `name`/`type_id`/`bound`) → `serde_json::to_string`
+(compact, defaults omitted) → canonical bytes.
+
+**Load:** bytes → `serde_json::from_str::<BlueprintDoc>` → check `format_version
+== 1` (else `LoadError::UnsupportedVersion`) → `reconstruct`: per primitive,
+`resolve(type_id)` → `PrimitiveBuilder`, re-apply `named` + `bind`; per nested
+composite, recurse → `Composite::new(...)` → caller runs
+`compile_with_params(point)` → `FlatGraph` → run.
+
+## Error handling
+
+`LoadError` (typed, each names its cause — never a panic, never a silent wrong
+graph):
+
+- `Json(serde_json::Error)` — malformed/structurally-invalid JSON.
+- `UnsupportedVersion { found, supported }` — `format_version` mismatch. This is
+  the *minimal* version-awareness #155 needs; the full must-understand horizon
+  (multiple versions, unknown-section policy) is #156.
+- `UnknownNodeType(String)` — `resolve` returned `None` for a `type_id`. (The
+  *richer* Tier-2 refusal semantics are #156; #155 fails cleanly and named.)
+
+`bind`'s existing preconditions (unknown/ambiguous param name, kind mismatch)
+panic by design at authoring (`node.rs:205`); a serialized blueprint that names a
+param the resolved builder does not declare is a corrupt/incompatible artifact —
+#156 owns turning that into a clean Tier-2 refusal; #155 inherits the existing
+`bind` panic as the honest first-cut failure and does not widen scope to catch
+it.
+
+## Testing strategy
+
+1. **Bit-identical run (the headline acceptance, C1).** Mirror
+   `composite_sma_cross_runs_bit_identical_to_hand_wired` (`blueprint.rs:1957`):
+   author a **sink-free** SMA-cross *signal* composite; `blueprint_to_json` it;
+   `blueprint_from_json` with `std_vocabulary`; attach **identical** recording
+   sinks post-load to both the Rust twin and the loaded blueprint; bootstrap both
+   with the **same** param point; assert the recorded traces are byte-for-byte
+   equal and non-empty.
+2. **Canonical round-trip idempotence.** `blueprint_to_json(loaded) ==
+   blueprint_to_json(original)` — serialize→load→serialize is byte-stable.
+3. **Omit-defaults canonicality.** A node with no instance name and no bound
+   params serializes with neither `name` nor `bound` key present (golden
+   substring), proving an absent optional is byte-identical to the pre-extension
+   form (#156's additive precondition).
+4. **Named failures.** An unknown `type_id` → `LoadError::UnknownNodeType`; a
+   `format_version` of `2` → `LoadError::UnsupportedVersion` — both asserted by
+   variant, neither panicking.
+5. **A richer composite round-trips** (e.g. a nested `risk_executor`-style
+   composite from `aura-composites`, sinks excluded) to exercise recursion +
+   bound params + roles, proving the loop is not SMA-specific.
+
+## Acceptance criteria
+
+- [ ] A serialized blueprint loads and runs to a **bit-identical** recorded trace
+      (C1) against its Rust-built twin — same wiring, same sink output (test 1).
+- [ ] Serializer and loader are **inverse** on the in-repo example blueprints —
+      canonical round-trip identity (tests 2, 5).
+- [ ] The format carries a top-level **`format_version`** and a **canonical
+      omit-defaults** encoding (test 3).
+- [ ] Unknown node type and unsupported version fail **cleanly and named**, never
+      panicking, never silently building a different graph (test 4).
+- [ ] The engine stays **domain-free** (`aura-engine → aura-core` only); the
+      vocabulary table lives in `aura-std`; the resolver is injected (no
+      engine-side node table — invariant 9, C24).
+- [ ] `cargo test --workspace` and `cargo clippy --workspace --all-targets -- -D
+      warnings` are green.