Aura/docs/specs/0087-blueprint-serialization-loader.md

# Blueprint serialization & loader (close the graph-as-data loop) — Design Spec

**Date:** 2026-06-29
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude
**Cycle:** 0087
**Issue:** #155 (milestone "Topology-as-data: blueprint serialization & loader (C24)")

## Goal

Close the graph-as-data loop mandated by **C24** (ledger, committed `f80571a`):
make a **blueprint** — the param-generic, named graph-as-data a Rust builder
produces (C9/C19) — a **first-class serializable data value** with a stable,
versioned, canonical format, *and* give the engine the missing **load path**
(`data → blueprint → FlatGraph`). Today only the forward render half exists
(`model_to_json`, `crates/aura-engine/src/graph_model.rs`); there is no faithful
serializer and no loader.

Scope is **format + serializer + loader only** (#155). Explicitly out of scope,
each its own milestone issue:

- the forward-compat *must-understand* refusal gate (Tier-2 unknown node type /
  too-new version) — #156;
- the introspectable construction service that *emits* blueprints — #157;
- content-addressed topology identity in the run manifest — #158.

### Acceptance criterion (applied prospectively)

aura's feature-acceptance default: the feature solves a real design-mandated
problem and contradicts no stated design commitment. This cycle:

- **Solves a mandated problem.** C24/C21/C18 cannot orchestrate, structurally
  search, or *reproduce* topology-varying families while topology stays opaque
  Rust source. The load path is the substrate they stand on.
- **Reintroduces no forbidden failure class.** The format carries **no node
  logic** (C17 — computation stays compiled Rust; no RustAst regression), is a
  **static DAG** (non-Turing by construction), and references nodes by
  **compiled-in type identity over a closed vocabulary** — *not* a by-name
  marketplace/registry (C24 forbid; domain invariant 9).
- **Empirical evidence** (this is an infrastructure cycle; the evidence is the
  delivered code + a must-pass test, per the no-surface-still-show-code rule):
  the round-trip test below, whose correct behaviour is a **bit-identical run**
  (C1) of a serialized-then-loaded blueprint against its Rust-built twin, plus
  the canonical JSON a real example blueprint serializes to (shown under
  *Concrete code shapes*).

## Architecture

Four pieces, placed to honour the engine-stays-domain-free boundary
(`aura-engine → aura-core` only; `aura-std` is a *test-only* dep of the engine —
see `crates/aura-engine/Cargo.toml`):

1. **A serde DTO layer** (new module `crates/aura-engine/src/blueprint_serde.rs`).
   `Composite` and `PrimitiveBuilder` cannot derive serde — they transitively
   hold `build: Box<dyn Fn(&[Cell]) -> Box<dyn Node>>` (`node.rs:115`). So the
   format is a **separate, faithful projection** to serde-derivable DTO mirrors,
   *not* an extension of the lossy render model `model_to_json` (which carries no
   constructor link and no param values — confirmed lossy). This mirrors the
   established read-widening idiom in `crates/aura-registry/src/compat.rs`.

2. **The serializer** `blueprint_to_json(&Composite) -> Result<String, _>`:
   projects a `&Composite` to the DTO (dropping the build closures, which the
   loader re-derives) and emits **canonical** JSON — compact, struct-declaration
   field order, defaults omitted via `skip_serializing_if`. Canonicality is the
   precondition for stable content-addressing (#158) and for aura's existing
   byte-identical-golden discipline.

3. **The loader** `blueprint_from_json(data, resolve) -> Result<Composite, _>`:
   reads the `format_version` envelope first, deserializes the DTO, and
   reconstructs the `Composite`. The build closures are recovered **by type
   identity** through an **injected node resolver** (next point); bound params
   and instance names are re-applied from the DTO. The terminal step downstream
   is the unchanged `Composite::compile_with_params` (`blueprint.rs:261`).

4. **The node resolver seam.** The loader is **generic over an injected
   resolver** `Fn(&str) -> Option<PrimitiveBuilder>`, defined engine-side
   (`aura-core`/`aura-engine`, so the engine stays domain-free). The **concrete
   closed dispatch over the `aura-std` vocabulary** lives in the crate that owns
   those nodes — a new `std_vocabulary(type_id)` in `aura-std`, a plain
   compiled-in `match` over its ~29 `builder()` factories. This is **not** the
   forbidden registry (no dynamic registration, no marketplace, no engine-side
   node table): it is exactly C24's "compiled-in closed-set referenced by type
   identity". The injected seam is also where a project `cdylib` later supplies
   its own vocabulary (C16/#157) — but loading a cdylib is out of #155's scope;
   this cut wires the seam and one `aura-std` implementation.

### The load-bearing fork and its resolution (recorded on #155)

**Fork:** the loader fundamentally needs a `type-id → constructor` resolution,
and domain invariant 9 forbids "a node registry inside aura"; the recon flagged
this as a possible ledger-level decision.

**Decision (derived, not a user preference):** the *closed, compiled-in,
injected* resolver is **not** the forbidden registry — it realises the mechanism
C24 *already* mandates ("compiled-in type identity — the closed, typed node
vocabulary of `aura-std` + the project `cdylib`"; forbids only "a by-name node
marketplace/registry … the format is not a distribution mechanism"). Invariant
9's "no node registry" governs *multi-project management / distribution*, not a
crate's compile-time `match` over its own known nodes. The resolution is further
**forced** by the crate graph: `aura-engine → aura-core` only, so the engine
*cannot* hold an `aura-std` node table — the resolver *must* be injected and the
vocabulary table *must* live outside the engine. This is a realisation of C24,
not a contract change, so it is a `specify` decision, not a `brainstorm` bounce.
(Basis: C24 body `docs/design/INDEX.md:1751-1766`, `f80571a`; invariant 9 in
`crates/aura/CLAUDE.md`; `crates/aura-engine/Cargo.toml` engine-domain-free
comment.)

### What is and is not in the serialized blueprint

- **In:** topology (nodes, edges, input roles, exposed outputs), each primitive
  node's **type identity**, its instance name, and its **bound params** (the
  knobs the author fixed). Nested composites recurse structurally.
- **Out (param point):** the *free* param-space is **not** baked to a single
  point — a blueprint is param-generic (C19). The free point is supplied at
  `compile_with_params` exactly as today; the run manifest records the point
  separately (C18, `report.rs`). The round-trip test binds the **same** point to
  both twins.
- **Out (sinks / observation):** a recording sink (`Recorder::builder(.., tx)`,
  `test_fixtures.rs:67`) captures an `mpsc::Sender` in its build closure —
  runtime identity, neither a param nor topology, so it structurally cannot live
  in a param-generic, value-empty blueprint (C19). Sink-destination addressing
  is the C18-registry / trace-persistence concern (#101), out of #155. The
  round-trip test attaches **identical sinks post-load** to both twins and
  compares the recorded traces. (How a serialized harness later re-acquires its
  sinks — observation is part of the harness in C12 — is #101/#158's question,
  not #155's; this cut serialises the closure-free topology and stops there.)

## Concrete code shapes

### Delivered API + the canonical artifact (the north-star slice)

What a World / CLI eventually does — serialize an authored signal graph, store
the bytes, reload and run them (the capability this cycle delivers the substrate
for):

```rust
let bp: Composite = sma_cross_signal();           // authored via the Rust builder (C17)
let json = blueprint_to_json(&bp)?;               // canonical, versioned bytes
std::fs::write("sma_cross.bp.json", &json)?;      // topology is now a stored value (C24)

// later / elsewhere — no Rust rebuild, just data:
let data = std::fs::read_to_string("sma_cross.bp.json")?;
let bp2 = blueprint_from_json(&data, &|t| aura_std::std_vocabulary(t))?;
let flat = bp2.compile_with_params(&[Scalar::i64(2), Scalar::i64(4)])?;   // runs (C1)
```

The canonical bytes for a minimal SMA-cross **signal** graph (`sma_fast`,
`sma_slow`, a `Sub`, a `Bias`; fast length bound to 2, slow open):

```json
{"format_version":1,"blueprint":{"name":"sma_cross","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow"}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias"}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}]}],"output":[{"node":3,"field":0,"name":"bias"}]}}
```

Note: `slow` carries no `name`-less-than-needed clutter — `bound` and `name` are
**omitted** when empty/None (`skip_serializing_if`), so an absent optional is
byte-identical to the pre-extension form (the additive-extension precondition
#156 builds on).

### DTO layer (new — `blueprint_serde.rs`), before → after

There is no "before"; this is new. Shape:

```rust
pub const BLUEPRINT_FORMAT_VERSION: u32 = 1;

#[derive(Serialize, Deserialize)]
pub struct BlueprintDoc { pub format_version: u32, pub blueprint: CompositeData }

#[derive(Serialize, Deserialize)]
pub struct CompositeData {
    pub name: String,
    pub nodes: Vec<NodeData>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub edges: Vec<Edge>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub input_roles: Vec<RoleData>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub output: Vec<OutField>,
}

#[derive(Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NodeData { Primitive(PrimitiveData), Composite(CompositeData) }

#[derive(Serialize, Deserialize)]
pub struct PrimitiveData {
    #[serde(rename = "type")] pub type_id: String,
    #[serde(default, skip_serializing_if = "Option::is_none")] pub name: Option<String>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub bound: Vec<BoundParam>,
}

pub fn blueprint_to_json(c: &Composite) -> Result<String, SerializeError>;
pub fn blueprint_from_json(data: &str, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
    -> Result<Composite, LoadError>;
```

`RoleData` mirrors `Role { name, targets: Vec<Target>, source: Option<ScalarKind> }`
(`source` skipped when `None`). The plain data types `Edge`, `Target`, `OutField`
(`harness.rs`, `blueprint.rs:30`) and `BoundParam`, `ScalarKind`, `Firing`
(`node.rs`) gain `#[derive(Serialize, Deserialize)]` — they hold no closures and
already derive `Clone/Debug/PartialEq`. `Scalar` already derives serde
(`scalar.rs:40`). **No `HashMap` enters any DTO** — every field is a `Vec` or
scalar, so struct-declaration field order is the only ordering and JSON output is
deterministic (canonical).

### Reconstruction (the loader's crux), shape

```rust
fn reconstruct(d: &CompositeData, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
    -> Result<Composite, LoadError>
{
    let mut nodes = Vec::new();
    for nd in &d.nodes {
        nodes.push(match nd {
            NodeData::Primitive(p) => {
                let mut b = resolve(&p.type_id)
                    .ok_or_else(|| LoadError::UnknownNodeType(p.type_id.clone()))?;
                if let Some(n) = &p.name { b = b.named(n); }
                // bound params re-applied by NAME; order-independent in the
                // effective param vector (each bind computes pos against the
                // shrunk schema) — ascending original pos is the canonical order.
                let mut bound: Vec<&BoundParam> = p.bound.iter().collect();
                bound.sort_by_key(|b| b.pos);
                for bp in bound { b = b.bind(&bp.name, bp.value); }
                BlueprintNode::Primitive(b)
            }
            NodeData::Composite(c) => BlueprintNode::Composite(reconstruct(c, resolve)?),
        });
    }
    Ok(Composite::new(d.name.clone(), nodes, d.edges.clone(),
                      d.input_roles.iter().map(Into::into).collect(),
                      d.output.clone()))
}
```

### Resolver (new — `aura-std/src/vocabulary.rs`), shape

```rust
/// Closed, compiled-in dispatch over aura-std's OWN node vocabulary. Not a
/// dynamic registry (invariant 9): a `match` the crate that owns the nodes
/// writes, referenced by compiled-in type identity (C24).
pub fn std_vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
    Some(match type_id {
        "SMA"       => Sma::builder(),
        "Sub"       => Sub::builder(),
        "Bias"      => Bias::builder(),
        "VolStop"   => VolStop::builder(),
        // … the remaining aura-std builder() factories …
        _ => return None,
    })
}
```

The `type_id` is the `PrimitiveBuilder.name` (`&'static str`, e.g. `"SMA"`,
`node.rs:108`) — the type label, *not* the lowercased instance default.

## Components

| Component | Crate / file | Responsibility |
|---|---|---|
| DTO mirrors + envelope | `aura-engine/src/blueprint_serde.rs` (new) | serde-derivable projection of `Composite`; `format_version` |
| `blueprint_to_json` | same | `&Composite` → canonical JSON |
| `blueprint_from_json` + `reconstruct` | same | JSON + resolver → `Composite` |
| `SerializeError` / `LoadError` | same | typed, named failures |
| serde derives on plain types | `aura-core/src/node.rs`, `aura-engine/src/harness.rs`, `blueprint.rs` | `Edge`, `Target`, `OutField`, `BoundParam`, `ScalarKind`, `Firing`, `RoleData` source |
| `std_vocabulary` | `aura-std/src/vocabulary.rs` (new) | closed `type_id → builder()` dispatch |
| round-trip test | `aura-engine` tests (test-only `aura-std` dep, mirroring `blueprint.rs:1957`) | bit-identity + canonical-idempotence |

## Data flow

**Serialize:** `&Composite` → walk nodes/edges/roles/output → `BlueprintDoc`
(drop build closures, keep `name`/`type_id`/`bound`) → `serde_json::to_string`
(compact, defaults omitted) → canonical bytes.

**Load:** bytes → `serde_json::from_str::<BlueprintDoc>` → check `format_version
== 1` (else `LoadError::UnsupportedVersion`) → `reconstruct`: per primitive,
`resolve(type_id)` → `PrimitiveBuilder`, re-apply `named` + `bind`; per nested
composite, recurse → `Composite::new(...)` → caller runs
`compile_with_params(point)` → `FlatGraph` → run.

## Error handling

`LoadError` (typed, each names its cause — never a panic, never a silent wrong
graph):

- `Json(serde_json::Error)` — malformed/structurally-invalid JSON.
- `UnsupportedVersion { found, supported }` — `format_version` mismatch. This is
  the *minimal* version-awareness #155 needs; the full must-understand horizon
  (multiple versions, unknown-section policy) is #156.
- `UnknownNodeType(String)` — `resolve` returned `None` for a `type_id`. (The
  *richer* Tier-2 refusal semantics are #156; #155 fails cleanly and named.)

`bind`'s existing preconditions (unknown/ambiguous param name, kind mismatch)
panic by design at authoring (`node.rs:205`); a serialized blueprint that names a
param the resolved builder does not declare is a corrupt/incompatible artifact —
#156 owns turning that into a clean Tier-2 refusal; #155 inherits the existing
`bind` panic as the honest first-cut failure and does not widen scope to catch
it.

## Testing strategy

1. **Bit-identical run (the headline acceptance, C1).** Mirror
   `composite_sma_cross_runs_bit_identical_to_hand_wired` (`blueprint.rs:1957`):
   author a **sink-free** SMA-cross *signal* composite; `blueprint_to_json` it;
   `blueprint_from_json` with `std_vocabulary`; attach **identical** recording
   sinks post-load to both the Rust twin and the loaded blueprint; bootstrap both
   with the **same** param point; assert the recorded traces are byte-for-byte
   equal and non-empty.
2. **Canonical round-trip idempotence.** `blueprint_to_json(loaded) ==
   blueprint_to_json(original)` — serialize→load→serialize is byte-stable.
3. **Omit-defaults canonicality.** A node with no instance name and no bound
   params serializes with neither `name` nor `bound` key present (golden
   substring), proving an absent optional is byte-identical to the pre-extension
   form (#156's additive precondition).
4. **Named failures.** An unknown `type_id` → `LoadError::UnknownNodeType`; a
   `format_version` of `2` → `LoadError::UnsupportedVersion` — both asserted by
   variant, neither panicking.
5. **A richer composite round-trips** (e.g. a nested `risk_executor`-style
   composite from `aura-composites`, sinks excluded) to exercise recursion +
   bound params + roles, proving the loop is not SMA-specific.

## Acceptance criteria

- [ ] A serialized blueprint loads and runs to a **bit-identical** recorded trace
      (C1) against its Rust-built twin — same wiring, same sink output (test 1).
- [ ] Serializer and loader are **inverse** on the in-repo example blueprints —
      canonical round-trip identity (tests 2, 5).
- [ ] The format carries a top-level **`format_version`** and a **canonical
      omit-defaults** encoding (test 3).
- [ ] Unknown node type and unsupported version fail **cleanly and named**, never
      panicking, never silently building a different graph (test 4).
- [ ] The engine stays **domain-free** (`aura-engine → aura-core` only); the
      vocabulary table lives in `aura-std`; the resolver is injected (no
      engine-side node table — invariant 9, C24).
- [ ] `cargo test --workspace` and `cargo clippy --workspace --all-targets -- -D
      warnings` are green.