Aura/docs/specs/0087-blueprint-serialization-loader.md

Blueprint serialization & loader (close the graph-as-data loop) — Design Spec

Date: 2026-06-29 Status: Draft — awaiting user spec review Authors: orchestrator + Claude Cycle: 0087 Issue: #155 (milestone "Topology-as-data: blueprint serialization & loader (C24)")

Goal

Close the graph-as-data loop mandated by C24 (ledger, committed f80571a): make a blueprint — the param-generic, named graph-as-data a Rust builder produces (C9/C19) — a first-class serializable data value with a stable, versioned, canonical format, and give the engine the missing load path (data → blueprint → FlatGraph). Today only the forward render half exists (model_to_json, crates/aura-engine/src/graph_model.rs); there is no faithful serializer and no loader.

Scope is format + serializer + loader only (#155). Explicitly out of scope, each its own milestone issue:

• the forward-compat must-understand refusal gate (Tier-2 unknown node type / too-new version) — #156;
• the introspectable construction service that emits blueprints — #157;
• content-addressed topology identity in the run manifest — #158.

Acceptance criterion (applied prospectively)

aura's feature-acceptance default: the feature solves a real design-mandated problem and contradicts no stated design commitment. This cycle:

• Solves a mandated problem. C24/C21/C18 cannot orchestrate, structurally search, or reproduce topology-varying families while topology stays opaque Rust source. The load path is the substrate they stand on.
• Reintroduces no forbidden failure class. The format carries no node logic (C17 — computation stays compiled Rust; no RustAst regression), is a static DAG (non-Turing by construction), and references nodes by compiled-in type identity over a closed vocabulary — not a by-name marketplace/registry (C24 forbid; domain invariant 9).
• Empirical evidence (this is an infrastructure cycle; the evidence is the delivered code + a must-pass test, per the no-surface-still-show-code rule): the round-trip test below, whose correct behaviour is a bit-identical run (C1) of a serialized-then-loaded blueprint against its Rust-built twin, plus the canonical JSON a real example blueprint serializes to (shown under Concrete code shapes).

Architecture

Four pieces, placed to honour the engine-stays-domain-free boundary (aura-engine → aura-core only; aura-std is a test-only dep of the engine — see crates/aura-engine/Cargo.toml):

1. A serde DTO layer (new module crates/aura-engine/src/blueprint_serde.rs). Composite and PrimitiveBuilder cannot derive serde — they transitively hold build: Box<dyn Fn(&[Cell]) -> Box<dyn Node>> (node.rs:115). So the format is a separate, faithful projection to serde-derivable DTO mirrors, not an extension of the lossy render model model_to_json (which carries no constructor link and no param values — confirmed lossy). This mirrors the established read-widening idiom in crates/aura-registry/src/compat.rs.

2. The serializer blueprint_to_json(&Composite) -> Result<String, _>: projects a &Composite to the DTO (dropping the build closures, which the loader re-derives) and emits canonical JSON — compact, struct-declaration field order, defaults omitted via skip_serializing_if. Canonicality is the precondition for stable content-addressing (#158) and for aura's existing byte-identical-golden discipline.

3. The loader blueprint_from_json(data, resolve) -> Result<Composite, _>: reads the format_version envelope first, deserializes the DTO, and reconstructs the Composite. The build closures are recovered by type identity through an injected node resolver (next point); bound params and instance names are re-applied from the DTO. The terminal step downstream is the unchanged Composite::compile_with_params (blueprint.rs:261).

4. The node resolver seam. The loader is generic over an injected resolver Fn(&str) -> Option<PrimitiveBuilder>, defined engine-side (aura-core/aura-engine, so the engine stays domain-free). The concrete closed dispatch over the aura-std vocabulary lives in the crate that owns those nodes — a new std_vocabulary(type_id) in aura-std, a plain compiled-in match over its ~29 builder() factories. This is not the forbidden registry (no dynamic registration, no marketplace, no engine-side node table): it is exactly C24's "compiled-in closed-set referenced by type identity". The injected seam is also where a project cdylib later supplies its own vocabulary (C16/#157) — but loading a cdylib is out of #155's scope; this cut wires the seam and one aura-std implementation.

The load-bearing fork and its resolution (recorded on #155)

Fork: the loader fundamentally needs a type-id → constructor resolution, and domain invariant 9 forbids "a node registry inside aura"; the recon flagged this as a possible ledger-level decision.

Decision (derived, not a user preference): the closed, compiled-in, injected resolver is not the forbidden registry — it realises the mechanism C24 already mandates ("compiled-in type identity — the closed, typed node vocabulary of aura-std + the project cdylib"; forbids only "a by-name node marketplace/registry … the format is not a distribution mechanism"). Invariant 9's "no node registry" governs multi-project management / distribution, not a crate's compile-time match over its own known nodes. The resolution is further forced by the crate graph: aura-engine → aura-core only, so the engine cannot hold an aura-std node table — the resolver must be injected and the vocabulary table must live outside the engine. This is a realisation of C24, not a contract change, so it is a specify decision, not a brainstorm bounce. (Basis: C24 body docs/design/INDEX.md:1751-1766, f80571a; invariant 9 in crates/aura/CLAUDE.md; crates/aura-engine/Cargo.toml engine-domain-free comment.)

What is and is not in the serialized blueprint

• In: topology (nodes, edges, input roles, exposed outputs), each primitive node's type identity, its instance name, and its bound params (the knobs the author fixed). Nested composites recurse structurally.
• Out (param point): the free param-space is not baked to a single point — a blueprint is param-generic (C19). The free point is supplied at compile_with_params exactly as today; the run manifest records the point separately (C18, report.rs). The round-trip test binds the same point to both twins.
• Out (sinks / observation): a recording sink (Recorder::builder(.., tx), test_fixtures.rs:67) captures an mpsc::Sender in its build closure — runtime identity, neither a param nor topology, so it structurally cannot live in a param-generic, value-empty blueprint (C19). Sink-destination addressing is the C18-registry / trace-persistence concern (#101), out of #155. The round-trip test attaches identical sinks post-load to both twins and compares the recorded traces. (How a serialized harness later re-acquires its sinks — observation is part of the harness in C12 — is #101/#158's question, not #155's; this cut serialises the closure-free topology and stops there.)

Concrete code shapes

Delivered API + the canonical artifact (the north-star slice)

What a World / CLI eventually does — serialize an authored signal graph, store the bytes, reload and run them (the capability this cycle delivers the substrate for):

let bp: Composite = sma_cross_signal();           // authored via the Rust builder (C17)
let json = blueprint_to_json(&bp)?;               // canonical, versioned bytes
std::fs::write("sma_cross.bp.json", &json)?;      // topology is now a stored value (C24)

// later / elsewhere — no Rust rebuild, just data:
let data = std::fs::read_to_string("sma_cross.bp.json")?;
let bp2 = blueprint_from_json(&data, &|t| aura_std::std_vocabulary(t))?;
let flat = bp2.compile_with_params(&[Scalar::i64(2), Scalar::i64(4)])?;   // runs (C1)

The canonical bytes for a minimal SMA-cross signal graph (sma_fast, sma_slow, a Sub, a Bias; fast length bound to 2, slow open):

{"format_version":1,"blueprint":{"name":"sma_cross","nodes":[{"primitive":{"type":"SMA","name":"fast","bound":[{"pos":0,"name":"length","kind":"I64","value":{"I64":2}}]}},{"primitive":{"type":"SMA","name":"slow"}},{"primitive":{"type":"Sub"}},{"primitive":{"type":"Bias"}}],"edges":[{"from":0,"to":2,"slot":0,"from_field":0},{"from":1,"to":2,"slot":1,"from_field":0},{"from":2,"to":3,"slot":0,"from_field":0}],"input_roles":[{"name":"price","targets":[{"node":0,"slot":0},{"node":1,"slot":0}]}],"output":[{"node":3,"field":0,"name":"bias"}]}}

Note: slow carries no name-less-than-needed clutter — bound and name are omitted when empty/None (skip_serializing_if), so an absent optional is byte-identical to the pre-extension form (the additive-extension precondition #156 builds on).

DTO layer (new — blueprint_serde.rs), before → after

There is no "before"; this is new. Shape:

pub const BLUEPRINT_FORMAT_VERSION: u32 = 1;

#[derive(Serialize, Deserialize)]
pub struct BlueprintDoc { pub format_version: u32, pub blueprint: CompositeData }

#[derive(Serialize, Deserialize)]
pub struct CompositeData {
    pub name: String,
    pub nodes: Vec<NodeData>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub edges: Vec<Edge>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub input_roles: Vec<RoleData>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub output: Vec<OutField>,
}

#[derive(Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NodeData { Primitive(PrimitiveData), Composite(CompositeData) }

#[derive(Serialize, Deserialize)]
pub struct PrimitiveData {
    #[serde(rename = "type")] pub type_id: String,
    #[serde(default, skip_serializing_if = "Option::is_none")] pub name: Option<String>,
    #[serde(default, skip_serializing_if = "Vec::is_empty")] pub bound: Vec<BoundParam>,
}

pub fn blueprint_to_json(c: &Composite) -> Result<String, SerializeError>;
pub fn blueprint_from_json(data: &str, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
    -> Result<Composite, LoadError>;

RoleData mirrors Role { name, targets: Vec<Target>, source: Option<ScalarKind> } (source skipped when None). The plain data types Edge, Target, OutField (harness.rs, blueprint.rs:30) and BoundParam, ScalarKind, Firing (node.rs) gain #[derive(Serialize, Deserialize)] — they hold no closures and already derive Clone/Debug/PartialEq. Scalar already derives serde (scalar.rs:40). No HashMap enters any DTO — every field is a Vec or scalar, so struct-declaration field order is the only ordering and JSON output is deterministic (canonical).

Reconstruction (the loader's crux), shape

fn reconstruct(d: &CompositeData, resolve: &dyn Fn(&str) -> Option<PrimitiveBuilder>)
    -> Result<Composite, LoadError>
{
    let mut nodes = Vec::new();
    for nd in &d.nodes {
        nodes.push(match nd {
            NodeData::Primitive(p) => {
                let mut b = resolve(&p.type_id)
                    .ok_or_else(|| LoadError::UnknownNodeType(p.type_id.clone()))?;
                if let Some(n) = &p.name { b = b.named(n); }
                // bound params re-applied by NAME; order-independent in the
                // effective param vector (each bind computes pos against the
                // shrunk schema) — ascending original pos is the canonical order.
                let mut bound: Vec<&BoundParam> = p.bound.iter().collect();
                bound.sort_by_key(|b| b.pos);
                for bp in bound { b = b.bind(&bp.name, bp.value); }
                BlueprintNode::Primitive(b)
            }
            NodeData::Composite(c) => BlueprintNode::Composite(reconstruct(c, resolve)?),
        });
    }
    Ok(Composite::new(d.name.clone(), nodes, d.edges.clone(),
                      d.input_roles.iter().map(Into::into).collect(),
                      d.output.clone()))
}

Resolver (new — aura-std/src/vocabulary.rs), shape

/// Closed, compiled-in dispatch over aura-std's OWN node vocabulary. Not a
/// dynamic registry (invariant 9): a `match` the crate that owns the nodes
/// writes, referenced by compiled-in type identity (C24).
pub fn std_vocabulary(type_id: &str) -> Option<PrimitiveBuilder> {
    Some(match type_id {
        "SMA"       => Sma::builder(),
        "Sub"       => Sub::builder(),
        "Bias"      => Bias::builder(),
        "VolStop"   => VolStop::builder(),
        // … the remaining aura-std builder() factories …
        _ => return None,
    })
}

The type_id is the PrimitiveBuilder.name (&'static str, e.g. "SMA", node.rs:108) — the type label, not the lowercased instance default.

Components

Component	Crate / file	Responsibility
DTO mirrors + envelope	aura-engine/src/blueprint_serde.rs (new)	serde-derivable projection of Composite; format_version
blueprint_to_json	same	&Composite → canonical JSON
blueprint_from_json + reconstruct	same	JSON + resolver → Composite
SerializeError / LoadError	same	typed, named failures
serde derives on plain types	aura-core/src/node.rs, aura-engine/src/harness.rs, blueprint.rs	Edge, Target, OutField, BoundParam, ScalarKind, Firing, RoleData source
std_vocabulary	aura-std/src/vocabulary.rs (new)	closed type_id → builder() dispatch
round-trip test	aura-engine tests (test-only aura-std dep, mirroring blueprint.rs:1957)	bit-identity + canonical-idempotence

Data flow

Serialize: &Composite → walk nodes/edges/roles/output → BlueprintDoc (drop build closures, keep name/type_id/bound) → serde_json::to_string (compact, defaults omitted) → canonical bytes.

Load: bytes → serde_json::from_str::<BlueprintDoc> → check format_version == 1 (else LoadError::UnsupportedVersion) → reconstruct: per primitive, resolve(type_id) → PrimitiveBuilder, re-apply named + bind; per nested composite, recurse → Composite::new(...) → caller runs compile_with_params(point) → FlatGraph → run.

Error handling

LoadError (typed, each names its cause — never a panic, never a silent wrong graph):

• Json(serde_json::Error) — malformed/structurally-invalid JSON.
• UnsupportedVersion { found, supported } — format_version mismatch. This is the minimal version-awareness #155 needs; the full must-understand horizon (multiple versions, unknown-section policy) is #156.
• UnknownNodeType(String) — resolve returned None for a type_id. (The richer Tier-2 refusal semantics are #156; #155 fails cleanly and named.)

bind's existing preconditions (unknown/ambiguous param name, kind mismatch) panic by design at authoring (node.rs:205); a serialized blueprint that names a param the resolved builder does not declare is a corrupt/incompatible artifact — #156 owns turning that into a clean Tier-2 refusal; #155 inherits the existing bind panic as the honest first-cut failure and does not widen scope to catch it.

Testing strategy

1. Bit-identical run (the headline acceptance, C1). Mirror composite_sma_cross_runs_bit_identical_to_hand_wired (blueprint.rs:1957): author a sink-free SMA-cross signal composite; blueprint_to_json it; blueprint_from_json with std_vocabulary; attach identical recording sinks post-load to both the Rust twin and the loaded blueprint; bootstrap both with the same param point; assert the recorded traces are byte-for-byte equal and non-empty.
2. Canonical round-trip idempotence. blueprint_to_json(loaded) == blueprint_to_json(original) — serialize→load→serialize is byte-stable.
3. Omit-defaults canonicality. A node with no instance name and no bound params serializes with neither name nor bound key present (golden substring), proving an absent optional is byte-identical to the pre-extension form (#156's additive precondition).
4. Named failures. An unknown type_id → LoadError::UnknownNodeType; a format_version of 2 → LoadError::UnsupportedVersion — both asserted by variant, neither panicking.
5. A richer composite round-trips (e.g. a nested risk_executor-style composite from aura-composites, sinks excluded) to exercise recursion + bound params + roles, proving the loop is not SMA-specific.

Acceptance criteria

• A serialized blueprint loads and runs to a bit-identical recorded trace (C1) against its Rust-built twin — same wiring, same sink output (test 1).
• Serializer and loader are inverse on the in-repo example blueprints — canonical round-trip identity (tests 2, 5).
• The format carries a top-level format_version and a canonical omit-defaults encoding (test 3).
• Unknown node type and unsupported version fail cleanly and named, never panicking, never silently building a different graph (test 4).
• The engine stays domain-free (aura-engine → aura-core only); the vocabulary table lives in aura-std; the resolver is injected (no engine-side node table — invariant 9, C24).
• cargo test --workspace and cargo clippy --workspace --all-targets -- -D warnings are green.