diff --git a/docs/specs/0088-construction-op-script.md b/docs/specs/0088-construction-op-script.md new file mode 100644 index 0000000..70352aa --- /dev/null +++ b/docs/specs/0088-construction-op-script.md @@ -0,0 +1,431 @@ +# Introspectable, fail-fast construction service — Design Spec + +**Date:** 2026-06-29 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude +**Issue:** #157 (depends on #155, landed `d5602ec`) + +## Goal + +Deliver an **introspectable, fail-fast construction service** that lets a +data-level author (Claude Code, or a small LLM) build a runnable blueprint +through validated ops — without hand-writing the raw-index serialized format +(#155) and without authoring a many-invariant artifact blind. The service emits +a blueprint (the #155 value), reusing the engine's *existing* construction gates +rather than introducing a second validator (C24). + +The surface, settled in #157's discussion, is a **declarative, replayable, +by-identifier op-script**: a document of construction ops, replayed op-by-op +from scratch, stopping at the first op that does not resolve and naming it. The +document *is* the artifact (a Strudel-style document-re-eval model — the script +replayed from scratch is deterministic (C1) and a World-owned topology value +(C24)), with no held process state that can drift from the history that produced +it. MCP is out of scope (deferred, consumer-strength-dependent). + +The work has a **surface-agnostic core** (the value of #157) and a thin CLI +shell over it: + +- **Core (engine):** split the construction gates into *eager* per-op checks + and *holistic* end-of-document checks, both calling the same extracted + predicates; a per-op-fallible construction surface that drives them; and a + build-free introspection API over the closed `aura-std` vocabulary and over a + partial script. +- **Shell (CLI):** a JSON op-list encoding and the `aura graph build` / + `aura graph introspect` subcommands. + +This retires, in concert with #159, the hard-wired `aura-cli` harness +(`sample_blueprint`, `crates/aura-cli/src/main.rs:3313`) — topology as Rust +source is exactly what C24 forbids; this issue supplies the data-authoring +surface that replaces it. + +## Feature-acceptance criterion (applied, aura CLAUDE.md) + +aura's deliverable is the **engine**, and its north star (C24) is *topology as +data for every author*. The criterion for this feature: + +1. **A data-level author naturally reaches for it.** The author writes a + document of ops (shown below) instead of Rust builder source or raw-index + JSON — and is never authoring blind, because introspection answers "what + nodes exist, what are this node's ports/kinds, what is still unwired" without + a build. +2. **It measurably improves correctness / removes redundancy.** A constructed + graph runs **identically** to its Rust-built twin (C1); the hard-wired CLI + harness it replaces is removed source. No second validator is introduced — + the per-op and holistic checks are the *same* engine predicates at two + cadences. +3. **It reintroduces no class of failure the core constraint exists to + eliminate.** The failure mode that sank the RustAst DSL was *unaided + authoring of a many-invariant artifact*. The op-script does not reintroduce + it: every op is checked fail-fast against the engine's own gates, and + introspection means the author never authors blind. The document carries no + logic — only constructive ops with literal params — so it is data, not an + applied/Turing-complete DSL (C24, invariant 10). + +The worked author program under **Concrete code shapes** is this criterion's +empirical evidence. + +## Architecture + +Three layers, named §A/§B/§C; §A+§B are the surface-agnostic engine core +(iteration 1), §C the CLI shell (iteration 2). + +### §A — Eager / holistic gate split (the shared predicates) + +Today, *all* structural validation runs at finalize, in two stages: + +- `GraphBuilder::build()` (`crates/aura-engine/src/builder.rs:131`) resolves + port/field **names** to indices, yielding `BuildError` for unknown/ambiguous + ports and bad handles. It does **not** check edge kinds — the module doc + (`builder.rs:50-52`) states a name-resolved edge with mismatched kinds + surfaces only downstream. +- `Composite::compile_with_params` →`compile_with_cells` + (`blueprint.rs:262`, `:207`) runs the three structural gates: + `check_param_namespace_injective` (`:566`), `validate_wiring` (`:628`, which + includes the edge **kind-match**, raised as `Bootstrap(KindMismatch)` at + `:644`), and `check_ports_connected` (`:688`). + +The op-script needs some of these checks to fire **per op** (acceptance 2: +"rejected at the op"), while others are only decidable once the document ends. +The split is forced by *what is decidable when*: + +| Check | Today (finalize) | Op-script cadence | Why | +|---|---|---|---| +| Name resolution (port/field → index) | `build()` | **eager** (per op) | the referenced node is already added | +| Edge kind-match (producer field kind == consumer slot kind) | `validate_wiring` `:638-649` | **eager** (per op) | both schemas known at the `connect` op | +| Slot covered **>1** (`DoubleWiredPort`) | `check_ports_connected` `:708` | **eager** (per op) | a second cover is visible the moment it happens | +| Param bind name/kind | `bind` panics (`node.rs`) | **eager** (per op) | the node's schema is known at `add` | +| Slot covered **0** (`UnconnectedPort`) | `check_ports_connected` `:707` | **holistic** (finalize) | a mid-document graph is *legitimately* incomplete | +| Param-namespace injective (`DuplicateParamPath`) | `check_param_namespace_injective` | **holistic** (finalize) | only decidable once all nodes + names exist | +| Role-kind consistency, root-role boundness, output range | `compile_with_*` | **holistic** (finalize) | whole-graph properties | + +The principle is sharpest at `check_ports_connected`, whose match has exactly +two failure arms: the **>1** arm (`DoubleWiredPort`) is eagerly decidable; the +**0** arm (`UnconnectedPort`) is not. The op-script front-runs the >1 arm and +defers the 0 arm. + +**No second validator.** The eager path does not re-implement these checks: the +edge-kind check is **extracted** from `validate_wiring`'s inline loop into a +shared predicate called by *both* the eager op and the unchanged holistic +`validate_wiring`; the eager coverage tracker shares the "exactly-once" +invariant with the unchanged `check_ports_connected` (which remains the holistic +backstop, still catching both arms). The name-resolution helpers +(`builder.rs:160-198`, `resolve_slot`/`resolve_field`) become shared +(`pub(crate)`) and are called eagerly per op. + +### §B — Per-op-fallible construction surface + introspection (engine) + +A new engine surface (planner names the type; provisionally `GraphSession`) +drives the eager checks. It is **not** an extension of `GraphBuilder`: +`GraphBuilder`'s documented contract is *infallible accumulate, one fallible +`build()`* (`builder.rs:1-9`, `:67-68`); the op-script's stop-at-first-failing-op +is a different, per-op-fallible discipline. The new surface reuses +`GraphBuilder`'s resolution helpers and the §A predicates over shared internals, +rather than overloading one type with two contracts. + +It owns: the in-progress nodes/schemas/edges/roles/outputs (as `GraphBuilder` +does), an **incremental coverage map** `(node, slot) → count` (the eager +double-wire tracker), and a **name→handle map** for the document's `as NAME` +identifiers. Each op is applied fallibly; a terminal `finish` runs the holistic +gates and returns the `Composite` (→ `blueprint_to_json` for the emitted +blueprint; → `compile_with_params` for the runnable `FlatGraph`). + +**Introspection** is read-only and build-free, in two modes: + +- **Static (type-level):** enumerate the vocabulary; for a type, its ports + + kinds (`NodeSchema.inputs`/`.output`, `PortSpec`/`FieldSpec` carry `name` + + `kind`) and its param paths (`PrimitiveBuilder::params`). All declared + pre-build (C8). This requires an **enumerable** companion to the closed + `std_vocabulary` match (see §Components) — a `fn(&str)->Option<…>` resolver + alone cannot be listed. +- **Partial-script (instance-level):** the still-**unwired** slots of an + in-progress session = every added node's declared input slots minus the + covered ones, read straight off the coverage map. + +### §C — JSON op-list encoding + CLI shell + +The document's canonical encoding is a **JSON op-list** (serde-native, mirroring +#155's blueprint JSON), so the core carries no hand-written parser/lexer (the +smallest "no new language" surface). A line-oriented human text front-end is a +**deferred** ergonomic sugar (not this cycle). + +`aura graph build < doc.json` deserializes the op-list, replays it through a +`GraphSession`, and on success emits the blueprint JSON (and/or compiles to a +`FlatGraph`); on the first failing op it prints the op index + cause and exits +non-zero. `aura graph introspect` answers the static + partial-script queries. +Both extend the existing flat argv `match` (`main.rs:3292`); the bare +`["graph"]` HTML-render arm is unaffected for now (its `sample_blueprint` +removal is #159's concern, with which this pairs). + +## Concrete code shapes + +### Worked author program (the acceptance evidence) + +The author builds the `sma_cross` **signal blueprint** — `price → fast/slow SMA +→ Sub → Bias`, exposed as `bias`. Every node is in the zero-arg vocabulary +(`SMA`, `Sub`, `Bias`), so it round-trips today. (A *full* harness adds +`SimBroker`/`Recorder`, whose construction-arg / `mpsc::Sender` builders are +deliberately outside the #155 vocabulary — #156; so the op-script's near-term +reach is signal blueprints, not full harnesses, exactly as #157's sequencing +note states.) + +Canonical document (`sma_cross.json`): + +```json +[ + {"op": "source", "role": "price", "kind": "f64"}, + {"op": "add", "type": "SMA", "as": "fast", "bind": {"length": 2}}, + {"op": "add", "type": "SMA", "as": "slow", "bind": {"length": 4}}, + {"op": "add", "type": "Sub"}, + {"op": "add", "type": "Bias"}, + {"op": "feed", "role": "price", "into": ["fast.series", "slow.series"]}, + {"op": "connect", "from": "fast.value", "to": "sub.lhs"}, + {"op": "connect", "from": "slow.value", "to": "sub.rhs"}, + {"op": "connect", "from": "sub.value", "to": "bias.signal"}, + {"op": "expose", "from": "bias.bias", "as": "bias"} +] +``` + +```console +$ aura graph build < sma_cross.json +# stdout: the #155 blueprint JSON (format_version + blueprint{nodes,edges,…}) +``` + +Deferred text-sugar equivalent (illustrative only, **not** in this cycle): + +``` +source price: f64 +add SMA(length=2) as fast +add SMA(length=4) as slow +add Sub +add Bias +feed price -> fast.series, slow.series +connect fast.value -> sub.lhs +connect slow.value -> sub.rhs +connect sub.value -> bias.signal +expose bias.bias as bias +``` + +Fail-fast rejection (acceptance 2) — each names the offending op + cause, and +nothing after it runs: + +```console +$ echo '[{"op":"add","type":"Nope"}]' | aura graph build +error: op 0 (add): unknown node type "Nope" # not in std_vocabulary + +# kind mismatch — a bool field into an f64 slot, rejected at the connect op +$ aura graph build < kindbad.json +error: op 4 (connect gt.value -> bias.signal): kind mismatch (producer bool, consumer f64) + +# double-wire — a second producer into an already-wired slot +$ aura graph build < doublewire.json +error: op 5 (connect slow.value -> sub.lhs): slot sub.lhs already wired +``` + +Introspection (acceptance 3) — build-free: + +```console +$ aura graph introspect --vocabulary +Add And Bias CarryCost ConstantCost Delay EMA EqConst FixedStop Gt Latch +LongOnly Mul PositionManagement Resample RollingMax RollingMin Sizer SMA +Sqrt Sub VolSlippageCost + +$ aura graph introspect --node SMA +SMA inputs: series:f64 output: value:f64 params: length:i64 + +# the still-unwired slots of a partial document +$ aura graph introspect --unwired < partial.json +sub.rhs:f64 bias.signal:f64 +``` + +### Engine shapes (secondary, before → after) + +**(1) Extract the edge-kind predicate** — shared by the eager op and the +unchanged holistic `validate_wiring`. + +```rust +// NEW shared predicate (pub(crate)); the body is lifted verbatim from +// validate_wiring's edge loop (blueprint.rs:638-649), so the variant is +// unchanged (Bootstrap(KindMismatch)) and existing compiled-graph tests stay green. +pub(crate) fn edge_kind_check( + from: &NodeSchema, from_field: usize, + to: &NodeSchema, slot: usize, +) -> Result<(), CompileError> { + let f = from.output.get(from_field).ok_or(CompileError::BadInteriorIndex)?; + let s = to.inputs.get(slot).ok_or(CompileError::BadInteriorIndex)?; + if f.kind != s.kind { + return Err(CompileError::Bootstrap(BootstrapError::KindMismatch { + producer: f.kind, consumer: s.kind, + })); + } + Ok(()) +} +// validate_wiring's edge loop now calls edge_kind_check (after its index lookups); +// the eager connect op calls the same predicate. ONE check, two cadences. +``` + +**(2) `check_ports_connected` is unchanged** (`blueprint.rs:688`) — it stays the +holistic backstop catching the **0** arm (`UnconnectedPort`) and, defensively, +the **>1** arm. The eager surface front-runs the **>1** arm via its incremental +coverage map: + +```rust +// inside the per-op connect/feed handler (eager half of the exactly-once invariant) +let n = *self.coverage.entry((to_node, slot)).or_insert(0); +if n >= 1 { return Err(OpError::SlotAlreadyWired { node: to_node, slot }); } // DoubleWiredPort, raised early +self.coverage.insert((to_node, slot), n + 1); +``` + +**(3) A non-panicking bind** — today `PrimitiveBuilder::bind` *panics* on an +unknown/ambiguous/kind-wrong param (`node.rs`, the `debug_assert`/`expect` +path). The `add … bind{…}` op needs a fallible variant so a bad param is +rejected *at the op*, not a panic: + +```rust +// NEW: the fallible twin of bind() (same exactly-one-match + kind rule, Result not panic) +pub fn try_bind(self, slot: &str, value: Scalar) -> Result { … } +``` + +**(4) The per-op surface (engine) — op enum + drive/finish:** + +```rust +pub enum Op { // serde-deserialized from the JSON op-list (§C) + Source { role: String, kind: ScalarKind }, + Input { role: String }, + Add { type_id: String, as_name: Option, bind: Vec<(String, Scalar)> }, + Feed { role: String, into: Vec }, // "node.slot" identifiers + Connect { from: String, to: String }, // "node.field" -> "node.slot" + Expose { from: String, as_name: String }, +} + +pub struct GraphSession { /* nodes, schemas, edges, roles, out, coverage, name->handle */ } + +impl GraphSession { + pub fn new(name: &str, vocab: &dyn Fn(&str) -> Option) -> Self; + pub fn apply(&mut self, op: Op) -> Result<(), OpError>; // eager checks (§A) + pub fn finish(self) -> Result; // holistic gates → Composite + // build-free introspection over the partial session: + pub fn unwired(&self) -> Vec<(String /*node.slot*/, ScalarKind)>; +} + +// replay driver: stop at first failing op, naming it (acceptance 2) +pub fn replay(name: &str, ops: Vec, vocab: …) -> Result; +``` + +`OpError` carries by-identifier causes (`UnknownNodeType(String)`, +`UnknownPort{node,name}`, `KindMismatch{…}`, `SlotAlreadyWired{…}`, +`DuplicateIdentifier(String)`, `BadParam(BindOpError)`, and a holistic +`Incomplete(CompileError)` wrapping the finalize gate). The eager variants reuse +§A's predicates and `builder.rs`'s resolution; `Incomplete` wraps the unchanged +`compile_with_params`/finalize path. + +### §C — enumerable vocabulary companion + +```rust +// aura-std/src/vocabulary.rs — companion to the existing std_vocabulary match (:30) +pub fn std_vocabulary_types() -> &'static [&'static str] { + &["Add","And","Bias","CarryCost","ConstantCost","Delay","EMA","EqConst", + "FixedStop","Gt","Latch","LongOnly","Mul","PositionManagement","Resample", + "RollingMax","RollingMin","Sizer","SMA","Sqrt","Sub","VolSlippageCost"] +} +// introspection maps each type → std_vocabulary(type).schema() for ports/kinds/params. +// (The same iterable surface would let #160's guard test iterate; #160 stays separate.) +``` + +## Components + +- **`aura-engine` (§A):** extract `edge_kind_check` (and the index-range lookups + it needs) from `validate_wiring`; make `resolve_slot`/`resolve_field` shared + (`pub(crate)`); add `PrimitiveBuilder::try_bind` (fallible twin of `bind`). + `check_ports_connected` and `compile_with_params` are **unchanged**. +- **`aura-engine` (§B):** the `GraphSession` per-op surface + `Op`/`OpError` + enums + the `replay` driver + `unwired` introspection. New module + (e.g. `construction.rs`). +- **`aura-std` (§C):** `std_vocabulary_types()` — the enumerable companion. +- **`aura-cli` (§C):** serde `Deserialize` for `Op` (the JSON op-list); the + `["graph","build", …]` and `["graph","introspect", …]` argv arms over + `main.rs:3292`; rendering of `OpError` / introspection results to + stdout/stderr with the op index. + +## Data flow + +``` +document (JSON op-list) [§C] + └─ serde → Vec + └─ replay(name, ops, std_vocabulary) ──── [§B] + ├─ per op: apply(op) → eager checks [§A predicates] ──┐ stop at first Err, + │ │ naming op index + cause + └─ finish() → holistic gates → Composite ┘ + ├─ blueprint_to_json(&composite) → emitted blueprint (#155) [title: "emits blueprints"] + └─ composite.compile_with_params(params) → FlatGraph + └─ identical to the Rust-built twin's FlatGraph [acceptance 1, C1] + +introspection (no build): + --vocabulary → std_vocabulary_types() + --node T → std_vocabulary(T).schema() (ports/kinds) + .params() + --unwired → replay partial → GraphSession::unwired() +``` + +## Error handling + +- **Eager (per op):** `apply` returns `OpError` with a by-identifier cause; the + `replay` driver returns `(op_index, OpError)`; the CLI prints `op N (kind): + cause` and exits non-zero. Stop-at-first — no op after the failing one runs. +- **Holistic (finish):** `Incomplete(CompileError)` wraps the unchanged finalize + gates (`UnconnectedPort`, `DuplicateParamPath`, `UnboundRootRole`, …). An + `UnconnectedPort` is reported against the document as a whole, not an op. +- **No second validator:** every eager rejection traces to a §A predicate that + the holistic gate also uses; the holistic gates remain the backstop. A bug in + a predicate fails both cadences identically (a single fix site). + +## Testing strategy + +Iteration 1 (engine, §A+§B) — driven directly through `GraphSession`/`replay`, +no CLI: + +- **Acceptance 1 (identical run):** author `sma_cross` (the worked example) via + `replay`; author the same via `GraphBuilder`; compile both with the same + params; assert `flat.edges` and `flat.sources` equal — modeled on the green + `builder_harness_compiles_identically_to_hand_wired` (`builder.rs:234`). +- **Acceptance 2 (per-op rejection):** one test per cause — unknown node + (`add Nope` → op 0), edge kind-mismatch (bool→f64 → at the connect op), + double-wire (second connect into a slot → at that op), bad bind param. Each + asserts the failing op index + the `OpError` variant. +- **Eager/holistic boundary:** a document missing a connection passes every + per-op `apply` but `finish` returns `Incomplete(UnconnectedPort{…})` — proving + totality is *not* front-run (the legitimately-incomplete mid-document state). +- **No-second-validator:** a kind-mismatch caught eagerly and the same wiring + caught by `validate_wiring` both resolve to `edge_kind_check` (the eager + `OpError::KindMismatch` and the holistic `Bootstrap(KindMismatch)` agree). +- **Introspection:** `std_vocabulary_types()` lists exactly the resolvable + keys (cross-check against `std_vocabulary` — a key present in one but not the + other fails); `--node SMA` shape; `unwired()` on a partial session. + +Iteration 2 (CLI, §C) — E2E: `aura graph build < doc.json` emits the blueprint; +an invalid op prints `op N` + cause and exits non-zero; `aura graph introspect` +answers each query. `cargo test --workspace` green; `clippy --all-targets -D +warnings` clean. + +## Acceptance criteria + +- [ ] **(1)** A `sma_cross` signal blueprint built purely through the ops + compiles to a `FlatGraph` identical (edges + sources) to its `GraphBuilder`-built + twin (C1). +- [ ] **(2)** An invalid op — unknown node, edge kind-mismatch, double-wire, bad + bind param — is rejected **at that op**, naming the op index + cause; no later + op runs. +- [ ] **(3)** Introspection answers the node vocabulary, a node's ports/kinds + + param paths, and a partial document's unwired slots — all without a build. +- [ ] No second validator: the eager and holistic checks call the same extracted + predicates (`edge_kind_check`, the exactly-once coverage invariant); the + finalize gates (`compile_with_params`) are unchanged. +- [ ] `aura graph build` emits the #155 blueprint JSON for a valid document. +- [ ] `cargo test --workspace` green; `cargo clippy --workspace --all-targets -D + warnings` clean. + +## Iteration scope (for the planner) + +- **Iteration 1 — engine core (§A + §B):** the gate split + shared predicates, + `GraphSession`/`Op`/`replay`, `try_bind`, the introspection API + enumerable + vocabulary companion. Fully testable through the engine API; no CLI. +- **Iteration 2 — CLI shell (§C):** the JSON op-list serde encoding and the + `aura graph build` / `aura graph introspect` subcommands; E2E tests.