diff --git a/docs/plans/0018-composite-multi-output-record.md b/docs/plans/0018-composite-multi-output-record.md new file mode 100644 index 0000000..eb358cb --- /dev/null +++ b/docs/plans/0018-composite-multi-output-record.md @@ -0,0 +1,623 @@ +# Composite multi-output record — Implementation Plan + +> **Parent spec:** `docs/specs/0018-composite-multi-output-record.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement` skill to run +> this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Make a composite's output a named, ordered, multi-field record +(`output: Vec`) so multi-line indicators +(MACD/Bollinger/Stochastic/Ichimoku) can be authored as a composition and +re-exported as a unit, selected downstream by `Edge::from_field`. + +**Architecture:** The substrate is already multi-field everywhere except the +composite boundary (`NodeSchema.output: Vec`, `Edge::from_field` +selects a column, leaf multi-output already works). This cycle replaces the +single `OutPort` with a `Vec` and lifts the three `field == 0` caps at +the boundary (`inline_composite` nested arm, `rewrite_edge` composite arm). Names +live at the blueprint boundary only and are dropped in the compilat (C23); C8/C7/C4 +are untouched — one port, one row, K columns. + +**Tech Stack:** `crates/aura-engine/src/blueprint.rs` (type + compile logic), +`crates/aura-engine/src/lib.rs` (re-export), `crates/aura-cli/src/graph.rs` +(render), `crates/aura-cli/src/main.rs` (author sites + render tests), +`fieldtests/milestone-construction-layer/*.rs` (non-workspace stale-ref sweep). + +**Files this plan creates or modifies:** + +- Modify: `crates/aura-engine/src/blueprint.rs` — `OutPort`→`OutField`, `Composite` + struct/`new`/`output()`, `ItemLowering::Composite.output`, `inline_composite`, + `rewrite_edge`, all `#[cfg(test)]` `OutPort` literals; new capability tests. +- Modify: `crates/aura-engine/src/lib.rs:38` — re-export `OutPort`→`OutField`. +- Modify: `crates/aura-cli/src/graph.rs:105–129` — `render_definition` K `[out:]` markers. +- Modify: `crates/aura-cli/src/main.rs` — import `:13`, `sma_cross` `:130`, `macd` `:206`, + nested-render-test literals `:391`/`:398`, needle-list test `:377`, `blueprint_view_golden`. +- Modify: `fieldtests/milestone-construction-layer/mc_1..mc_4*.rs` — `OutPort`→`OutField` (8 sites, non-workspace). + +**Naming decisions (orchestrator, fixed for this plan):** +- `sma_cross` single-field output name → `"cross"` (renders as `[out:cross]` in the re-captured golden). +- Engine test-fixture single-field outputs → `"out"`. New multi-output test fields → `"a"`, `"b"` (and `"c"` where a third is needed). +- Fieldtest single-field outputs → `"out"`. + +--- + +## Task 1: Engine — introduce `OutField`, widen the type (behaviour-preserving) + +This task replaces `OutPort` with `OutField` and widens `Composite.output` / +`ItemLowering::Composite.output` to vectors, threading **every** call site in +`blueprint.rs` so the crate compiles and **all existing tests stay green**. The +three `field == 0` caps are **kept** here (single-field composites only re-export +field 0, so behaviour is preserved); Task 2 lifts them test-first. + +**Files:** +- Modify: `crates/aura-engine/src/blueprint.rs` +- Modify: `crates/aura-engine/src/lib.rs:38` + +- [ ] **Step 1: Rename `OutPort` → `OutField` and add `name` (blueprint.rs:18–23)** + +Replace: + +```rust +/// Which interior `(node, output-field)` is a composite's single output port (C8). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct OutPort { + pub node: usize, + pub field: usize, +} +``` + +with: + +```rust +/// One re-exported field of a composite's output record: an interior +/// `(node, output-field)` surfaced at the boundary under `name`. `name` is a +/// non-load-bearing render/debug symbol (C23) — like `FieldSpec.name` and +/// `Composite.name`, it does not reach the compilat. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct OutField { + pub node: usize, + pub field: usize, + pub name: String, +} +``` + +(`Copy` is dropped — `String` is not `Copy`. Callers that relied on `output()` +returning by Copy are updated in Step 3 and Step 6.) + +- [ ] **Step 2: Widen `Composite.output` to a record (blueprint.rs:43–49)** + +In `struct Composite`, change the field: + +```rust + input_roles: Vec>, + output: Vec, +``` + +(was `output: OutPort,` at `:48`.) + +- [ ] **Step 3: Update `Composite::new` and `output()` (blueprint.rs:56–85)** + +In `Composite::new`, change the param type: + +```rust + input_roles: Vec>, + output: Vec, + ) -> Self { + Self { name: name.into(), nodes, edges, input_roles, output } + } +``` + +(was `output: OutPort,` at `:61`.) Change the accessor (`:83–85`): + +```rust + /// The exposed output record: each entry re-exports one interior + /// `(node, output-field)` under a boundary name (C8 — one port, K columns). + pub fn output(&self) -> &[OutField] { + &self.output + } +``` + +(was `pub fn output(&self) -> OutPort { self.output }`.) + +- [ ] **Step 4: Widen the lowering variant (blueprint.rs:243–249)** + +In `enum ItemLowering`, change the `Composite` variant: + +```rust + /// A composite lowered to its interior: its output record is these flat + /// `(node, field)` producers (one per re-exported field, declared order), and + /// input role `r` fans into `roles[r]` (flat targets). Names dropped (C23). + Composite { output: Vec<(usize, usize)>, roles: Vec> }, +``` + +(was `output: (usize, usize)`.) + +- [ ] **Step 5: Make `inline_composite` build the output record, caps kept (blueprint.rs:292–355)** + +Remove the single pre-check at `:306–308`: + +```rust + if output.node >= item_count { + return Err(CompileError::OutputPortOutOfRange); + } +``` + +Replace the single-output resolution block (`:318–333`, the `let out = match … ;`) +with a loop that builds the Vec. **The `!= 0` caps stay** (behaviour-preserving): + +```rust + // resolve each re-exported field to a flat (node, field), in declared order + let mut out: Vec<(usize, usize)> = Vec::with_capacity(output.len()); + for of in &output { + if of.node >= item_count { + return Err(CompileError::OutputPortOutOfRange); + } + let resolved = match &interior[of.node] { + ItemLowering::Leaf { index } => { + if of.field >= flat_nodes[*index].schema().output.len() { + return Err(CompileError::OutputPortOutOfRange); + } + (*index, of.field) + } + ItemLowering::Composite { output: nested, .. } => { + // cap kept in Task 1; Task 2 lifts this to nested.get(of.field) + if of.field != 0 { + return Err(CompileError::OutputPortOutOfRange); + } + nested[0] + } + }; + out.push(resolved); + } +``` + +The terminal `Ok(ItemLowering::Composite { output: out, roles })` at `:354` is +unchanged (`out` is now a `Vec`). + +- [ ] **Step 6: Thread `rewrite_edge`, cap kept (blueprint.rs:375–381)** + +Replace the composite arm: + +```rust + ItemLowering::Composite { output, .. } => { + // cap kept in Task 1; Task 2 lifts this to output.get(e.from_field) + if e.from_field != 0 { + return Err(CompileError::BadInteriorIndex); + } + output[0] + } +``` + +(was `*output` after the `!= 0` guard; `output` is now a `Vec`, so `output[0]`.) + +- [ ] **Step 7: Update `lib.rs` re-export (lib.rs:38)** + +```rust +pub use blueprint::{Blueprint, BlueprintNode, CompileError, Composite, OutField}; +``` + +(was `… Composite, OutPort};`.) + +- [ ] **Step 8: Migrate all existing `OutPort` test literals (blueprint.rs `#[cfg(test)]`)** + +Every existing test composite is single-field. Rewrite each literal at +`:532, :578, :611, :626, :641, :736, :916, :924, :967, :975` from +`OutPort { node: N, field: F }` to a one-element record: + +```rust +vec![OutField { node: N, field: F, name: "out".into() }] +``` + +preserving each site's existing `node`/`field` values. (The site at `:641` is +`OutPort { node: 0, field: 5 }` — the OHLCV-field re-export — becomes +`vec![OutField { node: 0, field: 5, name: "out".into() }]`.) The comment mention +at `:549` is prose; update its wording from "OutPort" to "OutField" if it names +the type, otherwise leave. + +- [ ] **Step 9: Build and test the engine crate** + +Run: `cargo test -p aura-engine` +Expected: PASS — the crate compiles and every existing test is green +(single-field records are behaviour-preserving; the caps still hold). + +- [ ] **Step 10: Clippy the engine crate** + +Run: `cargo clippy -p aura-engine --all-targets -- -D warnings` +Expected: PASS — no warnings (watch for a needless `Vec::with_capacity` or +`clone` lint; none expected). + +--- + +## Task 2: Engine — lift the caps, multi-output capability (RED-first) + +The type is in place (Task 1); now lift the two `field == 0` caps, each gated by +a failing test written first. Tests reuse the existing `#[cfg(test)]` helpers +`pass1()` (1-in/1-out f64 leaf), `sink_f64()` (f64 sink), `Blueprint::new`, +`SourceSpec`, and the `bp.compile()` / `bp.compile().err()` assertion idiom — no +new helper is introduced. Spec test 3 (out-of-range re-export) is already covered +by the migrated `output_port_out_of_range_rejected` (Task 1 Step 8), so it is not +re-added here. + +**Files:** +- Modify: `crates/aura-engine/src/blueprint.rs` (the two caps + three new `#[cfg(test)]` tests) + +- [ ] **Step 1: Write the multi-output happy-path test (RED)** + +Add to the `#[cfg(test)]` module, next to `single_composite_inlines_with_offset_fan_and_output`: + +```rust +#[test] +fn composite_reexports_two_fields_to_distinct_consumers() { + // composite: two independent Pass1 leaves; role 0 -> leaf 0, role 1 -> leaf 1; + // output record re-exports leaf 0 as "a", leaf 1 as "b". + let c = Composite::new( + "two_out", + vec![pass1(), pass1()], + vec![], + vec![ + vec![Target { node: 0, slot: 0 }], + vec![Target { node: 1, slot: 0 }], + ], + vec![ + OutField { node: 0, field: 0, name: "a".into() }, + OutField { node: 1, field: 0, name: "b".into() }, + ], + ); + // composite is item 0; two sinks (items 1, 2) read its two output fields by + // from_field; one source fans into both roles. + let bp = Blueprint::new( + vec![BlueprintNode::Composite(c), sink_f64(), sink_f64()], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![Target { node: 0, slot: 0 }, Target { node: 0, slot: 1 }], + }], + vec![ + Edge { from: 0, to: 1, slot: 0, from_field: 0 }, // field "a" -> sink 1 + Edge { from: 0, to: 2, slot: 0, from_field: 1 }, // field "b" -> sink 2 + ], + ); + let (nodes, sources, edges) = bp.compile().expect("valid multi-output composite"); + // flat layout: Pass1(0), Pass1(1), SinkF64(2), SinkF64(3) + assert_eq!(nodes.len(), 4); + // from_field 0 resolves to leaf 0, from_field 1 to leaf 1 — distinct producers + assert_eq!( + edges, + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, + Edge { from: 1, to: 3, slot: 0, from_field: 0 }, + ] + ); + // the source fanned into both interior leaves + assert_eq!( + sources[0].targets, + vec![Target { node: 0, slot: 0 }, Target { node: 1, slot: 0 }] + ); +} +``` + +Run: `cargo test -p aura-engine composite_reexports_two_fields_to_distinct_consumers` +Expected: FAIL — the consumer reading `from_field: 1` hits the `rewrite_edge` cap; +`compile` returns `Err(BadInteriorIndex)`, so `.expect(...)` panics. + +- [ ] **Step 2: Lift the `rewrite_edge` cap (blueprint.rs composite arm)** + +Replace the composite arm (the Step-6 Task-1 body) with the range-checked index: + +```rust + ItemLowering::Composite { output, .. } => { + *output.get(e.from_field).ok_or(CompileError::BadInteriorIndex)? + } +``` + +Run: `cargo test -p aura-engine composite_reexports_two_fields_to_distinct_consumers` +Expected: PASS. + +- [ ] **Step 3: Write the nested multi-output test (RED)** + +An **outer** composite re-exporting two fields of an **inner** multi-output +composite (exercises the nested arm). Add next to `nested_composite_inlines`: + +```rust +#[test] +fn outer_reexports_two_fields_of_inner_composite() { + // inner re-exports two leaves as "a","b"; outer re-exposes both inner roles + // and re-exports inner field 0 and field 1 (the latter exercises the nested arm). + let inner = Composite::new( + "inner_two", + vec![pass1(), pass1()], + vec![], + vec![ + vec![Target { node: 0, slot: 0 }], + vec![Target { node: 1, slot: 0 }], + ], + vec![ + OutField { node: 0, field: 0, name: "a".into() }, + OutField { node: 1, field: 0, name: "b".into() }, + ], + ); + let outer = Composite::new( + "outer_two", + vec![BlueprintNode::Composite(inner)], + vec![], + vec![ + vec![Target { node: 0, slot: 0 }], // outer role 0 -> inner role 0 + vec![Target { node: 0, slot: 1 }], // outer role 1 -> inner role 1 + ], + vec![ + OutField { node: 0, field: 0, name: "x".into() }, // inner field 0 + OutField { node: 0, field: 1, name: "y".into() }, // inner field 1 (nested arm) + ], + ); + let bp = Blueprint::new( + vec![BlueprintNode::Composite(outer), sink_f64(), sink_f64()], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![Target { node: 0, slot: 0 }, Target { node: 0, slot: 1 }], + }], + vec![ + Edge { from: 0, to: 1, slot: 0, from_field: 0 }, // outer field x -> sink 1 + Edge { from: 0, to: 2, slot: 0, from_field: 1 }, // outer field y -> sink 2 + ], + ); + let (nodes, _sources, edges) = bp.compile().expect("valid nested multi-output"); + assert_eq!(nodes.len(), 4); // Pass1, Pass1, SinkF64, SinkF64 + assert_eq!( + edges, + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, + Edge { from: 1, to: 3, slot: 0, from_field: 0 }, + ] + ); +} +``` + +Run: `cargo test -p aura-engine outer_reexports_two_fields_of_inner_composite` +Expected: FAIL — `inline_composite`'s nested arm caps `of.field != 0` on the +outer's `field: 1` re-export; `compile` returns `Err(OutputPortOutOfRange)`. + +- [ ] **Step 4: Lift the nested arm cap (blueprint.rs inline_composite)** + +Replace the nested arm in the Step-5 Task-1 loop: + +```rust + ItemLowering::Composite { output: nested, .. } => { + *nested.get(of.field).ok_or(CompileError::OutputPortOutOfRange)? + } +``` + +(was the `if of.field != 0 { … } nested[0]` cap.) + +Run: `cargo test -p aura-engine outer_reexports_two_fields_of_inner_composite` +Expected: PASS. + +- [ ] **Step 5: Write the out-of-range consume guard test** + +```rust +#[test] +fn consume_of_missing_output_field_is_rejected() { + // a single-field composite; a consumer reads from_field 1 (past the 1-field + // record) -> the rewrite_edge range-check rejects it. + let c = Composite::new( + "c", + vec![pass1()], + vec![], + vec![vec![Target { node: 0, slot: 0 }]], + vec![OutField { node: 0, field: 0, name: "a".into() }], + ); + let bp = Blueprint::new( + vec![BlueprintNode::Composite(c), sink_f64()], + vec![SourceSpec { kind: ScalarKind::F64, targets: vec![Target { node: 0, slot: 0 }] }], + vec![Edge { from: 0, to: 1, slot: 0, from_field: 1 }], // only field 0 exists + ); + assert_eq!(bp.compile().err(), Some(CompileError::BadInteriorIndex)); +} +``` + +Run: `cargo test -p aura-engine consume_of_missing_output_field_is_rejected` +Expected: PASS — `output.get(1)` on a 1-field record is `None` → `BadInteriorIndex`. +(This is a guard, not a RED gate: it also errored under the Task-1 cap, but now via +the range-check that replaced it.) + +- [ ] **Step 6: Full engine regression + clippy** + +Run: `cargo test -p aura-engine` +Expected: PASS — all old tests (single-field, behaviour-preserving) plus the four +new capability tests are green. + +Run: `cargo clippy -p aura-engine --all-targets -- -D warnings` +Expected: PASS. + +--- + +## Task 3: CLI — multi-output render + author sites + golden re-capture + +`aura-cli` depends on `aura-engine`; after Task 1 it no longer compiles (`OutPort` +removed from the import and four literals). This task threads every CLI site, +re-exports the MACD's three lines, renders K named output markers, and re-captures +the one drifting golden — one atomic compile-and-test unit. + +**Files:** +- Modify: `crates/aura-cli/src/main.rs` (import `:13`, `sma_cross` `:130`, `macd` `:206`, nested-render-test literals `:391`/`:398`, needle test `:377`, `blueprint_view_golden`) +- Modify: `crates/aura-cli/src/graph.rs:124–126` (`render_definition`) + +- [ ] **Step 1: Update the import (main.rs:13)** + +Change `OutPort` to `OutField` in the `use` list: + +```rust + f64_field, summarize, Blueprint, BlueprintNode, Composite, Edge, Harness, OutField, +``` + +- [ ] **Step 2: Render K named output markers (graph.rs:124–126)** + +Replace the single `[out]` marker block: + +```rust + for of in c.output() { + let out_id = labels.len(); + labels.push(format!("out:{}", of.name)); + edges.push((of.node, out_id)); + } +``` + +(was the three lines pushing one `"out"` label wired from `c.output().node`.) +Update the function's doc-comment at `:101–104` to say "an `[out:]` marker +per re-exported output field" (was "an `[out]` marker (wired from the output +port)"). + +- [ ] **Step 3: Migrate `sma_cross` to a single-field record (main.rs:130)** + +```rust + vec![OutField { node: 2, field: 0, name: "cross".into() }], +``` + +(was `OutPort { node: 2, field: 0 }`.) Behaviour-preserving: one re-exported +field; renders as `[out:cross]`. + +- [ ] **Step 4: Re-export all three MACD lines (main.rs:206)** + +Replace the `macd` composite's output (was `OutPort { node: 4, field: 0 }, // the histogram`): + +```rust + vec![ + OutField { node: 2, field: 0, name: "macd".into() }, // the MACD line + OutField { node: 3, field: 0, name: "signal".into() }, // the signal line + OutField { node: 4, field: 0, name: "histogram".into() }, // the histogram + ], +``` + +Update the `macd` doc-comment (`:178–184`): the composite now exposes the **three +MACD lines** as a record; the strategy trades the histogram by reading +`from_field: 2`. + +**Then update the consumer edge** in `macd_strategy_blueprint` (`:233`). The +histogram is now field index **2** of the 3-field record (was the sole output at +field 0), so the edge feeding `Exposure` must select it: + +```rust + Edge { from: 0, to: 1, slot: 0, from_field: 2 }, // histogram → Exposure +``` + +(was `from_field: 0`.) This keeps the strategy trading the **histogram** — leaving +it at `0` would silently feed the MACD line instead, changing the run and breaking +`run_macd_compiles_from_nested_composite_and_is_deterministic`. The other three +edges (`:234–236`) are unchanged. + +- [ ] **Step 5: Migrate the nested-render-test literals (main.rs:391, :398)** + +In `nested_composite_renders_without_panic`, both single-field composites: + +```rust + vec![OutField { node: 0, field: 0, name: "out".into() }], // :391 inner +``` +```rust + vec![OutField { node: 1, field: 0, name: "out".into() }], // :398 outer +``` + +- [ ] **Step 6: Build the CLI crate** + +Run: `cargo build -p aura-cli` +Expected: PASS — every `OutPort` site threaded; crate compiles. (Golden/needle +tests may still be red — handled next.) + +- [ ] **Step 7: Add the `[out:]` needle assertion (main.rs:377)** + +In `blueprint_view_defines_each_composite_once`, extend the needle list so the +multi-output marker is asserted. `sma_cross` is the rendered composite, so its +marker is `[out:cross]`: + +```rust + for needle in ["[SMA]", "[Sub]", "[in:0]", "[out:cross]"] { +``` + +(was `… "[out]"`.) + +Run: `cargo test -p aura-cli blueprint_view_defines_each_composite_once` +Expected: PASS — the render now emits `[out:cross]` for `sma_cross`. + +- [ ] **Step 8: Re-capture the `blueprint_view_golden` (main.rs:464–502)** + +The `[out]` → `[out:cross]` change drifts this golden. Follow the test's own +re-capture protocol (comment at `:466–468`): run the golden test, observe the +actual rendered string in the failure diff, and paste it verbatim into the golden +literal (`:497` region). Do **not** hand-edit the expected string field-by-field — +copy the actual output wholesale. + +Run: `cargo test -p aura-cli blueprint_view_golden` +Expected: FAIL first (golden drift: `[out]` → `[out:cross]`), then PASS after the +literal is re-captured. + +- [ ] **Step 9: Confirm the compilat golden is byte-identical (main.rs:505–530)** + +`compiled_view_golden` pins the flat compilat render; per C23 names are dropped at +inline, so it must **not** change (acceptance criterion 6 — the regression guard). + +Run: `cargo test -p aura-cli compiled_view_golden` +Expected: PASS with **no** edit to the golden literal. If it drifts, a name leaked +into the compilat — a bug to fix, not a golden to re-capture. + +- [ ] **Step 10: Full CLI test + clippy** + +Run: `cargo test -p aura-cli` +Expected: PASS — including `run_macd_compiles_from_nested_composite_and_is_deterministic` +(the MACD run still produces the histogram, now via `from_field: 2`). + +Run: `cargo clippy -p aura-cli --all-targets -- -D warnings` +Expected: PASS. + +--- + +## Task 4: Non-workspace fieldtest sweep + full workspace gate + +The construction-layer fieldtests are **not** workspace members, so +`cargo …--workspace` does not compile them — they hold stale `OutPort` references +to a now-deleted type. Sweep them for tree consistency (mechanical), then run the +full workspace gate. + +**Files:** +- Modify: `fieldtests/milestone-construction-layer/mc_1_composite_build_run.rs:6,20,47` +- Modify: `fieldtests/milestone-construction-layer/mc_2_miswire_render.rs:15,47` +- Modify: `fieldtests/milestone-construction-layer/mc_3_nested_composite.rs:18,35,52` +- Modify: `fieldtests/milestone-construction-layer/mc_4_introspect_graph.rs:13,34` + +- [ ] **Step 1: Sweep `OutPort` → `OutField` across the fieldtests** + +In each file: update the `use … OutPort` import to `OutField`, and rewrite each +`OutPort { node: N, field: F }` literal to `vec![OutField { node: N, field: F, name: "out".into() }]` +**at the `Composite::new` output-arg position** (these are all single-field +composites). Update comment mentions of `OutPort` (e.g. mc_1 `:6`) to `OutField`. + +- [ ] **Step 2: Verify no `OutPort` remains anywhere in the tree** + +Run: `rg -n "OutPort" --type rust` +Expected: **no matches** — the type is fully retired (every reference is now +`OutField`). + +- [ ] **Step 3: Full workspace build** + +Run: `cargo build --workspace` +Expected: PASS. + +- [ ] **Step 4: Full workspace test** + +Run: `cargo test --workspace` +Expected: PASS — every crate green; single-field composites behaviour-preserving, +multi-output capability covered by Task 2, render by Task 3. + +- [ ] **Step 5: Full workspace clippy** + +Run: `cargo clippy --workspace --all-targets -- -D warnings` +Expected: PASS — zero warnings. + +--- + +## Acceptance gate (whole plan) + +- [ ] `cargo build --workspace` green. +- [ ] `cargo test --workspace` green (incl. the four new engine capability tests and the re-captured `blueprint_view_golden`). +- [ ] `cargo clippy --workspace --all-targets -- -D warnings` green. +- [ ] `rg "OutPort" --type rust` returns nothing (type fully retired). +- [ ] `compiled_view_golden` unchanged (compilat byte-identical — C23 / acceptance criterion 6). +- [ ] The MACD PoC re-exports `macd`/`signal`/`histogram`; the run still trades the histogram (via `from_field: 2`). + +The implementation commit closes #40.