spec: 0028 param-sweep grid

A grid-enumerated family of disjoint instances from one value-empty blueprint (milestone The World, cycle C). Engine-side GridSpace + sweep() (std::thread::scope, lock-free disjoint per C1) + SweepFamily; closure-driven so harness-specific metrics glue stays author-side. refs #32
2026-06-10 18:21:23 +02:00
parent 776bd5432a
commit 0692637b60
1 changed files with 484 additions and 0 deletions
@@ -0,0 +1,484 @@
+# Param-sweep: a grid family of disjoint instances from one blueprint — Design Spec
+
+**Date:** 2026-06-10
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+## Goal
+
+Turn one value-empty blueprint into a **family** of frozen instances — one per
+point in a declared grid — run them **disjointly in parallel**, and collect each
+point's metrics into an ordered family. This is cycle C (#32) of the milestone
+"The World — parameter-space & sweep": the first orchestration axis (C12.1), the
+first "N instead of 1". The walking skeleton and the construction layer each
+produce a single instance; the World begins here.
+
+#30 landed the declarable param-space (`Composite::param_space() -> Vec<ParamSpec>`,
+flat, path-qualified, kind-typed). #31 landed the binding primitive
+(`Composite::bootstrap_with_params(Vec<Scalar>) -> Harness`, one point → one
+instance). This cycle is the axis *over* that primitive: enumerate a grid of
+points across the param-space, bootstrap and run each disjointly, and fold the
+results into a family that #33 (the run registry) will later index.
+
+Scope is the **sweep axis only**, and within it **grid enumeration only**: a
+cartesian product over per-slot discrete value-lists. Random enumeration
+(per-slot ranges + sample-count + seed + RNG), optimize (argmax, C12.2),
+walk-forward (C12.3), and Monte-Carlo (C12.4) are deferred to later cycles.
+
+## Architecture
+
+A new module `crates/aura-engine/src/sweep.rs` carries three parts:
+
+1. **Enumeration — `GridSpace`.** A validated cartesian grid over a blueprint's
+   param-space: one discrete value-list per param slot, kind-checked against the
+   `&[ParamSpec]` the blueprint reports. `points()` materializes the cartesian
+   product in a deterministic odometer order (last axis varies fastest).
+
+2. **Execution — `sweep()`.** A closure-driven primitive. It owns enumeration +
+   **disjoint parallel execution** + family collection; the **author** owns the
+   per-point closure `run_one: Fn(&[Scalar]) -> RunMetrics`. Each point's
+   `run_one` builds a fresh blueprint, bootstraps it under the point vector,
+   runs it, drains its sinks, and summarizes — entirely within one thread, so it
+   shares nothing mutable (C1). Execution is `std::thread::scope` over
+   `available_parallelism()` workers pulling point indices from a shared atomic
+   cursor (work-stealing load-balances uneven per-point cost). **No external
+   dependency** — `rayon` would violate C16 (the engine workspace is
+   zero-external-dependency by commitment; only `aura-ingest` links an external
+   tree). `std::thread::scope` is std and sufficient for a flat,
+   embarrassingly-parallel map.
+
+3. **Collection — `SweepFamily`.** An ordered `Vec<SweepPoint>`, each a
+   `{ params, metrics }` pair. The family order is the enumeration order,
+   **independent of thread completion order**: workers tag each result with its
+   point index, and the family is assembled by sorting on that index after the
+   scope joins. Only the cursor is shared (atomic); the results side is
+   lock-free.
+
+### Why closure-driven (the engine ↔ author boundary)
+
+Metrics extraction is **harness-specific glue**: a run's metrics are reduced by
+draining its recording sinks, and the engine cannot generically know which
+`Recorder` is equity vs exposure — the sink channels live *outside* the graph,
+held by the author (the run loop is oblivious to the side effect, by C8/C18).
+So "run one point → `RunMetrics`" stays author-side, and the sweep's engine-level
+job is purely enumeration + disjoint execution + collection.
+
+A fresh per-point build also resolves two structural facts from #31 cleanly:
+`bootstrap_with_params(self, …)` **consumes** the blueprint (N instances need N
+fresh builds), and a `Recorder` bakes an `mpsc::Sender` into the blueprint (a
+fresh build gives each point its **own** drainable channels). Both fall out of
+`run_one` constructing its blueprint per call.
+
+### Determinism and disjointness (C1)
+
+Each point's run is a pure function of its param vector and the shared read-only
+data window (`Arc<[…]>` / `&[…]`, `Sync`). No two points share mutable state, so
+they are lock-free disjoint — and the *family* is bit-identical regardless of how
+many threads ran it or in what order they finished, because the assembled order
+is the enumeration order, not the completion order.
+
+## Concrete code shapes
+
+### The user-facing program (the acceptance evidence)
+
+The author authors the topology once (a value-empty blueprint factory), declares
+a grid over its param-space, and sweeps — getting one `RunMetrics` per grid point.
+The engine's own SMA-cross sample is the worked example:
+
+```rust
+use aura_engine::{GridSpace, Scalar, sweep, summarize, f64_field};
+
+// 1. The value-empty blueprint factory (built fresh per point — fresh channels).
+//    `sma_cross_sample()` returns the blueprint plus a way to drain its two sinks.
+let param_space = sma_cross_sample().0.param_space();
+// param_space == ["sma_cross.fast": I64, "sma_cross.slow": I64, "scale": F64]
+
+// 2. Declare a grid: one discrete value-list per param slot, in param_space order.
+let grid = GridSpace::new(&param_space, vec![
+    vec![Scalar::I64(2), Scalar::I64(3)],   // fast  ∈ {2, 3}
+    vec![Scalar::I64(4), Scalar::I64(5)],   // slow  ∈ {4, 5}
+    vec![Scalar::F64(0.5)],                 // scale ∈ {0.5}
+])?;                                         // 2 × 2 × 1 = 4 points
+assert_eq!(grid.len(), 4);
+
+let prices = synthetic_prices();             // shared read-only data window
+
+// 3. Sweep: each point builds fresh, bootstraps, runs, drains, summarizes.
+let family = sweep(&grid, |point| {
+    let (bp, rx_eq, rx_ex) = sma_cross_sample();          // fresh blueprint + channels
+    let mut h = bp.bootstrap_with_params(point.to_vec())
+        .expect("grid points are kind-checked against param_space");
+    h.run(vec![prices.clone()]);
+    let equity   = f64_field(&rx_eq.try_iter().collect::<Vec<_>>(), 0);
+    let exposure = f64_field(&rx_ex.try_iter().collect::<Vec<_>>(), 0);
+    summarize(&equity, &exposure)
+});
+
+// 4. The family is ordered (odometer: last axis fastest), self-describing.
+assert_eq!(family.points.len(), 4);
+assert_eq!(family.points[0].params, vec![Scalar::I64(2), Scalar::I64(4), Scalar::F64(0.5)]);
+assert_eq!(family.points[3].params, vec![Scalar::I64(3), Scalar::I64(5), Scalar::F64(0.5)]);
+for pt in &family.points {
+    assert!(pt.metrics.total_pips.is_finite());   // every point produced metrics
+}
+```
+
+The CLI demonstrator (iteration 2) makes the same payoff visible without writing
+Rust — `aura sweep` runs the built-in sample over a small built-in grid and
+prints the family as one JSON line per point:
+
+```text
+$ aura sweep
+{"params":{"sma_cross.fast":2,"sma_cross.slow":4,"scale":0.5},"metrics":{"total_pips":…,"max_drawdown":…,"exposure_sign_flips":…}}
+{"params":{"sma_cross.fast":2,"sma_cross.slow":5,"scale":0.5},"metrics":{…}}
+{"params":{"sma_cross.fast":3,"sma_cross.slow":4,"scale":0.5},"metrics":{…}}
+{"params":{"sma_cross.fast":3,"sma_cross.slow":5,"scale":0.5},"metrics":{…}}
+```
+
+### Implementation shapes (secondary — the new module)
+
+The four public types + the free function, in `aura-engine/src/sweep.rs`:
+
+```rust
+use aura_core::{ParamSpec, Scalar, ScalarKind};
+use crate::RunMetrics;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+/// A validated cartesian grid over a blueprint's param-space: one discrete
+/// value-list per param slot, in `param_space()` order. Enumerates a family of
+/// points (C12.1 grid axis).
+pub struct GridSpace {
+    axes: Vec<Vec<Scalar>>,
+}
+
+impl GridSpace {
+    /// Validate `axes` against `space` (the blueprint's `param_space()`):
+    /// one axis per slot (`Arity`), every value the slot's declared kind
+    /// (`KindMismatch`), no empty axis (`EmptyAxis`, which would yield zero
+    /// points). On success the grid enumerates `∏ |axis_i|` points.
+    pub fn new(space: &[ParamSpec], axes: Vec<Vec<Scalar>>) -> Result<Self, SweepError> {
+        if axes.len() != space.len() {
+            return Err(SweepError::Arity { expected: space.len(), got: axes.len() });
+        }
+        for (slot, (axis, ps)) in axes.iter().zip(space).enumerate() {
+            if axis.is_empty() {
+                return Err(SweepError::EmptyAxis { slot });
+            }
+            for (value_index, v) in axis.iter().enumerate() {
+                if v.kind() != ps.kind {
+                    return Err(SweepError::KindMismatch {
+                        slot, value_index, expected: ps.kind, got: v.kind(),
+                    });
+                }
+            }
+        }
+        Ok(Self { axes })
+    }
+
+    /// The number of grid points (`∏ |axis_i|`), always `>= 1`.
+    pub fn len(&self) -> usize {
+        self.axes.iter().map(|a| a.len()).product()
+    }
+
+    /// Always `false` — a valid `GridSpace` rejects empty axes, so it has at
+    /// least one point. Present to satisfy clippy's `len_without_is_empty`.
+    pub fn is_empty(&self) -> bool {
+        false
+    }
+
+    /// The cartesian product, in odometer order: the **last** axis varies
+    /// fastest. Deterministic — the same grid yields the same point sequence.
+    pub fn points(&self) -> Vec<Vec<Scalar>> {
+        let mut out = Vec::with_capacity(self.len());
+        let mut idx = vec![0usize; self.axes.len()];
+        loop {
+            out.push(self.axes.iter().zip(&idx).map(|(a, &i)| a[i]).collect());
+            // odometer increment from the last axis
+            let mut k = self.axes.len();
+            loop {
+                if k == 0 {
+                    return out;
+                }
+                k -= 1;
+                idx[k] += 1;
+                if idx[k] < self.axes[k].len() {
+                    break;
+                }
+                idx[k] = 0;
+            }
+        }
+    }
+}
+
+/// One enumerated point and the metrics its run produced. Self-describing: the
+/// `params` vector is the point's coordinate in `param_space()` order.
+#[derive(Clone, Debug, PartialEq)]
+pub struct SweepPoint {
+    pub params: Vec<Scalar>,
+    pub metrics: RunMetrics,
+}
+
+/// The ordered result family of a sweep — one `SweepPoint` per grid point, in
+/// enumeration (odometer) order, independent of thread completion order.
+#[derive(Clone, Debug, PartialEq)]
+pub struct SweepFamily {
+    pub points: Vec<SweepPoint>,
+}
+
+/// Run `run_one` over every grid point, disjointly in parallel (C1), and collect
+/// the family in enumeration order. `run_one` builds + bootstraps + runs +
+/// summarizes one point; it shares nothing mutable, so it is `Sync` and the runs
+/// are lock-free. Workers pull point indices from a shared atomic cursor
+/// (work-stealing); each tags its result with the point index, and the family is
+/// assembled by sorting on that index after the scope joins — so the order is the
+/// enumeration order, not the completion order.
+pub fn sweep<F>(space: &GridSpace, run_one: F) -> SweepFamily
+where
+    F: Fn(&[Scalar]) -> RunMetrics + Sync,
+{
+    let points = space.points();
+    let cursor = AtomicUsize::new(0);
+    let nthreads = std::thread::available_parallelism()
+        .map(|n| n.get())
+        .unwrap_or(1)
+        .min(points.len().max(1));
+
+    let mut results: Vec<(usize, RunMetrics)> = std::thread::scope(|scope| {
+        let handles: Vec<_> = (0..nthreads)
+            .map(|_| {
+                scope.spawn(|| {
+                    let mut local: Vec<(usize, RunMetrics)> = Vec::new();
+                    loop {
+                        let i = cursor.fetch_add(1, Ordering::Relaxed);
+                        if i >= points.len() {
+                            break;
+                        }
+                        local.push((i, run_one(&points[i])));
+                    }
+                    local
+                })
+            })
+            .collect();
+        handles.into_iter().flat_map(|h| h.join().unwrap()).collect()
+    });
+
+    results.sort_by_key(|&(i, _)| i);
+    SweepFamily {
+        points: results
+            .into_iter()
+            .map(|(i, metrics)| SweepPoint { params: points[i].clone(), metrics })
+            .collect(),
+    }
+}
+
+/// A structural fault constructing a `GridSpace` — the typed gate before any run.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum SweepError {
+    /// The number of axes does not equal the param-space slot count.
+    Arity { expected: usize, got: usize },
+    /// A grid value's kind does not match its slot's declared kind.
+    KindMismatch { slot: usize, value_index: usize, expected: ScalarKind, got: ScalarKind },
+    /// A slot was given no values (would yield zero points).
+    EmptyAxis { slot: usize },
+}
+```
+
+The `lib.rs` re-export (one import surface, beside the existing engine exports):
+
+```rust
+// crates/aura-engine/src/lib.rs (new)
+pub mod sweep;
+pub use sweep::{sweep, GridSpace, SweepError, SweepFamily, SweepPoint};
+```
+
+### Implementation shapes (secondary — the CLI demonstrator, iteration 2)
+
+`aura sweep` reuses the existing sample wiring, lifted so the receivers reach the
+caller. Today `build_sample()` drops its `Recorder` receivers (`crates/aura-cli/
+src/main.rs:161`); the demonstrator factors a `sample_blueprint_with_sinks() ->
+(Composite, Receiver, Receiver)` that returns them, so `run_one` can drain per
+point. The subcommand declares a small built-in grid, sweeps, and prints one JSON
+line per point:
+
+```rust
+// crates/aura-cli/src/main.rs (new arm, beside `run` and `graph`)
+fn cmd_sweep() {
+    let space = sample_blueprint_with_sinks().0.param_space();
+    let grid = GridSpace::new(&space, vec![
+        vec![Scalar::I64(2), Scalar::I64(3)],
+        vec![Scalar::I64(4), Scalar::I64(5)],
+        vec![Scalar::F64(0.5)],
+    ])
+    .expect("the built-in grid matches the sample param-space");
+    let prices = synthetic_prices();
+    let family = sweep(&grid, |point| {
+        let (bp, rx_eq, rx_ex) = sample_blueprint_with_sinks();
+        let mut h = bp.bootstrap_with_params(point.to_vec()).expect("kind-checked point");
+        h.run(vec![prices.clone()]);
+        let equity = f64_field(&rx_eq.try_iter().collect::<Vec<_>>(), 0);
+        let exposure = f64_field(&rx_ex.try_iter().collect::<Vec<_>>(), 0);
+        summarize(&equity, &exposure)
+    });
+    for (pt, ps) in family.points.iter().zip(std::iter::repeat(&space)) {
+        println!("{}", sweep_point_to_json(pt, ps));   // hand-rolled JSON (C14)
+    }
+}
+```
+
+`sweep_point_to_json` renders `{"params":{name:value,…},"metrics":{…}}` with the
+param names from `param_space()` zipped onto the point vector, reusing the
+canonical `f64`/`i64` token rules already established for `RunReport::to_json`
+(`crates/aura-engine/src/report.rs:80`). The exact byte format is the planner's
+to pin against a golden test.
+
+## Components
+
+| Crate / file | Change |
+|---|---|
+| `crates/aura-engine/src/sweep.rs` (new) | `GridSpace` (`new` / `len` / `is_empty` / `points`), `SweepPoint`, `SweepFamily`, `SweepError`, and the free `sweep()` with the `std::thread::scope` execution. The only new module this cycle. |
+| `crates/aura-engine/src/lib.rs` | `pub mod sweep;` + re-export `sweep`, `GridSpace`, `SweepError`, `SweepFamily`, `SweepPoint`. |
+| `crates/aura-cli/src/main.rs` (iteration 2) | Factor `sample_blueprint_with_sinks() -> (Composite, Receiver, Receiver)` (returns the receivers `build_sample` drops); add the `sweep` subcommand arg parsing + `cmd_sweep()` + `sweep_point_to_json`. |
+
+No change to `Node`, the run loop, `bootstrap_with_params`, `param_space`, or
+`RunMetrics`/`summarize` — the sweep is a new layer *over* the unchanged #30/#31
+primitives.
+
+## Data flow
+
+```
+author: build_blueprint() factory  +  per-slot value-lists
+  │  GridSpace::new(bp.param_space(), axes)
+  │     ├─ axes.len() == space.len()        else SweepError::Arity
+  │     ├─ each axis non-empty               else SweepError::EmptyAxis { slot }
+  │     └─ each value.kind() == slot.kind    else SweepError::KindMismatch { slot, value_index, .. }
+  ▼  space.points()  ->  cartesian product, odometer order (last axis fastest), deterministic
+  ▼  sweep(&space, run_one):
+       cursor = AtomicUsize(0)
+       std::thread::scope over min(available_parallelism(), npoints) workers:
+         worker loop: i = cursor.fetch_add(1); if i >= npoints break
+                      metrics = run_one(&points[i])         ── disjoint, lock-free (C1)
+                        └ build fresh bp (fresh channels)
+                          -> bootstrap_with_params(points[i])  (#31, kind-checked -> .expect honest)
+                          -> run(prices)                        (#3 shared read-only window)
+                          -> drain sinks -> summarize           (#9 RunMetrics)
+                      local.push((i, metrics))
+       join all locals -> sort_by_key(i)                     ── order = enumeration order, not completion
+  ▼  SweepFamily { points: [SweepPoint { params: points[i], metrics }, … ] }
+C1: same grid -> same family, bit-identical, thread-count-independent.
+```
+
+## Error handling
+
+`GridSpace::new` is the **typed gate** — it rejects a malformed grid before any
+run, mirroring #31's `CompileError::{ParamKindMismatch, ParamArity}` discipline
+at the enumeration layer:
+
+- `SweepError::Arity { expected, got }` — the axis count does not equal
+  `param_space().len()`.
+- `SweepError::KindMismatch { slot, value_index, expected, got }` — a grid
+  value's `ScalarKind` does not match the slot's declared kind. `slot` is the
+  flat param-space index; `value_index` is the position within that axis.
+- `SweepError::EmptyAxis { slot }` — a slot was given no values (an empty axis
+  collapses the cartesian product to zero points; an author mistake, caught
+  rather than silently producing an empty family).
+
+`run_one` returns `RunMetrics` directly (no `Result`). The grid's points are
+kind-checked at `GridSpace::new`, and `bootstrap_with_params`'s arity matches
+`param_space()` by construction, so the `bootstrap_with_params(…).expect(…)`
+inside the closure is the honest gate — exactly how `run_sample` treats it today
+(`crates/aura-cli/src/main.rs`). The **value domain** (e.g. `length >= 1`) stays
+the node constructor's own `assert` (`Sma::new`); the grid author is responsible
+for in-domain values this iteration — consistent with #31's deliberate cut, where
+the search-range / valid domain belongs to the run (C20), and full domain
+validation is a later cycle's concern. The sweep does **not** catch a panicking
+point this iteration (a panic in `run_one` propagates through `scope`); introducing
+search-range validation or panic recovery here would be scope creep.
+
+## Testing strategy
+
+All in `crates/aura-engine/src/sweep.rs` `#[cfg(test)]`, except the CLI golden
+(iteration 2).
+
+**Enumeration (`GridSpace`).**
+- `points()` over `[[2,3],[4,5]]` is exactly `[[2,4],[2,5],[3,4],[3,5]]` — odometer
+  order, last axis fastest.
+- `len()` equals `∏ |axis_i|` (e.g. `2 × 2 × 1 == 4`); a single-axis and a
+  three-axis grid both enumerate correctly.
+
+**Faults (`GridSpace::new`).**
+- Wrong axis count → `SweepError::Arity { expected, got }`.
+- A value of the wrong kind (e.g. `F64` where the slot is `I64`) →
+  `SweepError::KindMismatch { slot, value_index, .. }`.
+- An empty axis → `SweepError::EmptyAxis { slot }`.
+
+**Sweep == N independent runs (central, C1).** Build a value-empty SMA-cross
+two-sink blueprint fixture (the existing `report.rs` / `blueprint.rs` two-sink
+pattern, value-empty). For a grid over the SMA lengths, each family point's
+`metrics` equals `run_one` called **directly** on that same point vector — the
+sweep adds enumeration + execution, never a metrics change.
+
+**Determinism under concurrency (C1).** The same grid swept twice yields an
+identical `SweepFamily` (`PartialEq`). The family is identical whether the sweep
+ran on one worker or many — pinned by comparing a sweep forced to a single point
+of parallelism against the default multi-worker run (e.g. a grid large enough to
+spread across workers), asserting equal families. Order is the enumeration order
+regardless of scheduling.
+
+**Distinct points produce distinct, populated metrics.** Over `fast ∈ {2,3}`,
+`slow ∈ {4,5}`, `scale ∈ {0.5}` (4 points), every point has a finite
+`total_pips`, and points whose lengths differ produce differing metrics — the
+family is not a constant.
+
+**CLI demonstrator (iteration 2).** A golden test on `aura sweep` stdout: four
+JSON lines, one per grid point, in odometer order, each
+`{"params":{…},"metrics":{…}}` with names from `param_space()`. Asserted as the
+deterministic contract (the JSON model, like `RunReport::to_json`).
+
+## Acceptance criteria
+
+The feature passes aura's acceptance criterion (CLAUDE.md):
+
+- **The intended author reaches for exactly this.** The World / a sweep is the
+  milestone's intended author, and the worked program above is precisely "one
+  blueprint, a grid of points, a family of metrics" — the C12.1 grid axis. It is
+  the first "N instead of 1", which #30/#31 set up but could not express.
+- **It measurably enables the next cycle.** #33 (the run registry) indexes and
+  compares a *sweep family*; without `sweep()` producing a `SweepFamily`, #33 has
+  nothing to index. This cycle is its precondition.
+- **It reintroduces no forbidden failure class.** Determinism (C1) is preserved —
+  the same grid yields a bit-identical family, thread-count-independent (disjoint,
+  lock-free; the assembled order is the enumeration order). No look-ahead is
+  introduced (each point is an ordinary `bootstrap_with_params` + `run`).
+  Topology is param-invariant (C19 — the grid varies values, never structure).
+  Zero-external-dependency (C16) holds — execution is `std::thread::scope`, not
+  `rayon`.
+
+Concretely, the cycle is accepted when:
+
+1. `GridSpace::new` validates arity / kind / non-empty and `points()` enumerates
+   the cartesian product in deterministic odometer order.
+2. `sweep()` runs a value-empty SMA-cross blueprint over a grid and returns a
+   `SweepFamily` whose per-point metrics each equal the single-run metrics for
+   that point.
+3. The family is bit-identical across repeated runs and across thread counts
+   (determinism under concurrency, C1).
+4. `aura sweep` prints the family as one JSON line per point in odometer order
+   (iteration 2).
+5. `cargo build/test --workspace` is green and `cargo clippy --workspace
+   --all-targets -- -D warnings` is clean.
+
+## Out of scope (deferred)
+
+- **Random enumeration** (per-slot ranges + sample-count + seed + hand-rolled
+  deterministic RNG, C12.1's other half) — a follow-up issue filed at cycle
+  close.
+- **Per-point `RunManifest` assembly** (commit / window / seed / broker per
+  point) and indexing the family — #33 (the run registry). This cycle's family
+  carries `params + metrics` only; the manifest constants are sweep-level, not
+  per-point.
+- **Value-domain / search-range validation** beyond the node constructor's own
+  `assert` — C20 (the range belongs to the run); a later cycle.
+- **Optimize (argmax), walk-forward, Monte-Carlo** — C12.2 / C12.3 / C12.4, later
+  milestones.
+- Any change to `Node::eval`, the run loop, `bootstrap_with_params`, or
+  `param_space`.