spec: 0009 run metrics + manifest

Cycle 0009 (Walking skeleton milestone, issue #6): a post-run pure-function reduction over the recorded pip-equity + exposure streams into summary metrics (total pips, peak-to-trough drawdown, exposure sign-flip count) plus a caller-supplied RunManifest (commit, params, data-window, seed, broker), paired as RunReport with canonical hand-rolled zero-dep JSON (C14). Lives in a new aura-engine report module; no engine/Harness/node-contract change. The reduction is post-run, not a node (C8 caps a node at one record per eval with no end-of-run eval); matches C18's manifest+metrics-per-run-from-day-one. The `aura run` subcommand that prints the report is deferred to #8. Grounding-check PASS (all load-bearing assumptions ratified by green tests / authoritative sources). Spec-validation parse pass is a documented no-op (no parser configured for the profile). refs #6
2026-06-04 18:42:55 +02:00
parent 521a16e56f
commit bf72924db7
1 changed files with 289 additions and 0 deletions
@@ -0,0 +1,289 @@
+# Run metrics + run manifest — Design Spec
+
+**Date:** 2026-06-04
+**Status:** Draft — awaiting user spec review
+**Authors:** orchestrator + Claude
+
+> Cycle 0009. Tracker: Gitea issue #6 (Brummel/Aura), milestone *Walking
+> skeleton*. Unblocks issue #8 (`aura run` CLI), which wires the surface this
+> cycle delivers into an actual subcommand.
+
+## Goal
+
+A run must produce the two artefacts C18 mandates "per run from day one": a
+small **manifest** (the reproducible descriptor — node identity/commit,
+param-set, data-window, seed, broker profile) and **summary metrics** reduced
+from the run's recorded streams (at least total pips, peak-to-trough drawdown,
+and an exposure sign-flip count as a turnover proxy). Determinism (C1/C12) makes
+the run reproducible from the manifest, so the pair `(manifest, metrics)` is the
+durable run record; the full result is re-derivable on demand.
+
+This cycle delivers the **library surface** for that pair — typed `RunManifest`
+/ `RunMetrics` / `RunReport` values, a pure `summarize` reduction over recorded
+streams, and canonical-JSON rendering for the structured (C14) face — plus an
+end-to-end test that runs a harness and produces the pair. Printing it from an
+actual `aura run` subcommand is the next cycle (#8); this cycle stops at the
+library surface and its demonstrating test.
+
+## Non-goals (out of scope)
+
+- The `aura run` subcommand / any CLI wiring — that is #8. This cycle's
+  end-to-end demonstration is a harness **test**, not a binary.
+- The queryable run **registry**, lineage, run-diff, promotion/status — the
+  later C18 milestone *over* manifests that already exist. This cycle emits the
+  manifest+metrics; it does not store or index them.
+- A typed/ranged **param-space** surface. `node.rs` deliberately defers tunable
+  params to the blueprint/optimizer layer (settled in the cycle-0008 audit). The
+  manifest records the params the harness author bound as plain `name → value`
+  pairs; it does not introduce the optimizer-facing param-space type.
+- Any new engine mechanism. `Harness` / the run loop / the node contract are
+  unchanged; this is a pure-additive consumer surface over the recorded streams
+  the cycle-0006 sink mechanism already produces.
+
+## Architecture
+
+The reduction is a **post-run pure function over recorded streams, not a node.**
+A node emits at most one record per `eval` and there is no end-of-run `eval`
+(C8), so an end-of-run reduction has no home as a node. The pipeline is exactly
+the one C18 describes:
+
+```
+recording sink (cycle 0006) captures a stream out of the graph
+        │  Harness::run(...) -> ()   [unchanged]
+        ▼
+World/runner drains the sink's channel -> Vec<(Timestamp, Vec<Scalar>)>
+        │  f64_field(rows, 0)
+        ▼
+(Timestamp, f64) samples  ──►  summarize(equity, exposure) -> RunMetrics
+                                        │
+        RunManifest (caller-supplied) ─┴─► RunReport { manifest, metrics }
+                                                  │ to_json()
+                                                  ▼
+                                          canonical JSON (C14)
+```
+
+`summarize` is total and pure: identical recorded streams → identical metrics,
+which is the `C1/C12` reproducibility acceptance (the surrounding run is already
+bit-identical under C1, and a pure reduction preserves that).
+
+**Home:** a new `report` module in `aura-engine`, beside `Harness`. Not
+`aura-core`: a hot-reloadable node cdylib never constructs a manifest or reduces
+a run — only the World/runner does, and the World links the engine, not just the
+core contract. `aura-cli` (which depends on `aura-engine`) renders the JSON to
+stdout in #8.
+
+## Concrete code shapes
+
+### Delivered library surface (`aura-engine/src/report.rs`)
+
+```rust
+use aura_core::{Scalar, Timestamp};
+
+/// Summary metrics reduced from a run's recorded streams — the "-> metrics"
+/// half of C12's atomic sim unit. Pure function of the recorded streams.
+#[derive(Clone, Debug, PartialEq)]
+pub struct RunMetrics {
+    /// Final cumulative pip equity — the last value of the (cumulative)
+    /// pip-equity curve. 0.0 if the curve is empty.
+    pub total_pips: f64,
+    /// Largest peak-to-trough drop on the cumulative pip curve:
+    /// `max_t (running_peak(t) - equity(t))`, always >= 0.0 (0.0 if the curve
+    /// is monotonic non-decreasing or empty).
+    pub max_drawdown: f64,
+    /// Count of adjacent recorded exposure samples whose `signum` differs
+    /// (`signum(0.0) == 0.0`) — a turnover proxy. Counts long<->short
+    /// reversals AND transitions into/out of flat; it is the plain
+    /// sign-change count over the recorded exposure series, not a
+    /// reversal-only count.
+    pub exposure_sign_flips: u64,
+}
+
+/// The reproducible run descriptor (C18). **Caller-supplied**: the engine
+/// cannot introspect a git commit, an RNG seed, or a broker label — the World
+/// that bootstraps and runs the harness holds those and fills them in.
+#[derive(Clone, Debug, PartialEq)]
+pub struct RunManifest {
+    /// Node/engine identity: the git commit of the frozen artifact
+    /// (C18 — commit = identity; the frozen bot *is* a commit).
+    pub commit: String,
+    /// The bound tuning params as ordered `name -> value` pairs. Precursor to
+    /// the eventual typed param-space (deferred — see Non-goals); honest about
+    /// what exists today.
+    pub params: Vec<(String, f64)>,
+    /// The data-window: inclusive `(from, to)` epoch-ns bounds (C12).
+    pub window: (Timestamp, Timestamp),
+    /// The RNG seed (C12 seed-as-input). 0 for a seed-free synthetic run.
+    pub seed: u64,
+    /// The broker profile label, e.g. `"sim-optimal(pip_size=1.0)"`.
+    pub broker: String,
+}
+
+/// A run's full structured result: the descriptor plus the metrics it
+/// reproduces. The durable run record of C18 ("stores manifests + metrics,
+/// re-derives full results on demand").
+#[derive(Clone, Debug, PartialEq)]
+pub struct RunReport {
+    pub manifest: RunManifest,
+    pub metrics: RunMetrics,
+}
+
+impl RunReport {
+    /// Render canonical, machine-readable JSON (C14). Hand-rolled: the schema
+    /// is tiny, closed, and flat, so the deliberately zero-dependency workspace
+    /// stays zero-dependency. Field order is fixed; `f64` is rendered with the
+    /// round-trippable `{}` shortest form (finite values only — pip equity and
+    /// exposure are always finite). `params` renders as a JSON object in
+    /// insertion order.
+    pub fn to_json(&self) -> String { /* shape; planner owns bytes */ }
+}
+
+/// Reduce a run's recorded pip-equity + exposure streams into summary metrics.
+/// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are
+/// carried in the input to match exactly what a sink records; the reduction
+/// itself is value-only (it does not read the timestamps).
+pub fn summarize(
+    equity: &[(Timestamp, f64)],
+    exposure: &[(Timestamp, f64)],
+) -> RunMetrics { /* shape; planner owns bytes */ }
+
+/// Bridge a sink's recorded `(ts, row)` stream to `summarize`: extract one
+/// `f64` field of each row into `(ts, f64)` samples. Panics if a row is too
+/// short or the field is not an `f64` scalar — a wiring bug (the sink's
+/// declared kinds are fixed at bootstrap), reported like the engine's other
+/// "kind checked at wiring" contract violations.
+pub fn f64_field(
+    rows: &[(Timestamp, Vec<Scalar>)],
+    field: usize,
+) -> Vec<(Timestamp, f64)> { /* shape; planner owns bytes */ }
+```
+
+### North-star slice — what the World/runner author writes (this is #8's seam)
+
+Shown minimal and honest: the harness wiring (SMA fast/slow → `Sub` → `Exposure`
+→ `SimBroker`, with a price tap into the broker) is the cycle-0007 pattern and
+is **not** re-specified here; the new bytes are the drain → reduce → report tail.
+
+```rust
+// ... bootstrap the harness with two recording sinks: one on the SimBroker
+// equity output, one on the Exposure output; run it ...
+h.run(streams);
+
+let equity   = f64_field(&equity_rows,   0); // drained from the equity sink
+let exposure = f64_field(&exposure_rows, 0); // drained from the exposure sink
+let metrics  = summarize(&equity, &exposure);
+
+let report = RunReport {
+    manifest: RunManifest {
+        commit: env!("AURA_COMMIT").to_string(),
+        params: vec![
+            ("sma_fast".into(), 2.0),
+            ("sma_slow".into(), 4.0),
+            ("exposure_scale".into(), 1.0),
+        ],
+        window: (Timestamp(1), Timestamp(6)),
+        seed: 0,
+        broker: "sim-optimal(pip_size=1.0)".into(),
+    },
+    metrics,
+};
+println!("{}", report.to_json()); // the structured C14 face (lands in #8)
+```
+
+### Canonical JSON shape (one line; whitespace fixed)
+
+```json
+{"manifest":{"commit":"abc123","params":{"sma_fast":2,"sma_slow":4,"exposure_scale":1},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=1.0)"},"metrics":{"total_pips":12,"max_drawdown":1,"exposure_sign_flips":1}}
+```
+
+## Components
+
+- **`RunMetrics`** — the three summary numbers. Plain data; `PartialEq` so tests
+  assert exact equality and determinism.
+- **`RunManifest`** — the reproducible descriptor; all fields caller-supplied.
+- **`RunReport`** — the `(manifest, metrics)` pair + `to_json`.
+- **`summarize(equity, exposure)`** — the pure reduction. `total_pips` = last
+  equity value; `max_drawdown` = running-peak minus value, maximized;
+  `exposure_sign_flips` = adjacent-`signum`-change count.
+- **`f64_field(rows, field)`** — the recorded-`Vec<Scalar>`-row → `(ts, f64)`
+  adapter; the one place that knows a recorded row is `Scalar`-typed.
+
+## Data flow
+
+The cycle-0006 sink mechanism is unchanged: a recording node pushes
+`(ctx.now(), row)` out of the graph each fired cycle. After `Harness::run`
+returns, the World holds each sink's drained `Vec<(Timestamp, Vec<Scalar>)>`.
+`f64_field` projects the single `f64` column (field 0) of the equity stream and
+of the exposure stream; `summarize` folds the two projections into `RunMetrics`;
+the World pairs it with a `RunManifest` it constructed and renders `to_json`. No
+data flows back into the graph; the reduction is strictly downstream of a
+completed run.
+
+## Error handling
+
+- `summarize` is **total**: empty streams → `total_pips = 0.0`,
+  `max_drawdown = 0.0`, `exposure_sign_flips = 0`. No panics, no `Result`.
+- `to_json` is **total**: it writes to a `String`; finite-float assumption is
+  documented (pip equity / exposure are finite by construction — `SimBroker`
+  integrates finite returns, `Exposure` clamps to `[-1, +1]`).
+- `f64_field` **panics** on a row shorter than `field` or a non-`f64` scalar at
+  `field`: that is a wiring bug (a sink's declared kinds are fixed at bootstrap,
+  so a correctly-wired equity/exposure sink always yields `f64` at field 0),
+  surfaced exactly like the engine's existing `expect("… checked at wiring")`
+  contract violations rather than silently dropped.
+
+## Testing strategy
+
+Unit tests in `report.rs` (`aura-engine`):
+
+- `summarize` total_pips = last cumulative value; 0.0 on empty.
+- `summarize` max_drawdown on a non-monotonic curve (up → down → up): asserts
+  the worst peak-to-trough drop, not the final drop; 0.0 on a monotonic curve.
+- `summarize` exposure_sign_flips on a crafted series
+  (`+ + - 0 -` → flips at `+→-`, `-→0`, `0→-` = 3) pinning the documented
+  signum-change definition; 0 on a constant-sign series.
+- `to_json` snapshot: a known `RunReport` renders the exact canonical string
+  above (field order, JSON number forms, `params`-as-object).
+- `f64_field` projects field 0 of multi-field rows; panics on a kind mismatch
+  (a `#[should_panic]` test).
+
+End-to-end determinism test (the C18 / acceptance demonstrator), reusing the
+cycle-0007 harness shape (`Sma` fast/slow → `Sub` → `Exposure` → `SimBroker`,
+price tapped into the broker) with two recording sinks:
+
+- build + run the harness twice on identical synthetic input;
+- drain, `f64_field`, `summarize` each run;
+- assert the two `RunMetrics` are bit-identical and the two `to_json` strings are
+  identical — "same manifest → same metrics" (C1/C12).
+
+Gates (profile `commands`): `cargo test --workspace`,
+`cargo clippy --workspace --all-targets -- -D warnings`,
+`RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps`.
+
+## Load-bearing decision flagged for review
+
+**Structured output is hand-rolled canonical JSON, zero-dependency — not serde.**
+The workspace today has *zero* external dependencies by deliberate policy
+("Add crates only when a boundary proves real"). The manifest+metrics schema is
+tiny, closed, and flat, so a hand-written `to_json` satisfies C14
+(machine-readable, stable, parseable) without pulling `serde`/`serde_json` and
+breaking that stance. serde is the natural choice *later*, when the run-registry
+milestone (C18) grows a larger, evolving, possibly-deserialized schema — that is
+an effort/scope judgement deferred to when the boundary is real, not a
+correctness need now. If you would rather adopt serde now (e.g. you expect the
+schema to churn immediately, or want `#[derive(Serialize)]` ergonomics from day
+one), say so at review and the spec flips to a serde-based `to_json`.
+
+## Acceptance criteria
+
+- [ ] A run produces `RunMetrics` with total pips, max drawdown, and exposure
+      sign-flips, reduced from its recorded streams (`summarize`).
+- [ ] A run produces a `RunManifest` carrying commit, params, data-window, seed,
+      and broker profile.
+- [ ] `RunReport::to_json` emits structured, machine-readable JSON (C14) in a
+      fixed canonical form.
+- [ ] An end-to-end test runs a harness twice and asserts identical metrics +
+      identical JSON — same manifest → same metrics (C1/C12).
+- [ ] No engine/`Harness`/node-contract change; the surface is pure-additive in
+      a new `aura-engine` `report` module; the workspace stays zero-dependency.
+- [ ] `cargo test --workspace`, clippy `-D warnings`, and `cargo doc -D warnings`
+      are clean.