diff --git a/docs/specs/0009-run-metrics-and-manifest.md b/docs/specs/0009-run-metrics-and-manifest.md new file mode 100644 index 0000000..b2eaee2 --- /dev/null +++ b/docs/specs/0009-run-metrics-and-manifest.md @@ -0,0 +1,289 @@ +# Run metrics + run manifest — Design Spec + +**Date:** 2026-06-04 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +> Cycle 0009. Tracker: Gitea issue #6 (Brummel/Aura), milestone *Walking +> skeleton*. Unblocks issue #8 (`aura run` CLI), which wires the surface this +> cycle delivers into an actual subcommand. + +## Goal + +A run must produce the two artefacts C18 mandates "per run from day one": a +small **manifest** (the reproducible descriptor — node identity/commit, +param-set, data-window, seed, broker profile) and **summary metrics** reduced +from the run's recorded streams (at least total pips, peak-to-trough drawdown, +and an exposure sign-flip count as a turnover proxy). Determinism (C1/C12) makes +the run reproducible from the manifest, so the pair `(manifest, metrics)` is the +durable run record; the full result is re-derivable on demand. + +This cycle delivers the **library surface** for that pair — typed `RunManifest` +/ `RunMetrics` / `RunReport` values, a pure `summarize` reduction over recorded +streams, and canonical-JSON rendering for the structured (C14) face — plus an +end-to-end test that runs a harness and produces the pair. Printing it from an +actual `aura run` subcommand is the next cycle (#8); this cycle stops at the +library surface and its demonstrating test. + +## Non-goals (out of scope) + +- The `aura run` subcommand / any CLI wiring — that is #8. This cycle's + end-to-end demonstration is a harness **test**, not a binary. +- The queryable run **registry**, lineage, run-diff, promotion/status — the + later C18 milestone *over* manifests that already exist. This cycle emits the + manifest+metrics; it does not store or index them. +- A typed/ranged **param-space** surface. `node.rs` deliberately defers tunable + params to the blueprint/optimizer layer (settled in the cycle-0008 audit). The + manifest records the params the harness author bound as plain `name → value` + pairs; it does not introduce the optimizer-facing param-space type. +- Any new engine mechanism. `Harness` / the run loop / the node contract are + unchanged; this is a pure-additive consumer surface over the recorded streams + the cycle-0006 sink mechanism already produces. + +## Architecture + +The reduction is a **post-run pure function over recorded streams, not a node.** +A node emits at most one record per `eval` and there is no end-of-run `eval` +(C8), so an end-of-run reduction has no home as a node. The pipeline is exactly +the one C18 describes: + +``` +recording sink (cycle 0006) captures a stream out of the graph + │ Harness::run(...) -> () [unchanged] + ▼ +World/runner drains the sink's channel -> Vec<(Timestamp, Vec)> + │ f64_field(rows, 0) + ▼ +(Timestamp, f64) samples ──► summarize(equity, exposure) -> RunMetrics + │ + RunManifest (caller-supplied) ─┴─► RunReport { manifest, metrics } + │ to_json() + ▼ + canonical JSON (C14) +``` + +`summarize` is total and pure: identical recorded streams → identical metrics, +which is the `C1/C12` reproducibility acceptance (the surrounding run is already +bit-identical under C1, and a pure reduction preserves that). + +**Home:** a new `report` module in `aura-engine`, beside `Harness`. Not +`aura-core`: a hot-reloadable node cdylib never constructs a manifest or reduces +a run — only the World/runner does, and the World links the engine, not just the +core contract. `aura-cli` (which depends on `aura-engine`) renders the JSON to +stdout in #8. + +## Concrete code shapes + +### Delivered library surface (`aura-engine/src/report.rs`) + +```rust +use aura_core::{Scalar, Timestamp}; + +/// Summary metrics reduced from a run's recorded streams — the "-> metrics" +/// half of C12's atomic sim unit. Pure function of the recorded streams. +#[derive(Clone, Debug, PartialEq)] +pub struct RunMetrics { + /// Final cumulative pip equity — the last value of the (cumulative) + /// pip-equity curve. 0.0 if the curve is empty. + pub total_pips: f64, + /// Largest peak-to-trough drop on the cumulative pip curve: + /// `max_t (running_peak(t) - equity(t))`, always >= 0.0 (0.0 if the curve + /// is monotonic non-decreasing or empty). + pub max_drawdown: f64, + /// Count of adjacent recorded exposure samples whose `signum` differs + /// (`signum(0.0) == 0.0`) — a turnover proxy. Counts long<->short + /// reversals AND transitions into/out of flat; it is the plain + /// sign-change count over the recorded exposure series, not a + /// reversal-only count. + pub exposure_sign_flips: u64, +} + +/// The reproducible run descriptor (C18). **Caller-supplied**: the engine +/// cannot introspect a git commit, an RNG seed, or a broker label — the World +/// that bootstraps and runs the harness holds those and fills them in. +#[derive(Clone, Debug, PartialEq)] +pub struct RunManifest { + /// Node/engine identity: the git commit of the frozen artifact + /// (C18 — commit = identity; the frozen bot *is* a commit). + pub commit: String, + /// The bound tuning params as ordered `name -> value` pairs. Precursor to + /// the eventual typed param-space (deferred — see Non-goals); honest about + /// what exists today. + pub params: Vec<(String, f64)>, + /// The data-window: inclusive `(from, to)` epoch-ns bounds (C12). + pub window: (Timestamp, Timestamp), + /// The RNG seed (C12 seed-as-input). 0 for a seed-free synthetic run. + pub seed: u64, + /// The broker profile label, e.g. `"sim-optimal(pip_size=1.0)"`. + pub broker: String, +} + +/// A run's full structured result: the descriptor plus the metrics it +/// reproduces. The durable run record of C18 ("stores manifests + metrics, +/// re-derives full results on demand"). +#[derive(Clone, Debug, PartialEq)] +pub struct RunReport { + pub manifest: RunManifest, + pub metrics: RunMetrics, +} + +impl RunReport { + /// Render canonical, machine-readable JSON (C14). Hand-rolled: the schema + /// is tiny, closed, and flat, so the deliberately zero-dependency workspace + /// stays zero-dependency. Field order is fixed; `f64` is rendered with the + /// round-trippable `{}` shortest form (finite values only — pip equity and + /// exposure are always finite). `params` renders as a JSON object in + /// insertion order. + pub fn to_json(&self) -> String { /* shape; planner owns bytes */ } +} + +/// Reduce a run's recorded pip-equity + exposure streams into summary metrics. +/// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are +/// carried in the input to match exactly what a sink records; the reduction +/// itself is value-only (it does not read the timestamps). +pub fn summarize( + equity: &[(Timestamp, f64)], + exposure: &[(Timestamp, f64)], +) -> RunMetrics { /* shape; planner owns bytes */ } + +/// Bridge a sink's recorded `(ts, row)` stream to `summarize`: extract one +/// `f64` field of each row into `(ts, f64)` samples. Panics if a row is too +/// short or the field is not an `f64` scalar — a wiring bug (the sink's +/// declared kinds are fixed at bootstrap), reported like the engine's other +/// "kind checked at wiring" contract violations. +pub fn f64_field( + rows: &[(Timestamp, Vec)], + field: usize, +) -> Vec<(Timestamp, f64)> { /* shape; planner owns bytes */ } +``` + +### North-star slice — what the World/runner author writes (this is #8's seam) + +Shown minimal and honest: the harness wiring (SMA fast/slow → `Sub` → `Exposure` +→ `SimBroker`, with a price tap into the broker) is the cycle-0007 pattern and +is **not** re-specified here; the new bytes are the drain → reduce → report tail. + +```rust +// ... bootstrap the harness with two recording sinks: one on the SimBroker +// equity output, one on the Exposure output; run it ... +h.run(streams); + +let equity = f64_field(&equity_rows, 0); // drained from the equity sink +let exposure = f64_field(&exposure_rows, 0); // drained from the exposure sink +let metrics = summarize(&equity, &exposure); + +let report = RunReport { + manifest: RunManifest { + commit: env!("AURA_COMMIT").to_string(), + params: vec![ + ("sma_fast".into(), 2.0), + ("sma_slow".into(), 4.0), + ("exposure_scale".into(), 1.0), + ], + window: (Timestamp(1), Timestamp(6)), + seed: 0, + broker: "sim-optimal(pip_size=1.0)".into(), + }, + metrics, +}; +println!("{}", report.to_json()); // the structured C14 face (lands in #8) +``` + +### Canonical JSON shape (one line; whitespace fixed) + +```json +{"manifest":{"commit":"abc123","params":{"sma_fast":2,"sma_slow":4,"exposure_scale":1},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=1.0)"},"metrics":{"total_pips":12,"max_drawdown":1,"exposure_sign_flips":1}} +``` + +## Components + +- **`RunMetrics`** — the three summary numbers. Plain data; `PartialEq` so tests + assert exact equality and determinism. +- **`RunManifest`** — the reproducible descriptor; all fields caller-supplied. +- **`RunReport`** — the `(manifest, metrics)` pair + `to_json`. +- **`summarize(equity, exposure)`** — the pure reduction. `total_pips` = last + equity value; `max_drawdown` = running-peak minus value, maximized; + `exposure_sign_flips` = adjacent-`signum`-change count. +- **`f64_field(rows, field)`** — the recorded-`Vec`-row → `(ts, f64)` + adapter; the one place that knows a recorded row is `Scalar`-typed. + +## Data flow + +The cycle-0006 sink mechanism is unchanged: a recording node pushes +`(ctx.now(), row)` out of the graph each fired cycle. After `Harness::run` +returns, the World holds each sink's drained `Vec<(Timestamp, Vec)>`. +`f64_field` projects the single `f64` column (field 0) of the equity stream and +of the exposure stream; `summarize` folds the two projections into `RunMetrics`; +the World pairs it with a `RunManifest` it constructed and renders `to_json`. No +data flows back into the graph; the reduction is strictly downstream of a +completed run. + +## Error handling + +- `summarize` is **total**: empty streams → `total_pips = 0.0`, + `max_drawdown = 0.0`, `exposure_sign_flips = 0`. No panics, no `Result`. +- `to_json` is **total**: it writes to a `String`; finite-float assumption is + documented (pip equity / exposure are finite by construction — `SimBroker` + integrates finite returns, `Exposure` clamps to `[-1, +1]`). +- `f64_field` **panics** on a row shorter than `field` or a non-`f64` scalar at + `field`: that is a wiring bug (a sink's declared kinds are fixed at bootstrap, + so a correctly-wired equity/exposure sink always yields `f64` at field 0), + surfaced exactly like the engine's existing `expect("… checked at wiring")` + contract violations rather than silently dropped. + +## Testing strategy + +Unit tests in `report.rs` (`aura-engine`): + +- `summarize` total_pips = last cumulative value; 0.0 on empty. +- `summarize` max_drawdown on a non-monotonic curve (up → down → up): asserts + the worst peak-to-trough drop, not the final drop; 0.0 on a monotonic curve. +- `summarize` exposure_sign_flips on a crafted series + (`+ + - 0 -` → flips at `+→-`, `-→0`, `0→-` = 3) pinning the documented + signum-change definition; 0 on a constant-sign series. +- `to_json` snapshot: a known `RunReport` renders the exact canonical string + above (field order, JSON number forms, `params`-as-object). +- `f64_field` projects field 0 of multi-field rows; panics on a kind mismatch + (a `#[should_panic]` test). + +End-to-end determinism test (the C18 / acceptance demonstrator), reusing the +cycle-0007 harness shape (`Sma` fast/slow → `Sub` → `Exposure` → `SimBroker`, +price tapped into the broker) with two recording sinks: + +- build + run the harness twice on identical synthetic input; +- drain, `f64_field`, `summarize` each run; +- assert the two `RunMetrics` are bit-identical and the two `to_json` strings are + identical — "same manifest → same metrics" (C1/C12). + +Gates (profile `commands`): `cargo test --workspace`, +`cargo clippy --workspace --all-targets -- -D warnings`, +`RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps`. + +## Load-bearing decision flagged for review + +**Structured output is hand-rolled canonical JSON, zero-dependency — not serde.** +The workspace today has *zero* external dependencies by deliberate policy +("Add crates only when a boundary proves real"). The manifest+metrics schema is +tiny, closed, and flat, so a hand-written `to_json` satisfies C14 +(machine-readable, stable, parseable) without pulling `serde`/`serde_json` and +breaking that stance. serde is the natural choice *later*, when the run-registry +milestone (C18) grows a larger, evolving, possibly-deserialized schema — that is +an effort/scope judgement deferred to when the boundary is real, not a +correctness need now. If you would rather adopt serde now (e.g. you expect the +schema to churn immediately, or want `#[derive(Serialize)]` ergonomics from day +one), say so at review and the spec flips to a serde-based `to_json`. + +## Acceptance criteria + +- [ ] A run produces `RunMetrics` with total pips, max drawdown, and exposure + sign-flips, reduced from its recorded streams (`summarize`). +- [ ] A run produces a `RunManifest` carrying commit, params, data-window, seed, + and broker profile. +- [ ] `RunReport::to_json` emits structured, machine-readable JSON (C14) in a + fixed canonical form. +- [ ] An end-to-end test runs a harness twice and asserts identical metrics + + identical JSON — same manifest → same metrics (C1/C12). +- [ ] No engine/`Harness`/node-contract change; the surface is pure-additive in + a new `aura-engine` `report` module; the workspace stays zero-dependency. +- [ ] `cargo test --workspace`, clippy `-D warnings`, and `cargo doc -D warnings` + are clean.