# Run metrics + run manifest — Design Spec

**Date:** 2026-06-04
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

> Cycle 0009. Tracker: Gitea issue #6 (Brummel/Aura), milestone *Walking
> skeleton*. Unblocks issue #8 (`aura run` CLI), which wires the surface this
> cycle delivers into an actual subcommand.

## Goal

A run must produce the two artefacts C18 mandates "per run from day one": a
small **manifest** (the reproducible descriptor — node identity/commit,
param-set, data-window, seed, broker profile) and **summary metrics** reduced
from the run's recorded streams (at least total pips, peak-to-trough drawdown,
and an exposure sign-flip count as a turnover proxy). Determinism (C1/C12) makes
the run reproducible from the manifest, so the pair `(manifest, metrics)` is the
durable run record; the full result is re-derivable on demand.

This cycle delivers the **library surface** for that pair — typed `RunManifest`
/ `RunMetrics` / `RunReport` values, a pure `summarize` reduction over recorded
streams, and canonical-JSON rendering for the structured (C14) face — plus an
end-to-end test that runs a harness and produces the pair. Printing it from an
actual `aura run` subcommand is the next cycle (#8); this cycle stops at the
library surface and its demonstrating test.

## Non-goals (out of scope)

- The `aura run` subcommand / any CLI wiring — that is #8. This cycle's
  end-to-end demonstration is a harness **test**, not a binary.
- The queryable run **registry**, lineage, run-diff, promotion/status — the
  later C18 milestone *over* manifests that already exist. This cycle emits the
  manifest+metrics; it does not store or index them.
- A typed/ranged **param-space** surface. `node.rs` deliberately defers tunable
  params to the blueprint/optimizer layer (settled in the cycle-0008 audit). The
  manifest records the params the harness author bound as plain `name → value`
  pairs; it does not introduce the optimizer-facing param-space type.
- Any new engine mechanism. `Harness` / the run loop / the node contract are
  unchanged; this is a pure-additive consumer surface over the recorded streams
  the cycle-0006 sink mechanism already produces.

## Architecture

The reduction is a **post-run pure function over recorded streams, not a node.**
A node emits at most one record per `eval` and there is no end-of-run `eval`
(C8), so an end-of-run reduction has no home as a node. The pipeline is exactly
the one C18 describes:

```
recording sink (cycle 0006) captures a stream out of the graph
        │  Harness::run(...) -> ()   [unchanged]
        ▼
World/runner drains the sink's channel -> Vec<(Timestamp, Vec<Scalar>)>
        │  f64_field(rows, 0)
        ▼
(Timestamp, f64) samples  ──►  summarize(equity, exposure) -> RunMetrics
                                        │
        RunManifest (caller-supplied) ─┴─► RunReport { manifest, metrics }
                                                  │ to_json()
                                                  ▼
                                          canonical JSON (C14)
```

`summarize` is total and pure: identical recorded streams → identical metrics,
which is the `C1/C12` reproducibility acceptance (the surrounding run is already
bit-identical under C1, and a pure reduction preserves that).

**Home:** a new `report` module in `aura-engine`, beside `Harness`. Not
`aura-core`: a hot-reloadable node cdylib never constructs a manifest or reduces
a run — only the World/runner does, and the World links the engine, not just the
core contract. `aura-cli` (which depends on `aura-engine`) renders the JSON to
stdout in #8.

## Concrete code shapes

### Delivered library surface (`aura-engine/src/report.rs`)

```rust
use aura_core::{Scalar, Timestamp};

/// Summary metrics reduced from a run's recorded streams — the "-> metrics"
/// half of C12's atomic sim unit. Pure function of the recorded streams.
#[derive(Clone, Debug, PartialEq)]
pub struct RunMetrics {
    /// Final cumulative pip equity — the last value of the (cumulative)
    /// pip-equity curve. 0.0 if the curve is empty.
    pub total_pips: f64,
    /// Largest peak-to-trough drop on the cumulative pip curve:
    /// `max_t (running_peak(t) - equity(t))`, always >= 0.0 (0.0 if the curve
    /// is monotonic non-decreasing or empty).
    pub max_drawdown: f64,
    /// Count of adjacent recorded exposure samples whose `signum` differs
    /// (`signum(0.0) == 0.0`) — a turnover proxy. Counts long<->short
    /// reversals AND transitions into/out of flat; it is the plain
    /// sign-change count over the recorded exposure series, not a
    /// reversal-only count.
    pub exposure_sign_flips: u64,
}

/// The reproducible run descriptor (C18). **Caller-supplied**: the engine
/// cannot introspect a git commit, an RNG seed, or a broker label — the World
/// that bootstraps and runs the harness holds those and fills them in.
#[derive(Clone, Debug, PartialEq)]
pub struct RunManifest {
    /// Node/engine identity: the git commit of the frozen artifact
    /// (C18 — commit = identity; the frozen bot *is* a commit).
    pub commit: String,
    /// The bound tuning params as ordered `name -> value` pairs. Precursor to
    /// the eventual typed param-space (deferred — see Non-goals); honest about
    /// what exists today.
    pub params: Vec<(String, f64)>,
    /// The data-window: inclusive `(from, to)` epoch-ns bounds (C12).
    pub window: (Timestamp, Timestamp),
    /// The RNG seed (C12 seed-as-input). 0 for a seed-free synthetic run.
    pub seed: u64,
    /// The broker profile label, e.g. `"sim-optimal(pip_size=1.0)"`.
    pub broker: String,
}

/// A run's full structured result: the descriptor plus the metrics it
/// reproduces. The durable run record of C18 ("stores manifests + metrics,
/// re-derives full results on demand").
#[derive(Clone, Debug, PartialEq)]
pub struct RunReport {
    pub manifest: RunManifest,
    pub metrics: RunMetrics,
}

impl RunReport {
    /// Render canonical, machine-readable JSON (C14). Hand-rolled: the schema
    /// is tiny, closed, and flat, so the deliberately zero-dependency workspace
    /// stays zero-dependency. Field order is fixed; `f64` is rendered with the
    /// round-trippable `{}` shortest form (finite values only — pip equity and
    /// exposure are always finite). `params` renders as a JSON object in
    /// insertion order.
    pub fn to_json(&self) -> String { /* shape; planner owns bytes */ }
}

/// Reduce a run's recorded pip-equity + exposure streams into summary metrics.
/// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are
/// carried in the input to match exactly what a sink records; the reduction
/// itself is value-only (it does not read the timestamps).
pub fn summarize(
    equity: &[(Timestamp, f64)],
    exposure: &[(Timestamp, f64)],
) -> RunMetrics { /* shape; planner owns bytes */ }

/// Bridge a sink's recorded `(ts, row)` stream to `summarize`: extract one
/// `f64` field of each row into `(ts, f64)` samples. Panics if a row is too
/// short or the field is not an `f64` scalar — a wiring bug (the sink's
/// declared kinds are fixed at bootstrap), reported like the engine's other
/// "kind checked at wiring" contract violations.
pub fn f64_field(
    rows: &[(Timestamp, Vec<Scalar>)],
    field: usize,
) -> Vec<(Timestamp, f64)> { /* shape; planner owns bytes */ }
```

### North-star slice — what the World/runner author writes (this is #8's seam)

Shown minimal and honest: the harness wiring (SMA fast/slow → `Sub` → `Exposure`
→ `SimBroker`, with a price tap into the broker) is the cycle-0007 pattern and
is **not** re-specified here; the new bytes are the drain → reduce → report tail.

```rust
// ... bootstrap the harness with two recording sinks: one on the SimBroker
// equity output, one on the Exposure output; run it ...
h.run(streams);

let equity   = f64_field(&equity_rows,   0); // drained from the equity sink
let exposure = f64_field(&exposure_rows, 0); // drained from the exposure sink
let metrics  = summarize(&equity, &exposure);

let report = RunReport {
    manifest: RunManifest {
        commit: env!("AURA_COMMIT").to_string(),
        params: vec![
            ("sma_fast".into(), 2.0),
            ("sma_slow".into(), 4.0),
            ("exposure_scale".into(), 1.0),
        ],
        window: (Timestamp(1), Timestamp(6)),
        seed: 0,
        broker: "sim-optimal(pip_size=1.0)".into(),
    },
    metrics,
};
println!("{}", report.to_json()); // the structured C14 face (lands in #8)
```

### Canonical JSON shape (one line; whitespace fixed)

```json
{"manifest":{"commit":"abc123","params":{"sma_fast":2,"sma_slow":4,"exposure_scale":1},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=1.0)"},"metrics":{"total_pips":12,"max_drawdown":1,"exposure_sign_flips":1}}
```

## Components

- **`RunMetrics`** — the three summary numbers. Plain data; `PartialEq` so tests
  assert exact equality and determinism.
- **`RunManifest`** — the reproducible descriptor; all fields caller-supplied.
- **`RunReport`** — the `(manifest, metrics)` pair + `to_json`.
- **`summarize(equity, exposure)`** — the pure reduction. `total_pips` = last
  equity value; `max_drawdown` = running-peak minus value, maximized;
  `exposure_sign_flips` = adjacent-`signum`-change count.
- **`f64_field(rows, field)`** — the recorded-`Vec<Scalar>`-row → `(ts, f64)`
  adapter; the one place that knows a recorded row is `Scalar`-typed.

## Data flow

The cycle-0006 sink mechanism is unchanged: a recording node pushes
`(ctx.now(), row)` out of the graph each fired cycle. After `Harness::run`
returns, the World holds each sink's drained `Vec<(Timestamp, Vec<Scalar>)>`.
`f64_field` projects the single `f64` column (field 0) of the equity stream and
of the exposure stream; `summarize` folds the two projections into `RunMetrics`;
the World pairs it with a `RunManifest` it constructed and renders `to_json`. No
data flows back into the graph; the reduction is strictly downstream of a
completed run.

## Error handling

- `summarize` is **total**: empty streams → `total_pips = 0.0`,
  `max_drawdown = 0.0`, `exposure_sign_flips = 0`. No panics, no `Result`.
- `to_json` is **total**: it writes to a `String`; finite-float assumption is
  documented (pip equity / exposure are finite by construction — `SimBroker`
  integrates finite returns, `Exposure` clamps to `[-1, +1]`).
- `f64_field` **panics** on a row shorter than `field` or a non-`f64` scalar at
  `field`: that is a wiring bug (a sink's declared kinds are fixed at bootstrap,
  so a correctly-wired equity/exposure sink always yields `f64` at field 0),
  surfaced exactly like the engine's existing `expect("… checked at wiring")`
  contract violations rather than silently dropped.

## Testing strategy

Unit tests in `report.rs` (`aura-engine`):

- `summarize` total_pips = last cumulative value; 0.0 on empty.
- `summarize` max_drawdown on a non-monotonic curve (up → down → up): asserts
  the worst peak-to-trough drop, not the final drop; 0.0 on a monotonic curve.
- `summarize` exposure_sign_flips on a crafted series
  (`+ + - 0 -` → flips at `+→-`, `-→0`, `0→-` = 3) pinning the documented
  signum-change definition; 0 on a constant-sign series.
- `to_json` snapshot: a known `RunReport` renders the exact canonical string
  above (field order, JSON number forms, `params`-as-object).
- `f64_field` projects field 0 of multi-field rows; panics on a kind mismatch
  (a `#[should_panic]` test).

End-to-end determinism test (the C18 / acceptance demonstrator), reusing the
cycle-0007 harness shape (`Sma` fast/slow → `Sub` → `Exposure` → `SimBroker`,
price tapped into the broker) with two recording sinks:

- build + run the harness twice on identical synthetic input;
- drain, `f64_field`, `summarize` each run;
- assert the two `RunMetrics` are bit-identical and the two `to_json` strings are
  identical — "same manifest → same metrics" (C1/C12).

Gates (profile `commands`): `cargo test --workspace`,
`cargo clippy --workspace --all-targets -- -D warnings`,
`RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps`.

## Load-bearing decision flagged for review

**Structured output is hand-rolled canonical JSON, zero-dependency — not serde.**
The workspace today has *zero* external dependencies by deliberate policy
("Add crates only when a boundary proves real"). The manifest+metrics schema is
tiny, closed, and flat, so a hand-written `to_json` satisfies C14
(machine-readable, stable, parseable) without pulling `serde`/`serde_json` and
breaking that stance. serde is the natural choice *later*, when the run-registry
milestone (C18) grows a larger, evolving, possibly-deserialized schema — that is
an effort/scope judgement deferred to when the boundary is real, not a
correctness need now. If you would rather adopt serde now (e.g. you expect the
schema to churn immediately, or want `#[derive(Serialize)]` ergonomics from day
one), say so at review and the spec flips to a serde-based `to_json`.

## Acceptance criteria

- [ ] A run produces `RunMetrics` with total pips, max drawdown, and exposure
      sign-flips, reduced from its recorded streams (`summarize`).
- [ ] A run produces a `RunManifest` carrying commit, params, data-window, seed,
      and broker profile.
- [ ] `RunReport::to_json` emits structured, machine-readable JSON (C14) in a
      fixed canonical form.
- [ ] An end-to-end test runs a harness twice and asserts identical metrics +
      identical JSON — same manifest → same metrics (C1/C12).
- [ ] No engine/`Harness`/node-contract change; the surface is pure-additive in
      a new `aura-engine` `report` module; the workspace stays zero-dependency.
- [ ] `cargo test --workspace`, clippy `-D warnings`, and `cargo doc -D warnings`
      are clean.
