Aura/docs/specs/0009-run-metrics-and-manifest.md

Run metrics + run manifest — Design Spec

Date: 2026-06-04 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Cycle 0009. Tracker: Gitea issue #6 (Brummel/Aura), milestone Walking skeleton. Unblocks issue #8 (aura run CLI), which wires the surface this cycle delivers into an actual subcommand.

Goal

A run must produce the two artefacts C18 mandates "per run from day one": a small manifest (the reproducible descriptor — node identity/commit, param-set, data-window, seed, broker profile) and summary metrics reduced from the run's recorded streams (at least total pips, peak-to-trough drawdown, and an exposure sign-flip count as a turnover proxy). Determinism (C1/C12) makes the run reproducible from the manifest, so the pair (manifest, metrics) is the durable run record; the full result is re-derivable on demand.

This cycle delivers the library surface for that pair — typed RunManifest / RunMetrics / RunReport values, a pure summarize reduction over recorded streams, and canonical-JSON rendering for the structured (C14) face — plus an end-to-end test that runs a harness and produces the pair. Printing it from an actual aura run subcommand is the next cycle (#8); this cycle stops at the library surface and its demonstrating test.

Non-goals (out of scope)

• The aura run subcommand / any CLI wiring — that is #8. This cycle's end-to-end demonstration is a harness test, not a binary.
• The queryable run registry, lineage, run-diff, promotion/status — the later C18 milestone over manifests that already exist. This cycle emits the manifest+metrics; it does not store or index them.
• A typed/ranged param-space surface. node.rs deliberately defers tunable params to the blueprint/optimizer layer (settled in the cycle-0008 audit). The manifest records the params the harness author bound as plain name → value pairs; it does not introduce the optimizer-facing param-space type.
• Any new engine mechanism. Harness / the run loop / the node contract are unchanged; this is a pure-additive consumer surface over the recorded streams the cycle-0006 sink mechanism already produces.

Architecture

The reduction is a post-run pure function over recorded streams, not a node. A node emits at most one record per eval and there is no end-of-run eval (C8), so an end-of-run reduction has no home as a node. The pipeline is exactly the one C18 describes:

recording sink (cycle 0006) captures a stream out of the graph
        │  Harness::run(...) -> ()   [unchanged]
        ▼
World/runner drains the sink's channel -> Vec<(Timestamp, Vec<Scalar>)>
        │  f64_field(rows, 0)
        ▼
(Timestamp, f64) samples  ──►  summarize(equity, exposure) -> RunMetrics
                                        │
        RunManifest (caller-supplied) ─┴─► RunReport { manifest, metrics }
                                                  │ to_json()
                                                  ▼
                                          canonical JSON (C14)

summarize is total and pure: identical recorded streams → identical metrics, which is the C1/C12 reproducibility acceptance (the surrounding run is already bit-identical under C1, and a pure reduction preserves that).

Home: a new report module in aura-engine, beside Harness. Not aura-core: a hot-reloadable node cdylib never constructs a manifest or reduces a run — only the World/runner does, and the World links the engine, not just the core contract. aura-cli (which depends on aura-engine) renders the JSON to stdout in #8.

Concrete code shapes

Delivered library surface (aura-engine/src/report.rs)

use aura_core::{Scalar, Timestamp};

/// Summary metrics reduced from a run's recorded streams — the "-> metrics"
/// half of C12's atomic sim unit. Pure function of the recorded streams.
#[derive(Clone, Debug, PartialEq)]
pub struct RunMetrics {
    /// Final cumulative pip equity — the last value of the (cumulative)
    /// pip-equity curve. 0.0 if the curve is empty.
    pub total_pips: f64,
    /// Largest peak-to-trough drop on the cumulative pip curve:
    /// `max_t (running_peak(t) - equity(t))`, always >= 0.0 (0.0 if the curve
    /// is monotonic non-decreasing or empty).
    pub max_drawdown: f64,
    /// Count of adjacent recorded exposure samples whose `signum` differs
    /// (`signum(0.0) == 0.0`) — a turnover proxy. Counts long<->short
    /// reversals AND transitions into/out of flat; it is the plain
    /// sign-change count over the recorded exposure series, not a
    /// reversal-only count.
    pub exposure_sign_flips: u64,
}

/// The reproducible run descriptor (C18). **Caller-supplied**: the engine
/// cannot introspect a git commit, an RNG seed, or a broker label — the World
/// that bootstraps and runs the harness holds those and fills them in.
#[derive(Clone, Debug, PartialEq)]
pub struct RunManifest {
    /// Node/engine identity: the git commit of the frozen artifact
    /// (C18 — commit = identity; the frozen bot *is* a commit).
    pub commit: String,
    /// The bound tuning params as ordered `name -> value` pairs. Precursor to
    /// the eventual typed param-space (deferred — see Non-goals); honest about
    /// what exists today.
    pub params: Vec<(String, f64)>,
    /// The data-window: inclusive `(from, to)` epoch-ns bounds (C12).
    pub window: (Timestamp, Timestamp),
    /// The RNG seed (C12 seed-as-input). 0 for a seed-free synthetic run.
    pub seed: u64,
    /// The broker profile label, e.g. `"sim-optimal(pip_size=1.0)"`.
    pub broker: String,
}

/// A run's full structured result: the descriptor plus the metrics it
/// reproduces. The durable run record of C18 ("stores manifests + metrics,
/// re-derives full results on demand").
#[derive(Clone, Debug, PartialEq)]
pub struct RunReport {
    pub manifest: RunManifest,
    pub metrics: RunMetrics,
}

impl RunReport {
    /// Render canonical, machine-readable JSON (C14). Hand-rolled: the schema
    /// is tiny, closed, and flat, so the deliberately zero-dependency workspace
    /// stays zero-dependency. Field order is fixed; `f64` is rendered with the
    /// round-trippable `{}` shortest form (finite values only — pip equity and
    /// exposure are always finite). `params` renders as a JSON object in
    /// insertion order.
    pub fn to_json(&self) -> String { /* shape; planner owns bytes */ }
}

/// Reduce a run's recorded pip-equity + exposure streams into summary metrics.
/// Pure — identical inputs yield identical metrics (C1/C12). Timestamps are
/// carried in the input to match exactly what a sink records; the reduction
/// itself is value-only (it does not read the timestamps).
pub fn summarize(
    equity: &[(Timestamp, f64)],
    exposure: &[(Timestamp, f64)],
) -> RunMetrics { /* shape; planner owns bytes */ }

/// Bridge a sink's recorded `(ts, row)` stream to `summarize`: extract one
/// `f64` field of each row into `(ts, f64)` samples. Panics if a row is too
/// short or the field is not an `f64` scalar — a wiring bug (the sink's
/// declared kinds are fixed at bootstrap), reported like the engine's other
/// "kind checked at wiring" contract violations.
pub fn f64_field(
    rows: &[(Timestamp, Vec<Scalar>)],
    field: usize,
) -> Vec<(Timestamp, f64)> { /* shape; planner owns bytes */ }

North-star slice — what the World/runner author writes (this is #8's seam)

Shown minimal and honest: the harness wiring (SMA fast/slow → Sub → Exposure → SimBroker, with a price tap into the broker) is the cycle-0007 pattern and is not re-specified here; the new bytes are the drain → reduce → report tail.

// ... bootstrap the harness with two recording sinks: one on the SimBroker
// equity output, one on the Exposure output; run it ...
h.run(streams);

let equity   = f64_field(&equity_rows,   0); // drained from the equity sink
let exposure = f64_field(&exposure_rows, 0); // drained from the exposure sink
let metrics  = summarize(&equity, &exposure);

let report = RunReport {
    manifest: RunManifest {
        commit: env!("AURA_COMMIT").to_string(),
        params: vec![
            ("sma_fast".into(), 2.0),
            ("sma_slow".into(), 4.0),
            ("exposure_scale".into(), 1.0),
        ],
        window: (Timestamp(1), Timestamp(6)),
        seed: 0,
        broker: "sim-optimal(pip_size=1.0)".into(),
    },
    metrics,
};
println!("{}", report.to_json()); // the structured C14 face (lands in #8)

Canonical JSON shape (one line; whitespace fixed)

{"manifest":{"commit":"abc123","params":{"sma_fast":2,"sma_slow":4,"exposure_scale":1},"window":[1,6],"seed":0,"broker":"sim-optimal(pip_size=1.0)"},"metrics":{"total_pips":12,"max_drawdown":1,"exposure_sign_flips":1}}

Components

• RunMetrics — the three summary numbers. Plain data; PartialEq so tests assert exact equality and determinism.
• RunManifest — the reproducible descriptor; all fields caller-supplied.
• RunReport — the (manifest, metrics) pair + to_json.
• summarize(equity, exposure) — the pure reduction. total_pips = last equity value; max_drawdown = running-peak minus value, maximized; exposure_sign_flips = adjacent-signum-change count.
• f64_field(rows, field) — the recorded-Vec<Scalar>-row → (ts, f64) adapter; the one place that knows a recorded row is Scalar-typed.

Data flow

The cycle-0006 sink mechanism is unchanged: a recording node pushes (ctx.now(), row) out of the graph each fired cycle. After Harness::run returns, the World holds each sink's drained Vec<(Timestamp, Vec<Scalar>)>. f64_field projects the single f64 column (field 0) of the equity stream and of the exposure stream; summarize folds the two projections into RunMetrics; the World pairs it with a RunManifest it constructed and renders to_json. No data flows back into the graph; the reduction is strictly downstream of a completed run.

Error handling

• summarize is total: empty streams → total_pips = 0.0, max_drawdown = 0.0, exposure_sign_flips = 0. No panics, no Result.
• to_json is total: it writes to a String; finite-float assumption is documented (pip equity / exposure are finite by construction — SimBroker integrates finite returns, Exposure clamps to [-1, +1]).
• f64_field panics on a row shorter than field or a non-f64 scalar at field: that is a wiring bug (a sink's declared kinds are fixed at bootstrap, so a correctly-wired equity/exposure sink always yields f64 at field 0), surfaced exactly like the engine's existing expect("… checked at wiring") contract violations rather than silently dropped.

Testing strategy

Unit tests in report.rs (aura-engine):

• summarize total_pips = last cumulative value; 0.0 on empty.
• summarize max_drawdown on a non-monotonic curve (up → down → up): asserts the worst peak-to-trough drop, not the final drop; 0.0 on a monotonic curve.
• summarize exposure_sign_flips on a crafted series (+ + - 0 - → flips at +→-, -→0, 0→- = 3) pinning the documented signum-change definition; 0 on a constant-sign series.
• to_json snapshot: a known RunReport renders the exact canonical string above (field order, JSON number forms, params-as-object).
• f64_field projects field 0 of multi-field rows; panics on a kind mismatch (a #[should_panic] test).

End-to-end determinism test (the C18 / acceptance demonstrator), reusing the cycle-0007 harness shape (Sma fast/slow → Sub → Exposure → SimBroker, price tapped into the broker) with two recording sinks:

• build + run the harness twice on identical synthetic input;
• drain, f64_field, summarize each run;
• assert the two RunMetrics are bit-identical and the two to_json strings are identical — "same manifest → same metrics" (C1/C12).

Gates (profile commands): cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps.

Load-bearing decision flagged for review

Structured output is hand-rolled canonical JSON, zero-dependency — not serde. The workspace today has zero external dependencies by deliberate policy ("Add crates only when a boundary proves real"). The manifest+metrics schema is tiny, closed, and flat, so a hand-written to_json satisfies C14 (machine-readable, stable, parseable) without pulling serde/serde_json and breaking that stance. serde is the natural choice later, when the run-registry milestone (C18) grows a larger, evolving, possibly-deserialized schema — that is an effort/scope judgement deferred to when the boundary is real, not a correctness need now. If you would rather adopt serde now (e.g. you expect the schema to churn immediately, or want #[derive(Serialize)] ergonomics from day one), say so at review and the spec flips to a serde-based to_json.

Acceptance criteria

• A run produces RunMetrics with total pips, max drawdown, and exposure sign-flips, reduced from its recorded streams (summarize).
• A run produces a RunManifest carrying commit, params, data-window, seed, and broker profile.
• RunReport::to_json emits structured, machine-readable JSON (C14) in a fixed canonical form.
• An end-to-end test runs a harness twice and asserts identical metrics + identical JSON — same manifest → same metrics (C1/C12).
• No engine/Harness/node-contract change; the surface is pure-additive in a new aura-engine report module; the workspace stays zero-dependency.
• cargo test --workspace, clippy -D warnings, and cargo doc -D warnings are clean.