From a982b96ecc8c4293b8ca857653e26a83e4d04efa Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 4 Jun 2026 19:06:13 +0200 Subject: [PATCH] =?UTF-8?q?fieldtest:=20cycle-0009=20=E2=80=94=204=20examp?= =?UTF-8?q?les,=206=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First fieldtest of the run-metrics + manifest report surface. A standalone downstream-consumer crate (fieldtests/cycle-0009-run-metrics/) path-depends on the engine crates and exercises the post-0009 surface from the public interface only (rustdoc + ledger + glossary + project layout, never crates/*/src). Primary axis empirically met: the north-star "a run emits metrics + manifest" move is reachable from rustdoc alone — drain two recording sinks -> f64_field -> summarize -> RunManifest -> to_json, metrics matching the hand model on the first run, deterministic across reruns. Findings: 0 bugs, 1 friction, 1 spec_gap, 4 working. - working x4: north-star reachable from rustdoc; SimBroker firing/slot docs (a resolved 0007 gap) now carry the example; summarize metric definitions exact on six degenerate inputs (incl. negative-curve drawdown + flat-as-sign-0); f64_field panics precise and well-located. - spec_gap: to_json's JSON key names + {manifest,metrics} nesting are not on the public surface — a consumer parsing the JSON (C18 registry, the deferred aura run printer) cannot author against it from rustdoc alone. - friction: to_json renders whole-valued f64 without a decimal point (3.0 -> "3"), so one f64 field appears as integer or decimal token within one schema. Both doc-level findings are the same doc pass and matter mainly for the deferred aura run (#8) and the C18 registry that will parse this JSON. Spec feeds the next plan as reference. refs #6 --- docs/specs/fieldtest-0009-run-metrics.md | 173 ++++++++++++++++++ fieldtests/cycle-0009-run-metrics/Cargo.lock | 30 +++ fieldtests/cycle-0009-run-metrics/Cargo.toml | 37 ++++ .../c0009_1_run_to_report.rs | 158 ++++++++++++++++ .../c0009_2_compare_two_runs.rs | 138 ++++++++++++++ .../c0009_3_degenerate_streams.rs | 84 +++++++++ .../c0009_4_f64field_and_json.rs | 94 ++++++++++ 7 files changed, 714 insertions(+) create mode 100644 docs/specs/fieldtest-0009-run-metrics.md create mode 100644 fieldtests/cycle-0009-run-metrics/Cargo.lock create mode 100644 fieldtests/cycle-0009-run-metrics/Cargo.toml create mode 100644 fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs create mode 100644 fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs create mode 100644 fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs create mode 100644 fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs diff --git a/docs/specs/fieldtest-0009-run-metrics.md b/docs/specs/fieldtest-0009-run-metrics.md new file mode 100644 index 0000000..0cbaef9 --- /dev/null +++ b/docs/specs/fieldtest-0009-run-metrics.md @@ -0,0 +1,173 @@ +# Fieldtest — cycle-0009 (run metrics + manifest) — 2026-06-04 + +**Status:** Draft — awaiting orchestrator triage +**Author:** fieldtester (dispatched by fieldtest skill) + +## Scope + +Cycle 0009 (Walking-skeleton milestone, issue #6) shipped a pure-additive +`report` module in `aura-engine`, re-exported at the crate root: +`summarize(equity, exposure) -> RunMetrics` (fields `total_pips`, +`max_drawdown`, `exposure_sign_flips`); `f64_field(rows, field)` bridging a +recording sink's `Vec` rows to `summarize`; `RunManifest` +(`commit`, `params`, `window`, `seed`, `broker`); `RunReport { manifest, +metrics }` with `to_json() -> String` (the structured C14 face). The intended +downstream move: after `Harness::run`, drain recording sinks, project f64 +columns with `f64_field`, `summarize` the pip-equity + exposure streams, pair +with a caller-built `RunManifest`, render `to_json`. The `aura run` CLI that +prints this is deferred to #8 and out of scope. Worked from the public interface +only (rustdoc via `cargo doc`, `docs/design/INDEX.md`, `docs/glossary.md`, +project layout, public re-exports); never `crates/*/src`. + +Build/run: workspace built clean from HEAD (`cargo build --workspace`); each +example run via `cargo run --manifest-path +fieldtests/cycle-0009-run-metrics/Cargo.toml --bin `, so HEAD source is +always what compiled and ran (a standalone non-workspace consumer crate +path-depending on the three engine crates, exactly as a C16 project would). + +## Examples + +### fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs — north-star run-to-report +- **What it does.** Bootstraps the full SMA(2)/SMA(4) → Sub → Exposure(4) → + SimBroker harness with TWO recording sinks (equity tapped on the broker, + exposure tapped on the Exposure node), runs a 7-tick rising-price stream, + drains both sinks, `f64_field`-projects each, `summarize`s, builds a + `RunManifest`, and renders `to_json`. +- **Why it fits.** This is the carrier's primary axis verbatim — the end-to-end + research-loop move the whole module exists to enable, reaching every new + surface element once. +- **Outcome.** Built, ran, matched the hand model exactly: equity + `[0,0,0,0,1,2,3]` → `total_pips=3.0`, `max_drawdown=0.0`; exposure all `+0.5` + → `exposure_sign_flips=0`. `to_json` emitted a single well-formed object. + +### fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs — comparison + determinism +- **What it does.** Runs the same harness twice at `Exposure` scale=4 (asserts + bit-identical metrics AND bit-identical `to_json` across the two runs), then a + third run at scale=2, and compares `total_pips`. +- **Why it fits.** The carrier's second axis: determinism (C1) plus a + smallest-slice tuning-sweep comparison (C12/C21) read off `RunMetrics`. +- **Outcome.** Built, ran, matched expected: scale=4 deterministic + (`total_pips=3`), scale=2 = `total_pips=6` (exactly double — exposure + saturates 0.5→1.0), both drawdown-free, no sign flips. + +### fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs — degenerate summarize semantics +- **What it does.** Calls `summarize` with six hand-built `&[(Timestamp,f64)]` + argument literals: empty, monotonic-up, dip-and-recover, long/short/flat + sign-flips, same-sign varying magnitude, and an all-negative curve. +- **Why it fits.** The carrier's third axis: verify the documented metric + definitions from docs alone. The slices are the public function's direct + argument type — calling the function with literals, not authoring an + intermediate representation. +- **Outcome.** Built, ran, every documented definition held exactly: empty→ + zeros; last-value `total_pips` (incl. negative); running-peak drawdown (incl. + on a negative curve, peak −1 → trough −5 = 4); sign-flips counting flat (sign + 0) as distinct from long/short (3 flips), magnitude changes not counted. + +### fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs — f64_field bridge + JSON shape +- **What it does.** Projects a non-zero column from a multi-column recorded row; + confirms (via `catch_unwind`) the documented panics on a non-f64 field and an + out-of-range field index; inspects the exact `to_json` byte shape under a + realistic FX `pip_size=0.0001`, fractional/negative metrics, and ns-scale + timestamps. +- **Why it fits.** Exercises the `f64_field` "checked at wiring" contract and + the `to_json` round-trippability claim — the corners of the north-star + surface a generic sink-drainer would hit. +- **Outcome.** Built, ran, matched: column-pick correct; both panics fired with + precise messages; JSON kept ns timestamps as bare integers (no sci-notation), + params in insertion order as a nested object. + +## Findings + +### [working] North-star run-to-report is reachable from rustdoc alone +- **Example(s):** c0009_1, c0009_2. +- **What happened.** The full drain → `f64_field` → `summarize` → + `RunManifest` → `to_json` chain was authored entirely from the crate-root + rustdoc (`fn.summarize`, `fn.f64_field`, the three structs, `to_json`) plus + the prior fieldtest corpus for the harness wiring. Metrics matched the hand + model on the first run; `to_json` produced a clean object. +- **Why working.** The cycle's headline move is reachable and correct without + reading `src` — the module rustdoc names the post-run drain-and-fold workflow + explicitly, and the field docs are precise enough to predict every value. +- **Recommended action:** carry-on. + +### [working] SimBroker firing/warm-up/slot-order docs (a resolved 0007 spec_gap) now carry the example +- **Example(s):** c0009_1. +- **What happened.** Predicting the equity sink's `[0,0,0,0,1,2,3]` shape + required knowing slot 0 = exposure / slot 1 = price, both `Firing::Any`, and + the "leading 0.0 rows, one row per price cycle" warm-up. All three are now on + `struct.SimBroker` rustdoc ("Input slots" / "Firing and warm-up"). The 0007 + fieldtest had to recover these from the ledger + commit body and recorded them + as a spec_gap; they are now on the surface. +- **Why working.** A prior gap is closed and directly enabled this cycle's + north-star example to be authored from rustdoc. +- **Recommended action:** carry-on. + +### [working] summarize metric definitions hold exactly on all degenerate inputs +- **Example(s):** c0009_3. +- **What happened.** All six hand-built cases matched the rustdoc field + definitions to the bit, including the two subtle ones: drawdown measured from + the running peak even when the whole curve is negative, and `exposure_sign_flips` + treating flat (0.0 → sign 0) as distinct from long/short. +- **Why working.** The `RunMetrics` field docs are unambiguous and empirically + exact; the "flat is sign 0" and "running-peak" subtleties are spelled out and + behave as written. +- **Recommended action:** carry-on. + +### [working] f64_field panics are precise and well-located +- **Example(s):** c0009_4. +- **What happened.** `f64_field` on an `I64` field panicked with + `f64_field: field 0 is not an f64 scalar: I64(7)`; on an out-of-range index + with `f64_field: row has no field 5 (row width 2)`. Both match the documented + "checked at wiring" panic contract and name the exact offending field/width. +- **Why working.** The diagnostic is actionable and matches the rustdoc promise; + a downstream consumer mis-wiring a sink column gets a precise message. +- **Recommended action:** carry-on. + +### [spec_gap] to_json key names / nesting are not on the public surface +- **Example(s):** c0009_1, c0009_4. +- **What happened.** `RunReport::to_json` rustdoc promises "field order is + fixed", "params renders as a JSON object in insertion order", "f64 uses the + round-trippable `{}` shortest form" — but it does **not** state the actual JSON + key names or the `{manifest:{...},metrics:{...}}` nesting. The observed shape + is `{"manifest":{"commit":..,"params":{..},"window":[from,to],"seed":.., + "broker":..},"metrics":{"total_pips":..,"max_drawdown":..,"exposure_sign_flips":..}}`. + I had to read the runtime output to learn the schema; a consumer parsing the + JSON (the C18 registry, the deferred `aura run` printer) cannot author against + it from rustdoc alone and would couple to an undocumented contract. +- **Why spec_gap.** The reading I guessed (keys = struct field names, `window` + as a 2-array, `params` as an object) was the natural one and proved correct, + but it is unconstrained by the public surface; another reading (e.g. `window` + as `{from,to}`, or top-level flattening) was equally plausible. For a + "machine-readable" (C14) face this schema is part of the contract. +- **Recommended action:** tighten the design ledger / rustdoc — add a documented + JSON schema example to `to_json` rustdoc (a one-line sample object suffices), + or ratify the field-name-mirroring as the stated contract. + +### [friction] to_json renders whole-valued f64 without a decimal point (3.0 → `3`) +- **Example(s):** c0009_1, c0009_2. +- **What happened.** `total_pips: 3.0_f64` serializes as `"total_pips":3` (and + `4.0` param → `4`), whereas `-12.5`/`2.5` keep the point. This is the + documented round-trippable `{}` shortest form and is valid JSON (a number is a + number), so it is not a bug. But a downstream consumer string-matching or + schema-typing the field could trip: the same logical f64 field appears + sometimes as an integer token, sometimes as a decimal token, within one + schema. My example had to substring-match `"3"` rather than `"3.0"`. +- **Why friction.** The task completed, but the mixed integer/decimal rendering + of a single f64 field is a small surprise for a consumer expecting stable + per-field token shape; it interacts with the spec_gap above (no documented + schema to set the expectation). +- **Recommended action:** plan (tidy) — either document explicitly in the + `to_json` rustdoc that f64 fields may render without a fractional part, or (if + a stable decimal shape is wanted for the registry) normalize. Low urgency; + fold into the same doc pass as the spec_gap. + +## Recommendation summary + +| Finding | Class | Action | +|---|---|---| +| North-star reachable from rustdoc | working | carry-on | +| SimBroker firing/slot docs now carry the example | working | carry-on | +| summarize definitions exact on degenerate inputs | working | carry-on | +| f64_field panics precise + well-located | working | carry-on | +| to_json key names / nesting not on public surface | spec_gap | tighten the design ledger / rustdoc (or ratify) | +| to_json renders whole f64 as integer token | friction | plan (tidy doc pass) | diff --git a/fieldtests/cycle-0009-run-metrics/Cargo.lock b/fieldtests/cycle-0009-run-metrics/Cargo.lock new file mode 100644 index 0000000..a347d23 --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/Cargo.lock @@ -0,0 +1,30 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aura-core" +version = "0.1.0" + +[[package]] +name = "aura-engine" +version = "0.1.0" +dependencies = [ + "aura-core", +] + +[[package]] +name = "aura-std" +version = "0.1.0" +dependencies = [ + "aura-core", +] + +[[package]] +name = "c0009-fieldtest" +version = "0.0.0" +dependencies = [ + "aura-core", + "aura-engine", + "aura-std", +] diff --git a/fieldtests/cycle-0009-run-metrics/Cargo.toml b/fieldtests/cycle-0009-run-metrics/Cargo.toml new file mode 100644 index 0000000..d270a5d --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/Cargo.toml @@ -0,0 +1,37 @@ +# Standalone downstream-consumer crate for the cycle-0009 fieldtest (run report). +# +# Like the cycle-0007/0008 fixtures, this is NOT a member of the aura workspace — +# it path-depends on the engine crates exactly as a real research project (C16) +# would, and is built via +# `cargo run --manifest-path fieldtests/cycle-0009-run-metrics/Cargo.toml --bin ` +# so HEAD source is always what runs. +# Empty [workspace] table: marks this fixture crate as its OWN workspace root +# (documented in docs/project-layout.md as the nested-project onboarding fix). +[workspace] + +[package] +name = "c0009-fieldtest" +version = "0.0.0" +edition = "2024" +publish = false + +[dependencies] +aura-core = { path = "../../crates/aura-core" } +aura-engine = { path = "../../crates/aura-engine" } +aura-std = { path = "../../crates/aura-std" } + +[[bin]] +name = "c0009_1_run_to_report" +path = "c0009_1_run_to_report.rs" + +[[bin]] +name = "c0009_2_compare_two_runs" +path = "c0009_2_compare_two_runs.rs" + +[[bin]] +name = "c0009_3_degenerate_streams" +path = "c0009_3_degenerate_streams.rs" + +[[bin]] +name = "c0009_4_f64field_and_json" +path = "c0009_4_f64field_and_json.rs" diff --git a/fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs b/fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs new file mode 100644 index 0000000..a22a4cf --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs @@ -0,0 +1,158 @@ +//! Fieldtest c0009 #1 — the north-star run-to-report move, end to end. +//! +//! Axis (carrier): bootstrap a harness (SMA fast/slow -> Sub -> Exposure -> +//! SimBroker, price tapped into the broker) with TWO recording sinks (equity on +//! the broker, exposure on the Exposure node), run it, drain both sinks, +//! f64_field + summarize + build a RunManifest + to_json. Question under test: +//! does the public rustdoc make this reachable WITHOUT reading crates/*/src? +//! +//! price ----+--> SMA(2) --\ +//! | Sub(fast - slow) --> Exposure(4) --+--> SimBroker --> equitySink +//! +--> SMA(4) --/ | ^ +//! | v | +//! | exposureSink | +//! +------------------------------------------------ price --+ (slot 1) +//! +//! Public-surface facts used (rustdoc only): +//! - aura_std::Exposure::new(scale) = clamp(signal/scale, -1, +1), None until warm. +//! - aura_std::SimBroker::new(pip_size): slot 0 = exposure, slot 1 = price; +//! both Firing::Any, leading 0.0 rows during warm-up, one row per price cycle +//! (rustdoc struct.SimBroker "Input slots"/"Firing and warm-up" — these were +//! a 0007 spec_gap, now ON the surface; recorded as a `working` for that). +//! - aura_engine::f64_field(rows, field) extracts one f64 column (rustdoc). +//! - aura_engine::summarize(equity, exposure) -> RunMetrics (rustdoc). +//! - RunManifest { commit, params, window, seed, broker }, RunReport, to_json. + +use std::sync::mpsc::{self, Sender}; + +use aura_core::{Ctx, Firing, InputSpec, Node, NodeSchema, Scalar, ScalarKind, Timestamp}; +use aura_engine::{Edge, Harness, RunManifest, RunReport, SourceSpec, Target, f64_field, summarize}; +use aura_std::{Exposure, SimBroker, Sma, Sub}; + +/// A recording sink: one f64 input, persists (ts, row) out of graph (C8/C22). +/// Records the WHOLE row (Vec) the way a real registry sink would, so +/// f64_field can later project a column — this is the shape summarize expects. +struct RowRecorder { + tx: Sender<(Timestamp, Vec)>, +} +impl Node for RowRecorder { + fn schema(&self) -> NodeSchema { + NodeSchema { + inputs: vec![InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Any }], + output: vec![], + } + } + fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> { + let w = ctx.f64_in(0); + if w.is_empty() { + return None; + } + let _ = self.tx.send((ctx.now(), vec![Scalar::F64(w[0])])); + None + } +} + +fn f64_stream(pairs: &[(i64, f64)]) -> Vec<(Timestamp, Scalar)> { + pairs.iter().map(|&(t, v)| (Timestamp(t), Scalar::F64(v))).collect() +} + +fn main() { + let (eq_tx, eq_rx) = mpsc::channel(); + let (exp_tx, exp_rx) = mpsc::channel(); + + // nodes: 0=SMA(2) fast, 1=SMA(4) slow, 2=Sub, 3=Exposure(4), + // 4=SimBroker(1.0), 5=equity sink (taps broker), 6=exposure sink (taps Exposure). + let mut h = Harness::bootstrap( + vec![ + Box::new(Sma::new(2)), + Box::new(Sma::new(4)), + Box::new(Sub::new()), + Box::new(Exposure::new(4.0)), + Box::new(SimBroker::new(1.0)), + Box::new(RowRecorder { tx: eq_tx }), + Box::new(RowRecorder { tx: exp_tx }), + ], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![ + Target { node: 0, slot: 0 }, // -> SMA fast + Target { node: 1, slot: 0 }, // -> SMA slow + Target { node: 4, slot: 1 }, // -> SimBroker price (slot 1) + ], + }], + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, // SMA2 -> Sub.in0 (fast) + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, // SMA4 -> Sub.in1 (slow) + Edge { from: 2, to: 3, slot: 0, from_field: 0 }, // Sub -> Exposure + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, // Exposure -> SimBroker.in0 + Edge { from: 3, to: 6, slot: 0, from_field: 0 }, // Exposure -> exposure sink + Edge { from: 4, to: 5, slot: 0, from_field: 0 }, // SimBroker -> equity sink + ], + ) + .expect("valid two-sink signal-quality DAG"); + + let prices: &[(i64, f64)] = &[ + (1, 100.0), + (2, 102.0), + (3, 104.0), + (4, 106.0), + (5, 108.0), + (6, 110.0), + (7, 112.0), + ]; + h.run(vec![f64_stream(prices)]); + + // Drain both sinks (the World's post-run step the rustdoc describes). + let equity_rows: Vec<(Timestamp, Vec)> = eq_rx.try_iter().collect(); + let exposure_rows: Vec<(Timestamp, Vec)> = exp_rx.try_iter().collect(); + println!("equity rows = {equity_rows:?}"); + println!("exposure rows = {exposure_rows:?}"); + + // Project field 0 of each (the documented bridge to summarize). + let equity = f64_field(&equity_rows, 0); + let exposure = f64_field(&exposure_rows, 0); + + let metrics = summarize(&equity, &exposure); + println!("metrics = {metrics:?}"); + + // Hand model (public-surface, from c0007_1 + SimBroker rustdoc): + // equity curve = [0,0,0,0,1,2,3] (leading zeros until exposure warms at t=5), + // so total_pips = 3.0, monotonic non-decreasing => max_drawdown = 0.0. + // exposure samples are recorded only from the Exposure node's first warm + // cycle (t=4 on): all +0.5 (long), no sign change => exposure_sign_flips = 0. + assert_eq!(metrics.total_pips, 3.0, "final cumulative pips"); + assert_eq!(metrics.max_drawdown, 0.0, "monotonic-up curve has no drawdown"); + assert_eq!(metrics.exposure_sign_flips, 0, "exposure stays long throughout"); + + // Pair with a caller-built manifest and render the structured C14 face. + let report = RunReport { + manifest: RunManifest { + commit: "fieldtest-c0009-synthetic".to_string(), + params: vec![ + ("sma_fast".to_string(), 2.0), + ("sma_slow".to_string(), 4.0), + ("exposure_scale".to_string(), 4.0), + ], + window: (Timestamp(1), Timestamp(7)), + seed: 0, + broker: "sim-optimal(pip_size=1)".to_string(), + }, + metrics, + }; + let json = report.to_json(); + println!("to_json() =\n{json}"); + + // The rustdoc says: machine-readable JSON, field order fixed, params a JSON + // object in insertion order, f64 via {} shortest form. It does NOT state the + // exact field NAMES or nesting on the public surface — recorded as a + // spec_gap. I assert only what the rustdoc promises that I can check without + // reading src: it must parse as a non-empty object string and carry the + // numbers I put in. (No serde in this crate either, by design — so I do a + // substring check, which is itself a small friction.) + assert!(json.starts_with('{') && json.ends_with('}'), "looks like a JSON object"); + assert!(json.contains("3"), "total_pips value present somewhere"); + assert!(json.contains("sma_fast"), "a param key present"); + assert!(json.contains("sim-optimal(pip_size=1)"), "broker label present"); + + println!("c0009_1 OK: two sinks -> f64_field -> summarize -> RunManifest -> to_json"); +} diff --git a/fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs b/fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs new file mode 100644 index 0000000..3d733d7 --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs @@ -0,0 +1,138 @@ +//! Fieldtest c0009 #2 — comparison + determinism over the report surface. +//! +//! Axis (carrier): run two harnesses differing in a tuning param (Exposure +//! scale), confirm each run is deterministic (bit-identical metrics across two +//! runs) and the two metrics differ as expected. This is the smallest slice of +//! the World's reason to exist (C21): compare runs by their RunMetrics. +//! +//! Same SMA-cross harness as #1, but the Exposure node's `scale` is the swept +//! tuning param. A smaller scale => larger |exposure| (clamp(signal/scale,..)), +//! so the same price move earns proportionally more pips — until the exposure +//! saturates at +1. So total_pips(scale=2) > total_pips(scale=4) here. + +use std::sync::mpsc::{self, Sender}; + +use aura_core::{Ctx, Firing, InputSpec, Node, NodeSchema, Scalar, ScalarKind, Timestamp}; +use aura_engine::{ + Edge, Harness, RunManifest, RunMetrics, RunReport, SourceSpec, Target, f64_field, summarize, +}; +use aura_std::{Exposure, SimBroker, Sma, Sub}; + +struct RowRecorder { + tx: Sender<(Timestamp, Vec)>, +} +impl Node for RowRecorder { + fn schema(&self) -> NodeSchema { + NodeSchema { + inputs: vec![InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Any }], + output: vec![], + } + } + fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> { + let w = ctx.f64_in(0); + if w.is_empty() { + return None; + } + let _ = self.tx.send((ctx.now(), vec![Scalar::F64(w[0])])); + None + } +} + +fn f64_stream(pairs: &[(i64, f64)]) -> Vec<(Timestamp, Scalar)> { + pairs.iter().map(|&(t, v)| (Timestamp(t), Scalar::F64(v))).collect() +} + +const PRICES: &[(i64, f64)] = &[ + (1, 100.0), + (2, 102.0), + (3, 104.0), + (4, 106.0), + (5, 108.0), + (6, 110.0), + (7, 112.0), +]; + +/// One full run: bootstrap a harness with the given exposure scale, run, drain +/// the two sinks, reduce to a RunReport. This is exactly the World's per-instance +/// step in a tuning sweep (C12/C19/C20), authored in plain Rust. +fn run_one(exposure_scale: f64) -> RunReport { + let (eq_tx, eq_rx) = mpsc::channel(); + let (exp_tx, exp_rx) = mpsc::channel(); + + let mut h = Harness::bootstrap( + vec![ + Box::new(Sma::new(2)), + Box::new(Sma::new(4)), + Box::new(Sub::new()), + Box::new(Exposure::new(exposure_scale)), + Box::new(SimBroker::new(1.0)), + Box::new(RowRecorder { tx: eq_tx }), + Box::new(RowRecorder { tx: exp_tx }), + ], + vec![SourceSpec { + kind: ScalarKind::F64, + targets: vec![ + Target { node: 0, slot: 0 }, + Target { node: 1, slot: 0 }, + Target { node: 4, slot: 1 }, + ], + }], + vec![ + Edge { from: 0, to: 2, slot: 0, from_field: 0 }, + Edge { from: 1, to: 2, slot: 1, from_field: 0 }, + Edge { from: 2, to: 3, slot: 0, from_field: 0 }, + Edge { from: 3, to: 4, slot: 0, from_field: 0 }, + Edge { from: 3, to: 6, slot: 0, from_field: 0 }, + Edge { from: 4, to: 5, slot: 0, from_field: 0 }, + ], + ) + .expect("valid harness"); + + h.run(vec![f64_stream(PRICES)]); + + let equity = f64_field(&eq_rx.try_iter().collect::>(), 0); + let exposure = f64_field(&exp_rx.try_iter().collect::>(), 0); + let metrics = summarize(&equity, &exposure); + + RunReport { + manifest: RunManifest { + commit: "fieldtest-c0009-synthetic".to_string(), + params: vec![("exposure_scale".to_string(), exposure_scale)], + window: (Timestamp(1), Timestamp(7)), + seed: 0, + broker: "sim-optimal(pip_size=1)".to_string(), + }, + metrics, + } +} + +fn main() { + // Determinism (C1): the same instance run twice yields bit-identical metrics. + let a1 = run_one(4.0); + let a2 = run_one(4.0); + assert_eq!(a1.metrics, a2.metrics, "scale=4 deterministic across runs"); + assert_eq!(a1.to_json(), a2.to_json(), "scale=4 JSON bit-identical"); + println!("determinism OK: scale=4 metrics = {:?}", a1.metrics); + + // Comparison: a second instance with a different tuning param. + let b = run_one(2.0); + println!("scale=4 report json = {}", a1.to_json()); + println!("scale=2 report json = {}", b.to_json()); + + // Expectation (public model): scale=2 doubles |exposure| (0.5 -> 1.0 saturated) + // versus scale=4 (0.5), so the same +2/cycle price move earns ~2x the pips. + let RunMetrics { total_pips: p4, .. } = a1.metrics; + let RunMetrics { total_pips: p2, .. } = b.metrics; + println!("total_pips scale=4 = {p4}, scale=2 = {p2}"); + assert!(p2 > p4, "smaller scale => larger exposure => more pips ({p2} > {p4})"); + + // Both should be drawdown-free monotonic-up curves on a steadily-rising price, + // and stay long throughout (no sign flips) — a sanity check the comparison is + // apples-to-apples, differing only on the swept axis. + assert_eq!(a1.metrics.max_drawdown, 0.0); + assert_eq!(b.metrics.max_drawdown, 0.0); + assert_eq!(a1.metrics.exposure_sign_flips, 0); + assert_eq!(b.metrics.exposure_sign_flips, 0); + + println!("c0009_2 OK: deterministic + comparable two-run metrics over the report surface"); +} diff --git a/fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs b/fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs new file mode 100644 index 0000000..1f41c86 --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs @@ -0,0 +1,84 @@ +//! Fieldtest c0009 #3 — degenerate-stream semantics from the docs alone. +//! +//! Axis (carrier): feed `summarize` empty / monotonic / sign-flipping HAND-BUILT +//! streams and verify the documented metric definitions hold: +//! - total_pips = last value of the cumulative pip curve, 0.0 if empty +//! - max_drawdown = max_t (running_peak(t) - equity(t)), >= 0, 0 if +//! monotonic-non-decreasing or empty +//! - exposure_sign_flips = count of adjacent samples whose SIGN differs, with +//! zero normalizing to sign 0 (flat distinct from +//! long/short) +//! +//! NOTE on fixture form: `summarize(equity, exposure)` takes `&[(Timestamp, f64)]` +//! DIRECTLY as its public signature. Hand-building those slices is calling the +//! public function with literal arguments — NOT hand-authoring an intermediate +//! representation. (f64_field, the sink->slice bridge, is exercised in #1/#2/#4.) + +use aura_engine::summarize; +use aura_core::Timestamp; + +fn eq(samples: &[(i64, f64)]) -> Vec<(Timestamp, f64)> { + samples.iter().map(|&(t, v)| (Timestamp(t), v)).collect() +} + +fn main() { + // --- (1) empty streams -> all zeros (documented "0.0 if empty"). --- + let m = summarize(&[], &[]); + println!("empty -> {m:?}"); + assert_eq!(m.total_pips, 0.0); + assert_eq!(m.max_drawdown, 0.0); + assert_eq!(m.exposure_sign_flips, 0); + + // --- (2) monotonic non-decreasing curve -> drawdown 0, last value total. --- + let equity = eq(&[(1, 0.0), (2, 1.0), (3, 3.0), (4, 6.0), (5, 10.0)]); + let exposure = eq(&[(1, 0.5), (2, 0.5), (3, 0.5), (4, 0.5), (5, 0.5)]); + let m = summarize(&equity, &exposure); + println!("monotonic-up -> {m:?}"); + assert_eq!(m.total_pips, 10.0, "last value of curve"); + assert_eq!(m.max_drawdown, 0.0, "monotonic => no drawdown"); + assert_eq!(m.exposure_sign_flips, 0, "constant-sign exposure"); + + // --- (3) a dip then recovery -> worst peak-to-trough drawdown. --- + // peak 10 at t=3, trough 4 at t=5 => max drawdown 6; recovers to 8 (final). + let equity = eq(&[(1, 0.0), (2, 5.0), (3, 10.0), (4, 7.0), (5, 4.0), (6, 8.0)]); + let flat = eq(&[(1, 0.0)]); + let m = summarize(&equity, &flat); + println!("dip-and-recover -> {m:?}"); + assert_eq!(m.total_pips, 8.0, "final value, not the peak"); + assert_eq!(m.max_drawdown, 6.0, "peak 10 -> trough 4"); + + // --- (4) sign-flipping exposure: long, short, flat, long. --- + // signs: +,+,-,-,0,+ -> adjacent changes at (+,-), (-,0)? wait: + // +0.5 -> +0.5 : same (0 flips) + // +0.5 -> -0.5 : differ (1) + // -0.5 -> -0.5 : same + // -0.5 -> 0.0 : differ (2) [flat distinct from short] + // 0.0 -> +0.5 : differ (3) [flat distinct from long] + // => 3 sign flips. + let exposure = eq(&[(1, 0.5), (2, 0.5), (3, -0.5), (4, -0.5), (5, 0.0), (6, 0.5)]); + let any_eq = eq(&[(1, 0.0), (2, 0.0)]); + let m = summarize(&any_eq, &exposure); + println!("sign-flips long/short/flat -> {m:?}"); + assert_eq!( + m.exposure_sign_flips, 3, + "long->short, short->flat, flat->long all count (flat is sign 0)" + ); + + // --- (5) does the SIGN matter, not the magnitude? +0.1 vs +0.9 = no flip. --- + let exposure = eq(&[(1, 0.1), (2, 0.9), (3, 0.3)]); + let m = summarize(&eq(&[(1, 0.0)]), &exposure); + println!("same-sign varying magnitude -> {m:?}"); + assert_eq!(m.exposure_sign_flips, 0, "magnitude changes are not sign flips"); + + // --- (6) all-negative equity curve: total_pips can be negative (a losing run); + // drawdown measured from the running peak (which starts at the first value). --- + // curve: -1, -3, -2, -5 => peak is -1 (the max so far never improves above -1), + // trough -5 => drawdown = (-1) - (-5) = 4; final = -5. + let equity = eq(&[(1, -1.0), (2, -3.0), (3, -2.0), (4, -5.0)]); + let m = summarize(&equity, &eq(&[(1, -1.0)])); + println!("all-negative -> {m:?}"); + assert_eq!(m.total_pips, -5.0, "a losing run has negative total_pips"); + assert_eq!(m.max_drawdown, 4.0, "peak -1 -> trough -5"); + + println!("c0009_3 OK: empty/monotonic/dip/sign-flip/negative summarize defs all hold"); +} diff --git a/fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs b/fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs new file mode 100644 index 0000000..d24ea3c --- /dev/null +++ b/fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs @@ -0,0 +1,94 @@ +//! Fieldtest c0009 #4 — the f64_field bridge + the to_json structured face, +//! probed from the docs alone. +//! +//! Axis (carrier, north-star sub-surface): f64_field is the documented bridge +//! from a recording sink's `Vec` rows to summarize. The rustdoc says: +//! "extract one f64 field of each row into (ts, f64) samples. Panics if a row +//! has no such field or the field is not an f64 scalar — a wiring bug ... +//! surfaced like the engine's other 'checked at wiring' contract violations." +//! And RunReport::to_json: "field order is fixed; f64 uses the round-trippable +//! {} shortest form; params renders as a JSON object in insertion order." +//! +//! This example exercises: +//! (a) projecting a NON-zero column out of a multi-column recorded row, +//! (b) the documented panic on a wrong-kind field (caught with catch_unwind so +//! the program reports it instead of aborting — a downstream consumer +//! wanting to validate a sink would want this), +//! (c) the exact to_json byte shape, and whether the documented "round-trippable +//! {}" claim survives a 0.0001-style pip_size and a NEGATIVE/fractional pip. + +use std::panic::{self, AssertUnwindSafe}; + +use aura_core::{Scalar, Timestamp}; +use aura_engine::{RunManifest, RunMetrics, RunReport, f64_field}; + +fn main() { + // (a) A recorded row may carry K columns (C7/C8 — a record is a bundle of + // base columns). f64_field(field=1) must pick the SECOND column. + let rows: Vec<(Timestamp, Vec)> = vec![ + (Timestamp(1), vec![Scalar::F64(10.0), Scalar::F64(-1.5)]), + (Timestamp(2), vec![Scalar::F64(20.0), Scalar::F64(2.5)]), + ]; + let col0 = f64_field(&rows, 0); + let col1 = f64_field(&rows, 1); + println!("col0 = {col0:?}"); + println!("col1 = {col1:?}"); + assert_eq!(col0, vec![(Timestamp(1), 10.0), (Timestamp(2), 20.0)]); + assert_eq!(col1, vec![(Timestamp(1), -1.5), (Timestamp(2), 2.5)]); + + // (b) Documented panic: a non-f64 field (here an i64 in the row) is a wiring + // bug. A real consumer building a generic "drain any sink" helper would hit + // this if it mis-declared a column kind. We confirm it PANICS (per docs) + // rather than silently coercing or dropping. + let bad_rows: Vec<(Timestamp, Vec)> = + vec![(Timestamp(1), vec![Scalar::I64(7)])]; + let r = panic::catch_unwind(AssertUnwindSafe(|| f64_field(&bad_rows, 0))); + match r { + Err(_) => println!("(b) f64_field on an i64 field PANICKED as documented (wiring bug)"), + Ok(v) => panic!("(b) expected panic on non-f64 field, got {v:?}"), + } + + // Also: field index out of range panics ("a row has no such field"). + let r = panic::catch_unwind(AssertUnwindSafe(|| f64_field(&rows, 5))); + assert!(r.is_err(), "(b) out-of-range field index panics as documented"); + println!("(b) f64_field on an out-of-range field index PANICKED as documented"); + + // (c) to_json byte shape + the "round-trippable {}" claim under a realistic + // FX pip_size (0.0001) and fractional/negative metric values. + let report = RunReport { + manifest: RunManifest { + commit: "abc123".to_string(), + params: vec![("len".to_string(), 14.0), ("k".to_string(), 2.5)], + window: (Timestamp(1_700_000_000_000_000_000), Timestamp(1_700_000_086_400_000_000)), + seed: 42, + broker: "sim-optimal(pip_size=0.0001)".to_string(), + }, + metrics: RunMetrics { + total_pips: -12.5, + max_drawdown: 33.25, + exposure_sign_flips: 7, + }, + }; + let json = report.to_json(); + println!("(c) to_json() = {json}"); + + // What the rustdoc lets me check WITHOUT reading src: it is one flat-ish JSON + // object, params is a nested object in insertion order, values are present in + // {} shortest form. I assert structure I can justify from the surface; the + // exact KEY NAMES are not on the public surface (recorded as a spec_gap), so + // I check the values I supplied appear, and that the ns timestamps survive as + // integers (round-trippable claim) rather than scientific notation. + assert!(json.starts_with('{') && json.ends_with('}')); + assert!(json.contains("-12.5"), "negative fractional total_pips present"); + assert!(json.contains("33.25"), "fractional drawdown present"); + assert!(json.contains("\"k\":2.5"), "fractional param in insertion order, object form"); + // params object ordering: len appears before k (insertion order, documented). + let li = json.find("\"len\"").expect("len key"); + let ki = json.find("\"k\":2.5").expect("k key"); + assert!(li < ki, "params keys render in insertion order (len before k)"); + // round-trip claim: the big ns window bound must be a bare integer, not 1.7e18. + assert!(json.contains("1700000000000000000"), "ns timestamp as integer, not sci-notation"); + assert!(!json.to_lowercase().contains("e1"), "no scientific-notation float leaked"); + + println!("c0009_4 OK: f64_field column-pick + documented panics + to_json shape"); +}