fieldtest: cycle-0009 — 4 examples, 6 findings
First fieldtest of the run-metrics + manifest report surface. A standalone
downstream-consumer crate (fieldtests/cycle-0009-run-metrics/) path-depends on
the engine crates and exercises the post-0009 surface from the public interface
only (rustdoc + ledger + glossary + project layout, never crates/*/src).
Primary axis empirically met: the north-star "a run emits metrics + manifest"
move is reachable from rustdoc alone — drain two recording sinks -> f64_field ->
summarize -> RunManifest -> to_json, metrics matching the hand model on the first
run, deterministic across reruns.
Findings: 0 bugs, 1 friction, 1 spec_gap, 4 working.
- working x4: north-star reachable from rustdoc; SimBroker firing/slot docs (a
resolved 0007 gap) now carry the example; summarize metric definitions exact
on six degenerate inputs (incl. negative-curve drawdown + flat-as-sign-0);
f64_field panics precise and well-located.
- spec_gap: to_json's JSON key names + {manifest,metrics} nesting are not on
the public surface — a consumer parsing the JSON (C18 registry, the deferred
aura run printer) cannot author against it from rustdoc alone.
- friction: to_json renders whole-valued f64 without a decimal point (3.0 ->
"3"), so one f64 field appears as integer or decimal token within one schema.
Both doc-level findings are the same doc pass and matter mainly for the deferred
aura run (#8) and the C18 registry that will parse this JSON. Spec feeds the next
plan as reference.
refs #6
This commit is contained in:
@@ -0,0 +1,173 @@
|
||||
# Fieldtest — cycle-0009 (run metrics + manifest) — 2026-06-04
|
||||
|
||||
**Status:** Draft — awaiting orchestrator triage
|
||||
**Author:** fieldtester (dispatched by fieldtest skill)
|
||||
|
||||
## Scope
|
||||
|
||||
Cycle 0009 (Walking-skeleton milestone, issue #6) shipped a pure-additive
|
||||
`report` module in `aura-engine`, re-exported at the crate root:
|
||||
`summarize(equity, exposure) -> RunMetrics` (fields `total_pips`,
|
||||
`max_drawdown`, `exposure_sign_flips`); `f64_field(rows, field)` bridging a
|
||||
recording sink's `Vec<Scalar>` rows to `summarize`; `RunManifest`
|
||||
(`commit`, `params`, `window`, `seed`, `broker`); `RunReport { manifest,
|
||||
metrics }` with `to_json() -> String` (the structured C14 face). The intended
|
||||
downstream move: after `Harness::run`, drain recording sinks, project f64
|
||||
columns with `f64_field`, `summarize` the pip-equity + exposure streams, pair
|
||||
with a caller-built `RunManifest`, render `to_json`. The `aura run` CLI that
|
||||
prints this is deferred to #8 and out of scope. Worked from the public interface
|
||||
only (rustdoc via `cargo doc`, `docs/design/INDEX.md`, `docs/glossary.md`,
|
||||
project layout, public re-exports); never `crates/*/src`.
|
||||
|
||||
Build/run: workspace built clean from HEAD (`cargo build --workspace`); each
|
||||
example run via `cargo run --manifest-path
|
||||
fieldtests/cycle-0009-run-metrics/Cargo.toml --bin <name>`, so HEAD source is
|
||||
always what compiled and ran (a standalone non-workspace consumer crate
|
||||
path-depending on the three engine crates, exactly as a C16 project would).
|
||||
|
||||
## Examples
|
||||
|
||||
### fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs — north-star run-to-report
|
||||
- **What it does.** Bootstraps the full SMA(2)/SMA(4) → Sub → Exposure(4) →
|
||||
SimBroker harness with TWO recording sinks (equity tapped on the broker,
|
||||
exposure tapped on the Exposure node), runs a 7-tick rising-price stream,
|
||||
drains both sinks, `f64_field`-projects each, `summarize`s, builds a
|
||||
`RunManifest`, and renders `to_json`.
|
||||
- **Why it fits.** This is the carrier's primary axis verbatim — the end-to-end
|
||||
research-loop move the whole module exists to enable, reaching every new
|
||||
surface element once.
|
||||
- **Outcome.** Built, ran, matched the hand model exactly: equity
|
||||
`[0,0,0,0,1,2,3]` → `total_pips=3.0`, `max_drawdown=0.0`; exposure all `+0.5`
|
||||
→ `exposure_sign_flips=0`. `to_json` emitted a single well-formed object.
|
||||
|
||||
### fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs — comparison + determinism
|
||||
- **What it does.** Runs the same harness twice at `Exposure` scale=4 (asserts
|
||||
bit-identical metrics AND bit-identical `to_json` across the two runs), then a
|
||||
third run at scale=2, and compares `total_pips`.
|
||||
- **Why it fits.** The carrier's second axis: determinism (C1) plus a
|
||||
smallest-slice tuning-sweep comparison (C12/C21) read off `RunMetrics`.
|
||||
- **Outcome.** Built, ran, matched expected: scale=4 deterministic
|
||||
(`total_pips=3`), scale=2 = `total_pips=6` (exactly double — exposure
|
||||
saturates 0.5→1.0), both drawdown-free, no sign flips.
|
||||
|
||||
### fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs — degenerate summarize semantics
|
||||
- **What it does.** Calls `summarize` with six hand-built `&[(Timestamp,f64)]`
|
||||
argument literals: empty, monotonic-up, dip-and-recover, long/short/flat
|
||||
sign-flips, same-sign varying magnitude, and an all-negative curve.
|
||||
- **Why it fits.** The carrier's third axis: verify the documented metric
|
||||
definitions from docs alone. The slices are the public function's direct
|
||||
argument type — calling the function with literals, not authoring an
|
||||
intermediate representation.
|
||||
- **Outcome.** Built, ran, every documented definition held exactly: empty→
|
||||
zeros; last-value `total_pips` (incl. negative); running-peak drawdown (incl.
|
||||
on a negative curve, peak −1 → trough −5 = 4); sign-flips counting flat (sign
|
||||
0) as distinct from long/short (3 flips), magnitude changes not counted.
|
||||
|
||||
### fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs — f64_field bridge + JSON shape
|
||||
- **What it does.** Projects a non-zero column from a multi-column recorded row;
|
||||
confirms (via `catch_unwind`) the documented panics on a non-f64 field and an
|
||||
out-of-range field index; inspects the exact `to_json` byte shape under a
|
||||
realistic FX `pip_size=0.0001`, fractional/negative metrics, and ns-scale
|
||||
timestamps.
|
||||
- **Why it fits.** Exercises the `f64_field` "checked at wiring" contract and
|
||||
the `to_json` round-trippability claim — the corners of the north-star
|
||||
surface a generic sink-drainer would hit.
|
||||
- **Outcome.** Built, ran, matched: column-pick correct; both panics fired with
|
||||
precise messages; JSON kept ns timestamps as bare integers (no sci-notation),
|
||||
params in insertion order as a nested object.
|
||||
|
||||
## Findings
|
||||
|
||||
### [working] North-star run-to-report is reachable from rustdoc alone
|
||||
- **Example(s):** c0009_1, c0009_2.
|
||||
- **What happened.** The full drain → `f64_field` → `summarize` →
|
||||
`RunManifest` → `to_json` chain was authored entirely from the crate-root
|
||||
rustdoc (`fn.summarize`, `fn.f64_field`, the three structs, `to_json`) plus
|
||||
the prior fieldtest corpus for the harness wiring. Metrics matched the hand
|
||||
model on the first run; `to_json` produced a clean object.
|
||||
- **Why working.** The cycle's headline move is reachable and correct without
|
||||
reading `src` — the module rustdoc names the post-run drain-and-fold workflow
|
||||
explicitly, and the field docs are precise enough to predict every value.
|
||||
- **Recommended action:** carry-on.
|
||||
|
||||
### [working] SimBroker firing/warm-up/slot-order docs (a resolved 0007 spec_gap) now carry the example
|
||||
- **Example(s):** c0009_1.
|
||||
- **What happened.** Predicting the equity sink's `[0,0,0,0,1,2,3]` shape
|
||||
required knowing slot 0 = exposure / slot 1 = price, both `Firing::Any`, and
|
||||
the "leading 0.0 rows, one row per price cycle" warm-up. All three are now on
|
||||
`struct.SimBroker` rustdoc ("Input slots" / "Firing and warm-up"). The 0007
|
||||
fieldtest had to recover these from the ledger + commit body and recorded them
|
||||
as a spec_gap; they are now on the surface.
|
||||
- **Why working.** A prior gap is closed and directly enabled this cycle's
|
||||
north-star example to be authored from rustdoc.
|
||||
- **Recommended action:** carry-on.
|
||||
|
||||
### [working] summarize metric definitions hold exactly on all degenerate inputs
|
||||
- **Example(s):** c0009_3.
|
||||
- **What happened.** All six hand-built cases matched the rustdoc field
|
||||
definitions to the bit, including the two subtle ones: drawdown measured from
|
||||
the running peak even when the whole curve is negative, and `exposure_sign_flips`
|
||||
treating flat (0.0 → sign 0) as distinct from long/short.
|
||||
- **Why working.** The `RunMetrics` field docs are unambiguous and empirically
|
||||
exact; the "flat is sign 0" and "running-peak" subtleties are spelled out and
|
||||
behave as written.
|
||||
- **Recommended action:** carry-on.
|
||||
|
||||
### [working] f64_field panics are precise and well-located
|
||||
- **Example(s):** c0009_4.
|
||||
- **What happened.** `f64_field` on an `I64` field panicked with
|
||||
`f64_field: field 0 is not an f64 scalar: I64(7)`; on an out-of-range index
|
||||
with `f64_field: row has no field 5 (row width 2)`. Both match the documented
|
||||
"checked at wiring" panic contract and name the exact offending field/width.
|
||||
- **Why working.** The diagnostic is actionable and matches the rustdoc promise;
|
||||
a downstream consumer mis-wiring a sink column gets a precise message.
|
||||
- **Recommended action:** carry-on.
|
||||
|
||||
### [spec_gap] to_json key names / nesting are not on the public surface
|
||||
- **Example(s):** c0009_1, c0009_4.
|
||||
- **What happened.** `RunReport::to_json` rustdoc promises "field order is
|
||||
fixed", "params renders as a JSON object in insertion order", "f64 uses the
|
||||
round-trippable `{}` shortest form" — but it does **not** state the actual JSON
|
||||
key names or the `{manifest:{...},metrics:{...}}` nesting. The observed shape
|
||||
is `{"manifest":{"commit":..,"params":{..},"window":[from,to],"seed":..,
|
||||
"broker":..},"metrics":{"total_pips":..,"max_drawdown":..,"exposure_sign_flips":..}}`.
|
||||
I had to read the runtime output to learn the schema; a consumer parsing the
|
||||
JSON (the C18 registry, the deferred `aura run` printer) cannot author against
|
||||
it from rustdoc alone and would couple to an undocumented contract.
|
||||
- **Why spec_gap.** The reading I guessed (keys = struct field names, `window`
|
||||
as a 2-array, `params` as an object) was the natural one and proved correct,
|
||||
but it is unconstrained by the public surface; another reading (e.g. `window`
|
||||
as `{from,to}`, or top-level flattening) was equally plausible. For a
|
||||
"machine-readable" (C14) face this schema is part of the contract.
|
||||
- **Recommended action:** tighten the design ledger / rustdoc — add a documented
|
||||
JSON schema example to `to_json` rustdoc (a one-line sample object suffices),
|
||||
or ratify the field-name-mirroring as the stated contract.
|
||||
|
||||
### [friction] to_json renders whole-valued f64 without a decimal point (3.0 → `3`)
|
||||
- **Example(s):** c0009_1, c0009_2.
|
||||
- **What happened.** `total_pips: 3.0_f64` serializes as `"total_pips":3` (and
|
||||
`4.0` param → `4`), whereas `-12.5`/`2.5` keep the point. This is the
|
||||
documented round-trippable `{}` shortest form and is valid JSON (a number is a
|
||||
number), so it is not a bug. But a downstream consumer string-matching or
|
||||
schema-typing the field could trip: the same logical f64 field appears
|
||||
sometimes as an integer token, sometimes as a decimal token, within one
|
||||
schema. My example had to substring-match `"3"` rather than `"3.0"`.
|
||||
- **Why friction.** The task completed, but the mixed integer/decimal rendering
|
||||
of a single f64 field is a small surprise for a consumer expecting stable
|
||||
per-field token shape; it interacts with the spec_gap above (no documented
|
||||
schema to set the expectation).
|
||||
- **Recommended action:** plan (tidy) — either document explicitly in the
|
||||
`to_json` rustdoc that f64 fields may render without a fractional part, or (if
|
||||
a stable decimal shape is wanted for the registry) normalize. Low urgency;
|
||||
fold into the same doc pass as the spec_gap.
|
||||
|
||||
## Recommendation summary
|
||||
|
||||
| Finding | Class | Action |
|
||||
|---|---|---|
|
||||
| North-star reachable from rustdoc | working | carry-on |
|
||||
| SimBroker firing/slot docs now carry the example | working | carry-on |
|
||||
| summarize definitions exact on degenerate inputs | working | carry-on |
|
||||
| f64_field panics precise + well-located | working | carry-on |
|
||||
| to_json key names / nesting not on public surface | spec_gap | tighten the design ledger / rustdoc (or ratify) |
|
||||
| to_json renders whole f64 as integer token | friction | plan (tidy doc pass) |
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "aura-core"
|
||||
version = "0.1.0"
|
||||
|
||||
[[package]]
|
||||
name = "aura-engine"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"aura-core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aura-std"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"aura-core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "c0009-fieldtest"
|
||||
version = "0.0.0"
|
||||
dependencies = [
|
||||
"aura-core",
|
||||
"aura-engine",
|
||||
"aura-std",
|
||||
]
|
||||
@@ -0,0 +1,37 @@
|
||||
# Standalone downstream-consumer crate for the cycle-0009 fieldtest (run report).
|
||||
#
|
||||
# Like the cycle-0007/0008 fixtures, this is NOT a member of the aura workspace —
|
||||
# it path-depends on the engine crates exactly as a real research project (C16)
|
||||
# would, and is built via
|
||||
# `cargo run --manifest-path fieldtests/cycle-0009-run-metrics/Cargo.toml --bin <name>`
|
||||
# so HEAD source is always what runs.
|
||||
# Empty [workspace] table: marks this fixture crate as its OWN workspace root
|
||||
# (documented in docs/project-layout.md as the nested-project onboarding fix).
|
||||
[workspace]
|
||||
|
||||
[package]
|
||||
name = "c0009-fieldtest"
|
||||
version = "0.0.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
|
||||
[dependencies]
|
||||
aura-core = { path = "../../crates/aura-core" }
|
||||
aura-engine = { path = "../../crates/aura-engine" }
|
||||
aura-std = { path = "../../crates/aura-std" }
|
||||
|
||||
[[bin]]
|
||||
name = "c0009_1_run_to_report"
|
||||
path = "c0009_1_run_to_report.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "c0009_2_compare_two_runs"
|
||||
path = "c0009_2_compare_two_runs.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "c0009_3_degenerate_streams"
|
||||
path = "c0009_3_degenerate_streams.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "c0009_4_f64field_and_json"
|
||||
path = "c0009_4_f64field_and_json.rs"
|
||||
@@ -0,0 +1,158 @@
|
||||
//! Fieldtest c0009 #1 — the north-star run-to-report move, end to end.
|
||||
//!
|
||||
//! Axis (carrier): bootstrap a harness (SMA fast/slow -> Sub -> Exposure ->
|
||||
//! SimBroker, price tapped into the broker) with TWO recording sinks (equity on
|
||||
//! the broker, exposure on the Exposure node), run it, drain both sinks,
|
||||
//! f64_field + summarize + build a RunManifest + to_json. Question under test:
|
||||
//! does the public rustdoc make this reachable WITHOUT reading crates/*/src?
|
||||
//!
|
||||
//! price ----+--> SMA(2) --\
|
||||
//! | Sub(fast - slow) --> Exposure(4) --+--> SimBroker --> equitySink
|
||||
//! +--> SMA(4) --/ | ^
|
||||
//! | v |
|
||||
//! | exposureSink |
|
||||
//! +------------------------------------------------ price --+ (slot 1)
|
||||
//!
|
||||
//! Public-surface facts used (rustdoc only):
|
||||
//! - aura_std::Exposure::new(scale) = clamp(signal/scale, -1, +1), None until warm.
|
||||
//! - aura_std::SimBroker::new(pip_size): slot 0 = exposure, slot 1 = price;
|
||||
//! both Firing::Any, leading 0.0 rows during warm-up, one row per price cycle
|
||||
//! (rustdoc struct.SimBroker "Input slots"/"Firing and warm-up" — these were
|
||||
//! a 0007 spec_gap, now ON the surface; recorded as a `working` for that).
|
||||
//! - aura_engine::f64_field(rows, field) extracts one f64 column (rustdoc).
|
||||
//! - aura_engine::summarize(equity, exposure) -> RunMetrics (rustdoc).
|
||||
//! - RunManifest { commit, params, window, seed, broker }, RunReport, to_json.
|
||||
|
||||
use std::sync::mpsc::{self, Sender};
|
||||
|
||||
use aura_core::{Ctx, Firing, InputSpec, Node, NodeSchema, Scalar, ScalarKind, Timestamp};
|
||||
use aura_engine::{Edge, Harness, RunManifest, RunReport, SourceSpec, Target, f64_field, summarize};
|
||||
use aura_std::{Exposure, SimBroker, Sma, Sub};
|
||||
|
||||
/// A recording sink: one f64 input, persists (ts, row) out of graph (C8/C22).
|
||||
/// Records the WHOLE row (Vec<Scalar>) the way a real registry sink would, so
|
||||
/// f64_field can later project a column — this is the shape summarize expects.
|
||||
struct RowRecorder {
|
||||
tx: Sender<(Timestamp, Vec<Scalar>)>,
|
||||
}
|
||||
impl Node for RowRecorder {
|
||||
fn schema(&self) -> NodeSchema {
|
||||
NodeSchema {
|
||||
inputs: vec![InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Any }],
|
||||
output: vec![],
|
||||
}
|
||||
}
|
||||
fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
|
||||
let w = ctx.f64_in(0);
|
||||
if w.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let _ = self.tx.send((ctx.now(), vec![Scalar::F64(w[0])]));
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn f64_stream(pairs: &[(i64, f64)]) -> Vec<(Timestamp, Scalar)> {
|
||||
pairs.iter().map(|&(t, v)| (Timestamp(t), Scalar::F64(v))).collect()
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let (eq_tx, eq_rx) = mpsc::channel();
|
||||
let (exp_tx, exp_rx) = mpsc::channel();
|
||||
|
||||
// nodes: 0=SMA(2) fast, 1=SMA(4) slow, 2=Sub, 3=Exposure(4),
|
||||
// 4=SimBroker(1.0), 5=equity sink (taps broker), 6=exposure sink (taps Exposure).
|
||||
let mut h = Harness::bootstrap(
|
||||
vec![
|
||||
Box::new(Sma::new(2)),
|
||||
Box::new(Sma::new(4)),
|
||||
Box::new(Sub::new()),
|
||||
Box::new(Exposure::new(4.0)),
|
||||
Box::new(SimBroker::new(1.0)),
|
||||
Box::new(RowRecorder { tx: eq_tx }),
|
||||
Box::new(RowRecorder { tx: exp_tx }),
|
||||
],
|
||||
vec![SourceSpec {
|
||||
kind: ScalarKind::F64,
|
||||
targets: vec![
|
||||
Target { node: 0, slot: 0 }, // -> SMA fast
|
||||
Target { node: 1, slot: 0 }, // -> SMA slow
|
||||
Target { node: 4, slot: 1 }, // -> SimBroker price (slot 1)
|
||||
],
|
||||
}],
|
||||
vec![
|
||||
Edge { from: 0, to: 2, slot: 0, from_field: 0 }, // SMA2 -> Sub.in0 (fast)
|
||||
Edge { from: 1, to: 2, slot: 1, from_field: 0 }, // SMA4 -> Sub.in1 (slow)
|
||||
Edge { from: 2, to: 3, slot: 0, from_field: 0 }, // Sub -> Exposure
|
||||
Edge { from: 3, to: 4, slot: 0, from_field: 0 }, // Exposure -> SimBroker.in0
|
||||
Edge { from: 3, to: 6, slot: 0, from_field: 0 }, // Exposure -> exposure sink
|
||||
Edge { from: 4, to: 5, slot: 0, from_field: 0 }, // SimBroker -> equity sink
|
||||
],
|
||||
)
|
||||
.expect("valid two-sink signal-quality DAG");
|
||||
|
||||
let prices: &[(i64, f64)] = &[
|
||||
(1, 100.0),
|
||||
(2, 102.0),
|
||||
(3, 104.0),
|
||||
(4, 106.0),
|
||||
(5, 108.0),
|
||||
(6, 110.0),
|
||||
(7, 112.0),
|
||||
];
|
||||
h.run(vec![f64_stream(prices)]);
|
||||
|
||||
// Drain both sinks (the World's post-run step the rustdoc describes).
|
||||
let equity_rows: Vec<(Timestamp, Vec<Scalar>)> = eq_rx.try_iter().collect();
|
||||
let exposure_rows: Vec<(Timestamp, Vec<Scalar>)> = exp_rx.try_iter().collect();
|
||||
println!("equity rows = {equity_rows:?}");
|
||||
println!("exposure rows = {exposure_rows:?}");
|
||||
|
||||
// Project field 0 of each (the documented bridge to summarize).
|
||||
let equity = f64_field(&equity_rows, 0);
|
||||
let exposure = f64_field(&exposure_rows, 0);
|
||||
|
||||
let metrics = summarize(&equity, &exposure);
|
||||
println!("metrics = {metrics:?}");
|
||||
|
||||
// Hand model (public-surface, from c0007_1 + SimBroker rustdoc):
|
||||
// equity curve = [0,0,0,0,1,2,3] (leading zeros until exposure warms at t=5),
|
||||
// so total_pips = 3.0, monotonic non-decreasing => max_drawdown = 0.0.
|
||||
// exposure samples are recorded only from the Exposure node's first warm
|
||||
// cycle (t=4 on): all +0.5 (long), no sign change => exposure_sign_flips = 0.
|
||||
assert_eq!(metrics.total_pips, 3.0, "final cumulative pips");
|
||||
assert_eq!(metrics.max_drawdown, 0.0, "monotonic-up curve has no drawdown");
|
||||
assert_eq!(metrics.exposure_sign_flips, 0, "exposure stays long throughout");
|
||||
|
||||
// Pair with a caller-built manifest and render the structured C14 face.
|
||||
let report = RunReport {
|
||||
manifest: RunManifest {
|
||||
commit: "fieldtest-c0009-synthetic".to_string(),
|
||||
params: vec![
|
||||
("sma_fast".to_string(), 2.0),
|
||||
("sma_slow".to_string(), 4.0),
|
||||
("exposure_scale".to_string(), 4.0),
|
||||
],
|
||||
window: (Timestamp(1), Timestamp(7)),
|
||||
seed: 0,
|
||||
broker: "sim-optimal(pip_size=1)".to_string(),
|
||||
},
|
||||
metrics,
|
||||
};
|
||||
let json = report.to_json();
|
||||
println!("to_json() =\n{json}");
|
||||
|
||||
// The rustdoc says: machine-readable JSON, field order fixed, params a JSON
|
||||
// object in insertion order, f64 via {} shortest form. It does NOT state the
|
||||
// exact field NAMES or nesting on the public surface — recorded as a
|
||||
// spec_gap. I assert only what the rustdoc promises that I can check without
|
||||
// reading src: it must parse as a non-empty object string and carry the
|
||||
// numbers I put in. (No serde in this crate either, by design — so I do a
|
||||
// substring check, which is itself a small friction.)
|
||||
assert!(json.starts_with('{') && json.ends_with('}'), "looks like a JSON object");
|
||||
assert!(json.contains("3"), "total_pips value present somewhere");
|
||||
assert!(json.contains("sma_fast"), "a param key present");
|
||||
assert!(json.contains("sim-optimal(pip_size=1)"), "broker label present");
|
||||
|
||||
println!("c0009_1 OK: two sinks -> f64_field -> summarize -> RunManifest -> to_json");
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
//! Fieldtest c0009 #2 — comparison + determinism over the report surface.
|
||||
//!
|
||||
//! Axis (carrier): run two harnesses differing in a tuning param (Exposure
|
||||
//! scale), confirm each run is deterministic (bit-identical metrics across two
|
||||
//! runs) and the two metrics differ as expected. This is the smallest slice of
|
||||
//! the World's reason to exist (C21): compare runs by their RunMetrics.
|
||||
//!
|
||||
//! Same SMA-cross harness as #1, but the Exposure node's `scale` is the swept
|
||||
//! tuning param. A smaller scale => larger |exposure| (clamp(signal/scale,..)),
|
||||
//! so the same price move earns proportionally more pips — until the exposure
|
||||
//! saturates at +1. So total_pips(scale=2) > total_pips(scale=4) here.
|
||||
|
||||
use std::sync::mpsc::{self, Sender};
|
||||
|
||||
use aura_core::{Ctx, Firing, InputSpec, Node, NodeSchema, Scalar, ScalarKind, Timestamp};
|
||||
use aura_engine::{
|
||||
Edge, Harness, RunManifest, RunMetrics, RunReport, SourceSpec, Target, f64_field, summarize,
|
||||
};
|
||||
use aura_std::{Exposure, SimBroker, Sma, Sub};
|
||||
|
||||
struct RowRecorder {
|
||||
tx: Sender<(Timestamp, Vec<Scalar>)>,
|
||||
}
|
||||
impl Node for RowRecorder {
|
||||
fn schema(&self) -> NodeSchema {
|
||||
NodeSchema {
|
||||
inputs: vec![InputSpec { kind: ScalarKind::F64, lookback: 1, firing: Firing::Any }],
|
||||
output: vec![],
|
||||
}
|
||||
}
|
||||
fn eval(&mut self, ctx: Ctx<'_>) -> Option<&[Scalar]> {
|
||||
let w = ctx.f64_in(0);
|
||||
if w.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let _ = self.tx.send((ctx.now(), vec![Scalar::F64(w[0])]));
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn f64_stream(pairs: &[(i64, f64)]) -> Vec<(Timestamp, Scalar)> {
|
||||
pairs.iter().map(|&(t, v)| (Timestamp(t), Scalar::F64(v))).collect()
|
||||
}
|
||||
|
||||
const PRICES: &[(i64, f64)] = &[
|
||||
(1, 100.0),
|
||||
(2, 102.0),
|
||||
(3, 104.0),
|
||||
(4, 106.0),
|
||||
(5, 108.0),
|
||||
(6, 110.0),
|
||||
(7, 112.0),
|
||||
];
|
||||
|
||||
/// One full run: bootstrap a harness with the given exposure scale, run, drain
|
||||
/// the two sinks, reduce to a RunReport. This is exactly the World's per-instance
|
||||
/// step in a tuning sweep (C12/C19/C20), authored in plain Rust.
|
||||
fn run_one(exposure_scale: f64) -> RunReport {
|
||||
let (eq_tx, eq_rx) = mpsc::channel();
|
||||
let (exp_tx, exp_rx) = mpsc::channel();
|
||||
|
||||
let mut h = Harness::bootstrap(
|
||||
vec![
|
||||
Box::new(Sma::new(2)),
|
||||
Box::new(Sma::new(4)),
|
||||
Box::new(Sub::new()),
|
||||
Box::new(Exposure::new(exposure_scale)),
|
||||
Box::new(SimBroker::new(1.0)),
|
||||
Box::new(RowRecorder { tx: eq_tx }),
|
||||
Box::new(RowRecorder { tx: exp_tx }),
|
||||
],
|
||||
vec![SourceSpec {
|
||||
kind: ScalarKind::F64,
|
||||
targets: vec![
|
||||
Target { node: 0, slot: 0 },
|
||||
Target { node: 1, slot: 0 },
|
||||
Target { node: 4, slot: 1 },
|
||||
],
|
||||
}],
|
||||
vec![
|
||||
Edge { from: 0, to: 2, slot: 0, from_field: 0 },
|
||||
Edge { from: 1, to: 2, slot: 1, from_field: 0 },
|
||||
Edge { from: 2, to: 3, slot: 0, from_field: 0 },
|
||||
Edge { from: 3, to: 4, slot: 0, from_field: 0 },
|
||||
Edge { from: 3, to: 6, slot: 0, from_field: 0 },
|
||||
Edge { from: 4, to: 5, slot: 0, from_field: 0 },
|
||||
],
|
||||
)
|
||||
.expect("valid harness");
|
||||
|
||||
h.run(vec![f64_stream(PRICES)]);
|
||||
|
||||
let equity = f64_field(&eq_rx.try_iter().collect::<Vec<_>>(), 0);
|
||||
let exposure = f64_field(&exp_rx.try_iter().collect::<Vec<_>>(), 0);
|
||||
let metrics = summarize(&equity, &exposure);
|
||||
|
||||
RunReport {
|
||||
manifest: RunManifest {
|
||||
commit: "fieldtest-c0009-synthetic".to_string(),
|
||||
params: vec![("exposure_scale".to_string(), exposure_scale)],
|
||||
window: (Timestamp(1), Timestamp(7)),
|
||||
seed: 0,
|
||||
broker: "sim-optimal(pip_size=1)".to_string(),
|
||||
},
|
||||
metrics,
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
// Determinism (C1): the same instance run twice yields bit-identical metrics.
|
||||
let a1 = run_one(4.0);
|
||||
let a2 = run_one(4.0);
|
||||
assert_eq!(a1.metrics, a2.metrics, "scale=4 deterministic across runs");
|
||||
assert_eq!(a1.to_json(), a2.to_json(), "scale=4 JSON bit-identical");
|
||||
println!("determinism OK: scale=4 metrics = {:?}", a1.metrics);
|
||||
|
||||
// Comparison: a second instance with a different tuning param.
|
||||
let b = run_one(2.0);
|
||||
println!("scale=4 report json = {}", a1.to_json());
|
||||
println!("scale=2 report json = {}", b.to_json());
|
||||
|
||||
// Expectation (public model): scale=2 doubles |exposure| (0.5 -> 1.0 saturated)
|
||||
// versus scale=4 (0.5), so the same +2/cycle price move earns ~2x the pips.
|
||||
let RunMetrics { total_pips: p4, .. } = a1.metrics;
|
||||
let RunMetrics { total_pips: p2, .. } = b.metrics;
|
||||
println!("total_pips scale=4 = {p4}, scale=2 = {p2}");
|
||||
assert!(p2 > p4, "smaller scale => larger exposure => more pips ({p2} > {p4})");
|
||||
|
||||
// Both should be drawdown-free monotonic-up curves on a steadily-rising price,
|
||||
// and stay long throughout (no sign flips) — a sanity check the comparison is
|
||||
// apples-to-apples, differing only on the swept axis.
|
||||
assert_eq!(a1.metrics.max_drawdown, 0.0);
|
||||
assert_eq!(b.metrics.max_drawdown, 0.0);
|
||||
assert_eq!(a1.metrics.exposure_sign_flips, 0);
|
||||
assert_eq!(b.metrics.exposure_sign_flips, 0);
|
||||
|
||||
println!("c0009_2 OK: deterministic + comparable two-run metrics over the report surface");
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
//! Fieldtest c0009 #3 — degenerate-stream semantics from the docs alone.
|
||||
//!
|
||||
//! Axis (carrier): feed `summarize` empty / monotonic / sign-flipping HAND-BUILT
|
||||
//! streams and verify the documented metric definitions hold:
|
||||
//! - total_pips = last value of the cumulative pip curve, 0.0 if empty
|
||||
//! - max_drawdown = max_t (running_peak(t) - equity(t)), >= 0, 0 if
|
||||
//! monotonic-non-decreasing or empty
|
||||
//! - exposure_sign_flips = count of adjacent samples whose SIGN differs, with
|
||||
//! zero normalizing to sign 0 (flat distinct from
|
||||
//! long/short)
|
||||
//!
|
||||
//! NOTE on fixture form: `summarize(equity, exposure)` takes `&[(Timestamp, f64)]`
|
||||
//! DIRECTLY as its public signature. Hand-building those slices is calling the
|
||||
//! public function with literal arguments — NOT hand-authoring an intermediate
|
||||
//! representation. (f64_field, the sink->slice bridge, is exercised in #1/#2/#4.)
|
||||
|
||||
use aura_engine::summarize;
|
||||
use aura_core::Timestamp;
|
||||
|
||||
fn eq(samples: &[(i64, f64)]) -> Vec<(Timestamp, f64)> {
|
||||
samples.iter().map(|&(t, v)| (Timestamp(t), v)).collect()
|
||||
}
|
||||
|
||||
fn main() {
|
||||
// --- (1) empty streams -> all zeros (documented "0.0 if empty"). ---
|
||||
let m = summarize(&[], &[]);
|
||||
println!("empty -> {m:?}");
|
||||
assert_eq!(m.total_pips, 0.0);
|
||||
assert_eq!(m.max_drawdown, 0.0);
|
||||
assert_eq!(m.exposure_sign_flips, 0);
|
||||
|
||||
// --- (2) monotonic non-decreasing curve -> drawdown 0, last value total. ---
|
||||
let equity = eq(&[(1, 0.0), (2, 1.0), (3, 3.0), (4, 6.0), (5, 10.0)]);
|
||||
let exposure = eq(&[(1, 0.5), (2, 0.5), (3, 0.5), (4, 0.5), (5, 0.5)]);
|
||||
let m = summarize(&equity, &exposure);
|
||||
println!("monotonic-up -> {m:?}");
|
||||
assert_eq!(m.total_pips, 10.0, "last value of curve");
|
||||
assert_eq!(m.max_drawdown, 0.0, "monotonic => no drawdown");
|
||||
assert_eq!(m.exposure_sign_flips, 0, "constant-sign exposure");
|
||||
|
||||
// --- (3) a dip then recovery -> worst peak-to-trough drawdown. ---
|
||||
// peak 10 at t=3, trough 4 at t=5 => max drawdown 6; recovers to 8 (final).
|
||||
let equity = eq(&[(1, 0.0), (2, 5.0), (3, 10.0), (4, 7.0), (5, 4.0), (6, 8.0)]);
|
||||
let flat = eq(&[(1, 0.0)]);
|
||||
let m = summarize(&equity, &flat);
|
||||
println!("dip-and-recover -> {m:?}");
|
||||
assert_eq!(m.total_pips, 8.0, "final value, not the peak");
|
||||
assert_eq!(m.max_drawdown, 6.0, "peak 10 -> trough 4");
|
||||
|
||||
// --- (4) sign-flipping exposure: long, short, flat, long. ---
|
||||
// signs: +,+,-,-,0,+ -> adjacent changes at (+,-), (-,0)? wait:
|
||||
// +0.5 -> +0.5 : same (0 flips)
|
||||
// +0.5 -> -0.5 : differ (1)
|
||||
// -0.5 -> -0.5 : same
|
||||
// -0.5 -> 0.0 : differ (2) [flat distinct from short]
|
||||
// 0.0 -> +0.5 : differ (3) [flat distinct from long]
|
||||
// => 3 sign flips.
|
||||
let exposure = eq(&[(1, 0.5), (2, 0.5), (3, -0.5), (4, -0.5), (5, 0.0), (6, 0.5)]);
|
||||
let any_eq = eq(&[(1, 0.0), (2, 0.0)]);
|
||||
let m = summarize(&any_eq, &exposure);
|
||||
println!("sign-flips long/short/flat -> {m:?}");
|
||||
assert_eq!(
|
||||
m.exposure_sign_flips, 3,
|
||||
"long->short, short->flat, flat->long all count (flat is sign 0)"
|
||||
);
|
||||
|
||||
// --- (5) does the SIGN matter, not the magnitude? +0.1 vs +0.9 = no flip. ---
|
||||
let exposure = eq(&[(1, 0.1), (2, 0.9), (3, 0.3)]);
|
||||
let m = summarize(&eq(&[(1, 0.0)]), &exposure);
|
||||
println!("same-sign varying magnitude -> {m:?}");
|
||||
assert_eq!(m.exposure_sign_flips, 0, "magnitude changes are not sign flips");
|
||||
|
||||
// --- (6) all-negative equity curve: total_pips can be negative (a losing run);
|
||||
// drawdown measured from the running peak (which starts at the first value). ---
|
||||
// curve: -1, -3, -2, -5 => peak is -1 (the max so far never improves above -1),
|
||||
// trough -5 => drawdown = (-1) - (-5) = 4; final = -5.
|
||||
let equity = eq(&[(1, -1.0), (2, -3.0), (3, -2.0), (4, -5.0)]);
|
||||
let m = summarize(&equity, &eq(&[(1, -1.0)]));
|
||||
println!("all-negative -> {m:?}");
|
||||
assert_eq!(m.total_pips, -5.0, "a losing run has negative total_pips");
|
||||
assert_eq!(m.max_drawdown, 4.0, "peak -1 -> trough -5");
|
||||
|
||||
println!("c0009_3 OK: empty/monotonic/dip/sign-flip/negative summarize defs all hold");
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
//! Fieldtest c0009 #4 — the f64_field bridge + the to_json structured face,
|
||||
//! probed from the docs alone.
|
||||
//!
|
||||
//! Axis (carrier, north-star sub-surface): f64_field is the documented bridge
|
||||
//! from a recording sink's `Vec<Scalar>` rows to summarize. The rustdoc says:
|
||||
//! "extract one f64 field of each row into (ts, f64) samples. Panics if a row
|
||||
//! has no such field or the field is not an f64 scalar — a wiring bug ...
|
||||
//! surfaced like the engine's other 'checked at wiring' contract violations."
|
||||
//! And RunReport::to_json: "field order is fixed; f64 uses the round-trippable
|
||||
//! {} shortest form; params renders as a JSON object in insertion order."
|
||||
//!
|
||||
//! This example exercises:
|
||||
//! (a) projecting a NON-zero column out of a multi-column recorded row,
|
||||
//! (b) the documented panic on a wrong-kind field (caught with catch_unwind so
|
||||
//! the program reports it instead of aborting — a downstream consumer
|
||||
//! wanting to validate a sink would want this),
|
||||
//! (c) the exact to_json byte shape, and whether the documented "round-trippable
|
||||
//! {}" claim survives a 0.0001-style pip_size and a NEGATIVE/fractional pip.
|
||||
|
||||
use std::panic::{self, AssertUnwindSafe};
|
||||
|
||||
use aura_core::{Scalar, Timestamp};
|
||||
use aura_engine::{RunManifest, RunMetrics, RunReport, f64_field};
|
||||
|
||||
fn main() {
|
||||
// (a) A recorded row may carry K columns (C7/C8 — a record is a bundle of
|
||||
// base columns). f64_field(field=1) must pick the SECOND column.
|
||||
let rows: Vec<(Timestamp, Vec<Scalar>)> = vec![
|
||||
(Timestamp(1), vec![Scalar::F64(10.0), Scalar::F64(-1.5)]),
|
||||
(Timestamp(2), vec![Scalar::F64(20.0), Scalar::F64(2.5)]),
|
||||
];
|
||||
let col0 = f64_field(&rows, 0);
|
||||
let col1 = f64_field(&rows, 1);
|
||||
println!("col0 = {col0:?}");
|
||||
println!("col1 = {col1:?}");
|
||||
assert_eq!(col0, vec![(Timestamp(1), 10.0), (Timestamp(2), 20.0)]);
|
||||
assert_eq!(col1, vec![(Timestamp(1), -1.5), (Timestamp(2), 2.5)]);
|
||||
|
||||
// (b) Documented panic: a non-f64 field (here an i64 in the row) is a wiring
|
||||
// bug. A real consumer building a generic "drain any sink" helper would hit
|
||||
// this if it mis-declared a column kind. We confirm it PANICS (per docs)
|
||||
// rather than silently coercing or dropping.
|
||||
let bad_rows: Vec<(Timestamp, Vec<Scalar>)> =
|
||||
vec![(Timestamp(1), vec![Scalar::I64(7)])];
|
||||
let r = panic::catch_unwind(AssertUnwindSafe(|| f64_field(&bad_rows, 0)));
|
||||
match r {
|
||||
Err(_) => println!("(b) f64_field on an i64 field PANICKED as documented (wiring bug)"),
|
||||
Ok(v) => panic!("(b) expected panic on non-f64 field, got {v:?}"),
|
||||
}
|
||||
|
||||
// Also: field index out of range panics ("a row has no such field").
|
||||
let r = panic::catch_unwind(AssertUnwindSafe(|| f64_field(&rows, 5)));
|
||||
assert!(r.is_err(), "(b) out-of-range field index panics as documented");
|
||||
println!("(b) f64_field on an out-of-range field index PANICKED as documented");
|
||||
|
||||
// (c) to_json byte shape + the "round-trippable {}" claim under a realistic
|
||||
// FX pip_size (0.0001) and fractional/negative metric values.
|
||||
let report = RunReport {
|
||||
manifest: RunManifest {
|
||||
commit: "abc123".to_string(),
|
||||
params: vec![("len".to_string(), 14.0), ("k".to_string(), 2.5)],
|
||||
window: (Timestamp(1_700_000_000_000_000_000), Timestamp(1_700_000_086_400_000_000)),
|
||||
seed: 42,
|
||||
broker: "sim-optimal(pip_size=0.0001)".to_string(),
|
||||
},
|
||||
metrics: RunMetrics {
|
||||
total_pips: -12.5,
|
||||
max_drawdown: 33.25,
|
||||
exposure_sign_flips: 7,
|
||||
},
|
||||
};
|
||||
let json = report.to_json();
|
||||
println!("(c) to_json() = {json}");
|
||||
|
||||
// What the rustdoc lets me check WITHOUT reading src: it is one flat-ish JSON
|
||||
// object, params is a nested object in insertion order, values are present in
|
||||
// {} shortest form. I assert structure I can justify from the surface; the
|
||||
// exact KEY NAMES are not on the public surface (recorded as a spec_gap), so
|
||||
// I check the values I supplied appear, and that the ns timestamps survive as
|
||||
// integers (round-trippable claim) rather than scientific notation.
|
||||
assert!(json.starts_with('{') && json.ends_with('}'));
|
||||
assert!(json.contains("-12.5"), "negative fractional total_pips present");
|
||||
assert!(json.contains("33.25"), "fractional drawdown present");
|
||||
assert!(json.contains("\"k\":2.5"), "fractional param in insertion order, object form");
|
||||
// params object ordering: len appears before k (insertion order, documented).
|
||||
let li = json.find("\"len\"").expect("len key");
|
||||
let ki = json.find("\"k\":2.5").expect("k key");
|
||||
assert!(li < ki, "params keys render in insertion order (len before k)");
|
||||
// round-trip claim: the big ns window bound must be a bare integer, not 1.7e18.
|
||||
assert!(json.contains("1700000000000000000"), "ns timestamp as integer, not sci-notation");
|
||||
assert!(!json.to_lowercase().contains("e1"), "no scientific-notation float leaked");
|
||||
|
||||
println!("c0009_4 OK: f64_field column-pick + documented panics + to_json shape");
|
||||
}
|
||||
Reference in New Issue
Block a user