Brummel
a982b96ecc
First fieldtest of the run-metrics + manifest report surface. A standalone
downstream-consumer crate (fieldtests/cycle-0009-run-metrics/) path-depends on
the engine crates and exercises the post-0009 surface from the public interface
only (rustdoc + ledger + glossary + project layout, never crates/*/src).
Primary axis empirically met: the north-star "a run emits metrics + manifest"
move is reachable from rustdoc alone — drain two recording sinks -> f64_field ->
summarize -> RunManifest -> to_json, metrics matching the hand model on the first
run, deterministic across reruns.
Findings: 0 bugs, 1 friction, 1 spec_gap, 4 working.
- working x4: north-star reachable from rustdoc; SimBroker firing/slot docs (a
resolved 0007 gap) now carry the example; summarize metric definitions exact
on six degenerate inputs (incl. negative-curve drawdown + flat-as-sign-0);
f64_field panics precise and well-located.
- spec_gap: to_json's JSON key names + {manifest,metrics} nesting are not on
the public surface — a consumer parsing the JSON (C18 registry, the deferred
aura run printer) cannot author against it from rustdoc alone.
- friction: to_json renders whole-valued f64 without a decimal point (3.0 ->
"3"), so one f64 field appears as integer or decimal token within one schema.
Both doc-level findings are the same doc pass and matter mainly for the deferred
aura run (#8) and the C18 registry that will parse this JSON. Spec feeds the next
plan as reference.
refs #6
174 lines
10 KiB
Markdown
174 lines
10 KiB
Markdown
# Fieldtest — cycle-0009 (run metrics + manifest) — 2026-06-04
|
||
|
||
**Status:** Draft — awaiting orchestrator triage
|
||
**Author:** fieldtester (dispatched by fieldtest skill)
|
||
|
||
## Scope
|
||
|
||
Cycle 0009 (Walking-skeleton milestone, issue #6) shipped a pure-additive
|
||
`report` module in `aura-engine`, re-exported at the crate root:
|
||
`summarize(equity, exposure) -> RunMetrics` (fields `total_pips`,
|
||
`max_drawdown`, `exposure_sign_flips`); `f64_field(rows, field)` bridging a
|
||
recording sink's `Vec<Scalar>` rows to `summarize`; `RunManifest`
|
||
(`commit`, `params`, `window`, `seed`, `broker`); `RunReport { manifest,
|
||
metrics }` with `to_json() -> String` (the structured C14 face). The intended
|
||
downstream move: after `Harness::run`, drain recording sinks, project f64
|
||
columns with `f64_field`, `summarize` the pip-equity + exposure streams, pair
|
||
with a caller-built `RunManifest`, render `to_json`. The `aura run` CLI that
|
||
prints this is deferred to #8 and out of scope. Worked from the public interface
|
||
only (rustdoc via `cargo doc`, `docs/design/INDEX.md`, `docs/glossary.md`,
|
||
project layout, public re-exports); never `crates/*/src`.
|
||
|
||
Build/run: workspace built clean from HEAD (`cargo build --workspace`); each
|
||
example run via `cargo run --manifest-path
|
||
fieldtests/cycle-0009-run-metrics/Cargo.toml --bin <name>`, so HEAD source is
|
||
always what compiled and ran (a standalone non-workspace consumer crate
|
||
path-depending on the three engine crates, exactly as a C16 project would).
|
||
|
||
## Examples
|
||
|
||
### fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs — north-star run-to-report
|
||
- **What it does.** Bootstraps the full SMA(2)/SMA(4) → Sub → Exposure(4) →
|
||
SimBroker harness with TWO recording sinks (equity tapped on the broker,
|
||
exposure tapped on the Exposure node), runs a 7-tick rising-price stream,
|
||
drains both sinks, `f64_field`-projects each, `summarize`s, builds a
|
||
`RunManifest`, and renders `to_json`.
|
||
- **Why it fits.** This is the carrier's primary axis verbatim — the end-to-end
|
||
research-loop move the whole module exists to enable, reaching every new
|
||
surface element once.
|
||
- **Outcome.** Built, ran, matched the hand model exactly: equity
|
||
`[0,0,0,0,1,2,3]` → `total_pips=3.0`, `max_drawdown=0.0`; exposure all `+0.5`
|
||
→ `exposure_sign_flips=0`. `to_json` emitted a single well-formed object.
|
||
|
||
### fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs — comparison + determinism
|
||
- **What it does.** Runs the same harness twice at `Exposure` scale=4 (asserts
|
||
bit-identical metrics AND bit-identical `to_json` across the two runs), then a
|
||
third run at scale=2, and compares `total_pips`.
|
||
- **Why it fits.** The carrier's second axis: determinism (C1) plus a
|
||
smallest-slice tuning-sweep comparison (C12/C21) read off `RunMetrics`.
|
||
- **Outcome.** Built, ran, matched expected: scale=4 deterministic
|
||
(`total_pips=3`), scale=2 = `total_pips=6` (exactly double — exposure
|
||
saturates 0.5→1.0), both drawdown-free, no sign flips.
|
||
|
||
### fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs — degenerate summarize semantics
|
||
- **What it does.** Calls `summarize` with six hand-built `&[(Timestamp,f64)]`
|
||
argument literals: empty, monotonic-up, dip-and-recover, long/short/flat
|
||
sign-flips, same-sign varying magnitude, and an all-negative curve.
|
||
- **Why it fits.** The carrier's third axis: verify the documented metric
|
||
definitions from docs alone. The slices are the public function's direct
|
||
argument type — calling the function with literals, not authoring an
|
||
intermediate representation.
|
||
- **Outcome.** Built, ran, every documented definition held exactly: empty→
|
||
zeros; last-value `total_pips` (incl. negative); running-peak drawdown (incl.
|
||
on a negative curve, peak −1 → trough −5 = 4); sign-flips counting flat (sign
|
||
0) as distinct from long/short (3 flips), magnitude changes not counted.
|
||
|
||
### fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs — f64_field bridge + JSON shape
|
||
- **What it does.** Projects a non-zero column from a multi-column recorded row;
|
||
confirms (via `catch_unwind`) the documented panics on a non-f64 field and an
|
||
out-of-range field index; inspects the exact `to_json` byte shape under a
|
||
realistic FX `pip_size=0.0001`, fractional/negative metrics, and ns-scale
|
||
timestamps.
|
||
- **Why it fits.** Exercises the `f64_field` "checked at wiring" contract and
|
||
the `to_json` round-trippability claim — the corners of the north-star
|
||
surface a generic sink-drainer would hit.
|
||
- **Outcome.** Built, ran, matched: column-pick correct; both panics fired with
|
||
precise messages; JSON kept ns timestamps as bare integers (no sci-notation),
|
||
params in insertion order as a nested object.
|
||
|
||
## Findings
|
||
|
||
### [working] North-star run-to-report is reachable from rustdoc alone
|
||
- **Example(s):** c0009_1, c0009_2.
|
||
- **What happened.** The full drain → `f64_field` → `summarize` →
|
||
`RunManifest` → `to_json` chain was authored entirely from the crate-root
|
||
rustdoc (`fn.summarize`, `fn.f64_field`, the three structs, `to_json`) plus
|
||
the prior fieldtest corpus for the harness wiring. Metrics matched the hand
|
||
model on the first run; `to_json` produced a clean object.
|
||
- **Why working.** The cycle's headline move is reachable and correct without
|
||
reading `src` — the module rustdoc names the post-run drain-and-fold workflow
|
||
explicitly, and the field docs are precise enough to predict every value.
|
||
- **Recommended action:** carry-on.
|
||
|
||
### [working] SimBroker firing/warm-up/slot-order docs (a resolved 0007 spec_gap) now carry the example
|
||
- **Example(s):** c0009_1.
|
||
- **What happened.** Predicting the equity sink's `[0,0,0,0,1,2,3]` shape
|
||
required knowing slot 0 = exposure / slot 1 = price, both `Firing::Any`, and
|
||
the "leading 0.0 rows, one row per price cycle" warm-up. All three are now on
|
||
`struct.SimBroker` rustdoc ("Input slots" / "Firing and warm-up"). The 0007
|
||
fieldtest had to recover these from the ledger + commit body and recorded them
|
||
as a spec_gap; they are now on the surface.
|
||
- **Why working.** A prior gap is closed and directly enabled this cycle's
|
||
north-star example to be authored from rustdoc.
|
||
- **Recommended action:** carry-on.
|
||
|
||
### [working] summarize metric definitions hold exactly on all degenerate inputs
|
||
- **Example(s):** c0009_3.
|
||
- **What happened.** All six hand-built cases matched the rustdoc field
|
||
definitions to the bit, including the two subtle ones: drawdown measured from
|
||
the running peak even when the whole curve is negative, and `exposure_sign_flips`
|
||
treating flat (0.0 → sign 0) as distinct from long/short.
|
||
- **Why working.** The `RunMetrics` field docs are unambiguous and empirically
|
||
exact; the "flat is sign 0" and "running-peak" subtleties are spelled out and
|
||
behave as written.
|
||
- **Recommended action:** carry-on.
|
||
|
||
### [working] f64_field panics are precise and well-located
|
||
- **Example(s):** c0009_4.
|
||
- **What happened.** `f64_field` on an `I64` field panicked with
|
||
`f64_field: field 0 is not an f64 scalar: I64(7)`; on an out-of-range index
|
||
with `f64_field: row has no field 5 (row width 2)`. Both match the documented
|
||
"checked at wiring" panic contract and name the exact offending field/width.
|
||
- **Why working.** The diagnostic is actionable and matches the rustdoc promise;
|
||
a downstream consumer mis-wiring a sink column gets a precise message.
|
||
- **Recommended action:** carry-on.
|
||
|
||
### [spec_gap] to_json key names / nesting are not on the public surface
|
||
- **Example(s):** c0009_1, c0009_4.
|
||
- **What happened.** `RunReport::to_json` rustdoc promises "field order is
|
||
fixed", "params renders as a JSON object in insertion order", "f64 uses the
|
||
round-trippable `{}` shortest form" — but it does **not** state the actual JSON
|
||
key names or the `{manifest:{...},metrics:{...}}` nesting. The observed shape
|
||
is `{"manifest":{"commit":..,"params":{..},"window":[from,to],"seed":..,
|
||
"broker":..},"metrics":{"total_pips":..,"max_drawdown":..,"exposure_sign_flips":..}}`.
|
||
I had to read the runtime output to learn the schema; a consumer parsing the
|
||
JSON (the C18 registry, the deferred `aura run` printer) cannot author against
|
||
it from rustdoc alone and would couple to an undocumented contract.
|
||
- **Why spec_gap.** The reading I guessed (keys = struct field names, `window`
|
||
as a 2-array, `params` as an object) was the natural one and proved correct,
|
||
but it is unconstrained by the public surface; another reading (e.g. `window`
|
||
as `{from,to}`, or top-level flattening) was equally plausible. For a
|
||
"machine-readable" (C14) face this schema is part of the contract.
|
||
- **Recommended action:** tighten the design ledger / rustdoc — add a documented
|
||
JSON schema example to `to_json` rustdoc (a one-line sample object suffices),
|
||
or ratify the field-name-mirroring as the stated contract.
|
||
|
||
### [friction] to_json renders whole-valued f64 without a decimal point (3.0 → `3`)
|
||
- **Example(s):** c0009_1, c0009_2.
|
||
- **What happened.** `total_pips: 3.0_f64` serializes as `"total_pips":3` (and
|
||
`4.0` param → `4`), whereas `-12.5`/`2.5` keep the point. This is the
|
||
documented round-trippable `{}` shortest form and is valid JSON (a number is a
|
||
number), so it is not a bug. But a downstream consumer string-matching or
|
||
schema-typing the field could trip: the same logical f64 field appears
|
||
sometimes as an integer token, sometimes as a decimal token, within one
|
||
schema. My example had to substring-match `"3"` rather than `"3.0"`.
|
||
- **Why friction.** The task completed, but the mixed integer/decimal rendering
|
||
of a single f64 field is a small surprise for a consumer expecting stable
|
||
per-field token shape; it interacts with the spec_gap above (no documented
|
||
schema to set the expectation).
|
||
- **Recommended action:** plan (tidy) — either document explicitly in the
|
||
`to_json` rustdoc that f64 fields may render without a fractional part, or (if
|
||
a stable decimal shape is wanted for the registry) normalize. Low urgency;
|
||
fold into the same doc pass as the spec_gap.
|
||
|
||
## Recommendation summary
|
||
|
||
| Finding | Class | Action |
|
||
|---|---|---|
|
||
| North-star reachable from rustdoc | working | carry-on |
|
||
| SimBroker firing/slot docs now carry the example | working | carry-on |
|
||
| summarize definitions exact on degenerate inputs | working | carry-on |
|
||
| f64_field panics precise + well-located | working | carry-on |
|
||
| to_json key names / nesting not on public surface | spec_gap | tighten the design ledger / rustdoc (or ratify) |
|
||
| to_json renders whole f64 as integer token | friction | plan (tidy doc pass) |
|