fieldtest: cycle-0009 — 4 examples, 6 findings

First fieldtest of the run-metrics + manifest report surface. A standalone
downstream-consumer crate (fieldtests/cycle-0009-run-metrics/) path-depends on
the engine crates and exercises the post-0009 surface from the public interface
only (rustdoc + ledger + glossary + project layout, never crates/*/src).

Primary axis empirically met: the north-star "a run emits metrics + manifest"
move is reachable from rustdoc alone — drain two recording sinks -> f64_field ->
summarize -> RunManifest -> to_json, metrics matching the hand model on the first
run, deterministic across reruns.

Findings: 0 bugs, 1 friction, 1 spec_gap, 4 working.
  - working x4: north-star reachable from rustdoc; SimBroker firing/slot docs (a
    resolved 0007 gap) now carry the example; summarize metric definitions exact
    on six degenerate inputs (incl. negative-curve drawdown + flat-as-sign-0);
    f64_field panics precise and well-located.
  - spec_gap: to_json's JSON key names + {manifest,metrics} nesting are not on
    the public surface — a consumer parsing the JSON (C18 registry, the deferred
    aura run printer) cannot author against it from rustdoc alone.
  - friction: to_json renders whole-valued f64 without a decimal point (3.0 ->
    "3"), so one f64 field appears as integer or decimal token within one schema.

Both doc-level findings are the same doc pass and matter mainly for the deferred
aura run (#8) and the C18 registry that will parse this JSON. Spec feeds the next
plan as reference.

refs #6
This commit is contained in:
2026-06-04 19:06:13 +02:00
parent 4dc1526196
commit a982b96ecc
7 changed files with 714 additions and 0 deletions
+173
View File
@@ -0,0 +1,173 @@
# Fieldtest — cycle-0009 (run metrics + manifest) — 2026-06-04
**Status:** Draft — awaiting orchestrator triage
**Author:** fieldtester (dispatched by fieldtest skill)
## Scope
Cycle 0009 (Walking-skeleton milestone, issue #6) shipped a pure-additive
`report` module in `aura-engine`, re-exported at the crate root:
`summarize(equity, exposure) -> RunMetrics` (fields `total_pips`,
`max_drawdown`, `exposure_sign_flips`); `f64_field(rows, field)` bridging a
recording sink's `Vec<Scalar>` rows to `summarize`; `RunManifest`
(`commit`, `params`, `window`, `seed`, `broker`); `RunReport { manifest,
metrics }` with `to_json() -> String` (the structured C14 face). The intended
downstream move: after `Harness::run`, drain recording sinks, project f64
columns with `f64_field`, `summarize` the pip-equity + exposure streams, pair
with a caller-built `RunManifest`, render `to_json`. The `aura run` CLI that
prints this is deferred to #8 and out of scope. Worked from the public interface
only (rustdoc via `cargo doc`, `docs/design/INDEX.md`, `docs/glossary.md`,
project layout, public re-exports); never `crates/*/src`.
Build/run: workspace built clean from HEAD (`cargo build --workspace`); each
example run via `cargo run --manifest-path
fieldtests/cycle-0009-run-metrics/Cargo.toml --bin <name>`, so HEAD source is
always what compiled and ran (a standalone non-workspace consumer crate
path-depending on the three engine crates, exactly as a C16 project would).
## Examples
### fieldtests/cycle-0009-run-metrics/c0009_1_run_to_report.rs — north-star run-to-report
- **What it does.** Bootstraps the full SMA(2)/SMA(4) → Sub → Exposure(4) →
SimBroker harness with TWO recording sinks (equity tapped on the broker,
exposure tapped on the Exposure node), runs a 7-tick rising-price stream,
drains both sinks, `f64_field`-projects each, `summarize`s, builds a
`RunManifest`, and renders `to_json`.
- **Why it fits.** This is the carrier's primary axis verbatim — the end-to-end
research-loop move the whole module exists to enable, reaching every new
surface element once.
- **Outcome.** Built, ran, matched the hand model exactly: equity
`[0,0,0,0,1,2,3]``total_pips=3.0`, `max_drawdown=0.0`; exposure all `+0.5`
`exposure_sign_flips=0`. `to_json` emitted a single well-formed object.
### fieldtests/cycle-0009-run-metrics/c0009_2_compare_two_runs.rs — comparison + determinism
- **What it does.** Runs the same harness twice at `Exposure` scale=4 (asserts
bit-identical metrics AND bit-identical `to_json` across the two runs), then a
third run at scale=2, and compares `total_pips`.
- **Why it fits.** The carrier's second axis: determinism (C1) plus a
smallest-slice tuning-sweep comparison (C12/C21) read off `RunMetrics`.
- **Outcome.** Built, ran, matched expected: scale=4 deterministic
(`total_pips=3`), scale=2 = `total_pips=6` (exactly double — exposure
saturates 0.5→1.0), both drawdown-free, no sign flips.
### fieldtests/cycle-0009-run-metrics/c0009_3_degenerate_streams.rs — degenerate summarize semantics
- **What it does.** Calls `summarize` with six hand-built `&[(Timestamp,f64)]`
argument literals: empty, monotonic-up, dip-and-recover, long/short/flat
sign-flips, same-sign varying magnitude, and an all-negative curve.
- **Why it fits.** The carrier's third axis: verify the documented metric
definitions from docs alone. The slices are the public function's direct
argument type — calling the function with literals, not authoring an
intermediate representation.
- **Outcome.** Built, ran, every documented definition held exactly: empty→
zeros; last-value `total_pips` (incl. negative); running-peak drawdown (incl.
on a negative curve, peak 1 → trough 5 = 4); sign-flips counting flat (sign
0) as distinct from long/short (3 flips), magnitude changes not counted.
### fieldtests/cycle-0009-run-metrics/c0009_4_f64field_and_json.rs — f64_field bridge + JSON shape
- **What it does.** Projects a non-zero column from a multi-column recorded row;
confirms (via `catch_unwind`) the documented panics on a non-f64 field and an
out-of-range field index; inspects the exact `to_json` byte shape under a
realistic FX `pip_size=0.0001`, fractional/negative metrics, and ns-scale
timestamps.
- **Why it fits.** Exercises the `f64_field` "checked at wiring" contract and
the `to_json` round-trippability claim — the corners of the north-star
surface a generic sink-drainer would hit.
- **Outcome.** Built, ran, matched: column-pick correct; both panics fired with
precise messages; JSON kept ns timestamps as bare integers (no sci-notation),
params in insertion order as a nested object.
## Findings
### [working] North-star run-to-report is reachable from rustdoc alone
- **Example(s):** c0009_1, c0009_2.
- **What happened.** The full drain → `f64_field``summarize`
`RunManifest``to_json` chain was authored entirely from the crate-root
rustdoc (`fn.summarize`, `fn.f64_field`, the three structs, `to_json`) plus
the prior fieldtest corpus for the harness wiring. Metrics matched the hand
model on the first run; `to_json` produced a clean object.
- **Why working.** The cycle's headline move is reachable and correct without
reading `src` — the module rustdoc names the post-run drain-and-fold workflow
explicitly, and the field docs are precise enough to predict every value.
- **Recommended action:** carry-on.
### [working] SimBroker firing/warm-up/slot-order docs (a resolved 0007 spec_gap) now carry the example
- **Example(s):** c0009_1.
- **What happened.** Predicting the equity sink's `[0,0,0,0,1,2,3]` shape
required knowing slot 0 = exposure / slot 1 = price, both `Firing::Any`, and
the "leading 0.0 rows, one row per price cycle" warm-up. All three are now on
`struct.SimBroker` rustdoc ("Input slots" / "Firing and warm-up"). The 0007
fieldtest had to recover these from the ledger + commit body and recorded them
as a spec_gap; they are now on the surface.
- **Why working.** A prior gap is closed and directly enabled this cycle's
north-star example to be authored from rustdoc.
- **Recommended action:** carry-on.
### [working] summarize metric definitions hold exactly on all degenerate inputs
- **Example(s):** c0009_3.
- **What happened.** All six hand-built cases matched the rustdoc field
definitions to the bit, including the two subtle ones: drawdown measured from
the running peak even when the whole curve is negative, and `exposure_sign_flips`
treating flat (0.0 → sign 0) as distinct from long/short.
- **Why working.** The `RunMetrics` field docs are unambiguous and empirically
exact; the "flat is sign 0" and "running-peak" subtleties are spelled out and
behave as written.
- **Recommended action:** carry-on.
### [working] f64_field panics are precise and well-located
- **Example(s):** c0009_4.
- **What happened.** `f64_field` on an `I64` field panicked with
`f64_field: field 0 is not an f64 scalar: I64(7)`; on an out-of-range index
with `f64_field: row has no field 5 (row width 2)`. Both match the documented
"checked at wiring" panic contract and name the exact offending field/width.
- **Why working.** The diagnostic is actionable and matches the rustdoc promise;
a downstream consumer mis-wiring a sink column gets a precise message.
- **Recommended action:** carry-on.
### [spec_gap] to_json key names / nesting are not on the public surface
- **Example(s):** c0009_1, c0009_4.
- **What happened.** `RunReport::to_json` rustdoc promises "field order is
fixed", "params renders as a JSON object in insertion order", "f64 uses the
round-trippable `{}` shortest form" — but it does **not** state the actual JSON
key names or the `{manifest:{...},metrics:{...}}` nesting. The observed shape
is `{"manifest":{"commit":..,"params":{..},"window":[from,to],"seed":..,
"broker":..},"metrics":{"total_pips":..,"max_drawdown":..,"exposure_sign_flips":..}}`.
I had to read the runtime output to learn the schema; a consumer parsing the
JSON (the C18 registry, the deferred `aura run` printer) cannot author against
it from rustdoc alone and would couple to an undocumented contract.
- **Why spec_gap.** The reading I guessed (keys = struct field names, `window`
as a 2-array, `params` as an object) was the natural one and proved correct,
but it is unconstrained by the public surface; another reading (e.g. `window`
as `{from,to}`, or top-level flattening) was equally plausible. For a
"machine-readable" (C14) face this schema is part of the contract.
- **Recommended action:** tighten the design ledger / rustdoc — add a documented
JSON schema example to `to_json` rustdoc (a one-line sample object suffices),
or ratify the field-name-mirroring as the stated contract.
### [friction] to_json renders whole-valued f64 without a decimal point (3.0 → `3`)
- **Example(s):** c0009_1, c0009_2.
- **What happened.** `total_pips: 3.0_f64` serializes as `"total_pips":3` (and
`4.0` param → `4`), whereas `-12.5`/`2.5` keep the point. This is the
documented round-trippable `{}` shortest form and is valid JSON (a number is a
number), so it is not a bug. But a downstream consumer string-matching or
schema-typing the field could trip: the same logical f64 field appears
sometimes as an integer token, sometimes as a decimal token, within one
schema. My example had to substring-match `"3"` rather than `"3.0"`.
- **Why friction.** The task completed, but the mixed integer/decimal rendering
of a single f64 field is a small surprise for a consumer expecting stable
per-field token shape; it interacts with the spec_gap above (no documented
schema to set the expectation).
- **Recommended action:** plan (tidy) — either document explicitly in the
`to_json` rustdoc that f64 fields may render without a fractional part, or (if
a stable decimal shape is wanted for the registry) normalize. Low urgency;
fold into the same doc pass as the spec_gap.
## Recommendation summary
| Finding | Class | Action |
|---|---|---|
| North-star reachable from rustdoc | working | carry-on |
| SimBroker firing/slot docs now carry the example | working | carry-on |
| summarize definitions exact on degenerate inputs | working | carry-on |
| f64_field panics precise + well-located | working | carry-on |
| to_json key names / nesting not on public surface | spec_gap | tighten the design ledger / rustdoc (or ratify) |
| to_json renders whole f64 as integer token | friction | plan (tidy doc pass) |