fieldtest: cycle-0010 — 3 examples, 6 findings
Public-interface-only field test of the walking-skeleton closing seam (#8): the `aura run` CLI and the reusable aura-std::Recorder sink. A standalone consumer crate (path-deps on the three engine crates, as a C16 project would) drives the real binary as a subprocess and wires Recorder via the raw Harness::bootstrap API. Examples (all built from HEAD and ran green): - c0010_1: drive `aura run` as a downstream CLI/jq consumer — single-line JSON report, byte-identical across two runs (C1), bad-args → exit 2 + exact usage on stderr with empty stdout. - c0010_2: wire Recorder onto Sma(3) via raw bootstrap, drain the mpsc::Receiver — rows match the rustdoc warm-up model exactly. - c0010_3: Recorder over [I64, Bool, Timestamp] — the four-kind claim (never exercised by the CLI's f64-only path) holds, verified from rustdoc alone. Findings: 0 bugs, 4 working, 1 spec_gap, 1 friction. - [spec_gap] `aura run <trailing-arg>` is accepted (exit 0), not rejected — the arg parse keys only on the first token being `run` and ignores the rest. The cycle_scope's "any other or missing subcommand -> exit 2" does not unambiguously cover `run <junk>`; the binary chose the lenient reading. Tracked for a ratify-or-tighten decision. - [friction] verifying the documented {manifest, metrics} JSON nesting needs a hand-rolled string walker — the zero-dep consumer has only to_json() and no typed read-path. Low-priority tidy. Both non-bug findings filed to the forward queue; neither blocks the cycle.
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# Fieldtest — cycle-0010 — 2026-06-04
|
||||
|
||||
**Status:** Draft — awaiting orchestrator triage
|
||||
**Author:** fieldtester (dispatched by fieldtest skill)
|
||||
|
||||
## Scope
|
||||
|
||||
Cycle 0010 is the Walking-skeleton milestone's closing seam (#8). It shipped two
|
||||
public surfaces:
|
||||
|
||||
1. **`aura_std::Recorder`** — a reusable recording-sink node. A pure consumer
|
||||
(`output: vec![]`) over `kinds.len()` input columns, holding an
|
||||
`mpsc::Sender<(Timestamp, Vec<Scalar>)>` as its out-of-graph destination
|
||||
(C7/C8/C22). Constructor: `Recorder::new(kinds: &[ScalarKind], firing: Firing,
|
||||
tx) -> Self`. Each fired cycle it sends `(ctx.now(), row)` with the newest
|
||||
value of every column; returns `None` and records nothing until all columns
|
||||
are warm. Supports all four base scalar kinds.
|
||||
2. **The `aura run` CLI** (crate `aura-cli`, bin `aura`) — bootstraps a built-in
|
||||
sample harness (synthetic source → SMA(2)/SMA(4) → Sub → Exposure → SimBroker
|
||||
→ two Recorder sinks), runs it deterministically (C1), and prints the
|
||||
cycle-0009 `RunReport` as canonical single-line JSON to stdout (exit 0). Any
|
||||
other or missing subcommand prints `aura: usage: aura run` to stderr, exit 2.
|
||||
|
||||
Tested from the public interface only (rustdoc, design ledger, glossary, project
|
||||
layout, the observable behaviour of the `aura` binary) — never `crates/*/src`.
|
||||
The CLI's own functions are `pub(crate)`, so axis (a) drove the real binary as a
|
||||
subprocess. The build exercised: workspace `cargo build` from HEAD
|
||||
(`target/debug/aura`) for the binary; `cargo run --manifest-path
|
||||
fieldtests/cycle-0010-aura-run/Cargo.toml` for the two library-consumer
|
||||
fixtures.
|
||||
|
||||
## Examples
|
||||
|
||||
### fieldtests/cycle-0010-aura-run/c0010_1_cli_report.rs — `aura run` as a downstream CLI consumer (axis a)
|
||||
- Spawns the real `aura` binary as a subprocess (path via `$AURA_BIN`, default
|
||||
the sibling workspace `target/debug/aura`); captures stdout/stderr/exit for
|
||||
`run`, no-args, an unknown subcommand, and `run` with a trailing junk arg.
|
||||
- Fits the cycle's headline C14 move: a shell/jq/registry consumer parses the
|
||||
single-line JSON report and relies on the exit-code + determinism contract.
|
||||
- Outcome: built (from HEAD) and ran; exit 0 with `{manifest:{commit,params,
|
||||
window,seed,broker}, metrics:{total_pips,max_drawdown,exposure_sign_flips}}`,
|
||||
byte-identical across two runs, clean stderr; bad-args → exit 2 + exact usage.
|
||||
Matched expected. Surfaced one spec_gap (trailing-arg leniency) and one
|
||||
friction (no JSON parser in the consumer).
|
||||
|
||||
### fieldtests/cycle-0010-aura-run/c0010_2_recorder_sink.rs — `aura_std::Recorder` as a reusable sink block (axis b)
|
||||
- Wires the shipped `Recorder` onto an `Sma(3)` output via the raw
|
||||
`Harness::bootstrap(nodes, sources, edges)` API (exactly as a C16 project
|
||||
would), runs 5 ticks, drains the `mpsc::Receiver`, asserts the recorded
|
||||
`(Timestamp, Vec<Scalar>)` rows against the rustdoc warm-up model.
|
||||
- Fits the cycle's first deliverable: Recorder used the way a downstream project
|
||||
consumes it, not the way the CLI's built-in harness does.
|
||||
- Outcome: built and ran first try; rows `[(3,[20.0]),(4,[30.0]),(5,[40.0])]`
|
||||
exactly matched the hand model (no warm-up rows, one row per warm cycle,
|
||||
ts = `ctx.now()`). Matched expected.
|
||||
|
||||
### fieldtests/cycle-0010-aura-run/c0010_3_recorder_four_kind.rs — Recorder four-kind support (axis c)
|
||||
- Authors a tiny custom 3-field producer node (i64 counter / bool even? /
|
||||
timestamp now) in Rust (C17), wires a multi-column
|
||||
`Recorder::new(&[I64, Bool, Timestamp], …)` over it via three field-wise edges,
|
||||
runs, drains, asserts the recorded Scalars are `I64`/`Bool`/`Ts`.
|
||||
- Fits the cycle's four-kind claim: the CLI only ever exercises the f64 path, so
|
||||
the non-f64 kinds are verifiable from the public rustdoc alone only this way.
|
||||
- Outcome: built (after one trivial authoring fix — see working finding) and
|
||||
ran; rows matched `[(100,[I64(1),Bool(false),Ts(100)]),(200,[I64(2),Bool(true),
|
||||
Ts(200)]),(300,[I64(3),Bool(false),Ts(300)])]` exactly. Matched expected.
|
||||
|
||||
## Findings
|
||||
|
||||
### [working] `aura run` end-to-end: clean single-line JSON, deterministic, jq-parseable
|
||||
- Example: c0010_1.
|
||||
- What happened: `aura run` → exit 0, one line of JSON on stdout, empty stderr.
|
||||
The documented `{manifest, metrics}` nesting and all eight documented fields
|
||||
(`commit, params, window, seed, broker`; `total_pips, max_drawdown,
|
||||
exposure_sign_flips`) are present. Two runs are byte-identical (C1). Piped
|
||||
through `jq`, `.metrics.total_pips` is a number, `.manifest.params | keys`
|
||||
resolves — a downstream C18/shell consumer can author against it.
|
||||
- Why working: the headline C14 "run a sim, emit structured metrics" move is
|
||||
reachable from a real binary, from rustdoc alone, on the first try, with the
|
||||
determinism and shape the ledger promises.
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [working] Bad-args contract exact and stdout-clean
|
||||
- Example: c0010_1.
|
||||
- What happened: `aura` (no args) and `aura play` (unknown) each → exit 2,
|
||||
stderr exactly `aura: usage: aura run\n`, stdout empty (0 bytes). A consumer
|
||||
keying off exit code never sees a half-report on stdout.
|
||||
- Why working: matches the cycle_scope contract precisely; the stdout/stderr
|
||||
split is clean (errors never contaminate the machine-readable channel).
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [working] Recorder as a reusable sink: warm-up + row shape match rustdoc
|
||||
- Example: c0010_2.
|
||||
- What happened: a downstream wiring of `Recorder` over `Sma(3)` via raw
|
||||
`Harness::bootstrap`, drained from `mpsc::Receiver`, produced exactly the rows
|
||||
the rustdoc model predicts — no warm-up rows, one one-element `Vec<Scalar>`
|
||||
per warm cycle, each tagged `ctx.now()`. The constructor signature on the
|
||||
surface (`&[ScalarKind]`, `Firing`, `Sender`) is enough to author against with
|
||||
no source reading.
|
||||
- Why working: the cycle's primary deliverable is reachable and correct from the
|
||||
public interface as a standalone consumer block (not just inside the CLI).
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [working] Recorder four-kind claim holds on a non-f64 column set
|
||||
- Example: c0010_3.
|
||||
- What happened: a `Recorder` over `[I64, Bool, Timestamp]` recorded
|
||||
`Scalar::I64` / `Scalar::Bool` / `Scalar::Ts` values verbatim, matching the
|
||||
rustdoc's four-kind claim — a path the CLI never exercises. The trivial
|
||||
authoring miss (`FieldSpec.name` is `&'static str`, I wrote `.to_string()`)
|
||||
was caught by the compiler with an exact fix-it; the public `FieldSpec`
|
||||
rustdoc states `pub name: &'static str` plainly. No source reading needed.
|
||||
- Why working: the four-kind claim, otherwise unverifiable from the CLI, is
|
||||
empirically true; the supporting types (`FieldSpec`, `Scalar`, multi-edge
|
||||
field-wise binding) author cleanly from rustdoc.
|
||||
- Recommended action: carry-on.
|
||||
|
||||
### [spec_gap] `aura run <trailing-arg>` is accepted (exit 0), not rejected
|
||||
- Example: c0010_1.
|
||||
- What happened: `aura run extra-garbage` printed the full JSON report and exited
|
||||
0 — trailing args after `run` are silently ignored. The cycle_scope states
|
||||
"Any **other** or missing subcommand prints `aura: usage: aura run` to stderr
|
||||
and exits 2."
|
||||
- Why spec_gap: `run <junk>` is neither a clean `run` nor a *different*
|
||||
subcommand, so the contract does not unambiguously cover it. The binary picked
|
||||
the lenient reading (key only on the first arg being `run`, ignore the rest);
|
||||
the strict reading (any extra/unexpected token → usage + exit 2) is equally
|
||||
plausible and is what a user who typos `aura run --sweep` would expect (they'd
|
||||
get a silent full run instead of a hint). I asserted the lenient reading the
|
||||
binary exhibits and recorded the ambiguity rather than treating it as a bug.
|
||||
- Recommended action: ratify (document the arg-parse contract as
|
||||
"first-arg-keyed, trailing args ignored") **or** tighten — decide whether
|
||||
unexpected trailing tokens should be a usage error. A one-line note on the
|
||||
intended arg contract in the `aura` crate rustdoc closes it.
|
||||
|
||||
### [friction] Verifying the documented JSON nesting needs a hand-rolled walker
|
||||
- Example: c0010_1.
|
||||
- What happened: the consumer crate is zero-external-dependency by design (no
|
||||
serde/serde_json), and `RunReport` exposes only `to_json() -> String` with no
|
||||
typed accessor. To confirm the documented `{manifest, metrics}` nesting and
|
||||
field presence, the fixture hand-walks the string (`find("\"key\":")`) — a
|
||||
consumer authoring real assertions against the report shape either pulls in a
|
||||
JSON dep or writes this dance.
|
||||
- Why friction: the task (confirm the structured-face shape) completed, but the
|
||||
surface offered no programmatic handle on the report's structure short of
|
||||
re-parsing the string the engine just serialized. The cycle's C14 framing
|
||||
("machine-readable") implies a consumer can *inspect* the structure, not only
|
||||
re-stringify it.
|
||||
- Recommended action: plan (tidy) — consider a minimal typed read-path
|
||||
(e.g. `RunReport` already round-trips its own fields; exposing the structured
|
||||
values, or shipping a tiny documented parse helper, would let consumers assert
|
||||
without a serde dep). Low priority; the JSON is correct and jq-parseable for
|
||||
shell consumers today.
|
||||
|
||||
## Recommendation summary
|
||||
|
||||
| Finding | Class | Action |
|
||||
|---|---|---|
|
||||
| `aura run` end-to-end JSON + determinism | working | carry-on |
|
||||
| Bad-args contract exact + stdout-clean | working | carry-on |
|
||||
| Recorder reusable sink matches rustdoc | working | carry-on |
|
||||
| Recorder four-kind claim holds | working | carry-on |
|
||||
| `aura run <trailing-arg>` accepted | spec_gap | ratify / tighten the design ledger |
|
||||
| JSON nesting needs a hand-rolled walker | friction | plan (tidy) |
|
||||
Reference in New Issue
Block a user