# Fieldtest — milestone World/C21 (run + orchestrate harness families from blueprint-data) — 2026-07-01

**Status:** Draft — awaiting orchestrator triage
**Author:** fieldtester (dispatched by fieldtest skill)
**Binary exercised:** `target/debug/aura`, built from HEAD `4710fcc` via
`cargo build --workspace` (build clean). All invocations are the debug binary of
the current working tree.

## Scope

The milestone promises that the **World** constructs and orchestrates *families*
of harnesses from serialized **blueprint-data** — not just one backtest. A
downstream researcher takes a loaded blueprint (a serialized signal graph) and
drives a coherent verb family over it: `graph build` (author a blueprint from a
JSON op-script), `sweep --list-axes` (discover sweepable knobs), `sweep --axis …`
(grid families), `mc --seeds N` (Monte-Carlo families), `walkforward --axis …`
(IS-refit walk-forward families, the capstone #173 / cycle 0097), `runs families`
+ `runs family <id> rank <metric>` (list + rank), and `reproduce <id>`
(content-addressed, bit-identical re-derivation across all three family kinds
through one shared store, C18). This fieldtest drove all of it **from the public
surface only** — the design ledger (C24/C18/C21/C9), the glossary, the documented
op-script grammar, and the two shipped `.json` blueprint fixtures as example data.
No implementation source (`crates/**/*.rs`) was read. Blueprints were **authored
as a consumer** via `aura graph build` from op-scripts, not copied from fixtures.

## Examples

### fieldtests/world-c21-milestone/w1_open_crossover.ops.json — discover → sweep → rank
- Op-script → `aura graph build` → an **open** SMA-crossover→Bias blueprint
  (`w1_open_crossover.bp.json`); then `sweep --list-axes` to discover the axis
  names, then `sweep --axis …` with exactly those names, then `runs families` /
  `runs family <id> rank <metric>`.
- Fits the milestone's grid-family + discovery + ranking axes; exercises the
  whole list-axes→sweep→rank composition on a consumer-authored blueprint.
- Outcome: **built** (exit 0, canonical, no trailing newline); `--list-axes`
  named 3 open knobs; a full sweep produced a 4-member `Sweep` family; `rank`
  ordered best-first by `sqn_normalized` / `expectancy_r`; `reproduce` re-derived
  4/4 bit-identically. One friction en route (see [spec_gap] mandatory-knobs).

### fieldtests/world-c21-milestone/w1 (reuse) — walk-forward → reproduce (CAPSTONE)
- `aura walkforward w1_open_crossover.bp.json --axis … --select argmax` →
  `WalkForward` family, then `aura reproduce w2_wf-0`.
- The capstone axis (#173): IS-refit walk-forward over a loaded blueprint +
  content-addressed reproduction of the third family kind.
- Outcome: **ran** (exit 0). 3 OOS members + a 4th
  `{"walkforward":{oos_r, param_stability, stitched_total_pips, windows:3}}`
  summary line, exactly as promised; `reproduce` re-derived 3/3 bit-identically.

### fieldtests/world-c21-milestone/w3_closed_crossover.ops.json — Monte-Carlo → reproduce
- Op-script → `aura graph build` → a **closed** (fully bound) blueprint; then
  `aura mc --seeds 5`, then `aura reproduce w3_mc-0`.
- Fits the Monte-Carlo-family + reproduce axes; needs the open/closed distinction
  inverted from sweep (mc requires a closed blueprint).
- Outcome: **ran** (exit 0). `--list-axes` empty (fully bound); `mc` produced a
  5-member `MonteCarlo` family (seeds 1..5, window [1,60], real trades);
  `reproduce` re-derived 5/5 bit-identically.

### fieldtests/world-c21-milestone/w4_* — cross-verb coherence & error actionability
- Closed→sweep, open→mc, closed→walkforward, unknown-axis, unknown-family-id,
  subset-of-open-knobs, and `run` on an open blueprint.
- Fits the "do the verbs compose / are errors actionable" axis.
- Outcome: mostly clean exit-2 diagnostics (see findings), **plus one panic bug**
  (`aura run` on an open blueprint).

## Findings

### [working] Consumer-authored blueprint round-trips through the whole family surface
- Examples: w1 (sweep), w2 (walkforward), w3 (mc).
- Authoring a blueprint from the **documented op-script grammar** (`aura graph
  build < ops.json`) was friction-free: exit 0, canonical bytes, no trailing
  newline (`tail -c` confirmed). The same authored `.bp.json` then fed `run`,
  `sweep`, `mc`, and `walkforward` unchanged. `graph introspect --vocabulary` /
  `--node SMA` give a clean author-discovery surface (ports, kinds, bind hint).
- This is the milestone's substrate working end-to-end from a real consumer path,
  not just the shipped fixtures.
- Recommended action: carry-on.

### [working] All three FamilyKinds reproduce bit-identically (the C18 core promise)
- Examples: w1 (`Sweep` 4/4), w2 (`WalkForward` 3/3), w3 (`MonteCarlo` 5/5).
- `aura reproduce <id>` re-derived every member of every family kind
  bit-identically (exit 0, `reproduced N/N members bit-identically`). Each member
  manifest carries the shared `topology_hash`; sweep/wf members echo their params
  in the reproduce line. This is exactly "every family kind persists AND
  reproduces through one shared content-addressed store."
- Recommended action: carry-on.

### [working] list-axes → sweep composes and is self-correcting; rank orders best-first
- Example: w1.
- `--list-axes` printed `<name>:<kind>` per open knob, prefixed by the
  blueprint's own root name (`graph.fast.length` for my `graph`-named blueprint;
  `stage1_signal.fast.length` for the fixture). Every printed name was accepted
  verbatim by `--axis`; a fully-bound knob is absent from the list (fixture's
  `scale`) and my unbound `scale` is present — the list tracks the actual open
  set. `runs family … rank sqn_normalized|expectancy_r` ordered members best-first.
- Note: the C24 ledger text uses the fixture-specific `stage1_signal.` prefix as
  its example; `--list-axes` makes the real prefix explicit, so the composition is
  self-correcting even though the doc example is fixture-bound.
- Recommended action: carry-on.

### [bug] `aura run <open-blueprint.json>` panics instead of erroring cleanly
- Example: w4 (`w4_run_open.err` / `.exit`).
- Verbatim:
  ```
  thread 'main' (…) panicked at crates/aura-cli/src/main.rs:3107:10:
  signal binds + wraps to a valid harness: ParamArity { expected: 3, got: 0 }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  ```
  Exit 101. The sibling verb `aura mc` on the *same* open blueprint gives a clean,
  actionable exit-2 error ("mc requires a closed blueprint … 3 free knob(s) — bind
  them or use `aura sweep --axis`"). `run` should likewise refuse an open
  (free-knob) blueprint at the dispatch boundary, not `.expect()`-panic — the
  cycle-0097 audit's own robustness claim is "a malformed blueprint is rejected at
  the dispatch boundary (exit 2, no panic)". A consumer who authors an open
  blueprint and runs `aura run` on it (a natural first move) hits a raw panic +
  backtrace note.
- Repro: `aura run <blueprint-with-≥1-free-knob>.json` → panic (exit 101).
- Recommended downstream action: debug (RED test: `run` on an open blueprint exits
  2 with a "requires a closed/bound blueprint" message, no panic).

### [bug] closed→walkforward with an axis prints the same diagnostic twice
- Example: w4 (`w4_4c_closed_to_wf.err`).
- `aura walkforward <closed bp> --axis graph.fast.length=2,3` emits
  `aura: UnknownKnob("graph.fast.length")` **twice** (exit 2). Exit code is
  correct and there is no panic; the diagnostic is just duplicated. Cosmetic but
  observable output defect (the sweep / mc paths emit their rejection once).
- Repro: `aura walkforward <closed bp> --axis <any>=<v>` → two identical stderr
  lines.
- Recommended downstream action: debug (low severity — dedupe the emit; likely a
  per-window closure re-raising the same top-level error).

### [friction] closed→sweep error is terse where the mc-side equivalent is exemplary
- Example: w4 (`w4_4a_closed_to_sweep.err` vs `w4_4b_open_to_mc.err`).
- `sweep <closed bp> --axis graph.fast.length=…` → `UnknownKnob("graph.fast.length")`.
  The knob *exists* in the blueprint but is bound out, so "UnknownKnob" is
  technically-true-but-misleading. The inverse case (`mc <open bp>`) instead says
  "mc requires a closed blueprint (no free parameters); 3 free knob(s) — bind them
  or use `aura sweep --axis`" — which names the count and the fix. The sweep side
  deserves the symmetric "this blueprint is fully bound; nothing to sweep (use
  `--list-axes`)" message.
- Why friction (not bug): task completes with the right exit code; the surface
  just under-explains a coherent rejection.
- Recommended downstream action: plan (tidy the sweep-side open/closed message to
  match the mc-side quality).

### [friction] `mc reproduce` lines omit the seed (blank member label)
- Example: w3 (`w3_reproduce.out`).
- MC reproduce prints `w3_mc-0  member   reproduced: bit-identical` — the member
  label between "member" and "reproduced" is blank, so a consumer cannot tell
  *which seed* reproduced (sweep/walkforward show the params). The `mc` **run**
  output does carry `"seed":N`, so the information exists; only the reproduce line
  drops it.
- Why friction: reproduction succeeds; the audit line is just less legible for the
  seed-varying family kind.
- Recommended downstream action: plan (label MC reproduce lines with `seed=N`).

### [spec_gap] Every open knob is MANDATORY on `sweep`/`walkforward`, but the docs frame list-axes as a menu
- Examples: w1 (first attempt), w4 (`w4_4f_missingknob_subset.err`).
- `--list-axes` presented 3 open knobs; sweeping the two "obvious" ones
  (`graph.fast.length`, `graph.slow.length`) failed with
  `MissingKnob("graph.bias.scale")` (exit 2). Every open knob must be assigned ≥1
  value; there is **no default** (consistent with C19's value-empty leaf / no
  baked default — but that consequence is nowhere in the family-verb docs). The
  glossary/ledger call `--list-axes` "discover sweepable knobs" and say the names
  are "exactly what `--axis` binds" — true, but silent that they are *all
  required*. I picked the reading "supply every listed knob; pin the ones you
  don't want to vary with a single-value axis" and it worked; an equally plausible
  reading — "unspecified knobs take a default / stay at their authored value" — is
  what the engine rejects. The diagnostic is actionable (names the missing knob).
- Recommended downstream action: ratify + tighten the docs (state that a sweep
  binds the blueprint's *entire* open param-space; a subset needs the rest bound
  out at authoring time or pinned via single-value axes). Optionally reconsider
  whether an un-swept open knob could error at *parse* time listing all missing
  knobs at once rather than one-at-a-time.

### [spec_gap] The whole blueprint-data verb family is undiscoverable from `--help` (no README)
- Examples: all — first thing a consumer runs is `aura --help`.
- `aura --help`, `aura graph --help`, `aura graph build --help`, and
  `aura graph introspect --help` **all print the identical generic usage string**,
  which lists only the hard-wired `--harness`/`--strategy` forms. None of the
  milestone verbs appear: `graph build`, `graph introspect`, `run <bp.json>`,
  `sweep <bp> --list-axes`, `sweep <bp> --axis`, `mc <bp> --seeds`,
  `walkforward <bp> --axis`. The repo has **no README**. So the entire milestone
  surface is discoverable *only* from the design ledger (a design doc, not a user
  guide) and the op-script grammar block inside C24. A downstream consumer with
  just the binary + `--help` cannot find any of it.
- This matches the deferred #159 (CLI discoverability of `build`/`introspect`),
  but empirically it means the milestone's own surface is invisible at the CLI.
- Recommended downstream action: plan (surface the blueprint-data verbs in
  `--help` and/or a README; tracked as #159). Until then, treat the ledger as the
  only entry point — a real gap for anyone but the author.

## Recommendation summary

| Finding | Class | Action |
|---|---|---|
| Consumer-authored blueprint round-trips whole family surface | working | carry-on |
| All three FamilyKinds reproduce bit-identically | working | carry-on |
| list-axes→sweep composes; rank best-first | working | carry-on |
| `aura run` on open blueprint panics (exit 101) | bug | debug |
| closed→walkforward prints diagnostic twice | bug | debug |
| closed→sweep error terser than mc-side equivalent | friction | plan |
| mc reproduce lines omit the seed | friction | plan |
| every open knob mandatory on sweep, undocumented | spec_gap | ratify + tighten docs |
| blueprint-data verbs undiscoverable from `--help` / no README | spec_gap | plan (#159) |