Aura/docs/specs/fieldtest-milestone-world-c21-blueprint-data-families.md

Fieldtest — milestone World/C21 (run + orchestrate harness families from blueprint-data) — 2026-07-01

Status: Draft — awaiting orchestrator triage Author: fieldtester (dispatched by fieldtest skill) Binary exercised: target/debug/aura, built from HEAD 4710fcc via cargo build --workspace (build clean). All invocations are the debug binary of the current working tree.

Scope

The milestone promises that the World constructs and orchestrates families of harnesses from serialized blueprint-data — not just one backtest. A downstream researcher takes a loaded blueprint (a serialized signal graph) and drives a coherent verb family over it: graph build (author a blueprint from a JSON op-script), sweep --list-axes (discover sweepable knobs), sweep --axis … (grid families), mc --seeds N (Monte-Carlo families), walkforward --axis … (IS-refit walk-forward families, the capstone #173 / cycle 0097), runs families

• runs family <id> rank <metric> (list + rank), and reproduce <id> (content-addressed, bit-identical re-derivation across all three family kinds through one shared store, C18). This fieldtest drove all of it from the public surface only — the design ledger (C24/C18/C21/C9), the glossary, the documented op-script grammar, and the two shipped .json blueprint fixtures as example data. No implementation source (crates/**/*.rs) was read. Blueprints were authored as a consumer via aura graph build from op-scripts, not copied from fixtures.

Examples

fieldtests/world-c21-milestone/w1_open_crossover.ops.json — discover → sweep → rank

• Op-script → aura graph build → an open SMA-crossover→Bias blueprint (w1_open_crossover.bp.json); then sweep --list-axes to discover the axis names, then sweep --axis … with exactly those names, then runs families / runs family <id> rank <metric>.
• Fits the milestone's grid-family + discovery + ranking axes; exercises the whole list-axes→sweep→rank composition on a consumer-authored blueprint.
• Outcome: built (exit 0, canonical, no trailing newline); --list-axes named 3 open knobs; a full sweep produced a 4-member Sweep family; rank ordered best-first by sqn_normalized / expectancy_r; reproduce re-derived 4/4 bit-identically. One friction en route (see [spec_gap] mandatory-knobs).

fieldtests/world-c21-milestone/w1 (reuse) — walk-forward → reproduce (CAPSTONE)

• aura walkforward w1_open_crossover.bp.json --axis … --select argmax → WalkForward family, then aura reproduce w2_wf-0.
• The capstone axis (#173): IS-refit walk-forward over a loaded blueprint + content-addressed reproduction of the third family kind.
• Outcome: ran (exit 0). 3 OOS members + a 4th {"walkforward":{oos_r, param_stability, stitched_total_pips, windows:3}} summary line, exactly as promised; reproduce re-derived 3/3 bit-identically.

fieldtests/world-c21-milestone/w3_closed_crossover.ops.json — Monte-Carlo → reproduce

• Op-script → aura graph build → a closed (fully bound) blueprint; then aura mc --seeds 5, then aura reproduce w3_mc-0.
• Fits the Monte-Carlo-family + reproduce axes; needs the open/closed distinction inverted from sweep (mc requires a closed blueprint).
• Outcome: ran (exit 0). --list-axes empty (fully bound); mc produced a 5-member MonteCarlo family (seeds 1..5, window [1,60], real trades); reproduce re-derived 5/5 bit-identically.

fieldtests/world-c21-milestone/w4_* — cross-verb coherence & error actionability

• Closed→sweep, open→mc, closed→walkforward, unknown-axis, unknown-family-id, subset-of-open-knobs, and run on an open blueprint.
• Fits the "do the verbs compose / are errors actionable" axis.
• Outcome: mostly clean exit-2 diagnostics (see findings), plus one panic bug (aura run on an open blueprint).

Findings

[working] Consumer-authored blueprint round-trips through the whole family surface

• Examples: w1 (sweep), w2 (walkforward), w3 (mc).
• Authoring a blueprint from the documented op-script grammar (aura graph build < ops.json) was friction-free: exit 0, canonical bytes, no trailing newline (tail -c confirmed). The same authored .bp.json then fed run, sweep, mc, and walkforward unchanged. graph introspect --vocabulary / --node SMA give a clean author-discovery surface (ports, kinds, bind hint).
• This is the milestone's substrate working end-to-end from a real consumer path, not just the shipped fixtures.
• Recommended action: carry-on.

[working] All three FamilyKinds reproduce bit-identically (the C18 core promise)

• Examples: w1 (Sweep 4/4), w2 (WalkForward 3/3), w3 (MonteCarlo 5/5).
• aura reproduce <id> re-derived every member of every family kind bit-identically (exit 0, reproduced N/N members bit-identically). Each member manifest carries the shared topology_hash; sweep/wf members echo their params in the reproduce line. This is exactly "every family kind persists AND reproduces through one shared content-addressed store."
• Recommended action: carry-on.

[working] list-axes → sweep composes and is self-correcting; rank orders best-first

• Example: w1.
• --list-axes printed <name>:<kind> per open knob, prefixed by the blueprint's own root name (graph.fast.length for my graph-named blueprint; stage1_signal.fast.length for the fixture). Every printed name was accepted verbatim by --axis; a fully-bound knob is absent from the list (fixture's scale) and my unbound scale is present — the list tracks the actual open set. runs family … rank sqn_normalized|expectancy_r ordered members best-first.
• Note: the C24 ledger text uses the fixture-specific stage1_signal. prefix as its example; --list-axes makes the real prefix explicit, so the composition is self-correcting even though the doc example is fixture-bound.
• Recommended action: carry-on.

[bug] aura run <open-blueprint.json> panics instead of erroring cleanly

• Example: w4 (w4_run_open.err / .exit).
• Verbatim:

  thread 'main' (…) panicked at crates/aura-cli/src/main.rs:3107:10:
  signal binds + wraps to a valid harness: ParamArity { expected: 3, got: 0 }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  

  Exit 101. The sibling verb aura mc on the same open blueprint gives a clean, actionable exit-2 error ("mc requires a closed blueprint … 3 free knob(s) — bind them or use aura sweep --axis"). run should likewise refuse an open (free-knob) blueprint at the dispatch boundary, not .expect()-panic — the cycle-0097 audit's own robustness claim is "a malformed blueprint is rejected at the dispatch boundary (exit 2, no panic)". A consumer who authors an open blueprint and runs aura run on it (a natural first move) hits a raw panic + backtrace note.
• Repro: aura run <blueprint-with-≥1-free-knob>.json → panic (exit 101).
• Recommended downstream action: debug (RED test: run on an open blueprint exits 2 with a "requires a closed/bound blueprint" message, no panic).

[bug] closed→walkforward with an axis prints the same diagnostic twice

• Example: w4 (w4_4c_closed_to_wf.err).
• aura walkforward <closed bp> --axis graph.fast.length=2,3 emits aura: UnknownKnob("graph.fast.length") twice (exit 2). Exit code is correct and there is no panic; the diagnostic is just duplicated. Cosmetic but observable output defect (the sweep / mc paths emit their rejection once).
• Repro: aura walkforward <closed bp> --axis <any>=<v> → two identical stderr lines.
• Recommended downstream action: debug (low severity — dedupe the emit; likely a per-window closure re-raising the same top-level error).

[friction] closed→sweep error is terse where the mc-side equivalent is exemplary

• Example: w4 (w4_4a_closed_to_sweep.err vs w4_4b_open_to_mc.err).
• sweep <closed bp> --axis graph.fast.length=… → UnknownKnob("graph.fast.length"). The knob exists in the blueprint but is bound out, so "UnknownKnob" is technically-true-but-misleading. The inverse case (mc <open bp>) instead says "mc requires a closed blueprint (no free parameters); 3 free knob(s) — bind them or use aura sweep --axis" — which names the count and the fix. The sweep side deserves the symmetric "this blueprint is fully bound; nothing to sweep (use --list-axes)" message.
• Why friction (not bug): task completes with the right exit code; the surface just under-explains a coherent rejection.
• Recommended downstream action: plan (tidy the sweep-side open/closed message to match the mc-side quality).

[friction] mc reproduce lines omit the seed (blank member label)

• Example: w3 (w3_reproduce.out).
• MC reproduce prints w3_mc-0 member reproduced: bit-identical — the member label between "member" and "reproduced" is blank, so a consumer cannot tell which seed reproduced (sweep/walkforward show the params). The mc run output does carry "seed":N, so the information exists; only the reproduce line drops it.
• Why friction: reproduction succeeds; the audit line is just less legible for the seed-varying family kind.
• Recommended downstream action: plan (label MC reproduce lines with seed=N).

[spec_gap] Every open knob is MANDATORY on sweep/walkforward, but the docs frame list-axes as a menu

• Examples: w1 (first attempt), w4 (w4_4f_missingknob_subset.err).
• --list-axes presented 3 open knobs; sweeping the two "obvious" ones (graph.fast.length, graph.slow.length) failed with MissingKnob("graph.bias.scale") (exit 2). Every open knob must be assigned ≥1 value; there is no default (consistent with C19's value-empty leaf / no baked default — but that consequence is nowhere in the family-verb docs). The glossary/ledger call --list-axes "discover sweepable knobs" and say the names are "exactly what --axis binds" — true, but silent that they are all required. I picked the reading "supply every listed knob; pin the ones you don't want to vary with a single-value axis" and it worked; an equally plausible reading — "unspecified knobs take a default / stay at their authored value" — is what the engine rejects. The diagnostic is actionable (names the missing knob).
• Recommended downstream action: ratify + tighten the docs (state that a sweep binds the blueprint's entire open param-space; a subset needs the rest bound out at authoring time or pinned via single-value axes). Optionally reconsider whether an un-swept open knob could error at parse time listing all missing knobs at once rather than one-at-a-time.

[spec_gap] The whole blueprint-data verb family is undiscoverable from --help (no README)

• Examples: all — first thing a consumer runs is aura --help.
• aura --help, aura graph --help, aura graph build --help, and aura graph introspect --help all print the identical generic usage string, which lists only the hard-wired --harness/--strategy forms. None of the milestone verbs appear: graph build, graph introspect, run <bp.json>, sweep <bp> --list-axes, sweep <bp> --axis, mc <bp> --seeds, walkforward <bp> --axis. The repo has no README. So the entire milestone surface is discoverable only from the design ledger (a design doc, not a user guide) and the op-script grammar block inside C24. A downstream consumer with just the binary + --help cannot find any of it.
• This matches the deferred #159 (CLI discoverability of build/introspect), but empirically it means the milestone's own surface is invisible at the CLI.
• Recommended downstream action: plan (surface the blueprint-data verbs in --help and/or a README; tracked as #159). Until then, treat the ledger as the only entry point — a real gap for anyone but the author.

Recommendation summary

Finding	Class	Action
Consumer-authored blueprint round-trips whole family surface	working	carry-on
All three FamilyKinds reproduce bit-identically	working	carry-on
list-axes→sweep composes; rank best-first	working	carry-on
aura run on open blueprint panics (exit 101)	bug	debug
closed→walkforward prints diagnostic twice	bug	debug
closed→sweep error terser than mc-side equivalent	friction	plan
mc reproduce lines omit the seed	friction	plan
every open knob mandatory on sweep, undocumented	spec_gap	ratify + tighten docs
blueprint-data verbs undiscoverable from --help / no README	spec_gap	plan (#159)