Aura/docs/specs/fieldtest-milestone-the-world-orchestration-families.md

Fieldtest — milestone "The World, part II — orchestration families" — 2026-06-17

Status: Draft — awaiting orchestrator triage Author: fieldtester (dispatched by fieldtest skill, milestone-close gate)

Scope

Milestone-18 promised the four C12 orchestration-family axes — parameter-sweep (grid + random), optimize/argmax, walk-forward, Monte-Carlo — delivered end-to-end through aura's public Rust API and CLI, resting on three substrate guarantees: a lazy Source ingestion seam (synthetic + real M1), seed-as-input reproducibility, and registry lineage that persists each family as a queryable unit (aura runs families / aura runs family <id> [rank <metric>]). This field test composes those axes top-down from the promise into four realistic downstream tasks, run from a standalone consumer crate (path-deps only; public interface = design ledger + glossary + specs 0041–0049 + cargo doc rustdoc; no crates/*/src was read) plus the aura CLI driven as a researcher would. Every binary was rebuilt from HEAD (8b22fa6) and run; the real-data scenario streamed the local EURUSD M1 archive.

Examples

fieldtests/milestone-the-world-orchestration-families/mw_3_walkforward_real.rs — real-data walk-forward (THE HEADLINE)

• Walk-forward over real EURUSD M1 close bars: roll 4-month IS / 1-month OOS over a 9-month span; per window, grid-sweep + optimize the in-sample slice by total_pips, run the chosen params out-of-sample, stitch the OOS months into one continuous pip curve, then persist the per-window OOS reports as a WalkForward lineage family.
• Fits the milestone: composes axes 1 (in-sample grid) + 2 (optimize) + 3 (walk-forward) over the real-data Source seam (M1FieldSource) plus registry lineage — exactly the promise's headline scenario.
• Outcome: built ✓, ran ✓, matched expected ✓. 5 windows; per-window optimize reacted to the data (window 4 chose slow=50, the others slow=20); stitched curve 145 200 points, final cumulative −2.2884 pips; C2 no-look-ahead held on every window; lineage round-tripped 5 members as eurusd-wfo-0.

fieldtests/milestone-the-world-orchestration-families/mw_2_monte_carlo_robustness.rs — Monte-Carlo robustness over a seed set

• monte_carlo over 128 seeds with a fixed strategy and SyntheticSpec::source(seed); read the aggregate metric distribution (mean + p5/p50/p95), confirm same seed set reproduces the whole McFamily, persist + round-trip the MC family.
• Fits the milestone: axis 4 + seed-as-input + lineage.
• Outcome: built ✓, ran ✓, matched expected ✓. 128 distinct draws (seed truly perturbs), ordered quantiles, byte-identical re-run (family == family2, C1), 128-member MonteCarlo lineage family.

fieldtests/milestone-the-world-orchestration-families/mw_1_random_tune_lineage.rs — random tune → optimize → lineage

• Draw 64 seeded random points over continuous ParamRanges, run the SAME sweep the grid uses (the "one unified sweep path"), optimize the best by total_pips, persist as a named Sweep family, round-trip, re-run mints a fresh id.
• Fits the milestone: axis 1 (random) + axis 2 + lineage over the synthetic seam.
• Outcome: built ✓, ran ✓, matched expected ✓ (after a fixture cleanup fix — see finding F8). 33 distinct total_pips over 64 points (family spreads); optimize == manual argmax; 64-member round-trip; re-run → eurusd-tune-1.

fieldtests/milestone-the-world-orchestration-families/out_mw_4_cli_lineage.txt — family-lineage CLI surface (CLI transcript)

• The family-lineage CLI driven end-to-end against the repo runs/ store: aura sweep --name, aura mc --name, aura runs families, aura runs family <id>, aura runs family <id> rank total_pips, unknown-id, and aura runs list.
• Fits the milestone: the registry-lineage CLI surface the promise names verbatim.
• Outcome: ran ✓; the family surface (sweep/mc persist, runs families, runs family, within-family rank) all worked; runs list / runs rank surfaced a bug (finding F5).

Findings

[working] Real-data walk-forward composes all four axes end-to-end

• Example: mw_3.
• What happened: walk_forward(roller, space, |bounds| { sweep+optimize in-sample; run OOS }) over M1FieldSource-streamed EURUSD M1 produced 5 windows, a 145 200-point stitched OOS curve, per-window chosen params that vary with the data, and a 5-member WalkForward lineage family — correct on first run.
• Why working: this is the milestone's central promise (the differentiator over a single backtest). The composition a researcher reaches for — roller + in-sample optimize + held-out OOS + stitch + persist — reads as ordinary code and runs deterministically over real data with no look-ahead (C2 structural).
• Recommended action: carry-on. Protect with a milestone-level integration test if one does not already exist.

[working] Monte-Carlo over a seed set is reproducible and yields a real distribution

• Example: mw_2.
• What happened: 128 seeds → 128 distinct draws, ordered aggregate quantiles, and monte_carlo(...) == monte_carlo(...) byte-for-byte (C1). Each draw's manifest.seed equals its input seed.
• Why working: seed-as-input (Fork B) plus the MC family delivers the robustness question directly; reproducibility is checkable in one assert_eq!.
• Recommended action: carry-on.

[working] Random sweep + optimize + lineage round-trip

• Example: mw_1.
• What happened: the random half of the sweep path drives the same sweep core, optimize returns the exact argmax, and the family persists + round-trips as a 64-member unit; a same-name re-run mints -1.
• Why working: confirms the "one unified sweep path" (grid and random both feed one enumeration-agnostic core) and the lineage round-trip empirically.
• Recommended action: carry-on.

[working] The family-lineage CLI surface lists and ranks families as units

• Example: mw_4 transcript.
• What happened: aura sweep --name mw-grid and aura mc --name mw-robust persist families; aura runs families lists {mw-grid-0 Sweep 4} + {mw-robust-0 MonteCarlo 3}; aura runs family mw-grid-0 rank total_pips ranks the members best-first; an unknown id prints nothing and exits 0 (per spec 0045).
• Why working: the CLI half of the lineage promise behaves exactly as the spec's worked console session shows.
• Recommended action: carry-on.

[bug] aura runs list / aura runs rank cannot read the existing runs.jsonl (typed-params schema break, unmigrated store)

• Example: mw_4 transcript (last two invocations).
• What happened, verbatim:

  $ aura runs list
  aura: registry parse error at line 1: expected value at line 1 column 94
  (exit 0)
  $ aura runs rank total_pips
  aura: registry parse error at line 1: expected value at line 1 column 94
  

  runs/runs.jsonl line 1 column 94 is the bare float 2.0 in ["sma_cross.fast",2.0]. RunManifest.params was migrated to typed Scalar ({"I64":2} / {"F64":2.0}) in cycle 0047 (commit 86746e3, inside the milestone range), but the persisted flat store was not migrated, so Registry::load fails to deserialize any pre-0047 line. The lineage path is unaffected (families.jsonl is written fresh in the new format — runs families / runs family succeeded).
• Why bug: aura runs list is documented to list the stored runs; on the repo's own checked-in store it instead errors out. The store is a real artefact (runs/runs.jsonl is in the tree, albeit under a gitignored dir), so a researcher who ran aura sweep/aura run before the 0047 upgrade loses read access to their run history. Secondary: a parse error prints to stderr but the process exits 0, so a script cannot detect the failure by status.
• Repro: cargo run -p aura-cli -- runs list against the current runs/runs.jsonl.
• Recommended action: debug. RED test: Registry::load over a pre-0047 Vec<(String, f64)>-params line. The fix is a migration/back-compat read (accept both the bare-float and the typed forms) or a one-shot store migration; separately decide whether a parse error should exit non-zero.

[friction] Real-data walk-forward must hand-invert the ms↔ns time convention

• Example: mw_3.
• What happened: WindowRoller / WindowBounds / RunManifest.window are in epoch-ns, but M1FieldSource::open(from_ms, to_ms) wants Unix-ms. aura-ingest exports unix_ms_to_epoch_ns (ms→ns) but no inverse, so the closure hand-writes ts.0 / 1_000_000 to feed the roller's ns bounds back into the ms-typed source — re-deriving a normalization the seam already owns.
• Why friction: the task completes, but a real-data walk-forward (the headline composition) forces the consumer to re-implement the ns→ms half of a convention the public surface only provides in one direction. A transposed divisor is a silent wrong-window bug class the seam was built to prevent.
• Recommended action: plan (tidy iteration) — export an epoch_ns_to_unix_ms inverse beside unix_ms_to_epoch_ns, or have M1FieldSource::open accept a Timestamp window directly so the roller's bounds feed straight in.

[friction] DataServer / DEFAULT_DATA_PATH are not re-exported by aura-ingest

• Example: mw_3 (Cargo.toml + use data_server::{DataServer, DEFAULT_DATA_PATH}).
• What happened: M1FieldSource::open takes &Arc<DataServer>, but aura-ingest re-exports neither DataServer nor the data path constant. To construct the required server, the consumer crate must add the external data-server git dependency directly and import from it — discovered only by reading the --real commit body (data_server::DEFAULT_DATA_PATH), not from any aura-ingest rustdoc or example.
• Why friction: the headline real-data task is reachable only if the consumer knows to pull and name a transitive external crate whose construction is undocumented in aura's public surface. The data path /mnt/tickdata/Pepperstone appears only in plan/spec prose, never in the API docs a consumer reads.
• Recommended action: plan — re-export data_server::DataServer (and a data-path accessor) from aura-ingest, or add an aura-ingest constructor (M1FieldSource::open_default_archive(symbol, from, to, field)) so a real-data source is buildable from aura-ingest alone.

[friction] The family store is keyed by the runs DIRECTORY, not the runs path; Registry::open rustdoc is silent

• Example: mw_1 (the round-trip assert first found 2 families instead of 1).
• What happened: append_family writes to path.with_file_name("families.jsonl") — a fixed filename in the runs path's directory. Two Registry::open calls with different runs filenames in the same directory therefore share one family store. A first fixture attempt isolated runs by a per-process filename (mw1-runs-<pid>.jsonl) and got a shared /tmp/families.jsonl, so the round-trip recovered 2 families. The sibling path is documented on append_family, but Registry::open's own rustdoc ("Bind to a JSONL path") does not mention that a second, directory-keyed store rides along.
• Why friction: the natural "isolate a temp store by unique filename" reflex silently fails for families; the consumer must know to isolate by directory. The fixture had to switch to a per-process temp directory.
• Recommended action: ratify or tighten the design ledger — note on Registry::open (and/or the C18 ledger entry) that a Registry owns two directory-co-located stores (runs.jsonl + families.jsonl), so isolation is per-directory. (Behaviour itself is a reasonable design; the silence is the gap.)

Recommendation summary

Finding	Class	Action
F1 real-data walk-forward composes all axes	working	carry-on
F2 Monte-Carlo reproducible + distribution	working	carry-on
F3 random sweep + optimize + lineage	working	carry-on
F4 family-lineage CLI lists/ranks units	working	carry-on
F5 runs list/rank break on pre-0047 store	bug	debug
F6 no ns→ms inverse for real-data WFO	friction	plan
F7 DataServer not re-exported by aura-ingest	friction	plan
F8 family store keyed by directory, undocumented on open	friction	ratify / tighten ledger