Aura/docs/specs/fieldtest-milestone-inferential-validation.md

Fieldtest — milestone "Inferential validation" — 2026-06-26

Status: Draft — awaiting orchestrator triage Author: fieldtester (dispatched by fieldtest skill, milestone-close gate)

Scope

The milestone "Inferential validation (defend against false discovery at sweep scale)" makes aura's massively-parallel sweep defensible rather than a false-discovery engine, via three recorded-provenance checks now all shipped: (a) trials-deflation — deflate the selected winner's metric for the number of configurations tried (#144); (b) plateau-over-peak — an opt-in selection objective preferring a robust parameter plateau over the in-sample peak (#145); (c) cross-instrument generalization — the new aura generalize subcommand grading how consistently a brought candidate holds across several instruments (#146). The promise field-tested: "given a parameter sweep, tell me whether an apparent edge is REAL or an artefact of luck / overfitting."

This fieldtest works only from the public surface: the aura CLI (--help), the design ledger (docs/design/INDEX.md), the glossary, and the aura-ingest example corpus. The implementation source was not read. Exact CLI syntax was discovered from the single usage string aura --help prints. Build exercised: cargo run -q --bin aura -- (debug, built from HEAD 15f5c0b at the start of the run; the workspace built clean before any scenario).

Real Pepperstone M1 bars from /mnt/tickdata/Pepperstone (GER40, USDJPY, EURUSD all carry Sept-2024; GER40 carries the full 2024 calendar year).

Examples

fieldtests/milestone-inferential-validation/scenario_1_trials_deflation.sh — trials-deflation provenance

• A walk-forward over a 2x2 stage1-r grid on real GER40 (full 2024, 9 IS+OOS windows). Asserts each window's recorded winner carries a selection block with deflated_score, overfit_probability, n_trials, and the resampling provenance (n_resamples, block_len, seed), default mode Argmax, and that the deflated score sits below the raw winner metric.
• Fits the (a) trials-deflation piece: the recorded selection is exactly the anti-false-discovery deflation #144 promised.
• Outcome: built, ran, matched expected. PASS. (raw_winner_metric 0.0618 → deflated_score -0.153, overfit_probability 0.685 for the family of 4.)

fieldtests/milestone-inferential-validation/scenario_2_plateau_over_peak.sh — plateau is opt-in, default unchanged

• The same walk-forward run with no flag, with --select plateau:worst, with --select plateau:mean, and with a bad token. Asserts default = Argmax + deflation annotation (plateau keys absent); plateau:worst = PlateauWorst + neighbourhood_score/n_neighbours (deflation keys absent); plateau:mean = PlateauMean; bad token rejected non-zero.
• Fits the (b) plateau-over-peak piece: opt-in, default-preserving, with the plateau annotation reaching the recorded winner (orthogonal rule × annotation).
• Outcome: built, ran, matched expected. PASS.

fieldtests/milestone-inferential-validation/scenario_3_generalize.sh — cross-instrument generalization

• Happy path: aura generalize over GER40,USDJPY,EURUSD (Sept-2024). Asserts the per-instrument breakdown, that worst_case equals the min over instruments, a sign_agreement count, and that the run persists as a CrossInstrument family whose 3 members each carry their instrument lineage field. Plus the three refusals (single --real symbol, non-R --metric, multi-value candidate flag), each exit-2.
• Fits the (c) cross-instrument piece — the newest surface.
• Outcome: built, ran, matched expected. PASS. (worst_case = -0.0055 = the EURUSD expectancy_r, sign_agreement 2/3; all refusals exit 2.)

fieldtests/milestone-inferential-validation/scenario_4_composition.sh — the end-to-end "real or overfit?" story

• The chain a researcher runs: sweep a stage1-r grid on GER40 → rank by sqn to read the in-sample winner → generalize that winner across GER40,USDJPY,EURUSD. Asserts the chain runs coherently, renders a verdict from worst_case + sign_agreement, and that the GER40 leg of generalize reproduces the swept winner's sqn (within 1e-9).
• Fits the milestone promise as a whole: it composes all three pieces into the single "is this edge real?" verdict.
• Outcome: built, ran, matched expected (within tolerance). PASS, with the verdict the toolkit is meant to give: the GER40 in-sample winner (sqn 0.994) does NOT generalize (worst_case -0.149 on EURUSD, sign_agreement 2/3) → "instrument-specific / likely overfit." The honest answer is the win.

Findings

[working] All three milestone pieces reach the output and read sensibly

• All four scenarios.
• Trials-deflation: the recorded selection block carries deflated_score, overfit_probability, n_trials, n_resamples, block_len, seed, selection_metric, raw_winner_metric, mode — a fully self-describing provenance record on the winning member's manifest, exactly the #144 contract. Default mode is Argmax; the deflated score sits below the raw metric.
• Plateau: --select plateau:worst/plateau:mean flips mode to PlateauWorst/PlateauMean and swaps the deflation annotation for neighbourhood_score + n_neighbours; the no-flag default stays Argmax+deflation byte-for-byte (C23). The orthogonal rule × annotation reshape is visible and correct from the JSON alone.
• Generalize: the per-instrument breakdown + worst-case floor + sign-agreement print live; the M members persist as a CrossInstrument family each stamped with its instrument lineage and the correct per-instrument pip geometry (pip_size=1/0.01/0.0001 for GER40/USDJPY/EURUSD). The non-R metric refusal carries a precise message naming the four allowed R-metrics.
• Why working: the new surface was reached for the way a researcher would, and the recorded provenance is legible and correct on the first try. These are the wins that protect the milestone from drift.
• Recommended downstream action: carry-on.

[bug] aura <family-command> | head panics with "Broken pipe (os error 32)" on stdout

• Surfaced in scenario 2 (walkforward … | head -1) and scenario 4 (runs family … rank sqn | head -1).
• What happened (verbatim):

  thread 'main' (…) panicked at …/library/std/src/io/stdio.rs:1165:9:
  failed printing to stdout: Broken pipe (os error 32)
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  

  Under set -o pipefail this propagated exit code 101 and aborted the script.
• Why a bug: piping a multi-line family/sweep listing into head (or any reader that closes early — head, less, a closed UI pane) is the single most natural thing a CLI consumer does. The family-listing subcommands stream many JSONL lines and do not install a SIGPIPE handler / tolerate EPIPE, so the writer panics instead of exiting quietly. The wrong-output is a Rust panic + backtrace note where a clean truncated stream is expected. aura run (single line) does not hit it; the family-emitting commands (walkforward, sweep, mc, runs family … rank, runs families) do whenever they emit more lines than the reader consumes.
• One-line repro: cargo run -q --bin aura -- walkforward --strategy stage1-r --real GER40 --from 1704067200000 --to 1735689599999 --fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0 | head -1
• Recommended downstream action: debug (RED test: a family-emitting command piped into a head-like early-closing reader exits 0/SIGPIPE with no panic — restore the default SIGPIPE disposition or swallow ErrorKind::BrokenPipe at the print loop).

[friction] Two of the three generalize refusals dump bare usage with no reason

• Scenario 3 refusals (b) single --real symbol and (d) multi-value candidate flag (--fast 3,5).
• What happened: both print only the usage string and exit 2: aura: generalize [--strategy stage1-r] --real <SYM1,SYM2,...> --fast <n> … whereas the (c) non-R metric refusal gives a precise, actionable message: aura: metric 'total_pips' is not comparable across instruments; cross-instrument scoring is R-only (sqn, sqn_normalized, expectancy_r, net_expectancy_r).
• Why friction: the task completes (correct exit code), but the consumer is not told why. --real GER40 (one symbol) is refused because generalization needs ≥2 instruments — the whole point of the worst-case-across-instruments framing — yet the message never says "needs at least two instruments." --fast 3,5 is refused because the candidate must be a single grid cell (generalize validates one brought candidate, not a sweep) — but the usage string does not convey "single value per candidate flag." The (c) message proves the project knows how to write a good refusal; (b) and (d) just fall through to usage.
• Recommended downstream action: plan (tidy iteration) — give the arity/min-symbol refusals a one-line reason mirroring the R-only message ("needs ≥2 instruments", "candidate flags take a single value; generalize validates one cell").

[friction] The --select plateau objective is invisible in aura --help

• Scenario 2 (the milestone's plateau-over-peak feature).
• What happened: aura --help (and every per-subcommand --help, which all print the same single usage string) shows the walkforward form as aura walkforward [--strategy …] [--real …] [--name <n>|--trace <n>] [--fast …] [--slow …] [--stop-length …] [--stop-k …] — with no --select <argmax|plateau:mean|plateau:worst>. The flag works (scenario 2 proves it) but a consumer reading the help would never discover the plateau objective exists.
• Why friction: a shipped, opt-in milestone feature is undiscoverable from the only help the CLI offers. The feature is reachable only if you already know the flag name (from the ledger / commit log, which a downstream consumer does not read). The mc subcommand's --block-len/--resamples/--seed and the generalize --metric are documented in the usage; --select is the gap.
• Recommended downstream action: plan (tidy iteration) — add [--select <argmax|plateau:mean|plateau:worst>] to the walkforward branch of the usage string.

[spec_gap] Two orchestration commands compute the same member's R metric with a 1-ULP difference

• Scenario 4 (and confirmed by a direct repeat probe).
• What happened: for the identical (params fast=3 slow=6 stop_length=3 stop_k=2.0, Sept-2024 window, GER40, metric sqn), the sweep path records sqn = 0.9941661927806448 while the generalize path records 0.9941661927806447 — a last-ULP difference, stable and reproducible across re-runs (each path is internally bit-deterministic; the two paths differ).
• Why spec_gap: the ledger's C1 guarantees "same input (incl. seed) → bit-identical run," and the cross-instrument realization note calls the generalization aggregate "recomputable from the members." But the ledger is silent on whether two different commands computing the same member's metric must agree to the bit. A naive consumer asking "did my swept winner's GER40 score survive into generalize?" by exact string-equality would conclude they differ. I chose the reading that this is a benign floating-point reassociation between two reduction paths (within 1e-9), not a determinism violation — but the equally-plausible reading is that "recomputable" should mean bit-identical, in which case the two paths should share one reduction kernel. The contract does not say which.
• Recommended downstream action: ratify (record in the ledger that cross-command recomputation of a member metric is bit-equal-up-to-reassociation, not bit-identical — or tighten it to share the kernel). Either resolution is a one-line ledger note, not code the fieldtest should write.

[spec_gap] The milestone's core vocabulary is absent from the glossary

• All scenarios (a documentation observation across the milestone surface).
• What happened: docs/glossary.md defines bias, R, R metrics, sweep, walk-forward, Monte-Carlo, etc., but has no entry for "trials-deflation," "deflated score," "overfit probability," "plateau (over peak)," "cross-instrument generalization," "worst-case floor," or "sign-agreement" — the entire vocabulary the milestone's JSON output uses (deflated_score, overfit_probability, neighbourhood_score, worst_case, sign_agreement).
• Why spec_gap: a downstream researcher reading overfit_probability off a winner manifest, or worst_case off a generalize run, has no public definition to look up — the terms are load-bearing for interpreting the output but undocumented on the public surface. The ledger's open-thread prose explains them, but the glossary (the consumer-facing term index) is silent, so the reading of each term is left to the consumer. I read the numbers as "deflated = luck-penalised, overfit_probability = P(this winner is noise), worst_case = the floor over instruments" — plausible, but the public surface does not pin it.
• Recommended downstream action: tighten the design ledger / glossary — add the five-to-seven milestone terms to docs/glossary.md (a docwriter task), so the output is self-explanatory from the public interface alone.

Recommendation summary

Finding	Class	Action
All three pieces reach output, read sensibly	working	carry-on
… | head panics with Broken pipe on stdout	bug	debug
Two generalize refusals dump bare usage, no reason	friction	plan
--select plateau objective absent from --help	friction	plan
Sweep vs generalize 1-ULP metric mismatch	spec_gap	ratify
Milestone vocabulary absent from the glossary	spec_gap	tighten the design ledger