# Milestone fieldtest — Inferential validation

These are downstream-consumer artefacts for the milestone
**"Inferential validation (defend against false discovery at sweep scale)"**.
Each scenario is a runnable shell script that drives only the public `aura`
CLI (discovered via `--help`) and asserts on its stdout / exit code — exactly
what a researcher with no access to the implementation source can do.

The milestone promise being field-tested:

> Given a parameter sweep, tell me whether an apparent edge is REAL or an
> artefact of luck / overfitting.

via three recorded-provenance checks, all shipped:

1. **trials-deflation** — deflate the selected winner's metric for the number
   of configurations tried (`scenario_1_trials_deflation.sh`).
2. **plateau-over-peak** — prefer a robust parameter plateau over the in-sample
   peak, opt-in via `--select` (`scenario_2_plateau_over_peak.sh`).
3. **cross-instrument generalization** — grade how consistently a candidate
   holds across several instruments, `aura generalize`
   (`scenario_3_generalize.sh`).

Plus the end-to-end composition a researcher actually runs
(`scenario_4_composition.sh`): sweep → pick a winner → validate it generalizes.

## Data

Real Pepperstone M1 bars from the local archive
(`/mnt/tickdata/Pepperstone`). GER40, USDJPY and EURUSD all carry a
Sept-2024 month (Unix-ms `1725148800000..1727740799999`); GER40 carries the
full 2024 calendar year, long enough for a walk-forward IS+OOS split.

## Running

```sh
cd /home/brummel/dev/aura
fieldtests/milestone-inferential-validation/scenario_1_trials_deflation.sh
fieldtests/milestone-inferential-validation/scenario_2_plateau_over_peak.sh
fieldtests/milestone-inferential-validation/scenario_3_generalize.sh
fieldtests/milestone-inferential-validation/scenario_4_composition.sh
```

Each script builds via `cargo run` so HEAD is always what runs. The
`AURA=` env var at the top of each script picks the invocation; default is
`cargo run -q --bin aura --`.

All findings are recorded in
`docs/specs/fieldtest-milestone-inferential-validation.md`.