Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
50 lines
2.0 KiB
Markdown
50 lines
2.0 KiB
Markdown
# Milestone fieldtest — Inferential validation
|
|
|
|
These are downstream-consumer artefacts for the milestone
|
|
**"Inferential validation (defend against false discovery at sweep scale)"**.
|
|
Each scenario is a runnable shell script that drives only the public `aura`
|
|
CLI (discovered via `--help`) and asserts on its stdout / exit code — exactly
|
|
what a researcher with no access to the implementation source can do.
|
|
|
|
The milestone promise being field-tested:
|
|
|
|
> Given a parameter sweep, tell me whether an apparent edge is REAL or an
|
|
> artefact of luck / overfitting.
|
|
|
|
via three recorded-provenance checks, all shipped:
|
|
|
|
1. **trials-deflation** — deflate the selected winner's metric for the number
|
|
of configurations tried (`scenario_1_trials_deflation.sh`).
|
|
2. **plateau-over-peak** — prefer a robust parameter plateau over the in-sample
|
|
peak, opt-in via `--select` (`scenario_2_plateau_over_peak.sh`).
|
|
3. **cross-instrument generalization** — grade how consistently a candidate
|
|
holds across several instruments, `aura generalize`
|
|
(`scenario_3_generalize.sh`).
|
|
|
|
Plus the end-to-end composition a researcher actually runs
|
|
(`scenario_4_composition.sh`): sweep → pick a winner → validate it generalizes.
|
|
|
|
## Data
|
|
|
|
Real Pepperstone M1 bars from the local archive
|
|
(`/mnt/tickdata/Pepperstone`). GER40, USDJPY and EURUSD all carry a
|
|
Sept-2024 month (Unix-ms `1725148800000..1727740799999`); GER40 carries the
|
|
full 2024 calendar year, long enough for a walk-forward IS+OOS split.
|
|
|
|
## Running
|
|
|
|
```sh
|
|
cd /home/brummel/dev/aura
|
|
fieldtests/milestone-inferential-validation/scenario_1_trials_deflation.sh
|
|
fieldtests/milestone-inferential-validation/scenario_2_plateau_over_peak.sh
|
|
fieldtests/milestone-inferential-validation/scenario_3_generalize.sh
|
|
fieldtests/milestone-inferential-validation/scenario_4_composition.sh
|
|
```
|
|
|
|
Each script builds via `cargo run` so HEAD is always what runs. The
|
|
`AURA=` env var at the top of each script picks the invocation; default is
|
|
`cargo run -q --bin aura --`.
|
|
|
|
All findings are recorded in
|
|
`docs/specs/fieldtest-milestone-inferential-validation.md`.
|