Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
Milestone fieldtest — Inferential validation
These are downstream-consumer artefacts for the milestone
"Inferential validation (defend against false discovery at sweep scale)".
Each scenario is a runnable shell script that drives only the public aura
CLI (discovered via --help) and asserts on its stdout / exit code — exactly
what a researcher with no access to the implementation source can do.
The milestone promise being field-tested:
Given a parameter sweep, tell me whether an apparent edge is REAL or an artefact of luck / overfitting.
via three recorded-provenance checks, all shipped:
- trials-deflation — deflate the selected winner's metric for the number
of configurations tried (
scenario_1_trials_deflation.sh). - plateau-over-peak — prefer a robust parameter plateau over the in-sample
peak, opt-in via
--select(scenario_2_plateau_over_peak.sh). - cross-instrument generalization — grade how consistently a candidate
holds across several instruments,
aura generalize(scenario_3_generalize.sh).
Plus the end-to-end composition a researcher actually runs
(scenario_4_composition.sh): sweep → pick a winner → validate it generalizes.
Data
Real Pepperstone M1 bars from the local archive
(/mnt/tickdata/Pepperstone). GER40, USDJPY and EURUSD all carry a
Sept-2024 month (Unix-ms 1725148800000..1727740799999); GER40 carries the
full 2024 calendar year, long enough for a walk-forward IS+OOS split.
Running
cd /home/brummel/dev/aura
fieldtests/milestone-inferential-validation/scenario_1_trials_deflation.sh
fieldtests/milestone-inferential-validation/scenario_2_plateau_over_peak.sh
fieldtests/milestone-inferential-validation/scenario_3_generalize.sh
fieldtests/milestone-inferential-validation/scenario_4_composition.sh
Each script builds via cargo run so HEAD is always what runs. The
AURA= env var at the top of each script picks the invocation; default is
cargo run -q --bin aura --.
All findings are recorded in
docs/specs/fieldtest-milestone-inferential-validation.md.