Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
64 lines
2.9 KiB
Bash
Executable File
64 lines
2.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Scenario 1 — trials-deflation provenance reaches the output.
|
|
#
|
|
# A researcher runs a walk-forward over a stage1-r parameter grid on real GER40
|
|
# bars (full 2024, long enough for an IS+OOS split). Each window's in-sample
|
|
# optimisation selects a winner; the milestone promise is that the recorded
|
|
# winner carries a trials-DEFLATED score + an overfit probability for the family
|
|
# size, NOT a bare argmax with no luck penalty.
|
|
#
|
|
# Asserts:
|
|
# - the default selection mode is "Argmax" (deflation is the default annotation)
|
|
# - the winner manifest carries a `selection` block with `deflated_score`,
|
|
# `overfit_probability`, `n_trials`, and the resampling provenance
|
|
# (`n_resamples`, `block_len`, `seed`)
|
|
# - the deflated score is BELOW the raw winner metric (the luck penalty bit)
|
|
set -euo pipefail
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
AURA="${AURA:-cargo run -q --bin aura --}"
|
|
|
|
# GER40, full calendar 2024 (Unix-ms). 9 IS+OOS windows.
|
|
FROM=1704067200000
|
|
TO=1735689599999
|
|
|
|
echo "### Scenario 1 — trials-deflation provenance"
|
|
echo "# cmd: aura walkforward --strategy stage1-r --real GER40 --from $FROM --to $TO \\"
|
|
echo "# --fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0 --name ft_deflation"
|
|
echo
|
|
|
|
OUT="$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
--fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0 --name ft_deflation)"
|
|
|
|
# Print the first window's selection block for the record.
|
|
FIRST="$(printf '%s\n' "$OUT" | head -1)"
|
|
echo "--- first window winner (verbatim) ---"
|
|
printf '%s\n' "$FIRST"
|
|
echo
|
|
|
|
fail() { echo "FAIL: $1" >&2; exit 1; }
|
|
|
|
printf '%s' "$FIRST" | grep -q '"mode":"Argmax"' \
|
|
|| fail 'default selection mode is not Argmax'
|
|
printf '%s' "$FIRST" | grep -q '"deflated_score"' \
|
|
|| fail 'no deflated_score in the selection block'
|
|
printf '%s' "$FIRST" | grep -q '"overfit_probability"' \
|
|
|| fail 'no overfit_probability in the selection block'
|
|
printf '%s' "$FIRST" | grep -q '"n_trials":4' \
|
|
|| fail 'n_trials is not the family size (4)'
|
|
printf '%s' "$FIRST" | grep -q '"n_resamples"' \
|
|
|| fail 'no n_resamples (bootstrap provenance) in the selection block'
|
|
printf '%s' "$FIRST" | grep -q '"block_len"' \
|
|
|| fail 'no block_len (moving-block provenance) in the selection block'
|
|
|
|
# raw_winner_metric > deflated_score (the penalty actually bites).
|
|
RAW="$(printf '%s' "$FIRST" | grep -oE '"raw_winner_metric":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
|
DEF="$(printf '%s' "$FIRST" | grep -oE '"deflated_score":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
|
echo "raw_winner_metric = $RAW deflated_score = $DEF"
|
|
awk -v r="$RAW" -v d="$DEF" 'BEGIN{ exit !(d < r) }' \
|
|
|| fail "deflated_score ($DEF) is not below raw_winner_metric ($RAW)"
|
|
|
|
echo
|
|
echo "PASS: deflation provenance (deflated_score + overfit_probability + n_trials"
|
|
echo " + resampling config) reaches the recorded winner, default mode Argmax,"
|
|
echo " and the deflated score sits below the raw winner metric."
|