Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
81 lines
3.5 KiB
Bash
Executable File
81 lines
3.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Scenario 2 — plateau-over-peak is opt-in; the default is unchanged.
|
|
#
|
|
# The milestone adds a robustness objective: prefer a parameter PLATEAU (a
|
|
# member whose grid neighbourhood is uniformly decent) over a sharp in-sample
|
|
# PEAK. The contract (ledger) is that this is OPT-IN via `--select`, the default
|
|
# stays the bare-argmax selection rule, and the plateau ANNOTATION reaches the
|
|
# recorded winner (orthogonal rule x annotation).
|
|
#
|
|
# Asserts:
|
|
# - default walk-forward records mode "Argmax" + deflation annotation
|
|
# (deflated_score present, plateau keys absent)
|
|
# - `--select plateau:worst` records mode "PlateauWorst" + plateau annotation
|
|
# (neighbourhood_score + n_neighbours present, deflation keys absent)
|
|
# - `--select plateau:mean` records mode "PlateauMean"
|
|
# - a bad --select token is rejected non-zero
|
|
set -euo pipefail
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
AURA="${AURA:-cargo run -q --bin aura --}"
|
|
|
|
FROM=1704067200000 # GER40 full 2024
|
|
TO=1735689599999
|
|
GRID=(--fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0)
|
|
|
|
fail() { echo "FAIL: $1" >&2; exit 1; }
|
|
# Capture the whole stream into a var, THEN slice the first window. Piping the
|
|
# subcommand straight into `head -1` closes stdout early and makes `aura` panic
|
|
# with a Broken-pipe on stdout (recorded as a finding) — a careful consumer
|
|
# captures first.
|
|
first_window() { printf '%s\n' "$1" | head -1; }
|
|
|
|
echo "### Scenario 2 — plateau-over-peak (opt-in; default unchanged)"
|
|
echo
|
|
|
|
echo "--- (a) default selection (no --select) ---"
|
|
DEF="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
"${GRID[@]}" --name ft_plateau_default)")"
|
|
printf '%s\n' "$DEF"
|
|
printf '%s' "$DEF" | grep -q '"mode":"Argmax"' \
|
|
|| fail 'default mode is not Argmax'
|
|
printf '%s' "$DEF" | grep -q '"deflated_score"' \
|
|
|| fail 'default did not carry the deflation annotation'
|
|
printf '%s' "$DEF" | grep -q '"neighbourhood_score"' \
|
|
&& fail 'default unexpectedly carried a plateau annotation'
|
|
echo
|
|
|
|
echo "--- (b) --select plateau:worst ---"
|
|
PW="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
"${GRID[@]}" --select plateau:worst --name ft_plateau_worst)")"
|
|
printf '%s\n' "$PW"
|
|
printf '%s' "$PW" | grep -q '"mode":"PlateauWorst"' \
|
|
|| fail 'plateau:worst did not record mode PlateauWorst'
|
|
printf '%s' "$PW" | grep -q '"neighbourhood_score"' \
|
|
|| fail 'plateau:worst did not record neighbourhood_score'
|
|
printf '%s' "$PW" | grep -q '"n_neighbours"' \
|
|
|| fail 'plateau:worst did not record n_neighbours'
|
|
printf '%s' "$PW" | grep -q '"deflated_score"' \
|
|
&& fail 'plateau:worst unexpectedly carried the deflation annotation'
|
|
echo
|
|
|
|
echo "--- (c) --select plateau:mean ---"
|
|
PM="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
"${GRID[@]}" --select plateau:mean --name ft_plateau_mean)")"
|
|
printf '%s' "$PM" | grep -q '"mode":"PlateauMean"' \
|
|
|| fail 'plateau:mean did not record mode PlateauMean'
|
|
echo "PlateauMean recorded OK"
|
|
echo
|
|
|
|
echo "--- (d) bad --select token is rejected ---"
|
|
set +e
|
|
$AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
"${GRID[@]}" --select plateau:median --name ft_plateau_bad >/dev/null 2>&1
|
|
RC=$?
|
|
set -e
|
|
[ "$RC" -ne 0 ] || fail 'bad --select token was not rejected (exit 0)'
|
|
echo "bad token --select plateau:median rejected, exit=$RC"
|
|
echo
|
|
|
|
echo "PASS: plateau is opt-in; default stays Argmax+deflation; plateau:worst /"
|
|
echo " plateau:mean record the plateau annotation; bad token rejected."
|