Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
85 lines
4.1 KiB
Bash
Executable File
85 lines
4.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Scenario 4 — the end-to-end "is this edge real or overfit?" story.
|
|
#
|
|
# The chain a researcher actually runs:
|
|
# 1. sweep a stage1-r grid on one instrument (GER40, Sept-2024)
|
|
# 2. rank the family to read off the in-sample winner
|
|
# 3. generalize that winner across instruments to see whether the edge holds
|
|
# out-of-instrument — the cross-instrument honesty check
|
|
#
|
|
# The point is the VERDICT: a winner that looks good in-sample on GER40 either
|
|
# generalizes (positive worst_case, full sign_agreement) or it does not (a
|
|
# negative worst_case / split sign_agreement = an instrument-specific artefact).
|
|
# Either way the toolkit gives a defensible answer; this script asserts the
|
|
# chain runs coherently and the winner's in-sample GER40 metric is reproduced
|
|
# byte-for-byte by the generalize step (determinism across paths, C1).
|
|
set -euo pipefail
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
AURA="${AURA:-cargo run -q --bin aura --}"
|
|
|
|
FROM=1725148800000 # Sept-2024 (sweep + generalize share this window)
|
|
TO=1727740799999
|
|
|
|
fail() { echo "FAIL: $1" >&2; exit 1; }
|
|
|
|
echo "### Scenario 4 — sweep -> pick winner -> generalize (end-to-end)"
|
|
echo
|
|
|
|
echo "--- (1) sweep stage1-r on GER40 ---"
|
|
$AURA sweep --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
|
--name ft_compose >/dev/null
|
|
echo "swept; family ft_compose-0 persisted"
|
|
|
|
echo "--- (2) rank the family by sqn, read the winner ---"
|
|
# Capture the full ranked output then slice; piping the subcommand directly into
|
|
# `head -1` makes `aura` panic with a Broken-pipe on stdout (recorded as a
|
|
# finding) — capture first, the careful-consumer pattern.
|
|
RANKED="$($AURA runs family ft_compose-0 rank sqn)"
|
|
TOP="$(printf '%s\n' "$RANKED" | head -1)"
|
|
echo "winner (verbatim manifest+metrics):"
|
|
printf '%s\n' "$TOP"
|
|
|
|
# extract the winner's signal knobs (fast.length, slow.length, stop_length, stop_k)
|
|
i64_of() { printf '%s' "$TOP" | grep -oE "\"$1\",\{\"I64\":[0-9]+\}" | sed -E 's/.*:([0-9]+)\}/\1/'; }
|
|
f64_of() { printf '%s' "$TOP" | grep -oE "\"$1\",\{\"F64\":[0-9.]+\}" | sed -E 's/.*:([0-9.]+)\}/\1/'; }
|
|
FAST="$(i64_of fast.length)"
|
|
SLOW="$(i64_of slow.length)"
|
|
SLEN="$(i64_of stop_length)"
|
|
SK="$(f64_of stop_k)"
|
|
WIN_SQN="$(printf '%s' "$TOP" | grep -oE '"sqn":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
|
echo "winner params: fast=$FAST slow=$SLOW stop_length=$SLEN stop_k=$SK (in-sample sqn=$WIN_SQN)"
|
|
[ -n "$FAST" ] && [ -n "$SLOW" ] && [ -n "$SLEN" ] && [ -n "$SK" ] \
|
|
|| fail 'could not read the winning params from the ranked family'
|
|
echo
|
|
|
|
echo "--- (3) generalize the winner across GER40,USDJPY,EURUSD by sqn ---"
|
|
GEN_ALL="$($AURA generalize --strategy stage1-r --real GER40,USDJPY,EURUSD \
|
|
--fast "$FAST" --slow "$SLOW" --stop-length "$SLEN" --stop-k "$SK" \
|
|
--metric sqn --from "$FROM" --to "$TO" --name ft_compose_gen)"
|
|
GEN="$(printf '%s\n' "$GEN_ALL" | head -1)"
|
|
printf '%s\n' "$GEN"
|
|
|
|
WORST="$(printf '%s' "$GEN" | grep -oE '"worst_case":[-0-9.eE]+' | cut -d: -f2)"
|
|
AGREE="$(printf '%s' "$GEN" | grep -oE '"sign_agreement":[0-9]+' | cut -d: -f2)"
|
|
GER_SQN="$(printf '%s' "$GEN" | grep -oE '\["GER40",[-0-9.eE]+\]' | sed -E 's/.*,([-0-9.eE]+)\]/\1/')"
|
|
echo
|
|
echo "verdict inputs: worst_case=$WORST sign_agreement=$AGREE GER40-sqn=$GER_SQN"
|
|
|
|
# The GER40 leg of generalize must reproduce the in-sample sweep winner's sqn
|
|
# (same window, same params, same engine -> bit-identical, C1).
|
|
awk -v a="$WIN_SQN" -v b="$GER_SQN" 'BEGIN{ d=a-b; if(d<0)d=-d; exit !(d < 1e-9) }' \
|
|
|| fail "generalize GER40 sqn ($GER_SQN) != sweep winner sqn ($WIN_SQN)"
|
|
echo "GER40 leg reproduces the sweep winner's sqn (determinism across paths OK)"
|
|
echo
|
|
|
|
# Render the human verdict from the floor + agreement.
|
|
if awk -v w="$WORST" 'BEGIN{ exit !(w > 0) }'; then
|
|
echo "VERDICT: edge holds across all instruments (worst_case > 0) -> defensible."
|
|
else
|
|
echo "VERDICT: edge does NOT hold on every instrument (worst_case <= 0,"
|
|
echo " sign_agreement=$AGREE/3) -> instrument-specific / likely overfit."
|
|
fi
|
|
echo
|
|
echo "PASS: the sweep -> rank -> generalize chain runs coherently and gives a"
|
|
echo " defensible real-vs-overfit verdict; the GER40 leg is bit-reproducible."
|