Brummel
ab04ed01c3
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
109 lines
4.7 KiB
Bash
Executable File
109 lines
4.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Scenario 3 — cross-instrument generalization (the newest piece).
|
|
#
|
|
# `aura generalize` runs a single brought candidate across an instrument list
|
|
# and grades how consistently it holds: a per-instrument R breakdown, a
|
|
# worst-case floor (min over instruments), and a sign-agreement count. The
|
|
# milestone framing is "an edge that holds on one instrument but not its
|
|
# neighbours is suspect" — so the worst-case-across-instruments min is the
|
|
# headline number.
|
|
#
|
|
# Happy path asserts (GER40,USDJPY,EURUSD share Sept-2024):
|
|
# - per_instrument breakdown carries all 3 symbols
|
|
# - worst_case == min over the per-instrument metric values
|
|
# - sign_agreement is reported
|
|
# - the family persists as a CrossInstrument family in `runs families`
|
|
# - each persisted member self-identifies via its `instrument` lineage field
|
|
#
|
|
# Refusal asserts (each exit non-zero):
|
|
# - a single --real symbol
|
|
# - a non-R --metric
|
|
# - a multi-value candidate flag (--fast 3,5)
|
|
set -euo pipefail
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
AURA="${AURA:-cargo run -q --bin aura --}"
|
|
|
|
FROM=1725148800000 # Sept-2024 shared window
|
|
TO=1727740799999
|
|
CAND=(--fast 3 --slow 8 --stop-length 14 --stop-k 2.0)
|
|
|
|
fail() { echo "FAIL: $1" >&2; exit 1; }
|
|
|
|
echo "### Scenario 3 — cross-instrument generalization"
|
|
echo
|
|
|
|
echo "--- (a) happy path: GER40,USDJPY,EURUSD ---"
|
|
OUT="$($AURA generalize --strategy stage1-r --real GER40,USDJPY,EURUSD "${CAND[@]}" \
|
|
--from "$FROM" --to "$TO" --name ft_generalize)"
|
|
printf '%s\n' "$OUT"
|
|
AGG="$(printf '%s\n' "$OUT" | head -1)"
|
|
for s in GER40 USDJPY EURUSD; do
|
|
printf '%s' "$AGG" | grep -q "\"$s\"" || fail "per_instrument missing $s"
|
|
done
|
|
printf '%s' "$AGG" | grep -q '"n_instruments":3' || fail 'n_instruments != 3'
|
|
printf '%s' "$AGG" | grep -q '"sign_agreement"' || fail 'no sign_agreement'
|
|
printf '%s' "$AGG" | grep -q '"worst_case"' || fail 'no worst_case'
|
|
|
|
# worst_case must equal the minimum of the three per-instrument metric values.
|
|
WORST="$(printf '%s' "$AGG" | grep -oE '"worst_case":[-0-9.eE]+' | cut -d: -f2)"
|
|
VALS="$(printf '%s' "$AGG" \
|
|
| grep -oE '\["[A-Z0-9]+",[-0-9.eE]+\]' | sed -E 's/.*,([-0-9.eE]+)\]/\1/')"
|
|
MIN="$(printf '%s\n' "$VALS" | sort -g | head -1)"
|
|
echo "per-instrument values: $(printf '%s ' $VALS)"
|
|
echo "worst_case = $WORST computed min = $MIN"
|
|
awk -v a="$WORST" -v b="$MIN" 'BEGIN{ exit !(a==b) }' \
|
|
|| fail "worst_case ($WORST) is not the min over instruments ($MIN)"
|
|
|
|
# the persisted family shows up as CrossInstrument, with 3 members.
|
|
# (re-running the family mints a fresh run index, per the ledger; read it back.)
|
|
FID="$(printf '%s' "$OUT" | grep -oE '"family_id":"ft_generalize-[0-9]+"' | head -1 | cut -d'"' -f4)"
|
|
[ -n "$FID" ] || fail 'no family_id minted for the generalize run'
|
|
echo "minted family id: $FID"
|
|
FAM="$($AURA runs families | grep "\"family_id\":\"$FID\"" || true)"
|
|
echo "$FAM"
|
|
printf '%s' "$FAM" | grep -q '"kind":"CrossInstrument"' \
|
|
|| fail 'persisted family kind is not CrossInstrument'
|
|
printf '%s' "$FAM" | grep -q '"members":3' \
|
|
|| fail 'CrossInstrument family does not have 3 members'
|
|
|
|
# each member self-identifies via its `instrument` lineage field.
|
|
MEMBERS="$($AURA runs family "$FID")"
|
|
for s in GER40 USDJPY EURUSD; do
|
|
printf '%s' "$MEMBERS" | grep -q "\"instrument\":\"$s\"" \
|
|
|| fail "member for $s missing its instrument lineage field"
|
|
done
|
|
echo "all 3 members carry their instrument lineage field"
|
|
echo
|
|
|
|
echo "--- (b) refusal: a single --real symbol ---"
|
|
set +e
|
|
$AURA generalize --strategy stage1-r --real GER40 "${CAND[@]}" --from "$FROM" --to "$TO" >/dev/null 2>&1
|
|
RC=$?; set -e
|
|
[ "$RC" -ne 0 ] || fail 'single-symbol generalize was not refused'
|
|
echo "single symbol refused, exit=$RC"
|
|
|
|
echo "--- (c) refusal: a non-R --metric ---"
|
|
set +e
|
|
MSG="$($AURA generalize --strategy stage1-r --real GER40,USDJPY "${CAND[@]}" \
|
|
--metric total_pips --from "$FROM" --to "$TO" 2>&1)"
|
|
RC=$?; set -e
|
|
[ "$RC" -ne 0 ] || fail 'non-R metric was not refused'
|
|
echo "non-R metric refused, exit=$RC; message:"
|
|
printf ' %s\n' "$MSG"
|
|
printf '%s' "$MSG" | grep -qi 'R-only\|not comparable' \
|
|
|| fail 'non-R refusal message does not explain the R-only rule'
|
|
|
|
echo "--- (d) refusal: a multi-value candidate flag (--fast 3,5) ---"
|
|
set +e
|
|
MSG2="$($AURA generalize --strategy stage1-r --real GER40,USDJPY --fast 3,5 --slow 8 \
|
|
--stop-length 14 --stop-k 2.0 --from "$FROM" --to "$TO" 2>&1)"
|
|
RC=$?; set -e
|
|
[ "$RC" -ne 0 ] || fail 'multi-value candidate flag was not refused'
|
|
echo "multi-value candidate refused, exit=$RC; message:"
|
|
printf ' %s\n' "$MSG2"
|
|
echo
|
|
|
|
echo "PASS: happy-path breakdown + worst-case floor + sign-agreement + persisted"
|
|
echo " CrossInstrument family with per-member instrument lineage; all three"
|
|
echo " refusals exit non-zero."
|