fieldtest: milestone inferential-validation — 4 scenarios, 6 findings (1 bug, 2 friction, 2 spec_gap, 1 working)
Milestone-close gate for "Inferential validation (defend against false discovery at sweep scale)". Downstream-consumer exercise from the public interface only (CLI --help, ledger, glossary, examples — never the source). All four end-to-end scenarios PASS — the milestone delivers its promise: trials-deflation (#144), plateau-over-peak (#145), and cross-instrument generalization (#146) all reach the output and read sensibly, and the composed "is this edge real or overfit?" verdict is honest (scenario 4: a GER40 in-sample winner sqn 0.994 does NOT generalize — worst_case -0.149 on EURUSD, sign_agreement 2/3 → instrument-specific / likely overfit). Milestone status roll-up: bugs_found — NOT green, so the milestone is NOT yet closeable (the green roll-up is the functional leg of the close gate). Findings: - [bug] a family-emitting command piped into an early-closing reader (`aura sweep|walkforward|mc|runs family … | head`) panics "Broken pipe (os error 32)" on stdout (exit 101 under pipefail) instead of exiting cleanly on EPIPE. Pre-existing (not a #146 regression — reproduced on `aura sweep | head -1`); surfaced by the milestone commands. -> debug (RED-first). - [friction] two generalize refusals (single --real symbol; multi-value candidate flag) dump bare usage with no reason. -> tidy iteration. - [friction] the --select plateau objective is absent from `aura --help`. -> tidy. - [spec_gap] sweep vs generalize recompute the same member's sqn with a stable 1-ULP difference; the ledger's C1 / "recomputable" is silent on cross-command metric agreement. -> ratify (ledger note). - [spec_gap] the milestone vocabulary (deflated_score / overfit_probability / worst_case / sign_agreement) is absent from docs/glossary.md. -> docwriter. Artefacts: docs/specs/fieldtest-milestone-inferential-validation.md + fieldtests/milestone-inferential-validation/ (README + 4 scenario scripts). The existing fieldtests/ tree was untouched. refs #146
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# Milestone fieldtest — Inferential validation
|
||||
|
||||
These are downstream-consumer artefacts for the milestone
|
||||
**"Inferential validation (defend against false discovery at sweep scale)"**.
|
||||
Each scenario is a runnable shell script that drives only the public `aura`
|
||||
CLI (discovered via `--help`) and asserts on its stdout / exit code — exactly
|
||||
what a researcher with no access to the implementation source can do.
|
||||
|
||||
The milestone promise being field-tested:
|
||||
|
||||
> Given a parameter sweep, tell me whether an apparent edge is REAL or an
|
||||
> artefact of luck / overfitting.
|
||||
|
||||
via three recorded-provenance checks, all shipped:
|
||||
|
||||
1. **trials-deflation** — deflate the selected winner's metric for the number
|
||||
of configurations tried (`scenario_1_trials_deflation.sh`).
|
||||
2. **plateau-over-peak** — prefer a robust parameter plateau over the in-sample
|
||||
peak, opt-in via `--select` (`scenario_2_plateau_over_peak.sh`).
|
||||
3. **cross-instrument generalization** — grade how consistently a candidate
|
||||
holds across several instruments, `aura generalize`
|
||||
(`scenario_3_generalize.sh`).
|
||||
|
||||
Plus the end-to-end composition a researcher actually runs
|
||||
(`scenario_4_composition.sh`): sweep → pick a winner → validate it generalizes.
|
||||
|
||||
## Data
|
||||
|
||||
Real Pepperstone M1 bars from the local archive
|
||||
(`/mnt/tickdata/Pepperstone`). GER40, USDJPY and EURUSD all carry a
|
||||
Sept-2024 month (Unix-ms `1725148800000..1727740799999`); GER40 carries the
|
||||
full 2024 calendar year, long enough for a walk-forward IS+OOS split.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
cd /home/brummel/dev/aura
|
||||
fieldtests/milestone-inferential-validation/scenario_1_trials_deflation.sh
|
||||
fieldtests/milestone-inferential-validation/scenario_2_plateau_over_peak.sh
|
||||
fieldtests/milestone-inferential-validation/scenario_3_generalize.sh
|
||||
fieldtests/milestone-inferential-validation/scenario_4_composition.sh
|
||||
```
|
||||
|
||||
Each script builds via `cargo run` so HEAD is always what runs. The
|
||||
`AURA=` env var at the top of each script picks the invocation; default is
|
||||
`cargo run -q --bin aura --`.
|
||||
|
||||
All findings are recorded in
|
||||
`docs/specs/fieldtest-milestone-inferential-validation.md`.
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scenario 1 — trials-deflation provenance reaches the output.
|
||||
#
|
||||
# A researcher runs a walk-forward over a stage1-r parameter grid on real GER40
|
||||
# bars (full 2024, long enough for an IS+OOS split). Each window's in-sample
|
||||
# optimisation selects a winner; the milestone promise is that the recorded
|
||||
# winner carries a trials-DEFLATED score + an overfit probability for the family
|
||||
# size, NOT a bare argmax with no luck penalty.
|
||||
#
|
||||
# Asserts:
|
||||
# - the default selection mode is "Argmax" (deflation is the default annotation)
|
||||
# - the winner manifest carries a `selection` block with `deflated_score`,
|
||||
# `overfit_probability`, `n_trials`, and the resampling provenance
|
||||
# (`n_resamples`, `block_len`, `seed`)
|
||||
# - the deflated score is BELOW the raw winner metric (the luck penalty bit)
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
AURA="${AURA:-cargo run -q --bin aura --}"
|
||||
|
||||
# GER40, full calendar 2024 (Unix-ms). 9 IS+OOS windows.
|
||||
FROM=1704067200000
|
||||
TO=1735689599999
|
||||
|
||||
echo "### Scenario 1 — trials-deflation provenance"
|
||||
echo "# cmd: aura walkforward --strategy stage1-r --real GER40 --from $FROM --to $TO \\"
|
||||
echo "# --fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0 --name ft_deflation"
|
||||
echo
|
||||
|
||||
OUT="$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
--fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0 --name ft_deflation)"
|
||||
|
||||
# Print the first window's selection block for the record.
|
||||
FIRST="$(printf '%s\n' "$OUT" | head -1)"
|
||||
echo "--- first window winner (verbatim) ---"
|
||||
printf '%s\n' "$FIRST"
|
||||
echo
|
||||
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
|
||||
printf '%s' "$FIRST" | grep -q '"mode":"Argmax"' \
|
||||
|| fail 'default selection mode is not Argmax'
|
||||
printf '%s' "$FIRST" | grep -q '"deflated_score"' \
|
||||
|| fail 'no deflated_score in the selection block'
|
||||
printf '%s' "$FIRST" | grep -q '"overfit_probability"' \
|
||||
|| fail 'no overfit_probability in the selection block'
|
||||
printf '%s' "$FIRST" | grep -q '"n_trials":4' \
|
||||
|| fail 'n_trials is not the family size (4)'
|
||||
printf '%s' "$FIRST" | grep -q '"n_resamples"' \
|
||||
|| fail 'no n_resamples (bootstrap provenance) in the selection block'
|
||||
printf '%s' "$FIRST" | grep -q '"block_len"' \
|
||||
|| fail 'no block_len (moving-block provenance) in the selection block'
|
||||
|
||||
# raw_winner_metric > deflated_score (the penalty actually bites).
|
||||
RAW="$(printf '%s' "$FIRST" | grep -oE '"raw_winner_metric":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
||||
DEF="$(printf '%s' "$FIRST" | grep -oE '"deflated_score":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
||||
echo "raw_winner_metric = $RAW deflated_score = $DEF"
|
||||
awk -v r="$RAW" -v d="$DEF" 'BEGIN{ exit !(d < r) }' \
|
||||
|| fail "deflated_score ($DEF) is not below raw_winner_metric ($RAW)"
|
||||
|
||||
echo
|
||||
echo "PASS: deflation provenance (deflated_score + overfit_probability + n_trials"
|
||||
echo " + resampling config) reaches the recorded winner, default mode Argmax,"
|
||||
echo " and the deflated score sits below the raw winner metric."
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scenario 2 — plateau-over-peak is opt-in; the default is unchanged.
|
||||
#
|
||||
# The milestone adds a robustness objective: prefer a parameter PLATEAU (a
|
||||
# member whose grid neighbourhood is uniformly decent) over a sharp in-sample
|
||||
# PEAK. The contract (ledger) is that this is OPT-IN via `--select`, the default
|
||||
# stays the bare-argmax selection rule, and the plateau ANNOTATION reaches the
|
||||
# recorded winner (orthogonal rule x annotation).
|
||||
#
|
||||
# Asserts:
|
||||
# - default walk-forward records mode "Argmax" + deflation annotation
|
||||
# (deflated_score present, plateau keys absent)
|
||||
# - `--select plateau:worst` records mode "PlateauWorst" + plateau annotation
|
||||
# (neighbourhood_score + n_neighbours present, deflation keys absent)
|
||||
# - `--select plateau:mean` records mode "PlateauMean"
|
||||
# - a bad --select token is rejected non-zero
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
AURA="${AURA:-cargo run -q --bin aura --}"
|
||||
|
||||
FROM=1704067200000 # GER40 full 2024
|
||||
TO=1735689599999
|
||||
GRID=(--fast 2,3 --slow 8,12 --stop-length 14 --stop-k 2.0)
|
||||
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
# Capture the whole stream into a var, THEN slice the first window. Piping the
|
||||
# subcommand straight into `head -1` closes stdout early and makes `aura` panic
|
||||
# with a Broken-pipe on stdout (recorded as a finding) — a careful consumer
|
||||
# captures first.
|
||||
first_window() { printf '%s\n' "$1" | head -1; }
|
||||
|
||||
echo "### Scenario 2 — plateau-over-peak (opt-in; default unchanged)"
|
||||
echo
|
||||
|
||||
echo "--- (a) default selection (no --select) ---"
|
||||
DEF="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
"${GRID[@]}" --name ft_plateau_default)")"
|
||||
printf '%s\n' "$DEF"
|
||||
printf '%s' "$DEF" | grep -q '"mode":"Argmax"' \
|
||||
|| fail 'default mode is not Argmax'
|
||||
printf '%s' "$DEF" | grep -q '"deflated_score"' \
|
||||
|| fail 'default did not carry the deflation annotation'
|
||||
printf '%s' "$DEF" | grep -q '"neighbourhood_score"' \
|
||||
&& fail 'default unexpectedly carried a plateau annotation'
|
||||
echo
|
||||
|
||||
echo "--- (b) --select plateau:worst ---"
|
||||
PW="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
"${GRID[@]}" --select plateau:worst --name ft_plateau_worst)")"
|
||||
printf '%s\n' "$PW"
|
||||
printf '%s' "$PW" | grep -q '"mode":"PlateauWorst"' \
|
||||
|| fail 'plateau:worst did not record mode PlateauWorst'
|
||||
printf '%s' "$PW" | grep -q '"neighbourhood_score"' \
|
||||
|| fail 'plateau:worst did not record neighbourhood_score'
|
||||
printf '%s' "$PW" | grep -q '"n_neighbours"' \
|
||||
|| fail 'plateau:worst did not record n_neighbours'
|
||||
printf '%s' "$PW" | grep -q '"deflated_score"' \
|
||||
&& fail 'plateau:worst unexpectedly carried the deflation annotation'
|
||||
echo
|
||||
|
||||
echo "--- (c) --select plateau:mean ---"
|
||||
PM="$(first_window "$($AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
"${GRID[@]}" --select plateau:mean --name ft_plateau_mean)")"
|
||||
printf '%s' "$PM" | grep -q '"mode":"PlateauMean"' \
|
||||
|| fail 'plateau:mean did not record mode PlateauMean'
|
||||
echo "PlateauMean recorded OK"
|
||||
echo
|
||||
|
||||
echo "--- (d) bad --select token is rejected ---"
|
||||
set +e
|
||||
$AURA walkforward --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
"${GRID[@]}" --select plateau:median --name ft_plateau_bad >/dev/null 2>&1
|
||||
RC=$?
|
||||
set -e
|
||||
[ "$RC" -ne 0 ] || fail 'bad --select token was not rejected (exit 0)'
|
||||
echo "bad token --select plateau:median rejected, exit=$RC"
|
||||
echo
|
||||
|
||||
echo "PASS: plateau is opt-in; default stays Argmax+deflation; plateau:worst /"
|
||||
echo " plateau:mean record the plateau annotation; bad token rejected."
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scenario 3 — cross-instrument generalization (the newest piece).
|
||||
#
|
||||
# `aura generalize` runs a single brought candidate across an instrument list
|
||||
# and grades how consistently it holds: a per-instrument R breakdown, a
|
||||
# worst-case floor (min over instruments), and a sign-agreement count. The
|
||||
# milestone framing is "an edge that holds on one instrument but not its
|
||||
# neighbours is suspect" — so the worst-case-across-instruments min is the
|
||||
# headline number.
|
||||
#
|
||||
# Happy path asserts (GER40,USDJPY,EURUSD share Sept-2024):
|
||||
# - per_instrument breakdown carries all 3 symbols
|
||||
# - worst_case == min over the per-instrument metric values
|
||||
# - sign_agreement is reported
|
||||
# - the family persists as a CrossInstrument family in `runs families`
|
||||
# - each persisted member self-identifies via its `instrument` lineage field
|
||||
#
|
||||
# Refusal asserts (each exit non-zero):
|
||||
# - a single --real symbol
|
||||
# - a non-R --metric
|
||||
# - a multi-value candidate flag (--fast 3,5)
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
AURA="${AURA:-cargo run -q --bin aura --}"
|
||||
|
||||
FROM=1725148800000 # Sept-2024 shared window
|
||||
TO=1727740799999
|
||||
CAND=(--fast 3 --slow 8 --stop-length 14 --stop-k 2.0)
|
||||
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
|
||||
echo "### Scenario 3 — cross-instrument generalization"
|
||||
echo
|
||||
|
||||
echo "--- (a) happy path: GER40,USDJPY,EURUSD ---"
|
||||
OUT="$($AURA generalize --strategy stage1-r --real GER40,USDJPY,EURUSD "${CAND[@]}" \
|
||||
--from "$FROM" --to "$TO" --name ft_generalize)"
|
||||
printf '%s\n' "$OUT"
|
||||
AGG="$(printf '%s\n' "$OUT" | head -1)"
|
||||
for s in GER40 USDJPY EURUSD; do
|
||||
printf '%s' "$AGG" | grep -q "\"$s\"" || fail "per_instrument missing $s"
|
||||
done
|
||||
printf '%s' "$AGG" | grep -q '"n_instruments":3' || fail 'n_instruments != 3'
|
||||
printf '%s' "$AGG" | grep -q '"sign_agreement"' || fail 'no sign_agreement'
|
||||
printf '%s' "$AGG" | grep -q '"worst_case"' || fail 'no worst_case'
|
||||
|
||||
# worst_case must equal the minimum of the three per-instrument metric values.
|
||||
WORST="$(printf '%s' "$AGG" | grep -oE '"worst_case":[-0-9.eE]+' | cut -d: -f2)"
|
||||
VALS="$(printf '%s' "$AGG" \
|
||||
| grep -oE '\["[A-Z0-9]+",[-0-9.eE]+\]' | sed -E 's/.*,([-0-9.eE]+)\]/\1/')"
|
||||
MIN="$(printf '%s\n' "$VALS" | sort -g | head -1)"
|
||||
echo "per-instrument values: $(printf '%s ' $VALS)"
|
||||
echo "worst_case = $WORST computed min = $MIN"
|
||||
awk -v a="$WORST" -v b="$MIN" 'BEGIN{ exit !(a==b) }' \
|
||||
|| fail "worst_case ($WORST) is not the min over instruments ($MIN)"
|
||||
|
||||
# the persisted family shows up as CrossInstrument, with 3 members.
|
||||
# (re-running the family mints a fresh run index, per the ledger; read it back.)
|
||||
FID="$(printf '%s' "$OUT" | grep -oE '"family_id":"ft_generalize-[0-9]+"' | head -1 | cut -d'"' -f4)"
|
||||
[ -n "$FID" ] || fail 'no family_id minted for the generalize run'
|
||||
echo "minted family id: $FID"
|
||||
FAM="$($AURA runs families | grep "\"family_id\":\"$FID\"" || true)"
|
||||
echo "$FAM"
|
||||
printf '%s' "$FAM" | grep -q '"kind":"CrossInstrument"' \
|
||||
|| fail 'persisted family kind is not CrossInstrument'
|
||||
printf '%s' "$FAM" | grep -q '"members":3' \
|
||||
|| fail 'CrossInstrument family does not have 3 members'
|
||||
|
||||
# each member self-identifies via its `instrument` lineage field.
|
||||
MEMBERS="$($AURA runs family "$FID")"
|
||||
for s in GER40 USDJPY EURUSD; do
|
||||
printf '%s' "$MEMBERS" | grep -q "\"instrument\":\"$s\"" \
|
||||
|| fail "member for $s missing its instrument lineage field"
|
||||
done
|
||||
echo "all 3 members carry their instrument lineage field"
|
||||
echo
|
||||
|
||||
echo "--- (b) refusal: a single --real symbol ---"
|
||||
set +e
|
||||
$AURA generalize --strategy stage1-r --real GER40 "${CAND[@]}" --from "$FROM" --to "$TO" >/dev/null 2>&1
|
||||
RC=$?; set -e
|
||||
[ "$RC" -ne 0 ] || fail 'single-symbol generalize was not refused'
|
||||
echo "single symbol refused, exit=$RC"
|
||||
|
||||
echo "--- (c) refusal: a non-R --metric ---"
|
||||
set +e
|
||||
MSG="$($AURA generalize --strategy stage1-r --real GER40,USDJPY "${CAND[@]}" \
|
||||
--metric total_pips --from "$FROM" --to "$TO" 2>&1)"
|
||||
RC=$?; set -e
|
||||
[ "$RC" -ne 0 ] || fail 'non-R metric was not refused'
|
||||
echo "non-R metric refused, exit=$RC; message:"
|
||||
printf ' %s\n' "$MSG"
|
||||
printf '%s' "$MSG" | grep -qi 'R-only\|not comparable' \
|
||||
|| fail 'non-R refusal message does not explain the R-only rule'
|
||||
|
||||
echo "--- (d) refusal: a multi-value candidate flag (--fast 3,5) ---"
|
||||
set +e
|
||||
MSG2="$($AURA generalize --strategy stage1-r --real GER40,USDJPY --fast 3,5 --slow 8 \
|
||||
--stop-length 14 --stop-k 2.0 --from "$FROM" --to "$TO" 2>&1)"
|
||||
RC=$?; set -e
|
||||
[ "$RC" -ne 0 ] || fail 'multi-value candidate flag was not refused'
|
||||
echo "multi-value candidate refused, exit=$RC; message:"
|
||||
printf ' %s\n' "$MSG2"
|
||||
echo
|
||||
|
||||
echo "PASS: happy-path breakdown + worst-case floor + sign-agreement + persisted"
|
||||
echo " CrossInstrument family with per-member instrument lineage; all three"
|
||||
echo " refusals exit non-zero."
|
||||
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scenario 4 — the end-to-end "is this edge real or overfit?" story.
|
||||
#
|
||||
# The chain a researcher actually runs:
|
||||
# 1. sweep a stage1-r grid on one instrument (GER40, Sept-2024)
|
||||
# 2. rank the family to read off the in-sample winner
|
||||
# 3. generalize that winner across instruments to see whether the edge holds
|
||||
# out-of-instrument — the cross-instrument honesty check
|
||||
#
|
||||
# The point is the VERDICT: a winner that looks good in-sample on GER40 either
|
||||
# generalizes (positive worst_case, full sign_agreement) or it does not (a
|
||||
# negative worst_case / split sign_agreement = an instrument-specific artefact).
|
||||
# Either way the toolkit gives a defensible answer; this script asserts the
|
||||
# chain runs coherently and the winner's in-sample GER40 metric is reproduced
|
||||
# byte-for-byte by the generalize step (determinism across paths, C1).
|
||||
set -euo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
AURA="${AURA:-cargo run -q --bin aura --}"
|
||||
|
||||
FROM=1725148800000 # Sept-2024 (sweep + generalize share this window)
|
||||
TO=1727740799999
|
||||
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
|
||||
echo "### Scenario 4 — sweep -> pick winner -> generalize (end-to-end)"
|
||||
echo
|
||||
|
||||
echo "--- (1) sweep stage1-r on GER40 ---"
|
||||
$AURA sweep --strategy stage1-r --real GER40 --from "$FROM" --to "$TO" \
|
||||
--name ft_compose >/dev/null
|
||||
echo "swept; family ft_compose-0 persisted"
|
||||
|
||||
echo "--- (2) rank the family by sqn, read the winner ---"
|
||||
# Capture the full ranked output then slice; piping the subcommand directly into
|
||||
# `head -1` makes `aura` panic with a Broken-pipe on stdout (recorded as a
|
||||
# finding) — capture first, the careful-consumer pattern.
|
||||
RANKED="$($AURA runs family ft_compose-0 rank sqn)"
|
||||
TOP="$(printf '%s\n' "$RANKED" | head -1)"
|
||||
echo "winner (verbatim manifest+metrics):"
|
||||
printf '%s\n' "$TOP"
|
||||
|
||||
# extract the winner's signal knobs (fast.length, slow.length, stop_length, stop_k)
|
||||
i64_of() { printf '%s' "$TOP" | grep -oE "\"$1\",\{\"I64\":[0-9]+\}" | sed -E 's/.*:([0-9]+)\}/\1/'; }
|
||||
f64_of() { printf '%s' "$TOP" | grep -oE "\"$1\",\{\"F64\":[0-9.]+\}" | sed -E 's/.*:([0-9.]+)\}/\1/'; }
|
||||
FAST="$(i64_of fast.length)"
|
||||
SLOW="$(i64_of slow.length)"
|
||||
SLEN="$(i64_of stop_length)"
|
||||
SK="$(f64_of stop_k)"
|
||||
WIN_SQN="$(printf '%s' "$TOP" | grep -oE '"sqn":[-0-9.eE]+' | head -1 | cut -d: -f2)"
|
||||
echo "winner params: fast=$FAST slow=$SLOW stop_length=$SLEN stop_k=$SK (in-sample sqn=$WIN_SQN)"
|
||||
[ -n "$FAST" ] && [ -n "$SLOW" ] && [ -n "$SLEN" ] && [ -n "$SK" ] \
|
||||
|| fail 'could not read the winning params from the ranked family'
|
||||
echo
|
||||
|
||||
echo "--- (3) generalize the winner across GER40,USDJPY,EURUSD by sqn ---"
|
||||
GEN_ALL="$($AURA generalize --strategy stage1-r --real GER40,USDJPY,EURUSD \
|
||||
--fast "$FAST" --slow "$SLOW" --stop-length "$SLEN" --stop-k "$SK" \
|
||||
--metric sqn --from "$FROM" --to "$TO" --name ft_compose_gen)"
|
||||
GEN="$(printf '%s\n' "$GEN_ALL" | head -1)"
|
||||
printf '%s\n' "$GEN"
|
||||
|
||||
WORST="$(printf '%s' "$GEN" | grep -oE '"worst_case":[-0-9.eE]+' | cut -d: -f2)"
|
||||
AGREE="$(printf '%s' "$GEN" | grep -oE '"sign_agreement":[0-9]+' | cut -d: -f2)"
|
||||
GER_SQN="$(printf '%s' "$GEN" | grep -oE '\["GER40",[-0-9.eE]+\]' | sed -E 's/.*,([-0-9.eE]+)\]/\1/')"
|
||||
echo
|
||||
echo "verdict inputs: worst_case=$WORST sign_agreement=$AGREE GER40-sqn=$GER_SQN"
|
||||
|
||||
# The GER40 leg of generalize must reproduce the in-sample sweep winner's sqn
|
||||
# (same window, same params, same engine -> bit-identical, C1).
|
||||
awk -v a="$WIN_SQN" -v b="$GER_SQN" 'BEGIN{ d=a-b; if(d<0)d=-d; exit !(d < 1e-9) }' \
|
||||
|| fail "generalize GER40 sqn ($GER_SQN) != sweep winner sqn ($WIN_SQN)"
|
||||
echo "GER40 leg reproduces the sweep winner's sqn (determinism across paths OK)"
|
||||
echo
|
||||
|
||||
# Render the human verdict from the floor + agreement.
|
||||
if awk -v w="$WORST" 'BEGIN{ exit !(w > 0) }'; then
|
||||
echo "VERDICT: edge holds across all instruments (worst_case > 0) -> defensible."
|
||||
else
|
||||
echo "VERDICT: edge does NOT hold on every instrument (worst_case <= 0,"
|
||||
echo " sign_agreement=$AGREE/3) -> instrument-specific / likely overfit."
|
||||
fi
|
||||
echo
|
||||
echo "PASS: the sweep -> rank -> generalize chain runs coherently and gives a"
|
||||
echo " defensible real-vs-overfit verdict; the GER40 leg is bit-reproducible."
|
||||
Reference in New Issue
Block a user