From cd18a750eb188de6d5416d40647532bb27659d9c Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 15 Jun 2026 13:55:08 +0200 Subject: [PATCH] Generalize inputs; add per-section external references --- Metric-Catalogue.md | 308 +++++++++++++++++++++++++++++--------------- 1 file changed, 201 insertions(+), 107 deletions(-) diff --git a/Metric-Catalogue.md b/Metric-Catalogue.md index 7958660..8939806 100644 --- a/Metric-Catalogue.md +++ b/Metric-Catalogue.md @@ -1,108 +1,107 @@ # Metric Catalogue -> Formula-level reference for aura's strategy-analysis layer. For each analysis -> tool and metric family: exact formula, what aura datum it is computed from, and -> the conventional benchmark threshold. Narrative and gap analysis on -> **[Home](Home)**; source-checking on **[Metric Verification Log](Metric-Verification-Log)**. +> Formula-level reference for trading-strategy analysis. For each analysis tool +> and metric family: exact formula, what input it is computed from, and the +> conventional benchmark threshold. Narrative on **[Home](Home)**; +> source-checking on **[Metric Verification Log](Metric-Verification-Log)**. -**Conventions.** **Per-period return** `r_t = pips[t] - pips[t-1]` = first -difference of the sim-optimal broker's cumulative pip-equity (the `equity` -argument to `summarize`). **Trade list** = derived from the C7 position-event -table. Pip returns are **additive**, so `total_pips = sum(r_t)` and CAGR-style -numerators use the arithmetic `mean(r)*P` form (no geometric compounding). `P` = -periods/year from the clock cadence (C12 window). Risk-free `r_f = 0` in -frictionless pip space. Markers: **[L]** law/exact, **[C]** convention, **[CORR]** -verification-corrected. +**Conventions.** **Per-period return** `r` (`r_t = E_t − E_{t-1}`) = first +difference of the cumulative **equity curve** `E`. **Trade list** = the set of +closed trades. When returns are expressed as additive P&L increments, +`total return = sum(r_t)` and CAGR-style numerators use `mean(r)·P`. `P` = +periods/year from the data cadence. The **risk-free rate** `r_f` is often taken as +`0` for frictionless P&L accounting. Markers: **[L]** law/exact, **[C]** +convention, **[CORR]** verification-corrected. --- -## A. Param-sweep (axis 1) — a metric *surface* +## A. Parameter sweep: the metric surface -Produces one metric vector per grid cell, laid on the param grid. Should emit per -cell the full single-run scorecard (§C/§D); across cells the surface geometry; and -the cell×time *performance matrix* M (one `r_t` series per cell) that PBO/DSR/CPCV -consume. +A sweep produces one metric vector per parameter cell, laid on the parameter grid. +Robustness is read from the *geometry* of that surface, not the single best cell. | metric | formula | computed from | threshold | |---|---|---|---| -| Plateau-to-peak ratio | `R = mean_{p∈Nbhd(p*)} f(p) / f(p*)` | smooth metric array over a k-step neighbourhood of argmax | `R ≳ 0.8–0.9` robust plateau; `< ~0.7` lucky spike **[C]** | +| Plateau-to-peak ratio | `R = mean_{p∈Nbhd(p*)} f(p) / f(p*)` | metric array over a k-step neighbourhood of the argmax | `R ≳ 0.8–0.9` robust plateau; `< ~0.7` lucky spike **[C]** | | Parameter sensitivity | `max(f over ±1-step) − min(...)` (or std) | perturb each axis one grid step around `p*` | small spread = robust; edge should survive ±10–20% param change **[C]** | | Surface smoothness | `mean|f(adjacent) − f|` (mean abs gradient) | adjacent-cell differences | lower = smoother; global-best `> ~20–30%` above neighbours = spike | -**Reading rule (load-bearing):** pick a param value from the *centre of the widest -plateau*, never the global peak. Pure post-processing over the existing -sweep-cell metric array — no engine change. +**Reading rule (load-bearing):** pick a parameter value from the *centre of the +widest plateau*, never the global peak. -## B. Optimization (axis 2) — the *selected* configuration +## B. Optimization: the selected configuration -Produces one winning param-set + `RunReport`. Should emit the winner's full -scorecard, the objective it won under, and (critically) the **deflated** -significance of the win (§J). **Objective choice (load-bearing): do not argmax raw -`total_pips`.** +Selection of the single best parameter set under an objective. The result should +carry the winner's full scorecard, the objective it won under, and the **deflated** +significance of the win (see §J). +**Objective choice (load-bearing): do not argmax raw net profit.** | metric (as objective) | formula | computed from | threshold | |---|---|---|---| -| Sharpe | `sqrt(P)·mean(r)/std(r)` | `r_t`, `r_f=0` | §C Sharpe | -| Calmar / CAR-over-MDD | `mean(r)·P / max_drawdown` | `r_t`, existing `max_drawdown` | §D Calmar | -| Ulcer Performance Index | `(R_p − R_f) / UlcerIndex` | `r_t` + drawdown series | more overfit-resistant than Calmar | +| Sharpe | `sqrt(P)·mean(r)/std(r)` | `r` | §C Sharpe | +| Calmar / CAR-over-MDD | `mean(r)·P / MaxDD` | `r`, `E` | §D Calmar | +| Ulcer Performance Index | `(R_p − R_f) / UlcerIndex` | `r`, `E` | more overfit-resistant than Calmar | | Profit factor | `gross_profit / gross_loss` | trade list | §E | **Caveat:** ratios with small denominators (low-variance / few-trade cells) explode → spurious winners; enforce a minimum trade count / minimum exposure. Optimizing any objective hard enough still overfits; optimization *shifts* the -problem to the multiple-testing layer (§J), it does not remove it. +problem to the multiple-testing layer (§J), +it does not remove it. -## C. Core performance & risk-adjusted ratios (single-run scorecard) +## C. Core performance & risk-adjusted ratios + +The single-run scorecard. All are pure functions of the per-period return series. | metric | formula | computed from | threshold | |---|---|---|---| -| Total return (pips) | `sum_t r_t` | `= total_pips` (emitted) | none in isolation | -| CAGR / annualized | pip: `mean(r)·P`; geometric: `(prod(1+r))^(P/T)−1` | `r_t`, `P` | asset-class relative | -| Volatility (ann.) | `std(r)·sqrt(P)`, sample std `1/(T−1)` | `r_t`, `P` | magnitude; sqrt(P) iid-only | -| **Sharpe** | `sqrt(P)·(mean(r)−r_f)/std(r−r_f)` | `r_t`, `r_f=0`, `P` | `<0` bad, `0–1` acceptable, `1–2` good, `2–3` very good, `>3` excellent **[C]**; sustained `>2` short-sample = overfit flag. **[CORR:** denominator = std of the *excess* return `std(r−r_f)`; equals `std(r)` only when `r_f` constant — exact here since `r_f=0`**]** | -| **Sortino** | `sqrt(P)·(mean(r)−MAR)/DD`, `DD=sqrt(Σ min(0,r−MAR)²/N)` | `r_t`, MAR (default 0), `P` | `<0` losing, `0–1` sub-optimal (*one* band), `1–2` good, `2–3` very good, `>3` excellent; "good" = `>1`. **[CORR:** no 0.5 split; "good" bar `>1` not `>2`; divisor total N**]** | -| Calmar | `mean(r)·P / |max_drawdown|`; strict = trailing-36mo monthly | `r_t`, `max_drawdown` | `<1` poor, `1–3` acceptable/good, `>3` excellent; HF ~1.5–3.0. **[CORR:** `>2` alone is NOT excellent (reserve for `>3`); `0.5–1.0` is "weak"**]** | -| MAR ratio | `CAGR(inception)/|max_drawdown(inception)|` | full curve | ~`>0.5` acc, `>1` good **[C]**; differs from Calmar in *window AND numerator* (avg-annual vs CAGR) **[CORR]** | -| Sterling | `CompoundROR / |avgAnnualMaxDD − 10%|` | annual DD episodes | no band; lower than Calmar. **[CORR:** subtract 10% *inside* the abs; 10% ≈ 1981 T-bill rate, not purely arbitrary**]** | +| Total return | `sum_t r_t` | `r` | none in isolation | +| CAGR / annualized | pip/additive: `mean(r)·P`; geometric: `(prod(1+r))^(P/T)−1` | `r`, `P` | asset-class relative | +| Volatility (ann.) | `std(r)·sqrt(P)`, sample std `1/(T−1)` | `r`, `P` | magnitude; sqrt(P) iid-only | +| **Sharpe** | `sqrt(P)·(mean(r)−r_f)/std(r−r_f)` | `r`, `r_f`, `P` | `<0` bad, `0–1` acceptable, `1–2` good, `2–3` very good, `>3` excellent **[C]**; sustained `>2` short-sample = overfit flag. **[CORR:** denominator = std of the *excess* return `std(r−r_f)`; equals `std(r)` only when `r_f` constant**]** | +| **Sortino** | `sqrt(P)·(mean(r)−MAR)/DD`, `DD=sqrt(Σ min(0,r−MAR)²/N)` | `r`, MAR (default 0), `P` | `<0` losing, `0–1` sub-optimal (*one* band), `1–2` good, `2–3` very good, `>3` excellent; "good" = `>1`. **[CORR:** no 0.5 split; "good" bar `>1` not `>2`; divisor total N**]** | +| Calmar | `mean(r)·P / |MaxDD|`; strict = trailing-36mo monthly | `r`, `E` | `<1` poor, `1–3` acceptable/good, `>3` excellent; HF ~1.5–3.0. **[CORR:** `>2` alone is NOT excellent (reserve for `>3`); `0.5–1.0` is "weak"**]** | +| MAR ratio | `CAGR(inception)/|MaxDD(inception)|` | full `E` | ~`>0.5` acc, `>1` good **[C]**; differs from Calmar in *window AND numerator* (avg-annual vs CAGR) **[CORR]** | +| Sterling | `CompoundROR / |avgAnnualMaxDD − 10%|` | annual DD episodes | no band; lower than Calmar. **[CORR:** subtract 10% *inside* the abs; 10% ≈ 1981 T-bill rate**]** | | Burke | `(r_P−r_F)/sqrt(Σ_t D_t²)`; modified divides `Σ D_t²` by n | DD-episode list | no band; higher better **[L]** | -| Omega(θ) | `Σ max(0,r−θ) / Σ max(0,θ−r)` | `r_t`, θ | `=1` breakeven (θ=mean), `>1` preferred **[L]** | -| Information ratio | `mean(r−b)/std(r−b)`, ann. ×`sqrt(P)` | `r_t`, benchmark `b_t` | `0.4–0.6` good, `0.61–1.0` very good, `≥1.0` exceptional **[formula L; thresholds C]** — N/A no bench | -| Treynor | `(E(R_p)−R_f)/beta`, `beta=Cov(R_p,R_m)/Var(R_m)` | `r_t`, `R_m` | no abs threshold; ranking only **[L]** — N/A no market | +| Omega(θ) | `Σ max(0,r−θ) / Σ max(0,θ−r)` | `r`, θ | `=1` breakeven (θ=mean), `>1` preferred **[L]** | +| Information ratio | `mean(r−b)/std(r−b)`, ann. ×`sqrt(P)` | `r`, benchmark `b` | `0.4–0.6` good, `0.61–1.0` very good, `≥1.0` exceptional **[formula L; thresholds C]** | +| Treynor | `(E(R_p)−R_f)/beta`, `beta=Cov(R_p,R_m)/Var(R_m)` | `r`, market `R_m` | no abs threshold; ranking only **[L]** | **Annualization rule [L as convention]:** mean ×`P`, std ×`sqrt(P)`, ratio ×`sqrt(P)`; `P=252` daily (trading days, not 365), 52 weekly, 12 monthly. Valid only under iid; Lo (2002) shows annualized Sharpe overstated up to ~65% under serial correlation. -## D. Drawdown & recovery family (equity-curve functionals — NO trade list) +## D. Drawdown & recovery (equity-curve functionals) -`E(t)` = cumulative pip curve, `HWM(t)` = running peak, `DD(t) = E(t)−HWM(t)`. +`E(t)` = cumulative equity, `HWM(t)` = running peak, `DD(t) = E(t)−HWM(t)`. No +trade list required. | metric | formula | computed from | threshold | |---|---|---|---| -| Max drawdown | `max_t(HWM−E)` (pip) or `/HWM` (%) | `= max_drawdown` (emitted) | stat-arb/MN `<10–15%`, HF `<15–20%`, L/S unacceptable `>25–30%`, `>40%` very high **[C; no universal cutoff]** | +| Max drawdown | `max_t(HWM−E)` (level) or `/HWM` (%) | `E` | stat-arb/MN `<10–15%`, HF `<15–20%`, L/S unacceptable `>25–30%`, `>40%` very high **[C; no universal cutoff]** | | Recovery asymmetry | DD fraction `d` needs gain `d/(1−d)` | a DD depth | exact: −20%→+25%, −50%→+100% **[L, algebraic]** | -| Average drawdown | `(1/n)Σ|DD|` (=Pain Index) OR episode-mean of trough depths | E(t), episode segmentation | comparative; `AvgDD/MaxDD≈1` → worst typical | -| Duration / time-to-recovery / time-underwater | per-episode index arithmetic; total UW = #(DD<0)/n | E(t) + cycle index | heuristic recovery ≈ `MDD/per-period-return`; flag censored open DD **[C]** | +| Average drawdown | `(1/n)Σ|DD|` (=Pain Index) OR episode-mean of trough depths | `E`, episode segmentation | comparative; `AvgDD/MaxDD≈1` → worst typical | +| Duration / time-to-recovery / time-underwater | per-episode index arithmetic; total UW = #(DD<0)/n | `E` + time index | heuristic recovery ≈ `MDD/per-period-return`; flag censored open DD **[C]** | | Longest drawdown | `max_k(t_recover − t_peak)` | episode segmentation | shorter better; often a *different* episode than max-depth | -| Ulcer Index | `sqrt((Σ_i D'_i²)/n)`, `D'_i = 100·(E_i−max_so_far)/max_so_far` | E(t) | `=0` no DD; `> ~10` severe; comparative **[C]** | -| Ulcer Perf Index / Martin | `(R_p−R_f)/UlcerIndex` | `r_t`+Ulcer | higher better; no band; more overfit-robust objective **[L]** | -| Pain Index | `(Σ_i |D'_i|)/n`, n = *total* obs | E(t) | comparative; `< MaxDD`. **[CORR:** it EMBEDS duration — a *remedy*, NOT a metric that "ignores time"; Bacon 2008 p.89; n=total obs else it collapses to Average Drawdown**]** | -| Pain Ratio | `(R_p−R_f)/PainIndex` | `r_t`+Pain | higher better; Bacon 2008 p.91 **[CORR page]** | -| Recovery Factor | `NetProfit/|MaxDrawdown|` | `= total_pips/max_drawdown` (both emitted!) | poor `<1`, good `2–5`, excellent `≥5` **[formula L; thresholds C]** | -| Expected MaxDD (Magdon-Ismail) | Brownian: linear (μ<0), `~sqrt(T)` (μ=0, exact `sqrt(π/2)·σ·sqrt(T)`), log (μ>0) | μ,σ from `r_t`; classify regime | not graded — a yardstick; observed ≫ E[MaxDD] = regime break / fat tails **[L asymptotics]** | +| Ulcer Index | `sqrt((Σ_i D'_i²)/n)`, `D'_i = 100·(E_i−max_so_far)/max_so_far` | `E` | `=0` no DD; `> ~10` severe; comparative **[C]** | +| Ulcer Perf Index / Martin | `(R_p−R_f)/UlcerIndex` | `r`, `E` | higher better; no band; more overfit-robust objective **[L]** | +| Pain Index | `(Σ_i |D'_i|)/n`, n = *total* obs | `E` | comparative; `< MaxDD`. **[CORR:** it EMBEDS duration — a *remedy*, NOT a metric that "ignores time"; Bacon 2008 p.89; n=total obs else it collapses to Average Drawdown**]** | +| Pain Ratio | `(R_p−R_f)/PainIndex` | `r`, `E` | higher better; Bacon 2008 p.91 **[CORR page]** | +| Recovery Factor | `NetProfit/|MaxDrawdown|` | final `E` / MaxDD | poor `<1`, good `2–5`, excellent `≥5` **[formula L; thresholds C]** | +| Expected MaxDD (Magdon-Ismail) | Brownian: linear (μ<0), `~sqrt(T)` (μ=0, exact `sqrt(π/2)·σ·sqrt(T)`), log (μ>0) | μ,σ from `r`; classify regime | not graded — a yardstick; observed ≫ E[MaxDD] = regime break / fat tails **[L asymptotics]** | -**Load-bearing axis interaction:** MaxDD is a downward-biased, sample-length- -dependent point estimate — the cure is axis-4 MC (report the *distribution* of -MaxDD, not one point) and axis-3 WFA (report OOS DD separately). Do NOT argmax +**Load-bearing point:** MaxDD is a downward-biased, sample-length-dependent point +estimate — the cure is Monte-Carlo (report the *distribution* of MaxDD, not one +point) and walk-forward (report OOS DD separately). Do NOT argmax Calmar/Recovery-Factor (single-point fragile); Ulcer/Pain-based ratios integrate the whole path and are more robust objectives. -## E. Trade-level & efficiency metrics (require the C7 position-event table) +## E. Trade-level & efficiency metrics -A "trade" = the span from exposure non-zero (or sign-flip) to it returning to zero -(or flipping); pip P&L = `Σ r_t` over the span; direction = sign of the held -exposure. +Require a **trade list**. A "trade" spans from a position opening (or sign-flip) to +its close; P&L = `Σ r_t` over the span; direction = sign of the held position. | metric | formula | computed from | threshold | |---|---|---|---| @@ -113,40 +112,41 @@ exposure. | **PF↔win-rate identity** | `PF=(p·R)/(1−p)`, inverse `p=PF/(PF+R)` | p, R | exact identity (holds iff R = ratio of *averages*) **[L]** | | Breakeven win rate | `1/(1+R)` | R | 50%@1:1, 33.3%@1:2, 25%@1:3, 16.7%@1:5 **[L]** | | SQN (Van Tharp) | `sqrt(N)·mean(R-mult)/std(R-mult)`, N often capped 100 | per-trade R-multiples | `<1.6` below, `1.6–2.0` avg, `2.0–2.5` good, `2.5–3.0` excellent, `>7` overfit **[C]** | -| MAE / MFE | max adverse / favorable excursion = `−min`/`max(running unreal. PnL)` | per-cycle running pip-since-entry | no cutoff; winners cluster at small MAE; large MFE-vs-realized gap → exiting early | +| MAE / MFE | max adverse / favorable excursion = `−min`/`max(running unreal. PnL)` | per-bar running P&L since entry | no cutoff; winners cluster at small MAE; large MFE-vs-realized gap → exiting early | | Trade efficiency | exit `ActualGain/MFE`; range `(Exit−Entry)/(High−Low)` | per-trade MFE/MAE/range | 0–100%; `<50%` exit eff poor **[C]** | -| Turnover | `Σ|Δexposure|` (or SEC `min(buys,sells)/avg_assets`) | exposure (`exposure_sign_flips` is the coarse proxy) | style-dependent; net-of-cost expectancy must stay `>0` | -| R-multiple / expectancy-in-R | per trade `PnL/1R`; mean `p·R−(1−p)` | trades + defined 1R | sign `>0`; ~`+0.5R` strong **[C]** — needs a stop; reserve for realistic broker / config reference-risk | +| Turnover | `Σ|Δposition|` (or SEC `min(buys,sells)/avg_assets`) | position series | style-dependent; net-of-cost expectancy must stay `>0` | +| R-multiple / expectancy-in-R | per trade `PnL/1R`; mean `p·R−(1−p)` | trades + defined 1R | sign `>0`; ~`+0.5R` strong **[C]** — needs a defined risk/stop | -## F. Tail, distribution & risk-of-ruin (from `r_t`; ruin from MC or trades) +## F. Tail, distribution & risk-of-ruin -Real returns are negatively skewed and leptokurtic, so iid-normal measures -*understate* tail risk. Prefer empirical / MC. +From `r` (ruin from Monte-Carlo or the trade list). Real returns are negatively +skewed and leptokurtic, so iid-normal measures *understate* tail risk. Prefer +empirical / Monte-Carlo estimates. | metric | formula | computed from | threshold | |---|---|---|---| -| Historical VaR_α | `−(1−α)` empirical quantile of r (5th @95%, 1st @99%) | sort `r_t`, index `floor((1−α)N)` | Basel traffic-light @250d/99%: GREEN 0–4, YELLOW 5–9, RED 10+ exceptions **[L]** | -| Parametric VaR | `−(μ+σ·Φ⁻¹(1−α))·W`; z = 1.645@95, 2.326@99, 1.282@90 | μ,σ of `r_t` | same grading; **iid-normal trap** **[L multipliers]** | -| Historical ES/CVaR | `mean of worst (1−α)N returns` | sorted `r_t`, `k=ceil((1−α)N)` | coherent (VaR is not); FRTB uses 97.5% ES ≈ 99% VaR under normality **[L]** | +| Historical VaR_α | `−(1−α)` empirical quantile of r (5th @95%, 1st @99%) | sort `r`, index `floor((1−α)N)` | Basel traffic-light @250d/99%: GREEN 0–4, YELLOW 5–9, RED 10+ exceptions **[L]** | +| Parametric VaR | `−(μ+σ·Φ⁻¹(1−α))·W`; z = 1.645@95, 2.326@99, 1.282@90 | μ,σ of `r` | same grading; **iid-normal trap** **[L multipliers]** | +| Historical ES/CVaR | `mean of worst (1−α)N returns` | sorted `r`, `k=ceil((1−α)N)` | coherent (VaR is not); FRTB uses 97.5% ES ≈ 99% VaR under normality **[L]** | | Parametric ES | `μ+σ·φ(Φ⁻¹(α))/(1−α)`; mult ~2.063@95, 2.665@99 | μ,σ | worse than VaR for fat tails **[L]** | -| Tail ratio | `q95(r)/|q05(r)|` | percentiles of `r_t` | `>1` favourable, `<1` dangerous **[L 95/5]** | -| Skewness | `(1/N)Σ(r−μ)³/σ³` | `r_t` | sign-driven; persistent negative = red flag even with high Sharpe | -| Excess kurtosis | `(1/N)Σ(r−μ)⁴/σ⁴ − 3` | `r_t` | `0`=normal; `>0` leptokurtic (typical, dangerous); daily 3–10+ **[L]** | -| Downside deviation | `sqrt((1/N)Σ[min(0,r−T)]²)`, divisor total N | `r_t`, T | feeds Sortino **[L, divisor=total N]** | +| Tail ratio | `q95(r)/|q05(r)|` | percentiles of `r` | `>1` favourable, `<1` dangerous **[L 95/5]** | +| Skewness | `(1/N)Σ(r−μ)³/σ³` | `r` | sign-driven; persistent negative = red flag even with high Sharpe | +| Excess kurtosis | `(1/N)Σ(r−μ)⁴/σ⁴ − 3` | `r` | `0`=normal; `>0` leptokurtic (typical, dangerous); daily 3–10+ **[L]** | +| Downside deviation | `sqrt((1/N)Σ[min(0,r−T)]²)`, divisor total N | `r`, T | feeds Sortino **[L, divisor=total N]** | | Risk of ruin (closed) | `((1−A)/(1+A))^U`, `A=p·R−q`, `U=capital/risk-per-trade` | trades p, R | `<1–5%`, desks `<1%`, pros `<0.1%`; `>5–10%` reject **[C]** | -| Risk of ruin (empirical) | `fraction of N MC paths breaching the ruin threshold` | axis-4 seeds | **preferred**; sidesteps iid-normal; default R=50% capital | -| Kelly / optimal-f | `f*=(b·p−q)/b`; `f*=(μ−r)/σ²` | `r_t`/odds | deploy `0 < f ≤ ~0.5·f*`; above full Kelly strictly dominated **[L]** — maps to exposure scale | -| Gain-to-Pain | `Σ(r)/|Σ(neg r)|` | `r_t` | (monthly) ~1 acc, ~2 outstanding, 3 excellent, ~4 world-class, `<1` poor **[C; state the basis]** | +| Risk of ruin (empirical) | `fraction of N Monte-Carlo paths breaching the ruin threshold` | multi-run dist. | **preferred**; sidesteps iid-normal; default R=50% capital | +| Kelly / optimal-f | `f*=(b·p−q)/b`; `f*=(μ−r)/σ²` | `r`/odds | deploy `0 < f ≤ ~0.5·f*`; above full Kelly strictly dominated **[L]** | +| Gain-to-Pain | `Σ(r)/|Σ(neg r)|` | `r` | (monthly) ~1 acc, ~2 outstanding, 3 excellent, ~4 world-class, `<1` poor **[C; state the basis]** | | Common-Sense ratio | `TailRatio · GainToPain` | the two above | `>1` robust, `<1` fragile **[L breakpoint]** | -## G. Walk-forward analysis (axis 3) +## G. Walk-forward analysis | metric | formula | computed from | threshold | |---|---|---|---| -| WFE (return) | `(OOS_pips/OOS_bars)/(IS_pips/IS_bars)` | per-fold IS-optimal/OOS `total_pips`, bar counts | `≥0.5` robust, `~1.0` ideal, `>0.7` excellent, `0.5–0.7` good, `0.3–0.5` mediocre, `<0.3` poor; `>1.0` is a *warning* (under-fit IS / lucky OOS), not superiority | -| WFE (Sharpe) | `OOS_Sharpe/IS_Sharpe` | per-fold Sharpe over `r_t` | `>0.5` robust, `~1.0` ideal, `<0.3` red flag | +| WFE (return) | `(OOS_profit/OOS_bars)/(IS_profit/IS_bars)` | per-fold IS-optimal/OOS profit, bar counts | `≥0.5` robust, `~1.0` ideal, `>0.7` excellent, `0.5–0.7` good, `0.3–0.5` mediocre, `<0.3` poor; `>1.0` is a *warning* (under-fit IS / lucky OOS), not superiority | +| WFE (Sharpe) | `OOS_Sharpe/IS_Sharpe` | per-fold Sharpe over `r` | `>0.5` robust, `~1.0` ideal, `<0.3` red flag | | IS-OOS gap | `IS_Sharpe − OOS_Sharpe` per fold, averaged | per-fold Sharpes | `>0.5`/cycle = overfitting; healthy mean `<0.3` | -| Stitched OOS curve | chain each fold's OOS segment, re-based | family of OOS runs (C18 sink) | net profitable; max DD `<40%` (TradeStation) | +| Stitched OOS curve | chain each fold's OOS segment, re-based | family of OOS runs | net profitable; max DD `<40%` (TradeStation) | | Parameter stability | dispersion of argmax params across folds; plateau check | per-fold winners | qualitative — broad plateau, clustered | | Fraction profitable OOS windows | `#(OOS>0)/#windows` | per-window OOS P&L | majority `>50%`; worst-window OOS Sharpe `>0` | | Pardo verdict | net-profitable AND WFE `≥50%` AND maxDD `<40%`; filter %Prof `≥40`, PF `≥1.5` | the above | composite pass/fail **[software-default convention]** | @@ -156,48 +156,51 @@ clean stitch; trade floors IS `≥100`, OOS `≥30`. Rolling vs anchored. Advanc sweep of WFA configs (a "family of families") proves the verdict is not a lucky schedule. -## H. Monte-Carlo (axis 4) +## H. Monte-Carlo | metric | formula | computed from | threshold | |---|---|---|---| -| Trade-order shuffle | permute `r_t`, rebuild equity | seeded reorder | additive sum INVARIANT — only path/DD stats vary **[load-bearing for pip wiring]** | -| Bootstrap (with replacement) | resample T from `r_t` | seeded resample | changes total → genuine `total_pips` distribution; CI=[2.5, 97.5] pct | +| Trade-order shuffle | permute `r`, rebuild equity | seeded reorder | additive sum INVARIANT — only path/DD stats vary | +| Bootstrap (with replacement) | resample T from `r` | seeded resample | changes total → genuine return distribution; CI=[2.5, 97.5] pct | | Equity percentile fan | per-horizon {5,25,50,75,95} pct | N curves | descriptive; viability: 5th-pct ending equity `>0` | -| Stress (95th-pct) MaxDD | 95th pct of N MaxDD | per-seed MaxDD | `<25%` good, `>35%` red flag; size capital off this **[C]** | -| Empirical VaR/ES | 5th-pct (1st @99) final P&L; ES = mean beyond | per-seed terminal | 5th-pct ending equity `>0` | -| Probability of ruin | `ruin_count/N` (MaxDD>R, default 50%) | per-seed MaxDD | `<1%` excellent, `1–5%` acceptable, `>5%` not recommended **[C]** | -| Prob. of a losing period | `#(period<0)/N` | per-seed period returns | `>~30–40%` hard to hold | +| Stress (95th-pct) MaxDD | 95th pct of N MaxDD | per-run MaxDD | `<25%` good, `>35%` red flag; size capital off this **[C]** | +| Empirical VaR/ES | 5th-pct (1st @99) final P&L; ES = mean beyond | per-run terminal | 5th-pct ending equity `>0` | +| Probability of ruin | `ruin_count/N` (MaxDD>R, default 50%) | per-run MaxDD | `<1%` excellent, `1–5%` acceptable, `>5%` not recommended **[C]** | +| Prob. of a losing period | `#(period<0)/N` | per-run period returns | `>~30–40%` hard to hold | | **MCPT p-value** | `(z+1)/(N+1)`, z=#perm `≥` real | permute bar-to-bar log returns, keep first price, re-run | `<0.05` min, `<0.01` strict, `>0.10` red flag, `~0.5` discard **[L: +1 in numerator and denominator]** | -| White RC / Hansen SPA | bootstrap max-stat over L rules, recentered | per-rule rel-perf + benchmark | `p<0.05`; SPA more powerful than conservative RC | +| White RC / Hansen SPA | bootstrap max-stat over L rules, recentered | per-rule rel-perf + benchmark | `p<0.05`; SPA more powerful than the conservative RC | **Conventions:** N = 1,000 min, 5,000 recommended, 10,000+ for tail stability. -Plain shuffle assumes serial independence — if `r_t` is autocorrelated use the -**block/stationary bootstrap** (Politis-Romano). MCPT permutes *log returns* -(preserving the marginal, destroying ordering) → tests specifically time-ordering -edges. +Plain shuffle assumes serial independence — if `r` is autocorrelated use the +**block/stationary bootstrap** (Politis-Romano). The permutation test permutes +*log returns* (preserving the marginal, destroying ordering) → tests specifically +time-ordering edges. ## I. Stress/scenario, regime, cost/capacity, benchmark-relative, sanity, live-vs-backtest | tool | produces | key metric + formula | computed from | threshold | |---|---|---|---|---| -| Stress/scenario | conditional worst-case | per-scenario MaxDD, total_pips, recover-time | frozen topology over crisis windows / shocked synthetic source (C6) | fail if stress DD > risk limit; excellent if net-positive through crises **[C; Fed CCAR]** | -| Regime | per-state metrics + fragility verdict | total_pips/Sharpe/DD masked by regime label | regime-label stream (vol/trend classifier, recorded HMM C6) joined at C3 | edge only in one regime + negative elsewhere = fragile **[qualitative]** | -| Cost/slippage | net-vs-cost curve + break-even | `c* where net=0 ≈ gross_profit/n_trades`; `d(net Sharpe)/d(cost)` | realistic broker (C7) over position-event table, cost swept | reject if edge survives only at zero cost; break-even ≳2× real cost **[C]** | -| Capacity/impact | AUM ceiling + turnover | `impact ≈ Y·σ·sqrt(Q/V)`, Y~O(1); capacity = AUM where impact = gross alpha | position volumes + ADV (C4) | capacity ≫ target = scalable; per-day order `< ~5–10%` ADV **[C]** | -| Benchmark-relative | alpha/beta/IR + significance | `beta=Cov/Var`; Jensen alpha = regression intercept; `IR=alpha/TE` | `r_t` regressed on ingested `R_m` (C3) | alpha positive + significant (`|t|>2`); IR 0.4/0.6/1.0 **[beta/CAPM L; IR C]** — N/A no bench | -| Null/sanity | "beat random?" | MCPT `p=(#perm≥real)/N`; random-entry: real in right tail | permuted source + null random-exposure node | `p<0.05` min, `≤0.01` strict, `>0.10` red flag, N`≥1000` **[L]** | -| Live-vs-backtest | overfitting/decay verdict | haircut `=(BT_Sharpe−live_Sharpe)/BT_Sharpe`; reconciliation diff≈0 | C8 live series vs frozen re-run | `<30%` acceptable, `30–50%` typical, `>50–70%` largely overfit; drift≠0 = a bug **[C; verify primary source]** | +| Stress/scenario | conditional worst-case | per-scenario MaxDD, profit, recover-time | strategy over crisis windows / shocked synthetic data | fail if stress DD > risk limit; excellent if net-positive through crises **[C; Fed CCAR]** | +| Regime | per-state metrics + fragility verdict | profit/Sharpe/DD masked by regime label | a regime-label series (vol/trend classifier or HMM) | edge only in one regime + negative elsewhere = fragile **[qualitative]** | +| Cost/slippage | net-vs-cost curve + break-even | `c* where net=0 ≈ gross_profit/n_trades`; `d(net Sharpe)/d(cost)` | a cost-modelling backtest, cost swept | reject if edge survives only at zero cost; break-even ≳2× real cost **[C]** | +| Capacity/impact | AUM ceiling + turnover | `impact ≈ Y·σ·sqrt(Q/V)`, Y~O(1); capacity = AUM where impact = gross alpha | order volumes + ADV | capacity ≫ target = scalable; per-day order `< ~5–10%` ADV **[C]** | +| Benchmark-relative | alpha/beta/IR + significance | `beta=Cov/Var`; Jensen alpha = regression intercept; `IR=alpha/TE` | `r` regressed on a market series `R_m` | alpha positive + significant (`|t|>2`); IR 0.4/0.6/1.0 **[beta/CAPM L; IR C]** | +| Null/sanity | "beat random?" | MCPT `p=(#perm≥real)/N`; random-entry: real in right tail | permuted input + a random-entry null strategy | `p<0.05` min, `≤0.01` strict, `>0.10` red flag, N`≥1000` **[L]** | +| Live-vs-backtest | overfitting/decay verdict | haircut `=(BT_Sharpe−live_Sharpe)/BT_Sharpe` | live series vs backtest prediction | `<30%` acceptable, `30–50%` typical, `>50–70%` largely overfit **[C; verify primary source]** | -## J. Overfitting / multiple-testing + CPCV (the significance layer over axes 1–4) +## J. Overfitting, multiple-testing & CPCV + +The significance layer over the analysis axes. Treats "I ran many variants and +kept the best" as the statistical problem it is. | metric | formula | computed from | threshold | |---|---|---|---| -| PSR(SR*) | `Φ[(SR_hat−SR*)·sqrt(T−1)/sqrt(1−g3·SR_hat+((g4−1)/4)·SR_hat²)]` | per-period `SR_hat`, T, g3=skew, g4=kurtosis (full, normal=3) of `r_t` | `>0.95` significant **[L]** | +| PSR(SR*) | `Φ[(SR_hat−SR*)·sqrt(T−1)/sqrt(1−g3·SR_hat+((g4−1)/4)·SR_hat²)]` | per-period `SR_hat`, T, g3=skew, g4=kurtosis (full, normal=3) of `r` | `>0.95` significant **[L]** | | MinTRL | `1+(1−g3·SR_hat+((g4−1)/4)·SR_hat²)·(Z_α/(SR_hat−SR*))²` | as PSR + target α | need T `≥` MinTRL **[L]** | | E[max SR_N] | `sqrt(V)·((1−γ)Φ⁻¹(1−1/N)+γΦ⁻¹(1−1/(Ne)))`, γ≈0.5772 | cross-trial Sharpe variance V, effective N | the moving hurdle; grows with N **[L]** | -| **DSR** | PSR at `SR*=SR_0=E[max SR_N]` | winner SR_hat/T/g3/g4 + V across cells + effective N | `>0.95` survives deflation **[L; ex: N=100,V=½,T=1250,g3=−3,g4=10 → DSR~0.90<0.95 REJECT]** | +| **DSR** | PSR at `SR*=SR_0=E[max SR_N]` | winner SR_hat/T/g3/g4 + V across trials + effective N | `>0.95` survives deflation **[L; ex: N=100,V=½,T=1250,g3=−3,g4=10 → DSR~0.90<0.95 REJECT]** | | MinBTL | `≤ 2·ln(N)/E[max]²` years | N, target SR | 5y + `>~45` independent trials → guaranteed overfit **[C/L]** | -| **PBO (CSCV)** | matrix `(T×M)`; S even blocks; `C(S,S/2)` combos; `ω=(rank−0.5)/M`; `λ=ln(ω/(1−ω))`; `PBO=P[λ≤0]` | per-cell `r_t` series | `<0.5` sanity, `~0.5` alarm, `<0.1` excellent **[L]** | +| **PBO (CSCV)** | matrix `(T×M)`; S even blocks; `C(S,S/2)` combos; `ω=(rank−0.5)/M`; `λ=ln(ω/(1−ω))`; `PBO=P[λ≤0]` | per-trial `r` series | `<0.5` sanity, `~0.5` alarm, `<0.1` excellent **[L]** | | Harvey-Liu-Zhu haircut | `t=SR·sqrt(T)`; adjust p (Bonferroni `min(M·p,1)`/Holm/BHY); `HSR=SR·(adj_t/orig_t)` | SR, T, M | required new-factor t `~3.0` (not 2.0) | | CPCV paths | splits `=C(N,k)`; paths `φ=(k/N)C(N,k)` | grouped purged+embargoed windows | N=5–10 groups, k=2; purge: drop train obs whose label span overlaps test; embargo h~1% of T (forward only) **[L]** | @@ -205,8 +208,8 @@ edges. ## Master catalogue (compact index) -Inputs: **r** = `diff(equity)`; **E** = cumulative pip-equity; **trades** = C7 -table; **bench** = ingested benchmark; **multi** = multi-run distribution. +Inputs: **r** = per-period returns; **E** = equity curve; **trades** = closed +trades; **b** = benchmark; **multi** = multi-run distribution. **Return** — Total `sum r` [C] · CAGR `mean(r)·P` [C] · Volatility `std(r)·sqrt(P)` [C]. @@ -216,7 +219,7 @@ table; **bench** = ingested benchmark; **multi** = multi-run distribution. `CompoundROR/|avgAnnMaxDD−10%|` [CORR] · Burke `(r_P−r_F)/sqrt(ΣD²)` [L] · Omega `Σmax(0,r−θ)/Σmax(0,θ−r)` [L] · Info-Ratio `mean(r−b)/std(r−b)·sqrt(P)` [L/C] · UPI `(R_p−R_f)/UlcerIndex` [L] · Pain-Ratio `(R_p−R_f)/PainIndex` [CORR page] · -Recovery-Factor `total_pips/|MaxDD|` [L/C] · Gain-to-Pain `Σr/|Σ neg r|` [C] · +Recovery-Factor `NetProfit/|MaxDD|` [L/C] · Gain-to-Pain `Σr/|Σ neg r|` [C] · Common-Sense `TailRatio·GainToPain` [L] · SQN `sqrt(N)·mean(R)/std(R)` [C]. **Drawdown** — MaxDD `max(HWM−E)` [C] · Recovery-asymmetry `d/(1−d)` [L] · AvgDD @@ -227,7 +230,7 @@ duration] · Expected-MaxDD μ=0 `sqrt(π/2)·σ·sqrt(T)` [L]. `<1`/`1.5–2`/`>3 overfit` [CORR swing>day] · Expectancy `p·AvgWin−(1−p)·AvgLoss`, sign>0 [L] · Payoff `AvgWin/AvgLoss` [L] · PF-identity `PF=(p·R)/(1−p)` [L] · Breakeven-win `1/(1+R)` [L] · MAE/MFE [C] · Trade-eff `ActualGain/MFE` [C] · Turnover -`Σ|Δexposure|` [C]. +`Σ|Δposition|` [C]. **Tail/distribution** — Hist-VaR `−q_{1−α}(r)`, Basel traffic-light [L] · Param-VaR `−(μ+σ·z)`, z 1.645/2.326 [L] · Hist-ES `mean(worst (1−α)N)` [L] · Tail-Ratio @@ -239,3 +242,94 @@ empirical, `<1%` pro [C] · Kelly `(μ−r)/σ²`, `≤0.5·f*` [L]. · MinTRL [L] · DSR = PSR at `SR*=E[maxSR_N]`, `>0.95` [L] · MinBTL `≤2ln(N)/E[max]²` [C/L] · PBO `P[ln(ω/(1−ω))≤0]`, `<0.5` sanity / `<0.1` excellent [L] · HLZ-haircut `HSR=SR·(adj_t/orig_t)`, t`~3.0` [C/L] · CPCV `φ=(k/N)C(N,k)`, embargo h~1% [L]. + +--- + +## References + +External, source-checked references per section (each link verified reachable and +on-topic). Where a formula or threshold was corrected, the citation is in the +[Metric Verification Log](Metric-Verification-Log). + +**A / B — Parameter sweep, optimization & robustness:** +[Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis) · +[Hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) · +[Pseudo-mathematics and financial charlatanism — Bailey, Borwein, López de Prado, Zhu](https://scholarworks.wmich.edu/math_pubs/42/) · +[Deflated Sharpe Ratio (objective choice)](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) + +**C — Core & risk-adjusted ratios:** +[Sharpe](https://en.wikipedia.org/wiki/Sharpe_ratio) · +[Sortino](https://en.wikipedia.org/wiki/Sortino_ratio) · +[Calmar & MAR](https://en.wikipedia.org/wiki/Calmar_ratio) · +[Sterling (PerformanceAnalytics)](https://rdrr.io/rforge/PerformanceAnalytics/man/CalmarRatio.html) · +[Burke (PerformanceAnalytics)](https://www.rdocumentation.org/packages/PerformanceAnalytics/versions/2.0.4/topics/BurkeRatio) · +[Omega](https://en.wikipedia.org/wiki/Omega_ratio) · +[Information ratio](https://en.wikipedia.org/wiki/Information_ratio) · +[Treynor](https://en.wikipedia.org/wiki/Treynor_ratio) · +[Volatility / sqrt-of-time](https://en.wikipedia.org/wiki/Volatility_(finance)) · +[Lo (2002), The Statistics of Sharpe Ratios — CFA Institute](https://rpc.cfainstitute.org/research/financial-analysts-journal/2002/the-statistics-of-sharpe-ratios) + +**D — Drawdown & recovery:** +[Drawdown (max / average / duration)](https://en.wikipedia.org/wiki/Drawdown_(economics)) · +[Ulcer Index & UPI](https://en.wikipedia.org/wiki/Ulcer_index) · +[Pain Index](https://rdrr.io/cran/PerformanceAnalytics/man/PainIndex.html) · +[Pain Ratio](https://rdrr.io/cran/PerformanceAnalytics/man/PainRatio.html) · +[Average drawdown](https://rdrr.io/cran/PerformanceAnalytics/man/AverageDrawdown.html) · +[Recovery Factor (Fidelity glossary)](https://www.fidelity.com/research/backtesting/glossary.html) · +[Expected MaxDD — Magdon-Ismail et al.](https://www.cambridge.org/core/journals/journal-of-applied-probability/article/abs/on-the-maximum-drawdown-of-a-brownian-motion/F9E3B8A454B020DDEBF0AC3390EF7807) + +**E — Trade-level & efficiency:** +[Win rate vs profit factor](https://www.backtestbase.com/education/win-rate-vs-profit-factor) · +[Expectancy](https://www.tradesviz.com/glossary/expectancy/) · +[Payoff & breakeven](https://learningcenter.fxstreet.com/education/learning-center/unit-3/chapter-2/revenue-statistics/index.html) · +[PF↔win-rate identity](https://www.pnlledger.com/profit-factor-vs-win-rate-vs-payoff-ratio/) · +[SQN (Van Tharp)](https://indextrader.com.au/van-tharps-sqn/) · +[MAE](https://www.tradesviz.com/glossary/maximum-adverse-excursion/) · +[MFE](https://www.tradesviz.com/glossary/maximum-favorable-excursion/) · +[Trade efficiency (TradeStation)](https://help.tradestation.com/10_00/eng/tradestationhelp/subsystems/spr_topics/report/entry_efficiency_exit_efficiency__strategy_performance_report_.htm) · +[Turnover (CFI)](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/portfolio-turnover-ratio/) + +**F — Tail, distribution & risk-of-ruin:** +[Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) · +[Expected Shortfall / CVaR](https://en.wikipedia.org/wiki/Expected_shortfall) · +[Skewness](https://en.wikipedia.org/wiki/Skewness) · +[Kurtosis](https://en.wikipedia.org/wiki/Kurtosis) · +[Downside deviation](https://rdrr.io/cran/PerformanceAnalytics/man/DownsideDeviation.html) · +[Risk of ruin](https://en.wikipedia.org/wiki/Risk_of_ruin) · +[Kelly criterion](https://en.wikipedia.org/wiki/Kelly_criterion) · +[Gain-to-pain](https://www.wallstreetmojo.com/gain-to-pain-ratio/) · +[Basel backtesting / traffic-light (BIS)](https://www.bis.org/publ/bcbs22.htm) + +**G — Walk-forward analysis:** +[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) · +[WFE / parameter stability / criteria (TradeStation)](https://help.tradestation.com/09_01/tswfo/topics/about_wfo.htm) · +[Anchored vs rolling windows](https://www.susanpotter.net/quant/walk-forward-optimization/) + +**H — Monte-Carlo & permutation tests:** +[Bootstrapping (incl. block/stationary)](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) · +[Permutation test](https://en.wikipedia.org/wiki/Permutation_test) · +[Monte-Carlo Permutation Test (BuildAlpha)](https://www.buildalpha.com/monte-carlo-permutation/) · +[White's Reality Check (1999)](https://ideas.repec.org/a/bla/jfinan/v54y1999i5p1647-1691.html) · +[Hansen's SPA test (2005)](https://ideas.repec.org/a/bes/jnlbes/v23y2005p365-380.html) + +**I — Stress, regime, cost/capacity, benchmark-relative, sanity:** +[Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) · +[Regime / Markov switching](https://en.wikipedia.org/wiki/Markov_switching_multifractal) · +[Slippage](https://en.wikipedia.org/wiki/Slippage_(finance)) · +[Transaction-cost backtesting (QuantStart)](https://www.quantstart.com/articles/Successful-Backtesting-of-Algorithmic-Trading-Strategies-Part-II/) · +[Square-root market-impact law (arXiv)](https://arxiv.org/abs/1602.03043) · +[CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model) · +[Jensen's alpha](https://en.wikipedia.org/wiki/Jensen's_alpha) · +[Live-vs-backtest haircut — Bailey & López de Prado, Deflated Sharpe (PDF)](https://www.davidhbailey.com/dhbpapers/deflated-sharpe.pdf) + +**J — Overfitting, multiple-testing & CPCV:** +[Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) · +[Probabilistic Sharpe Ratio (PerformanceAnalytics)](https://rdrr.io/cran/PerformanceAnalytics/man/ProbSharpeRatio.html) · +[Minimum Track Record Length](https://rdrr.io/cran/PerformanceAnalytics/man/MinTrackRecord.html) · +[PBO / MinBTL via CSCV (pbo vignette)](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) · +[Harvey-Liu-Zhu haircut (quantstrat)](https://rdrr.io/github/braverock/quantstrat/man/SharpeRatio.haircut.html) · +[Bonferroni](https://en.wikipedia.org/wiki/Bonferroni_correction) · +[Holm-Bonferroni](https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method) · +[Benjamini-Hochberg / FDR](https://en.wikipedia.org/wiki/False_discovery_rate) · +[Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) · +[López de Prado, Advances in Financial ML, ch. 7](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml)