Brummel/Aura
1
0
Fork 0
Code Issues 33 Pull Requests Actions Packages Projects Releases Wiki Activity

Generalize inputs; add per-section external references

Brummel
2026-06-15 13:55:08 +02:00
parent 9cf1f13547
commit cd18a750eb
1 changed files with 201 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Metric-Catalogue.md
+201 -107
@@ -1,108 +1,107 @@
# Metric Catalogue
> Formula-level reference for aura's strategy-analysis layer. For each analysis
> tool and metric family: exact formula, what aura datum it is computed from, and
> the conventional benchmark threshold. Narrative and gap analysis on
> **[Home](Home)**; source-checking on **[Metric Verification Log](Metric-Verification-Log)**.
> Formula-level reference for trading-strategy analysis. For each analysis tool
> and metric family: exact formula, what input it is computed from, and the
> conventional benchmark threshold. Narrative on **[Home](Home)**;
> source-checking on **[Metric Verification Log](Metric-Verification-Log)**.
**Conventions.** **Per-period return** `r_t = pips[t] - pips[t-1]` = first
difference of the sim-optimal broker's cumulative pip-equity (the `equity`
argument to `summarize`). **Trade list** = derived from the C7 position-event
table. Pip returns are **additive**, so `total_pips = sum(r_t)` and CAGR-style
numerators use the arithmetic `mean(r)*P` form (no geometric compounding). `P` =
periods/year from the clock cadence (C12 window). Risk-free `r_f = 0` in
frictionless pip space. Markers: **[L]** law/exact, **[C]** convention, **[CORR]**
verification-corrected.
**Conventions.** **Per-period return** `r` (`r_t = E_t − E_{t-1}`) = first
difference of the cumulative **equity curve** `E`. **Trade list** = the set of
closed trades. When returns are expressed as additive P&L increments,
`total return = sum(r_t)` and CAGR-style numerators use `mean(r)·P`. `P` =
periods/year from the data cadence. The **risk-free rate** `r_f` is often taken as
`0` for frictionless P&L accounting. Markers: **[L]** law/exact, **[C]**
convention, **[CORR]** verification-corrected.
---
## A. Param-sweep (axis 1) — a metric *surface*
## A. Parameter sweep: the metric surface
Produces one metric vector per grid cell, laid on the param grid. Should emit per
cell the full single-run scorecard (§C/§D); across cells the surface geometry; and
the cell×time *performance matrix* M (one `r_t` series per cell) that PBO/DSR/CPCV
consume.
A sweep produces one metric vector per parameter cell, laid on the parameter grid.
Robustness is read from the *geometry* of that surface, not the single best cell.
| metric | formula | computed from | threshold |
|---|---|---|---|
| Plateau-to-peak ratio | `R = mean_{p∈Nbhd(p*)} f(p) / f(p*)` | smooth metric array over a k-step neighbourhood of argmax | `R ≳ 0.8–0.9` robust plateau; `< ~0.7` lucky spike **[C]** |
| Plateau-to-peak ratio | `R = mean_{p∈Nbhd(p*)} f(p) / f(p*)` | metric array over a k-step neighbourhood of the argmax | `R ≳ 0.8–0.9` robust plateau; `< ~0.7` lucky spike **[C]** |
| Parameter sensitivity | `max(f over ±1-step) − min(...)` (or std) | perturb each axis one grid step around `p*` | small spread = robust; edge should survive ±10–20% param change **[C]** |
| Surface smoothness | `mean|f(adjacent) − f|` (mean abs gradient) | adjacent-cell differences | lower = smoother; global-best `> ~20–30%` above neighbours = spike |
**Reading rule (load-bearing):** pick a param value from the *centre of the widest
plateau*, never the global peak. Pure post-processing over the existing
sweep-cell metric array — no engine change.
**Reading rule (load-bearing):** pick a parameter value from the *centre of the
widest plateau*, never the global peak.
## B. Optimization (axis 2) — the *selected* configuration
## B. Optimization: the selected configuration
Produces one winning param-set + `RunReport`. Should emit the winner's full
scorecard, the objective it won under, and (critically) the **deflated**
significance of the win (§J). **Objective choice (load-bearing): do not argmax raw
`total_pips`.**
Selection of the single best parameter set under an objective. The result should
carry the winner's full scorecard, the objective it won under, and the **deflated**
significance of the win (see §J).
**Objective choice (load-bearing): do not argmax raw net profit.**
| metric (as objective) | formula | computed from | threshold |
|---|---|---|---|
| Sharpe | `sqrt(P)·mean(r)/std(r)` | `r_t`, `r_f=0` | §C Sharpe |
| Calmar / CAR-over-MDD | `mean(r)·P / max_drawdown` | `r_t`, existing `max_drawdown` | §D Calmar |
| Ulcer Performance Index | `(R_p − R_f) / UlcerIndex` | `r_t` + drawdown series | more overfit-resistant than Calmar |
| Sharpe | `sqrt(P)·mean(r)/std(r)` | `r` | §C Sharpe |
| Calmar / CAR-over-MDD | `mean(r)·P / MaxDD` | `r`, `E` | §D Calmar |
| Ulcer Performance Index | `(R_p − R_f) / UlcerIndex` | `r`, `E` | more overfit-resistant than Calmar |
| Profit factor | `gross_profit / gross_loss` | trade list | §E |
**Caveat:** ratios with small denominators (low-variance / few-trade cells)
explode → spurious winners; enforce a minimum trade count / minimum exposure.
Optimizing any objective hard enough still overfits; optimization *shifts* the
problem to the multiple-testing layer (§J), it does not remove it.
problem to the multiple-testing layer (§J),
it does not remove it.
## C. Core performance & risk-adjusted ratios (single-run scorecard)
## C. Core performance & risk-adjusted ratios
The single-run scorecard. All are pure functions of the per-period return series.
| metric | formula | computed from | threshold |
|---|---|---|---|
| Total return (pips) | `sum_t r_t` | `= total_pips` (emitted) | none in isolation |
| CAGR / annualized | pip: `mean(r)·P`; geometric: `(prod(1+r))^(P/T)−1` | `r_t`, `P` | asset-class relative |
| Volatility (ann.) | `std(r)·sqrt(P)`, sample std `1/(T−1)` | `r_t`, `P` | magnitude; sqrt(P) iid-only |
| **Sharpe** | `sqrt(P)·(mean(r)−r_f)/std(r−r_f)` | `r_t`, `r_f=0`, `P` | `<0` bad, `0–1` acceptable, `1–2` good, `2–3` very good, `>3` excellent **[C]**; sustained `>2` short-sample = overfit flag. **[CORR:** denominator = std of the *excess* return `std(r−r_f)`; equals `std(r)` only when `r_f` constant — exact here since `r_f=0`**]** |
| **Sortino** | `sqrt(P)·(mean(r)−MAR)/DD`, `DD=sqrt(Σ min(0,r−MAR)²/N)` | `r_t`, MAR (default 0), `P` | `<0` losing, `0–1` sub-optimal (*one* band), `1–2` good, `2–3` very good, `>3` excellent; "good" = `>1`. **[CORR:** no 0.5 split; "good" bar `>1` not `>2`; divisor total N**]** |
| Calmar | `mean(r)·P / |max_drawdown|`; strict = trailing-36mo monthly | `r_t`, `max_drawdown` | `<1` poor, `1–3` acceptable/good, `>3` excellent; HF ~1.5–3.0. **[CORR:** `>2` alone is NOT excellent (reserve for `>3`); `0.5–1.0` is "weak"**]** |
| MAR ratio | `CAGR(inception)/|max_drawdown(inception)|` | full curve | ~`>0.5` acc, `>1` good **[C]**; differs from Calmar in *window AND numerator* (avg-annual vs CAGR) **[CORR]** |
| Sterling | `CompoundROR / |avgAnnualMaxDD − 10%|` | annual DD episodes | no band; lower than Calmar. **[CORR:** subtract 10% *inside* the abs; 10% ≈ 1981 T-bill rate, not purely arbitrary**]** |
| Total return | `sum_t r_t` | `r` | none in isolation |
| CAGR / annualized | pip/additive: `mean(r)·P`; geometric: `(prod(1+r))^(P/T)−1` | `r`, `P` | asset-class relative |
| Volatility (ann.) | `std(r)·sqrt(P)`, sample std `1/(T−1)` | `r`, `P` | magnitude; sqrt(P) iid-only |
| **Sharpe** | `sqrt(P)·(mean(r)−r_f)/std(r−r_f)` | `r`, `r_f`, `P` | `<0` bad, `0–1` acceptable, `1–2` good, `2–3` very good, `>3` excellent **[C]**; sustained `>2` short-sample = overfit flag. **[CORR:** denominator = std of the *excess* return `std(r−r_f)`; equals `std(r)` only when `r_f` constant**]** |
| **Sortino** | `sqrt(P)·(mean(r)−MAR)/DD`, `DD=sqrt(Σ min(0,r−MAR)²/N)` | `r`, MAR (default 0), `P` | `<0` losing, `0–1` sub-optimal (*one* band), `1–2` good, `2–3` very good, `>3` excellent; "good" = `>1`. **[CORR:** no 0.5 split; "good" bar `>1` not `>2`; divisor total N**]** |
| Calmar | `mean(r)·P / |MaxDD|`; strict = trailing-36mo monthly | `r`, `E` | `<1` poor, `1–3` acceptable/good, `>3` excellent; HF ~1.5–3.0. **[CORR:** `>2` alone is NOT excellent (reserve for `>3`); `0.5–1.0` is "weak"**]** |
| MAR ratio | `CAGR(inception)/|MaxDD(inception)|` | full `E` | ~`>0.5` acc, `>1` good **[C]**; differs from Calmar in *window AND numerator* (avg-annual vs CAGR) **[CORR]** |
| Sterling | `CompoundROR / |avgAnnualMaxDD − 10%|` | annual DD episodes | no band; lower than Calmar. **[CORR:** subtract 10% *inside* the abs; 10% ≈ 1981 T-bill rate**]** |
| Burke | `(r_P−r_F)/sqrt(Σ_t D_t²)`; modified divides `Σ D_t²` by n | DD-episode list | no band; higher better **[L]** |
| Omega(θ) | `Σ max(0,r−θ) / Σ max(0,θ−r)` | `r_t`, θ | `=1` breakeven (θ=mean), `>1` preferred **[L]** |
| Information ratio | `mean(r−b)/std(r−b)`, ann. ×`sqrt(P)` | `r_t`, benchmark `b_t` | `0.4–0.6` good, `0.61–1.0` very good, `≥1.0` exceptional **[formula L; thresholds C]** — N/A no bench |
| Treynor | `(E(R_p)−R_f)/beta`, `beta=Cov(R_p,R_m)/Var(R_m)` | `r_t`, `R_m` | no abs threshold; ranking only **[L]** — N/A no market |
| Omega(θ) | `Σ max(0,r−θ) / Σ max(0,θ−r)` | `r`, θ | `=1` breakeven (θ=mean), `>1` preferred **[L]** |
| Information ratio | `mean(r−b)/std(r−b)`, ann. ×`sqrt(P)` | `r`, benchmark `b` | `0.4–0.6` good, `0.61–1.0` very good, `≥1.0` exceptional **[formula L; thresholds C]** |
| Treynor | `(E(R_p)−R_f)/beta`, `beta=Cov(R_p,R_m)/Var(R_m)` | `r`, market `R_m` | no abs threshold; ranking only **[L]** |
**Annualization rule [L as convention]:** mean ×`P`, std ×`sqrt(P)`, ratio
×`sqrt(P)`; `P=252` daily (trading days, not 365), 52 weekly, 12 monthly. Valid
only under iid; Lo (2002) shows annualized Sharpe overstated up to ~65% under
serial correlation.
## D. Drawdown & recovery family (equity-curve functionals — NO trade list)
## D. Drawdown & recovery (equity-curve functionals)
`E(t)` = cumulative pip curve, `HWM(t)` = running peak, `DD(t) = E(t)−HWM(t)`.
`E(t)` = cumulative equity, `HWM(t)` = running peak, `DD(t) = E(t)−HWM(t)`. No
trade list required.
| metric | formula | computed from | threshold |
|---|---|---|---|
| Max drawdown | `max_t(HWM−E)` (pip) or `/HWM` (%) | `= max_drawdown` (emitted) | stat-arb/MN `<10–15%`, HF `<15–20%`, L/S unacceptable `>25–30%`, `>40%` very high **[C; no universal cutoff]** |
| Max drawdown | `max_t(HWM−E)` (level) or `/HWM` (%) | `E` | stat-arb/MN `<10–15%`, HF `<15–20%`, L/S unacceptable `>25–30%`, `>40%` very high **[C; no universal cutoff]** |
| Recovery asymmetry | DD fraction `d` needs gain `d/(1−d)` | a DD depth | exact: −20%→+25%, −50%→+100% **[L, algebraic]** |
| Average drawdown | `(1/n)Σ|DD|` (=Pain Index) OR episode-mean of trough depths | E(t), episode segmentation | comparative; `AvgDD/MaxDD≈1` → worst typical |
| Duration / time-to-recovery / time-underwater | per-episode index arithmetic; total UW = #(DD<0)/n | E(t) + cycle index | heuristic recovery ≈ `MDD/per-period-return`; flag censored open DD **[C]** |
| Average drawdown | `(1/n)Σ|DD|` (=Pain Index) OR episode-mean of trough depths | `E`, episode segmentation | comparative; `AvgDD/MaxDD≈1` → worst typical |
| Duration / time-to-recovery / time-underwater | per-episode index arithmetic; total UW = #(DD<0)/n | `E` + time index | heuristic recovery ≈ `MDD/per-period-return`; flag censored open DD **[C]** |
| Longest drawdown | `max_k(t_recover − t_peak)` | episode segmentation | shorter better; often a *different* episode than max-depth |
| Ulcer Index | `sqrt((Σ_i D'_i²)/n)`, `D'_i = 100·(E_i−max_so_far)/max_so_far` | E(t) | `=0` no DD; `> ~10` severe; comparative **[C]** |
| Ulcer Perf Index / Martin | `(R_p−R_f)/UlcerIndex` | `r_t`+Ulcer | higher better; no band; more overfit-robust objective **[L]** |
| Pain Index | `(Σ_i |D'_i|)/n`, n = *total* obs | E(t) | comparative; `< MaxDD`. **[CORR:** it EMBEDS duration — a *remedy*, NOT a metric that "ignores time"; Bacon 2008 p.89; n=total obs else it collapses to Average Drawdown**]** |
| Pain Ratio | `(R_p−R_f)/PainIndex` | `r_t`+Pain | higher better; Bacon 2008 p.91 **[CORR page]** |
| Recovery Factor | `NetProfit/|MaxDrawdown|` | `= total_pips/max_drawdown` (both emitted!) | poor `<1`, good `2–5`, excellent `≥5` **[formula L; thresholds C]** |
| Expected MaxDD (Magdon-Ismail) | Brownian: linear (μ<0), `~sqrt(T)` (μ=0, exact `sqrt(π/2)·σ·sqrt(T)`), log (μ>0) | μ,σ from `r_t`; classify regime | not graded — a yardstick; observed ≫ E[MaxDD] = regime break / fat tails **[L asymptotics]** |
| Ulcer Index | `sqrt((Σ_i D'_i²)/n)`, `D'_i = 100·(E_i−max_so_far)/max_so_far` | `E` | `=0` no DD; `> ~10` severe; comparative **[C]** |
| Ulcer Perf Index / Martin | `(R_p−R_f)/UlcerIndex` | `r`, `E` | higher better; no band; more overfit-robust objective **[L]** |
| Pain Index | `(Σ_i |D'_i|)/n`, n = *total* obs | `E` | comparative; `< MaxDD`. **[CORR:** it EMBEDS duration — a *remedy*, NOT a metric that "ignores time"; Bacon 2008 p.89; n=total obs else it collapses to Average Drawdown**]** |
| Pain Ratio | `(R_p−R_f)/PainIndex` | `r`, `E` | higher better; Bacon 2008 p.91 **[CORR page]** |
| Recovery Factor | `NetProfit/|MaxDrawdown|` | final `E` / MaxDD | poor `<1`, good `2–5`, excellent `≥5` **[formula L; thresholds C]** |
| Expected MaxDD (Magdon-Ismail) | Brownian: linear (μ<0), `~sqrt(T)` (μ=0, exact `sqrt(π/2)·σ·sqrt(T)`), log (μ>0) | μ,σ from `r`; classify regime | not graded — a yardstick; observed ≫ E[MaxDD] = regime break / fat tails **[L asymptotics]** |
**Load-bearing axis interaction:** MaxDD is a downward-biased, sample-length-
dependent point estimate — the cure is axis-4 MC (report the *distribution* of
MaxDD, not one point) and axis-3 WFA (report OOS DD separately). Do NOT argmax
**Load-bearing point:** MaxDD is a downward-biased, sample-length-dependent point
estimate — the cure is Monte-Carlo (report the *distribution* of MaxDD, not one
point) and walk-forward (report OOS DD separately). Do NOT argmax
Calmar/Recovery-Factor (single-point fragile); Ulcer/Pain-based ratios integrate
the whole path and are more robust objectives.
## E. Trade-level & efficiency metrics (require the C7 position-event table)
## E. Trade-level & efficiency metrics
A "trade" = the span from exposure non-zero (or sign-flip) to it returning to zero
(or flipping); pip P&L = `Σ r_t` over the span; direction = sign of the held
exposure.
Require a **trade list**. A "trade" spans from a position opening (or sign-flip) to
its close; P&L = `Σ r_t` over the span; direction = sign of the held position.
| metric | formula | computed from | threshold |
|---|---|---|---|
@@ -113,40 +112,41 @@ exposure.
| **PF↔win-rate identity** | `PF=(p·R)/(1−p)`, inverse `p=PF/(PF+R)` | p, R | exact identity (holds iff R = ratio of *averages*) **[L]** |
| Breakeven win rate | `1/(1+R)` | R | 50%@1:1, 33.3%@1:2, 25%@1:3, 16.7%@1:5 **[L]** |
| SQN (Van Tharp) | `sqrt(N)·mean(R-mult)/std(R-mult)`, N often capped 100 | per-trade R-multiples | `<1.6` below, `1.6–2.0` avg, `2.0–2.5` good, `2.5–3.0` excellent, `>7` overfit **[C]** |
| MAE / MFE | max adverse / favorable excursion = `−min`/`max(running unreal. PnL)` | per-cycle running pip-since-entry | no cutoff; winners cluster at small MAE; large MFE-vs-realized gap → exiting early |
| MAE / MFE | max adverse / favorable excursion = `−min`/`max(running unreal. PnL)` | per-bar running P&L since entry | no cutoff; winners cluster at small MAE; large MFE-vs-realized gap → exiting early |
| Trade efficiency | exit `ActualGain/MFE`; range `(Exit−Entry)/(High−Low)` | per-trade MFE/MAE/range | 0–100%; `<50%` exit eff poor **[C]** |
| Turnover | `Σ|Δexposure|` (or SEC `min(buys,sells)/avg_assets`) | exposure (`exposure_sign_flips` is the coarse proxy) | style-dependent; net-of-cost expectancy must stay `>0` |
| R-multiple / expectancy-in-R | per trade `PnL/1R`; mean `p·R−(1−p)` | trades + defined 1R | sign `>0`; ~`+0.5R` strong **[C]** — needs a stop; reserve for realistic broker / config reference-risk |
| Turnover | `Σ|Δposition|` (or SEC `min(buys,sells)/avg_assets`) | position series | style-dependent; net-of-cost expectancy must stay `>0` |
| R-multiple / expectancy-in-R | per trade `PnL/1R`; mean `p·R−(1−p)` | trades + defined 1R | sign `>0`; ~`+0.5R` strong **[C]** — needs a defined risk/stop |
## F. Tail, distribution & risk-of-ruin (from `r_t`; ruin from MC or trades)
## F. Tail, distribution & risk-of-ruin
Real returns are negatively skewed and leptokurtic, so iid-normal measures
*understate* tail risk. Prefer empirical / MC.
From `r` (ruin from Monte-Carlo or the trade list). Real returns are negatively
skewed and leptokurtic, so iid-normal measures *understate* tail risk. Prefer
empirical / Monte-Carlo estimates.
| metric | formula | computed from | threshold |
|---|---|---|---|
| Historical VaR_α | `−(1−α)` empirical quantile of r (5th @95%, 1st @99%) | sort `r_t`, index `floor((1−α)N)` | Basel traffic-light @250d/99%: GREEN 0–4, YELLOW 5–9, RED 10+ exceptions **[L]** |
| Parametric VaR | `−(μ+σ·Φ⁻¹(1−α))·W`; z = 1.645@95, 2.326@99, 1.282@90 | μ,σ of `r_t` | same grading; **iid-normal trap** **[L multipliers]** |
| Historical ES/CVaR | `mean of worst (1−α)N returns` | sorted `r_t`, `k=ceil((1−α)N)` | coherent (VaR is not); FRTB uses 97.5% ES ≈ 99% VaR under normality **[L]** |
| Historical VaR_α | `−(1−α)` empirical quantile of r (5th @95%, 1st @99%) | sort `r`, index `floor((1−α)N)` | Basel traffic-light @250d/99%: GREEN 0–4, YELLOW 5–9, RED 10+ exceptions **[L]** |
| Parametric VaR | `−(μ+σ·Φ⁻¹(1−α))·W`; z = 1.645@95, 2.326@99, 1.282@90 | μ,σ of `r` | same grading; **iid-normal trap** **[L multipliers]** |
| Historical ES/CVaR | `mean of worst (1−α)N returns` | sorted `r`, `k=ceil((1−α)N)` | coherent (VaR is not); FRTB uses 97.5% ES ≈ 99% VaR under normality **[L]** |
| Parametric ES | `μ+σ·φ(Φ⁻¹(α))/(1−α)`; mult ~2.063@95, 2.665@99 | μ,σ | worse than VaR for fat tails **[L]** |
| Tail ratio | `q95(r)/|q05(r)|` | percentiles of `r_t` | `>1` favourable, `<1` dangerous **[L 95/5]** |
| Skewness | `(1/N)Σ(r−μ)³/σ³` | `r_t` | sign-driven; persistent negative = red flag even with high Sharpe |
| Excess kurtosis | `(1/N)Σ(r−μ)⁴/σ⁴ − 3` | `r_t` | `0`=normal; `>0` leptokurtic (typical, dangerous); daily 3–10+ **[L]** |
| Downside deviation | `sqrt((1/N)Σ[min(0,r−T)]²)`, divisor total N | `r_t`, T | feeds Sortino **[L, divisor=total N]** |
| Tail ratio | `q95(r)/|q05(r)|` | percentiles of `r` | `>1` favourable, `<1` dangerous **[L 95/5]** |
| Skewness | `(1/N)Σ(r−μ)³/σ³` | `r` | sign-driven; persistent negative = red flag even with high Sharpe |
| Excess kurtosis | `(1/N)Σ(r−μ)⁴/σ⁴ − 3` | `r` | `0`=normal; `>0` leptokurtic (typical, dangerous); daily 3–10+ **[L]** |
| Downside deviation | `sqrt((1/N)Σ[min(0,r−T)]²)`, divisor total N | `r`, T | feeds Sortino **[L, divisor=total N]** |
| Risk of ruin (closed) | `((1−A)/(1+A))^U`, `A=p·R−q`, `U=capital/risk-per-trade` | trades p, R | `<1–5%`, desks `<1%`, pros `<0.1%`; `>5–10%` reject **[C]** |
| Risk of ruin (empirical) | `fraction of N MC paths breaching the ruin threshold` | axis-4 seeds | **preferred**; sidesteps iid-normal; default R=50% capital |
| Kelly / optimal-f | `f*=(b·p−q)/b`; `f*=(μ−r)/σ²` | `r_t`/odds | deploy `0 < f ≤ ~0.5·f*`; above full Kelly strictly dominated **[L]** — maps to exposure scale |
| Gain-to-Pain | `Σ(r)/|Σ(neg r)|` | `r_t` | (monthly) ~1 acc, ~2 outstanding, 3 excellent, ~4 world-class, `<1` poor **[C; state the basis]** |
| Risk of ruin (empirical) | `fraction of N Monte-Carlo paths breaching the ruin threshold` | multi-run dist. | **preferred**; sidesteps iid-normal; default R=50% capital |
| Kelly / optimal-f | `f*=(b·p−q)/b`; `f*=(μ−r)/σ²` | `r`/odds | deploy `0 < f ≤ ~0.5·f*`; above full Kelly strictly dominated **[L]** |
| Gain-to-Pain | `Σ(r)/|Σ(neg r)|` | `r` | (monthly) ~1 acc, ~2 outstanding, 3 excellent, ~4 world-class, `<1` poor **[C; state the basis]** |
| Common-Sense ratio | `TailRatio · GainToPain` | the two above | `>1` robust, `<1` fragile **[L breakpoint]** |
## G. Walk-forward analysis (axis 3)
## G. Walk-forward analysis
| metric | formula | computed from | threshold |
|---|---|---|---|
| WFE (return) | `(OOS_pips/OOS_bars)/(IS_pips/IS_bars)` | per-fold IS-optimal/OOS `total_pips`, bar counts | `≥0.5` robust, `~1.0` ideal, `>0.7` excellent, `0.5–0.7` good, `0.3–0.5` mediocre, `<0.3` poor; `>1.0` is a *warning* (under-fit IS / lucky OOS), not superiority |
| WFE (Sharpe) | `OOS_Sharpe/IS_Sharpe` | per-fold Sharpe over `r_t` | `>0.5` robust, `~1.0` ideal, `<0.3` red flag |
| WFE (return) | `(OOS_profit/OOS_bars)/(IS_profit/IS_bars)` | per-fold IS-optimal/OOS profit, bar counts | `≥0.5` robust, `~1.0` ideal, `>0.7` excellent, `0.5–0.7` good, `0.3–0.5` mediocre, `<0.3` poor; `>1.0` is a *warning* (under-fit IS / lucky OOS), not superiority |
| WFE (Sharpe) | `OOS_Sharpe/IS_Sharpe` | per-fold Sharpe over `r` | `>0.5` robust, `~1.0` ideal, `<0.3` red flag |
| IS-OOS gap | `IS_Sharpe − OOS_Sharpe` per fold, averaged | per-fold Sharpes | `>0.5`/cycle = overfitting; healthy mean `<0.3` |
| Stitched OOS curve | chain each fold's OOS segment, re-based | family of OOS runs (C18 sink) | net profitable; max DD `<40%` (TradeStation) |
| Stitched OOS curve | chain each fold's OOS segment, re-based | family of OOS runs | net profitable; max DD `<40%` (TradeStation) |
| Parameter stability | dispersion of argmax params across folds; plateau check | per-fold winners | qualitative — broad plateau, clustered |
| Fraction profitable OOS windows | `#(OOS>0)/#windows` | per-window OOS P&L | majority `>50%`; worst-window OOS Sharpe `>0` |
| Pardo verdict | net-profitable AND WFE `≥50%` AND maxDD `<40%`; filter %Prof `≥40`, PF `≥1.5` | the above | composite pass/fail **[software-default convention]** |
@@ -156,48 +156,51 @@ clean stitch; trade floors IS `≥100`, OOS `≥30`. Rolling vs anchored. Advanc
sweep of WFA configs (a "family of families") proves the verdict is not a lucky
schedule.
## H. Monte-Carlo (axis 4)
## H. Monte-Carlo
| metric | formula | computed from | threshold |
|---|---|---|---|
| Trade-order shuffle | permute `r_t`, rebuild equity | seeded reorder | additive sum INVARIANT — only path/DD stats vary **[load-bearing for pip wiring]** |
| Bootstrap (with replacement) | resample T from `r_t` | seeded resample | changes total → genuine `total_pips` distribution; CI=[2.5, 97.5] pct |
| Trade-order shuffle | permute `r`, rebuild equity | seeded reorder | additive sum INVARIANT — only path/DD stats vary |
| Bootstrap (with replacement) | resample T from `r` | seeded resample | changes total → genuine return distribution; CI=[2.5, 97.5] pct |
| Equity percentile fan | per-horizon {5,25,50,75,95} pct | N curves | descriptive; viability: 5th-pct ending equity `>0` |
| Stress (95th-pct) MaxDD | 95th pct of N MaxDD | per-seed MaxDD | `<25%` good, `>35%` red flag; size capital off this **[C]** |
| Empirical VaR/ES | 5th-pct (1st @99) final P&L; ES = mean beyond | per-seed terminal | 5th-pct ending equity `>0` |
| Probability of ruin | `ruin_count/N` (MaxDD>R, default 50%) | per-seed MaxDD | `<1%` excellent, `1–5%` acceptable, `>5%` not recommended **[C]** |
| Prob. of a losing period | `#(period<0)/N` | per-seed period returns | `>~30–40%` hard to hold |
| Stress (95th-pct) MaxDD | 95th pct of N MaxDD | per-run MaxDD | `<25%` good, `>35%` red flag; size capital off this **[C]** |
| Empirical VaR/ES | 5th-pct (1st @99) final P&L; ES = mean beyond | per-run terminal | 5th-pct ending equity `>0` |
| Probability of ruin | `ruin_count/N` (MaxDD>R, default 50%) | per-run MaxDD | `<1%` excellent, `1–5%` acceptable, `>5%` not recommended **[C]** |
| Prob. of a losing period | `#(period<0)/N` | per-run period returns | `>~30–40%` hard to hold |
| **MCPT p-value** | `(z+1)/(N+1)`, z=#perm `≥` real | permute bar-to-bar log returns, keep first price, re-run | `<0.05` min, `<0.01` strict, `>0.10` red flag, `~0.5` discard **[L: +1 in numerator and denominator]** |
| White RC / Hansen SPA | bootstrap max-stat over L rules, recentered | per-rule rel-perf + benchmark | `p<0.05`; SPA more powerful than conservative RC |
| White RC / Hansen SPA | bootstrap max-stat over L rules, recentered | per-rule rel-perf + benchmark | `p<0.05`; SPA more powerful than the conservative RC |
**Conventions:** N = 1,000 min, 5,000 recommended, 10,000+ for tail stability.
Plain shuffle assumes serial independence — if `r_t` is autocorrelated use the
**block/stationary bootstrap** (Politis-Romano). MCPT permutes *log returns*
(preserving the marginal, destroying ordering) → tests specifically time-ordering
edges.
Plain shuffle assumes serial independence — if `r` is autocorrelated use the
**block/stationary bootstrap** (Politis-Romano). The permutation test permutes
*log returns* (preserving the marginal, destroying ordering) → tests specifically
time-ordering edges.
## I. Stress/scenario, regime, cost/capacity, benchmark-relative, sanity, live-vs-backtest
| tool | produces | key metric + formula | computed from | threshold |
|---|---|---|---|---|
| Stress/scenario | conditional worst-case | per-scenario MaxDD, total_pips, recover-time | frozen topology over crisis windows / shocked synthetic source (C6) | fail if stress DD > risk limit; excellent if net-positive through crises **[C; Fed CCAR]** |
| Regime | per-state metrics + fragility verdict | total_pips/Sharpe/DD masked by regime label | regime-label stream (vol/trend classifier, recorded HMM C6) joined at C3 | edge only in one regime + negative elsewhere = fragile **[qualitative]** |
| Cost/slippage | net-vs-cost curve + break-even | `c* where net=0 ≈ gross_profit/n_trades`; `d(net Sharpe)/d(cost)` | realistic broker (C7) over position-event table, cost swept | reject if edge survives only at zero cost; break-even ≳2× real cost **[C]** |
| Capacity/impact | AUM ceiling + turnover | `impact ≈ Y·σ·sqrt(Q/V)`, Y~O(1); capacity = AUM where impact = gross alpha | position volumes + ADV (C4) | capacity ≫ target = scalable; per-day order `< ~5–10%` ADV **[C]** |
| Benchmark-relative | alpha/beta/IR + significance | `beta=Cov/Var`; Jensen alpha = regression intercept; `IR=alpha/TE` | `r_t` regressed on ingested `R_m` (C3) | alpha positive + significant (`|t|>2`); IR 0.4/0.6/1.0 **[beta/CAPM L; IR C]** — N/A no bench |
| Null/sanity | "beat random?" | MCPT `p=(#perm≥real)/N`; random-entry: real in right tail | permuted source + null random-exposure node | `p<0.05` min, `≤0.01` strict, `>0.10` red flag, N`≥1000` **[L]** |
| Live-vs-backtest | overfitting/decay verdict | haircut `=(BT_Sharpe−live_Sharpe)/BT_Sharpe`; reconciliation diff≈0 | C8 live series vs frozen re-run | `<30%` acceptable, `30–50%` typical, `>50–70%` largely overfit; drift≠0 = a bug **[C; verify primary source]** |
| Stress/scenario | conditional worst-case | per-scenario MaxDD, profit, recover-time | strategy over crisis windows / shocked synthetic data | fail if stress DD > risk limit; excellent if net-positive through crises **[C; Fed CCAR]** |
| Regime | per-state metrics + fragility verdict | profit/Sharpe/DD masked by regime label | a regime-label series (vol/trend classifier or HMM) | edge only in one regime + negative elsewhere = fragile **[qualitative]** |
| Cost/slippage | net-vs-cost curve + break-even | `c* where net=0 ≈ gross_profit/n_trades`; `d(net Sharpe)/d(cost)` | a cost-modelling backtest, cost swept | reject if edge survives only at zero cost; break-even ≳2× real cost **[C]** |
| Capacity/impact | AUM ceiling + turnover | `impact ≈ Y·σ·sqrt(Q/V)`, Y~O(1); capacity = AUM where impact = gross alpha | order volumes + ADV | capacity ≫ target = scalable; per-day order `< ~5–10%` ADV **[C]** |
| Benchmark-relative | alpha/beta/IR + significance | `beta=Cov/Var`; Jensen alpha = regression intercept; `IR=alpha/TE` | `r` regressed on a market series `R_m` | alpha positive + significant (`|t|>2`); IR 0.4/0.6/1.0 **[beta/CAPM L; IR C]** |
| Null/sanity | "beat random?" | MCPT `p=(#perm≥real)/N`; random-entry: real in right tail | permuted input + a random-entry null strategy | `p<0.05` min, `≤0.01` strict, `>0.10` red flag, N`≥1000` **[L]** |
| Live-vs-backtest | overfitting/decay verdict | haircut `=(BT_Sharpe−live_Sharpe)/BT_Sharpe` | live series vs backtest prediction | `<30%` acceptable, `30–50%` typical, `>50–70%` largely overfit **[C; verify primary source]** |
## J. Overfitting / multiple-testing + CPCV (the significance layer over axes 1–4)
## J. Overfitting, multiple-testing & CPCV
The significance layer over the analysis axes. Treats "I ran many variants and
kept the best" as the statistical problem it is.
| metric | formula | computed from | threshold |
|---|---|---|---|
| PSR(SR*) | `Φ[(SR_hat−SR*)·sqrt(T−1)/sqrt(1−g3·SR_hat+((g4−1)/4)·SR_hat²)]` | per-period `SR_hat`, T, g3=skew, g4=kurtosis (full, normal=3) of `r_t` | `>0.95` significant **[L]** |
| PSR(SR*) | `Φ[(SR_hat−SR*)·sqrt(T−1)/sqrt(1−g3·SR_hat+((g4−1)/4)·SR_hat²)]` | per-period `SR_hat`, T, g3=skew, g4=kurtosis (full, normal=3) of `r` | `>0.95` significant **[L]** |
| MinTRL | `1+(1−g3·SR_hat+((g4−1)/4)·SR_hat²)·(Z_α/(SR_hat−SR*))²` | as PSR + target α | need T `≥` MinTRL **[L]** |
| E[max SR_N] | `sqrt(V)·((1−γ)Φ⁻¹(1−1/N)+γΦ⁻¹(1−1/(Ne)))`, γ≈0.5772 | cross-trial Sharpe variance V, effective N | the moving hurdle; grows with N **[L]** |
| **DSR** | PSR at `SR*=SR_0=E[max SR_N]` | winner SR_hat/T/g3/g4 + V across cells + effective N | `>0.95` survives deflation **[L; ex: N=100,V=½,T=1250,g3=−3,g4=10 → DSR~0.90<0.95 REJECT]** |
| **DSR** | PSR at `SR*=SR_0=E[max SR_N]` | winner SR_hat/T/g3/g4 + V across trials + effective N | `>0.95` survives deflation **[L; ex: N=100,V=½,T=1250,g3=−3,g4=10 → DSR~0.90<0.95 REJECT]** |
| MinBTL | `≤ 2·ln(N)/E[max]²` years | N, target SR | 5y + `>~45` independent trials → guaranteed overfit **[C/L]** |
| **PBO (CSCV)** | matrix `(T×M)`; S even blocks; `C(S,S/2)` combos; `ω=(rank−0.5)/M`; `λ=ln(ω/(1−ω))`; `PBO=P[λ≤0]` | per-cell `r_t` series | `<0.5` sanity, `~0.5` alarm, `<0.1` excellent **[L]** |
| **PBO (CSCV)** | matrix `(T×M)`; S even blocks; `C(S,S/2)` combos; `ω=(rank−0.5)/M`; `λ=ln(ω/(1−ω))`; `PBO=P[λ≤0]` | per-trial `r` series | `<0.5` sanity, `~0.5` alarm, `<0.1` excellent **[L]** |
| Harvey-Liu-Zhu haircut | `t=SR·sqrt(T)`; adjust p (Bonferroni `min(M·p,1)`/Holm/BHY); `HSR=SR·(adj_t/orig_t)` | SR, T, M | required new-factor t `~3.0` (not 2.0) |
| CPCV paths | splits `=C(N,k)`; paths `φ=(k/N)C(N,k)` | grouped purged+embargoed windows | N=5–10 groups, k=2; purge: drop train obs whose label span overlaps test; embargo h~1% of T (forward only) **[L]** |
@@ -205,8 +208,8 @@ edges.
## Master catalogue (compact index)
Inputs: **r** = `diff(equity)`; **E** = cumulative pip-equity; **trades** = C7
table; **bench** = ingested benchmark; **multi** = multi-run distribution.
Inputs: **r** = per-period returns; **E** = equity curve; **trades** = closed
trades; **b** = benchmark; **multi** = multi-run distribution.
**Return** — Total `sum r` [C] · CAGR `mean(r)·P` [C] · Volatility `std(r)·sqrt(P)` [C].
@@ -216,7 +219,7 @@ table; **bench** = ingested benchmark; **multi** = multi-run distribution.
`CompoundROR/|avgAnnMaxDD−10%|` [CORR] · Burke `(r_P−r_F)/sqrt(ΣD²)` [L] · Omega
`Σmax(0,r−θ)/Σmax(0,θ−r)` [L] · Info-Ratio `mean(r−b)/std(r−b)·sqrt(P)` [L/C] · UPI
`(R_p−R_f)/UlcerIndex` [L] · Pain-Ratio `(R_p−R_f)/PainIndex` [CORR page] ·
Recovery-Factor `total_pips/|MaxDD|` [L/C] · Gain-to-Pain `Σr/|Σ neg r|` [C] ·
Recovery-Factor `NetProfit/|MaxDD|` [L/C] · Gain-to-Pain `Σr/|Σ neg r|` [C] ·
Common-Sense `TailRatio·GainToPain` [L] · SQN `sqrt(N)·mean(R)/std(R)` [C].
**Drawdown** — MaxDD `max(HWM−E)` [C] · Recovery-asymmetry `d/(1−d)` [L] · AvgDD
@@ -227,7 +230,7 @@ duration] · Expected-MaxDD μ=0 `sqrt(π/2)·σ·sqrt(T)` [L].
`<1`/`1.5–2`/`>3 overfit` [CORR swing>day] · Expectancy `p·AvgWin−(1−p)·AvgLoss`,
sign>0 [L] · Payoff `AvgWin/AvgLoss` [L] · PF-identity `PF=(p·R)/(1−p)` [L] ·
Breakeven-win `1/(1+R)` [L] · MAE/MFE [C] · Trade-eff `ActualGain/MFE` [C] · Turnover
`Σ|Δexposure|` [C].
`Σ|Δposition|` [C].
**Tail/distribution** — Hist-VaR `−q_{1−α}(r)`, Basel traffic-light [L] · Param-VaR
`−(μ+σ·z)`, z 1.645/2.326 [L] · Hist-ES `mean(worst (1−α)N)` [L] · Tail-Ratio
@@ -239,3 +242,94 @@ empirical, `<1%` pro [C] · Kelly `(μ−r)/σ²`, `≤0.5·f*` [L].
· MinTRL [L] · DSR = PSR at `SR*=E[maxSR_N]`, `>0.95` [L] · MinBTL `≤2ln(N)/E[max]²`
[C/L] · PBO `P[ln(ω/(1−ω))≤0]`, `<0.5` sanity / `<0.1` excellent [L] · HLZ-haircut
`HSR=SR·(adj_t/orig_t)`, t`~3.0` [C/L] · CPCV `φ=(k/N)C(N,k)`, embargo h~1% [L].
---
## References
External, source-checked references per section (each link verified reachable and
on-topic). Where a formula or threshold was corrected, the citation is in the
[Metric Verification Log](Metric-Verification-Log).
**A / B — Parameter sweep, optimization & robustness:**
[Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis) ·
[Hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) ·
[Pseudo-mathematics and financial charlatanism — Bailey, Borwein, López de Prado, Zhu](https://scholarworks.wmich.edu/math_pubs/42/) ·
[Deflated Sharpe Ratio (objective choice)](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio)
**C — Core & risk-adjusted ratios:**
[Sharpe](https://en.wikipedia.org/wiki/Sharpe_ratio) ·
[Sortino](https://en.wikipedia.org/wiki/Sortino_ratio) ·
[Calmar & MAR](https://en.wikipedia.org/wiki/Calmar_ratio) ·
[Sterling (PerformanceAnalytics)](https://rdrr.io/rforge/PerformanceAnalytics/man/CalmarRatio.html) ·
[Burke (PerformanceAnalytics)](https://www.rdocumentation.org/packages/PerformanceAnalytics/versions/2.0.4/topics/BurkeRatio) ·
[Omega](https://en.wikipedia.org/wiki/Omega_ratio) ·
[Information ratio](https://en.wikipedia.org/wiki/Information_ratio) ·
[Treynor](https://en.wikipedia.org/wiki/Treynor_ratio) ·
[Volatility / sqrt-of-time](https://en.wikipedia.org/wiki/Volatility_(finance)) ·
[Lo (2002), The Statistics of Sharpe Ratios — CFA Institute](https://rpc.cfainstitute.org/research/financial-analysts-journal/2002/the-statistics-of-sharpe-ratios)
**D — Drawdown & recovery:**
[Drawdown (max / average / duration)](https://en.wikipedia.org/wiki/Drawdown_(economics)) ·
[Ulcer Index & UPI](https://en.wikipedia.org/wiki/Ulcer_index) ·
[Pain Index](https://rdrr.io/cran/PerformanceAnalytics/man/PainIndex.html) ·
[Pain Ratio](https://rdrr.io/cran/PerformanceAnalytics/man/PainRatio.html) ·
[Average drawdown](https://rdrr.io/cran/PerformanceAnalytics/man/AverageDrawdown.html) ·
[Recovery Factor (Fidelity glossary)](https://www.fidelity.com/research/backtesting/glossary.html) ·
[Expected MaxDD — Magdon-Ismail et al.](https://www.cambridge.org/core/journals/journal-of-applied-probability/article/abs/on-the-maximum-drawdown-of-a-brownian-motion/F9E3B8A454B020DDEBF0AC3390EF7807)
**E — Trade-level & efficiency:**
[Win rate vs profit factor](https://www.backtestbase.com/education/win-rate-vs-profit-factor) ·
[Expectancy](https://www.tradesviz.com/glossary/expectancy/) ·
[Payoff & breakeven](https://learningcenter.fxstreet.com/education/learning-center/unit-3/chapter-2/revenue-statistics/index.html) ·
[PF↔win-rate identity](https://www.pnlledger.com/profit-factor-vs-win-rate-vs-payoff-ratio/) ·
[SQN (Van Tharp)](https://indextrader.com.au/van-tharps-sqn/) ·
[MAE](https://www.tradesviz.com/glossary/maximum-adverse-excursion/) ·
[MFE](https://www.tradesviz.com/glossary/maximum-favorable-excursion/) ·
[Trade efficiency (TradeStation)](https://help.tradestation.com/10_00/eng/tradestationhelp/subsystems/spr_topics/report/entry_efficiency_exit_efficiency__strategy_performance_report_.htm) ·
[Turnover (CFI)](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/portfolio-turnover-ratio/)
**F — Tail, distribution & risk-of-ruin:**
[Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) ·
[Expected Shortfall / CVaR](https://en.wikipedia.org/wiki/Expected_shortfall) ·
[Skewness](https://en.wikipedia.org/wiki/Skewness) ·
[Kurtosis](https://en.wikipedia.org/wiki/Kurtosis) ·
[Downside deviation](https://rdrr.io/cran/PerformanceAnalytics/man/DownsideDeviation.html) ·
[Risk of ruin](https://en.wikipedia.org/wiki/Risk_of_ruin) ·
[Kelly criterion](https://en.wikipedia.org/wiki/Kelly_criterion) ·
[Gain-to-pain](https://www.wallstreetmojo.com/gain-to-pain-ratio/) ·
[Basel backtesting / traffic-light (BIS)](https://www.bis.org/publ/bcbs22.htm)
**G — Walk-forward analysis:**
[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) ·
[WFE / parameter stability / criteria (TradeStation)](https://help.tradestation.com/09_01/tswfo/topics/about_wfo.htm) ·
[Anchored vs rolling windows](https://www.susanpotter.net/quant/walk-forward-optimization/)
**H — Monte-Carlo & permutation tests:**
[Bootstrapping (incl. block/stationary)](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) ·
[Permutation test](https://en.wikipedia.org/wiki/Permutation_test) ·
[Monte-Carlo Permutation Test (BuildAlpha)](https://www.buildalpha.com/monte-carlo-permutation/) ·
[White's Reality Check (1999)](https://ideas.repec.org/a/bla/jfinan/v54y1999i5p1647-1691.html) ·
[Hansen's SPA test (2005)](https://ideas.repec.org/a/bes/jnlbes/v23y2005p365-380.html)
**I — Stress, regime, cost/capacity, benchmark-relative, sanity:**
[Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) ·
[Regime / Markov switching](https://en.wikipedia.org/wiki/Markov_switching_multifractal) ·
[Slippage](https://en.wikipedia.org/wiki/Slippage_(finance)) ·
[Transaction-cost backtesting (QuantStart)](https://www.quantstart.com/articles/Successful-Backtesting-of-Algorithmic-Trading-Strategies-Part-II/) ·
[Square-root market-impact law (arXiv)](https://arxiv.org/abs/1602.03043) ·
[CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model) ·
[Jensen's alpha](https://en.wikipedia.org/wiki/Jensen's_alpha) ·
[Live-vs-backtest haircut — Bailey & López de Prado, Deflated Sharpe (PDF)](https://www.davidhbailey.com/dhbpapers/deflated-sharpe.pdf)
**J — Overfitting, multiple-testing & CPCV:**
[Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) ·
[Probabilistic Sharpe Ratio (PerformanceAnalytics)](https://rdrr.io/cran/PerformanceAnalytics/man/ProbSharpeRatio.html) ·
[Minimum Track Record Length](https://rdrr.io/cran/PerformanceAnalytics/man/MinTrackRecord.html) ·
[PBO / MinBTL via CSCV (pbo vignette)](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) ·
[Harvey-Liu-Zhu haircut (quantstrat)](https://rdrr.io/github/braverock/quantstrat/man/SharpeRatio.haircut.html) ·
[Bonferroni](https://en.wikipedia.org/wiki/Bonferroni_correction) ·
[Holm-Bonferroni](https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method) ·
[Benjamini-Hochberg / FDR](https://en.wikipedia.org/wiki/False_discovery_rate) ·
[Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) ·
[López de Prado, Advances in Financial ML, ch. 7](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml)