Metric Verification Log
Source-checking of the load-bearing formulas and thresholds in the Metric Catalogue. Narrative on Strategy Analysis & Validation.
Method
The metric research was produced by a multi-stage workflow: 10 topic agents swept the quant-finance literature, then 23 load-bearing formulas/thresholds were adversarially re-checked by independent fact-checker agents — each tasked to refute the claim and to mark a threshold uncertain rather than confirmed when no authoritative source backed the specific number. Result: 14 confirmed, 9 corrected, 0 uncertain. The corrected values are the ones used in the catalogue.
A recurring finding: almost every grading threshold is a practitioner convention, not a law. Formulas are laws; "Sharpe > 2 is excellent" is a rule-of-thumb that varies by source, asset class, and market regime. The catalogue marks these [C]. Treat [C] thresholds as relative-comparison guidelines — do not hard-code them as pass/fail gates without revisiting the primary source.
Corrections applied (9)
| # | Claim (as first drafted) | Correction applied | Sources |
|---|---|---|---|
| 1 | Sharpe denominator = total std \operatorname{std}(r) |
Denominator is std of the excess return \operatorname{std}(r-r_f); equals \operatorname{std}(r) only when r_f is constant over the window (e.g. r_f=0 in frictionless P&L accounting). Numerator (excess return) and \sqrt{P} annualization confirmed; annualization is a convention (square-root-of-time, iid-only), not part of the core definition. |
Wikipedia: Sharpe ratio, CFI, QuantInsti |
| 2 | Sortino grading: 0–0.5 weak, 0.5–1 acceptable; "good" bar >2 |
No 0.5 split — 0–1 is one sub-optimal band. The "good" bar is >1 (not >2); >2 very good, >3 excellent. Divisor of downside deviation is total N. All bands are conventions tied to the chosen MAR. |
optiml, fe.training, portfolioslab |
| 3 | Calmar grading: >2 excellent |
>2 alone is not excellent — reserve "excellent" for >3; 0.5–1.0 is "weak" not "acceptable". Common scheme <1 poor / 1–3 acc-good / >3 excellent. No authoritative source defines any numeric band — all convention. |
FinanceStrategists, Quantt, NumberAnalytics |
| 4 | Calmar vs MAR differ "only in window" | They differ in two ways: window (trailing 36mo vs inception) and numerator (Calmar = average annual rate of return; MAR = compound/CAGR). | Wikipedia: Calmar ratio, SuperMoney: MAR, QuantifiedStrategies |
| 5 | Sterling = \mathrm{CompoundROR}/(\mathrm{avgMaxDD}+10\%) |
Canonical form subtracts 10% inside an absolute value: \mathrm{CompoundROR}/\lvert\mathrm{avgAnnualMaxDD}-10\%\rvert. The 10% ≈ the ~1981 T-bill rate (also guards a zero-DD divide-by-zero), not purely arbitrary. Modern variants often drop the 10% entirely (collapsing toward Calmar). |
Wikipedia: Sterling ratio, RCM Alternatives |
| 6 | Pain Index / Pain Ratio both at Bacon 2008 p.91 | Pain Index is Bacon 2008 p.89, Pain Ratio p.91. n in the Pain Index = total return observations over the whole series, not the count of drawdown episodes (else it silently becomes Average Drawdown). |
PerformanceAnalytics: PainIndex, PainRatio |
| 7 | Pain Index listed among metrics that "ignore time" | The Pain Index embeds duration (it is the time-average of the underwater curve) — it belongs in the remedy list, not the critique list. The metrics that ignore time/non-worst episodes are Calmar/MaxDD only. | PerformanceAnalytics, Wikipedia: Drawdown |
| 8 | Profit-factor targets: day-trade >1.5, swing ≥1.3 |
Ordering is backwards: swing targets are higher (~1.5–2.5+) than day-trading (~1.3–2.0), because fewer trades demand more edge per trade. Bands: <1 losing / 1.0–1.2 breakeven / 1.2–1.5 realistic / 1.5–2.0 good / 2.0–3.0 exceptional-but-scrutinize / >3 overfit. Needs ~200+ trades; live degrades PF ~10–20%. |
TradeZella, CrossTrade, QuantifiedStrategies |
| 9 | Calmar >1/>2/>3 three-band scheme as standard |
The widely-published convention uses two cuts only — <1 poor / 1–3 good / >3 excellent — with no distinct "good at 2". The 1.5–3.0 hedge-fund target has no primary backing. Formula and the Calmar-vs-MAR distinction are solid; the bands are unsourced convention. |
Wikipedia: Calmar ratio, CFI |
Confirmed as law / exact (used as-is)
Annualization (mean×P, std/ratio×√P; P=252/52/12), Sharpe-as-t-stat
(t = \mathrm{SR}\cdot\sqrt{T}), PSR / DSR / E[max SR_N], PBO via CSCV, Omega, Burke, Information &
Treynor ratio formulas, recovery asymmetry d/(1-d), Ulcer Index & UPI, Recovery
Factor formula, Magdon-Ismail E[MaxDD] three-regime asymptotics, VaR/ES
multipliers & the Basel traffic-light, tail ratio (95/5), excess kurtosis,
downside-deviation divisor (total N), risk-of-ruin closed form, Kelly / optimal-f,
gain-to-pain, common-sense ratio, MCPT p-value (z+1)/(N+1) (the +1 appears in
both numerator and denominator), the profit-factor↔win-rate identity
\mathrm{PF}=(p\cdot R)/(1-p), CPCV path count \varphi=(k/N)\cdot C(N,k).
Confirmed as convention (flagged [C]; verify primary source before hard-coding)
Information-ratio bands (0.4/0.6/1.0), Calmar / Sortino / SQN / Recovery-Factor /
profit-factor bands, MaxDD acceptability bands, capacity impact-coefficient Y,
the live-vs-backtest Sharpe haircut figures, and the Harvey-Liu-Zhu required
t-stat ~3.0 (and ~3.18 with publication-bias adjustment). These are
practitioner rules-of-thumb that vary by source and regime.
Known follow-ups for the implementer
- Verify the Harvey-Liu-Zhu required-t and the live-vs-backtest haircut figures against their primary papers before any code hard-codes them.
- Confirm the MinBTL closed form
\le 2\cdot\ln(N)/\operatorname{E}[\max]^2against the Bailey-Borwein-López de Prado-Zhu paper (the agent flagged the constant).
Backtesting Engine Architecture
- Architecture overview
- Event-Driven Backtesting
- Look-Ahead Bias & Causality
- Determinism & Reproducibility
- Reactive Streaming Dataflow
- Columnar Data Layout
- Signal, Exposure & Execution
- Graph Compilation & Optimization
- Authoring & Deployment Lifecycle
Strategy Analysis & Validation