diff --git a/Metric-Verification-Log.md b/Metric-Verification-Log.md new file mode 100644 index 0000000..3439b85 --- /dev/null +++ b/Metric-Verification-Log.md @@ -0,0 +1,60 @@ +# Metric Verification Log + +> Source-checking of the load-bearing formulas and thresholds in the +> **[Metric Catalogue](Metric-Catalogue)**. Narrative on **[Home](Home)**. + +## Method + +The metric research was produced by a multi-stage workflow: 10 topic agents +swept the quant-finance literature, then **23 load-bearing formulas/thresholds +were adversarially re-checked** by independent fact-checker agents — each tasked to +*refute* the claim and to mark a threshold *uncertain* rather than *confirmed* when +no authoritative source backed the specific number. Result: **14 confirmed, 9 +corrected, 0 uncertain.** The corrected values are the ones used in the catalogue. + +A recurring finding: **almost every grading threshold is a practitioner +convention, not a law.** Formulas are laws; "Sharpe > 2 is excellent" is a +rule-of-thumb that varies by source, asset class, and market regime. The catalogue +marks these **[C]**. Treat **[C]** thresholds as relative-comparison guidelines — +do not hard-code them as pass/fail gates without revisiting the primary source. + +## Corrections applied (9) + +| # | Claim (as first drafted) | Correction applied | Sources | +|---|---|---|---| +| 1 | Sharpe denominator = total std `std(r)` | Denominator is std of the **excess** return `std(r−r_f)`; equals `std(r)` only when `r_f` is constant over the window. In aura's pip space `r_f=0`, so `std(r)` is exact here. Numerator (excess return) and `sqrt(P)` annualization confirmed; annualization is a convention (square-root-of-time, iid-only), not part of the core definition. | [Wikipedia: Sharpe ratio](https://en.wikipedia.org/wiki/Sharpe_ratio), [CFI](https://corporatefinanceinstitute.com/resources/career-map/sell-side/risk-management/sharpe-ratio-definition-formula/), [QuantInsti](https://blog.quantinsti.com/sharpe-ratio-applications-algorithmic-trading/) | +| 2 | Sortino grading: 0–0.5 weak, 0.5–1 acceptable; "good" bar `>2` | No 0.5 split — `0–1` is **one** sub-optimal band. The "good" bar is `>1` (not `>2`); `>2` very good, `>3` excellent. Divisor of downside deviation is total N. All bands are conventions tied to the chosen MAR. | [optiml](https://optiml.co.uk/guides/invest/ratios/sortino.html), [fe.training](https://www.fe.training/free-resources/portfolio-management/sortino-ratio-formula-how-to-calculate-it-in-excel/), [portfolioslab](https://portfolioslab.com/tools/sortino-ratio) | +| 3 | Calmar grading: `>2` excellent | `>2` alone is **not** excellent — reserve "excellent" for `>3`; `0.5–1.0` is "weak" not "acceptable". Common scheme `<1` poor / `1–3` acc-good / `>3` excellent. No authoritative source defines any numeric band — all convention. | [FinanceStrategists](https://www.financestrategists.com/wealth-management/financial-ratios/calmar-ratio/), [Quantt](https://www.quantt.co.uk/resources/calmar-ratio-explained), [NumberAnalytics](https://www.numberanalytics.com/blog/ultimate-guide-calmar-ratio-capital-markets) | +| 4 | Calmar vs MAR differ "only in window" | They differ in **two** ways: window (trailing 36mo vs inception) **and** numerator (Calmar = average annual rate of return; MAR = compound/CAGR). | [Wikipedia: Calmar ratio](https://en.wikipedia.org/wiki/Calmar_ratio), [SuperMoney: MAR](https://www.supermoney.com/encyclopedia/mar-ratio), [QuantifiedStrategies](https://www.quantifiedstrategies.com/mar-ratio/) | +| 5 | Sterling = `CompoundROR / (avgMaxDD + 10%)` | Canonical form **subtracts** 10% **inside** an absolute value: `CompoundROR / |avgAnnualMaxDD − 10%|`. The 10% ≈ the ~1981 T-bill rate (also guards a zero-DD divide-by-zero), not purely arbitrary. Modern variants often drop the 10% entirely (collapsing toward Calmar). | [Wikipedia: Sterling ratio](https://en.wikipedia.org/wiki/Sterling_ratio), [RCM Alternatives](https://www.rcmalternatives.com/2014/03/the-sterling-ratio-explained/) | +| 6 | Pain Index / Pain Ratio both at Bacon 2008 p.91 | Pain **Index** is Bacon 2008 **p.89**, Pain **Ratio** p.91. `n` in the Pain Index = total return observations over the whole series, **not** the count of drawdown episodes (else it silently becomes Average Drawdown). | [PerformanceAnalytics: PainIndex](https://rdrr.io/cran/PerformanceAnalytics/man/PainIndex.html), [PainRatio](https://rdrr.io/cran/PerformanceAnalytics/man/PainRatio.html) | +| 7 | Pain Index listed among metrics that "ignore time" | The Pain Index **embeds** duration (it is the time-average of the underwater curve) — it belongs in the *remedy* list, not the critique list. The metrics that ignore time/non-worst episodes are Calmar/MaxDD only. | [PerformanceAnalytics](https://rdrr.io/rforge/PerformanceAnalytics/man/PainIndex.html), [Wikipedia: Drawdown](https://en.wikipedia.org/wiki/Drawdown_(economics)) | +| 8 | Profit-factor targets: day-trade `>1.5`, swing `≥1.3` | Ordering is **backwards**: swing targets are **higher** (~1.5–2.5+) than day-trading (~1.3–2.0), because fewer trades demand more edge per trade. Bands: `<1` losing / `1.0–1.2` breakeven / `1.2–1.5` realistic / `1.5–2.0` good / `2.0–3.0` exceptional-but-scrutinize / `>3` overfit. Needs ~200+ trades; live degrades PF ~10–20%. | [TradeZella](https://www.tradezella.com/blog/profit-factor), [CrossTrade](https://crosstrade.io/learn/performance-metrics/profit-factor), [QuantifiedStrategies](https://www.quantifiedstrategies.com/profit-factor/) | +| 9 | Calmar `>1`/`>2`/`>3` three-band scheme as standard | The widely-published convention uses two cuts only — `<1` poor / `1–3` good / `>3` excellent — with no distinct "good at 2". The `1.5–3.0` hedge-fund target has no primary backing. Formula and the Calmar-vs-MAR distinction are solid; the bands are unsourced convention. | [Wikipedia: Calmar ratio](https://en.wikipedia.org/wiki/Calmar_ratio), [CFI](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/calmar-ratio/) | + +## Confirmed as law / exact (used as-is) + +Annualization (mean×P, std/ratio×√P; P=252/52/12), Sharpe-as-t-stat +(`t = SR·√T`), PSR / DSR / E[max SR_N], PBO via CSCV, Omega, Burke, Information & +Treynor ratio formulas, recovery asymmetry `d/(1−d)`, Ulcer Index & UPI, Recovery +Factor formula, Magdon-Ismail E[MaxDD] three-regime asymptotics, VaR/ES +multipliers & the Basel traffic-light, tail ratio (95/5), excess kurtosis, +downside-deviation divisor (total N), risk-of-ruin closed form, Kelly / optimal-f, +gain-to-pain, common-sense ratio, MCPT p-value `(z+1)/(N+1)` (the `+1` appears in +**both** numerator and denominator), the profit-factor↔win-rate identity +`PF=(p·R)/(1−p)`, CPCV path count `φ=(k/N)·C(N,k)`. + +## Confirmed as convention (flagged [C]; verify primary source before hard-coding) + +Information-ratio bands (0.4/0.6/1.0), Calmar / Sortino / SQN / Recovery-Factor / +profit-factor bands, MaxDD acceptability bands, capacity impact-coefficient `Y`, +the live-vs-backtest Sharpe haircut figures, and the Harvey-Liu-Zhu required +t-stat `~3.0` (and `~3.18` with publication-bias adjustment). These are +practitioner rules-of-thumb that vary by source and regime. + +## Known follow-ups for the implementer + +- Verify the **Harvey-Liu-Zhu** required-t and the **live-vs-backtest haircut** + figures against their primary papers before any code hard-codes them. +- Confirm the **MinBTL** closed form `≤ 2·ln(N)/E[max]²` against the + Bailey-Borwein-López de Prado-Zhu paper (the agent flagged the constant).