Strategy Analysis & Validation
A knowledge base on how trading strategies are analysed, validated, and graded in quantitative finance. Project-neutral and durable: it records the methods, formulas, and benchmark thresholds, not the state of any one implementation.
Part of the Trading-System Engineering & Strategy Validation wiki; the sibling subject area is Backtesting Engine Architecture (how the engine that produces these results is built). The formula-level reference is in Metric Catalogue; the source-checking of every load-bearing number is in Metric Verification Log.
Why validation, not just backtesting
Running a single backtest and reading off the equity curve is commodity — every quant system does it. The hard, value-adding part is validation: deciding whether an apparent edge is real and robust or an artefact of luck and overfitting. A backtest produces one number; validation asks how that number would hold up under different parameters, different random draws, unseen data, and the statistics of having tried many variants. This page surveys the analysis methods that answer that question and how their results are professionally classified.
How to read the thresholds
Two kinds of statement appear throughout, and the catalogue marks each:
- [L] law / exact — a definition or identity (e.g. the Sharpe ratio formula, the profit-factor↔win-rate identity). These do not vary.
- [C] convention — a practitioner rule-of-thumb (e.g. "Sharpe > 2 is excellent"). These vary by source, asset class, and market regime; treat them as relative-comparison guidelines, never as hard pass/fail gates without checking the primary source.
- [CORR] marks a value that was corrected during source verification — see the Metric Verification Log.
Common inputs
Most metrics are computed from one of a few generic inputs, defined once here and referenced throughout the catalogue:
- per-period return
r— the return of one clock period; the first difference of the cumulative equity curve (r_t = E_t − E_{t-1}). - equity curve
E— the running cumulative P&L (or cumulative return). - trade list — the set of closed trades, each with entry/exit, P&L, and excursions.
- benchmark
b— a market/benchmark return series, for relative metrics. - multi-run distribution — the set of results across many resamples, seeds, parameter cells, or folds.
The four classic analysis axes
These are the standard ways to put a strategy through its paces. Each generates a family of backtests from one strategy and reduces it to a verdict. The engine-side counterpart — how a system constructs and runs these families efficiently — is in Determinism and Reproducibility (Monte-Carlo as a sweep over seeds) and Graph Compilation and Optimization (computing a sweep-invariant sub-graph once).
| Axis | What it is | Produces | Financial question |
|---|---|---|---|
| Parameter sweep (grid / random) | run the strategy across a grid (or random sample) of its tunable parameters | a surface: one result per parameter cell | "Where does it make money — a broad robust plateau, or a single lucky spike?" |
| Optimization (argmax of a metric) | select the parameter set that maximizes an objective | the winning configuration + its scorecard | "Which configuration is best by my chosen yardstick?" (and: is that yardstick robust, or does it reward luck?) |
| Walk-forward analysis | chain (optimize on an in-sample window) then (test on the next out-of-sample window), marched forward; stitch the OOS segments | a stitched out-of-sample curve + Walk-Forward Efficiency + parameter stability | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
| Monte-Carlo analysis | re-run the strategy across N randomized realizations (resampled returns/trade order, permuted prices, or a synthetic generator) | a distribution of every metric: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, a permutation-test p-value | "Given that one backtest is just one draw, what is the distribution, the tail risk, and the chance the edge is luck?" |
A recurring trap sits inside the optimization axis: maximizing raw net profit selects the cell most contaminated by luck and a single fat-tail trade. The professional move is to optimize a risk-adjusted objective and read robustness from the neighbourhood of the optimum, not its peak — see §A and §B of the Metric Catalogue.
The broader analysis landscape
The four axes generate families of backtests; the following are the reads and tests layered over them — the part that separates a rigorous validation from a pretty equity curve.
- Overfitting / selection-bias statistics (PSR → DSR, PBO, Minimum Backtest Length, the Harvey-Liu-Zhu haircut). A high in-sample Sharpe over many trials is expected by chance even at zero true skill (the False Strategy Theorem). These deflate the observed Sharpe for sample length, non-normality, and the number of trials, turning a sweep/optimize result into a significance verdict.
- Cross-validation for time series (CPCV, purging, embargo). Plain k-fold leaks information on serially-correlated, overlapping-label series. Purging and embargo remove the leakage; Combinatorial Purged Cross-Validation builds many fully-out-of-sample paths → a distribution of the Sharpe rather than one number.
- Sensitivity / robustness. Read the sweep surface geometrically: plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness, parameter-perturbation dispersion. A robust edge is a broad smooth plateau; an overfit one is an isolated island.
- Stress / scenario analysis. Replay curated crisis windows (2008, 2015 CHF, 2020) and inject stylized shocks (vol ×3, spread blow-out). A conditional worst-case profile — "if X then Y", not "X is likely".
- Regime analysis. Slice performance by market state (bull/bear, high/low-vol, trend/range). "Where does the money come from, and does the edge invert in some regime?"
- Cost / capacity analysis. A frictionless backtest hides execution reality. Sweeping cost assumptions yields the break-even cost; the square-root impact law yields the capacity ceiling (the AUM before the strategy's own market impact erodes the edge). The frictionless-vs-realistic distinction is detailed in Signal, Exposure, and Execution.
- Benchmark-relative significance. CAPM alpha/beta, the information ratio, and the t-statistic of the mean return test whether the return is skill (alpha) or just market exposure (beta).
- Null / random-entry sanity. A permutation test or a random-entry null strategy asks "does it beat random?" — permute the input return series (keep the marginal, destroy the time-ordering an edge would exploit) and re-run; the real result must sit in the right tail.
- Live-vs-backtest divergence. The one analysis outside the simulation: diff realized live performance against the backtest prediction (the Sharpe haircut) to separate genuine alpha decay from a code/data bug.
Where the detail lives
- Metric Catalogue — for each analysis tool and metric family: exact formula, what input it is computed from, and the conventional benchmark threshold.
- Metric Verification Log — how the formulas and thresholds were source-checked, which were corrected, and the citations.
This knowledge base was compiled from the quant-finance literature and adversarially source-checked. Primary references include Bailey & López de Prado (PSR / DSR / PBO / CPCV), Pardo (walk-forward), Magdon-Ismail et al. (expected maximum drawdown), the Basel/FRTB market-risk standards (VaR / ES), and Bacon (drawdown metrics). Per-topic citations are listed on each page.
References
Overview-level external references (each verified reachable and on-topic). The full per-metric citation list is in the Metric Catalogue → References; corrected formulas and thresholds are sourced in the Metric Verification Log.
- Analysis axes: Walk-forward optimization · Monte-Carlo method · Permutation test · Sensitivity analysis
- Overfitting & validation statistics: Deflated Sharpe Ratio & False Strategy Theorem · Probability of Backtest Overfitting / CSCV · Purged cross-validation (CPCV, purging, embargo) · Pseudo-mathematics and financial charlatanism (Bailey et al.) · López de Prado, Advances in Financial ML
- Risk & benchmark-relative: Value-at-Risk · Expected Shortfall · Stress testing (financial) · CAPM (alpha/beta)
Backtesting Engine Architecture
- Architecture overview
- Event-Driven Backtesting
- Look-Ahead Bias & Causality
- Determinism & Reproducibility
- Reactive Streaming Dataflow
- Columnar Data Layout
- Signal, Exposure & Execution
- Graph Compilation & Optimization
- Authoring & Deployment Lifecycle
Strategy Analysis & Validation