diff --git a/Home.md b/Home.md index 25508da..167019f 100644 --- a/Home.md +++ b/Home.md @@ -1,209 +1,144 @@ # Strategy Analysis & Validation -> Design analysis of aura's strategy-analysis / validation layer: what analyses -> aura plans, what the wider quant-finance landscape offers, and how results are -> professionally classified. This page is the narrative; the formula-level -> reference lives in **[Metric Catalogue](Metric-Catalogue)** and the -> source-checking of every load-bearing number in **[Metric Verification Log](Metric-Verification-Log)**. +> A knowledge base on how trading strategies are analysed, validated, and graded +> in quantitative finance. Project-neutral and durable: it records the *methods, +> formulas, and benchmark thresholds*, not the state of any one implementation. +> +> The formula-level reference is in **[Metric Catalogue](Metric-Catalogue)**; the +> source-checking of every load-bearing number is in +> **[Metric Verification Log](Metric-Verification-Log)**. -**Status:** design groundwork (not an implementation order). Milestone *The World, -part II — orchestration families*. +## Why validation, not just backtesting -## Why this exists +Running a single backtest and reading off the equity curve is commodity — every +quant system does it. The hard, value-adding part is **validation**: deciding +whether an apparent edge is *real and robust* or an artefact of luck and +overfitting. A backtest produces *one* number; validation asks how that number +would hold up under different parameters, different random draws, unseen data, and +the statistics of having tried many variants. This page surveys the analysis +methods that answer that question and how their results are professionally +classified. -aura's differentiator is not the single backtest but the **World** (C21): -orchestrating *families* of harnesses. C12 names the four axes those families run -along — but *how the results of such a family are professionally classified* -(which metrics, which benchmarks, how computed) is not yet specified. Today -`RunMetrics` is exactly three scalars (`crates/aura-engine/src/report.rs:16`) and -`metric_cmp` (`crates/aura-registry/src/lib.rs:83`) knows exactly those three -names. There is **no** tracker issue for a metric catalogue, overfitting -statistics (PBO/DSR), CPCV, or sensitivity scoring. This page fills that gap. +### How to read the thresholds -Verification markers used throughout: **[L]** = law / exact, **[C]** = convention -(no authoritative standard), **[CORR]** = corrected during source verification -(corrected value in use). 23 load-bearing formulas/thresholds were adversarially -verified against primary sources; 9 were corrected — see -[Metric Verification Log](Metric-Verification-Log). +Two kinds of statement appear throughout, and the catalogue marks each: -## aura grounding (read from the working tree) +- **[L] law / exact** — a definition or identity (e.g. the Sharpe ratio formula, + the profit-factor↔win-rate identity). These do not vary. +- **[C] convention** — a practitioner rule-of-thumb (e.g. "Sharpe > 2 is + excellent"). These vary by source, asset class, and market regime; treat them as + relative-comparison guidelines, never as hard pass/fail gates without checking + the primary source. +- **[CORR]** marks a value that was corrected during source verification — see the + [Metric Verification Log](Metric-Verification-Log). -- `RunMetrics` today is exactly `{ total_pips: f64, max_drawdown: f64, exposure_sign_flips: u64 }` (`report.rs:16`). -- `summarize(equity, exposure) -> RunMetrics` (`report.rs:75`) consumes the - **cumulative pip-equity samples** and reduces them to those three scalars. The - per-period return series is `diff(equity)` — **present in the input, discarded - by the reduction.** This is the single most load-bearing observation for the - gap analysis below. -- `RunManifest` already carries `seed: u64` (live, C12 seed-as-input), plus - `commit`, `params`, `window`, `broker` — the full reproducibility manifest - exists today. -- Sweep (axis 1) is built: `GridSpace` + `sweep()` → odometer-ordered - `SweepFamily` (`crates/aura-engine/src/sweep.rs`). Optimize (axis 2) is partial: - `optimize(family, metric)` is a single-objective argmax keyed by one of the - three metric names. Registry (C18) is an append-only JSONL of - `RunReport = (manifest, metrics)`. +### Common inputs -## 1. What aura already plans — the four C12 orchestration axes +Most metrics are computed from one of a few generic inputs, defined once here and +referenced throughout the catalogue: -Each axis is a *family generator* over the atomic unit `(frozen topology + -param-set + data-window + RNG-seed) → deterministic run → metrics`. C1 -determinism makes every cell reproducible from its `RunManifest`; C1 disjointness -makes the family embarrassingly parallel. +- **per-period return** `r` — the return of one clock period; the first difference + of the cumulative equity curve (`r_t = E_t − E_{t-1}`). +- **equity curve** `E` — the running cumulative P&L (or cumulative return). +- **trade list** — the set of closed trades, each with entry/exit, P&L, and + excursions. +- **benchmark** `b` — a market/benchmark return series, for relative metrics. +- **multi-run distribution** — the set of results across many resamples, seeds, + parameter cells, or folds. -| Axis | Status | Produces | Financial question | +## The four classic analysis axes + +These are the standard ways to put a strategy through its paces. Each generates a +*family* of backtests from one strategy and reduces it to a verdict. + +| Axis | What it is | Produces | Financial question | |---|---|---|---| -| **1 — Param-sweep** (grid/random) | **built** (grid: `sweep()`→`SweepFamily`; random = [#52](http://192.168.178.103:3000/Brummel/Aura/issues/52)) | a *surface*: one `RunReport` per grid cell, indexed by param coordinate | "Over the tuning region, where does it make money — broad robust plateau or a single lucky spike?" | -| **2 — Optimization** (argmax metric) | **partial** (`optimize`/`rank_by`) | one winning `SweepPoint` (coordinate + `RunReport`) | "Which configuration is best by my chosen yardstick?" | -| **3 — Walk-forward** (rolling IS-optimize + OOS-test) | **unbuilt** ([#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) `WalkForwardResult`) | stitched OOS equity curve + Walk-Forward Efficiency + parameter-stability + Pardo verdict | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" | -| **4 — Monte-Carlo** (N seeded realizations) | **unbuilt but enabled** (seed plumbing landed in 0042; [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68) `McFamily`) | a *distribution* of every metric across N seeds: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, MCPT p-value | "Given that one backtest is one draw, what is the distribution, the tail risk, the chance the edge is luck?" | +| **Parameter sweep** (grid / random) | run the strategy across a grid (or random sample) of its tunable parameters | a *surface*: one result per parameter cell | "Where does it make money — a broad robust plateau, or a single lucky spike?" | +| **Optimization** (argmax of a metric) | select the parameter set that maximizes an objective | the winning configuration + its scorecard | "Which configuration is best by my chosen yardstick?" (and: is that yardstick robust, or does it reward luck?) | +| **Walk-forward analysis** | chain *(optimize on an in-sample window)* then *(test on the next out-of-sample window)*, marched forward; stitch the OOS segments | a stitched out-of-sample curve + Walk-Forward Efficiency + parameter stability | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" | +| **Monte-Carlo analysis** | re-run the strategy across N randomized realizations (resampled returns/trade order, permuted prices, or a synthetic generator) | a *distribution* of every metric: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, a permutation-test p-value | "Given that one backtest is just one draw, what is the distribution, the tail risk, and the chance the edge is luck?" | -**"Partial" at axis 2** means three things: (a) single-objective only (no -constraints like *argmax return s.t. ES ≤ cap*, no Pareto front); (b) argmax over -a pre-enumerated grid, not search over a continuous space (Bayesian/genetic are -explicitly *pluggable*, undesigned); (c) the objective vocabulary is the three -implemented metrics, **none risk-adjusted**. This is the load-bearing weakness: -`argmax total_pips` selects the cell most contaminated by luck and a single -fat-tail trade. +A recurring trap sits inside the optimization axis: maximizing raw net profit +selects the cell most contaminated by luck and a single fat-tail trade. The +professional move is to optimize a *risk-adjusted* objective and read robustness +from the *neighbourhood* of the optimum, not its peak — see §A and §B of the +[Metric Catalogue](Metric-Catalogue). -## 2. The broader analysis landscape (beyond the four axes) +## The broader analysis landscape -The four axes *generate* families; the following are the *reads* and *tests* -layered over them — none has a place in aura today. +The four axes *generate* families of backtests; the following are the *reads* and +*tests* layered over them — the part that separates a rigorous validation from a +pretty equity curve. -- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, MinBTL, - Harvey-Liu-Zhu haircut). A high in-sample Sharpe over N trials is *expected by - chance even at zero true skill* (False Strategy Theorem). These deflate the - observed Sharpe for sample length, non-normality, and the **number of trials** — - turning a sweep/optimize result into a significance verdict. **aura is unusually - well placed: N *is* the sweep cardinality — exactly known and logged per run.** -- **CPCV + purging + embargo** (López de Prado). Fixes the two leakage channels - that make plain k-fold lie on time series (serial correlation; overlapping - labels). Generates many purged/embargoed splits stitched into `φ(N,k)` fully-OOS - *paths* → a *distribution* of Sharpe feeding PBO. Structurally a C12 family - (combinatorial data-window assignment) composed with walk-forward. -- **Sensitivity / robustness.** Reads the sweep *surface* geometrically: +- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, Minimum Backtest + Length, the Harvey-Liu-Zhu haircut). A high in-sample Sharpe over many trials is + *expected by chance even at zero true skill* (the False Strategy Theorem). These + deflate the observed Sharpe for sample length, non-normality, and the **number + of trials**, turning a sweep/optimize result into a significance verdict. +- **Cross-validation for time series (CPCV, purging, embargo).** Plain k-fold + leaks information on serially-correlated, overlapping-label series. Purging and + embargo remove the leakage; Combinatorial Purged Cross-Validation builds many + fully-out-of-sample paths → a *distribution* of the Sharpe rather than one + number. +- **Sensitivity / robustness.** Read the sweep *surface* geometrically: plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness, - parameter-perturbation dispersion. Robust edge = broad smooth plateau; overfit = - isolated island. -- **Stress / scenario.** Replays curated crisis windows (2008 / 2015-CHF / 2020) - and injects stylized shocks (vol ×3, spread blow-out). A *conditional* - worst-case profile ("if X then Y", not "X is likely"). -- **Regime analysis.** Slices performance by market state (bull/bear, - high/low-vol, trend/range), optionally with a detector (HMM). "Where does the - money come from, and does the edge invert in some regime?" -- **Cost / capacity.** The frictionless sim-optimal broker hides execution - reality; sweeping a realistic broker's cost params yields the *break-even cost*; - the square-root impact law yields the *capacity ceiling*. -- **Benchmark-relative significance.** CAPM alpha/beta, information ratio, and the - t-stat of mean return test whether return is skill (alpha) or just market - exposure (beta). Needs a benchmark source ingested alongside the strategy - (C3 one-merge). -- **Null / random-entry sanity.** MCPT and a random-entry null strategy ask "does - it beat random?" — permute the input return series (keep the marginal, destroy - time-ordering) and re-run; the edge must sit in the right tail. -- **Live-vs-backtest divergence.** The one analysis *outside* the sim: diff - realized live performance against the backtest prediction (Sharpe haircut), and - reconcile a deterministic re-run against live fills to separate a code/data bug - from genuine alpha decay. + parameter-perturbation dispersion. A robust edge is a broad smooth plateau; an + overfit one is an isolated island. +- **Stress / scenario analysis.** Replay curated crisis windows (2008, 2015 CHF, + 2020) and inject stylized shocks (vol ×3, spread blow-out). A *conditional* + worst-case profile — "if X then Y", not "X is likely". +- **Regime analysis.** Slice performance by market state (bull/bear, + high/low-vol, trend/range). "Where does the money come from, and does the edge + invert in some regime?" +- **Cost / capacity analysis.** A frictionless backtest hides execution reality. + Sweeping cost assumptions yields the *break-even cost*; the square-root impact + law yields the *capacity ceiling* (the AUM before the strategy's own market + impact erodes the edge). +- **Benchmark-relative significance.** CAPM alpha/beta, the information ratio, and + the t-statistic of the mean return test whether the return is skill (alpha) or + just market exposure (beta). +- **Null / random-entry sanity.** A permutation test or a random-entry null + strategy asks "does it beat random?" — permute the input return series (keep the + marginal, destroy the time-ordering an edge would exploit) and re-run; the real + result must sit in the right tail. +- **Live-vs-backtest divergence.** The one analysis *outside* the simulation: diff + realized live performance against the backtest prediction (the Sharpe haircut) to + separate genuine alpha decay from a code/data bug. -## 3. Per-tool result classification +## Where the detail lives -The formula-level reference — for each analysis tool and each metric family: exact -formula, what aura datum it is computed from, and the conventional benchmark -threshold — lives in **[Metric Catalogue](Metric-Catalogue)**. Throughout it: -**per-period return** `r_t = pips[t] - pips[t-1]` = first difference of the -sim-optimal broker's cumulative pip-equity; **trade list** = derived from the C7 -position-event table. Pip returns are additive, so `total_pips = sum(r_t)`. +- **[Metric Catalogue](Metric-Catalogue)** — for each analysis tool and metric + family: exact formula, what input it is computed from, and the conventional + benchmark threshold. +- **[Metric Verification Log](Metric-Verification-Log)** — how the formulas and + thresholds were source-checked, which were corrected, and the citations. -## 4. Mapping to aura's data model & the gaps +*This knowledge base was compiled from the quant-finance literature and +adversarially source-checked. Primary references include Bailey & López de Prado +(PSR / DSR / PBO / CPCV), Pardo (walk-forward), Magdon-Ismail et al. (expected +maximum drawdown), the Basel/FRTB market-risk standards (VaR / ES), and Bacon +(drawdown metrics). Per-topic citations are listed on each page.* -### What aura emits today vs what the catalogue needs +## References -- **total_pips** = `sum(r_t)` = total return in pips, and the numerator of - Recovery Factor, Calmar, MAR, UPI, Pain ratio. **Already emitted.** -- **max_drawdown** = the denominator of Calmar, MAR, Recovery Factor. **Already - emitted** — Recovery Factor is literally `total_pips / max_drawdown`, computable - with **zero new inputs**. -- **exposure_sign_flips** = coarse turnover proxy; true turnover = `sum|Δexposure|`. - **Partially covered.** +Overview-level external references (each verified reachable and on-topic). The +full per-metric citation list is in the +[Metric Catalogue → References](Metric-Catalogue#references); corrected formulas +and thresholds are sourced in the [Metric Verification Log](Metric-Verification-Log). -### New inputs the rest of the catalogue requires (dependency order) - -1. **The per-period return series `r_t = diff(E)`.** The single highest-leverage - gap. `summarize` *already receives the full cumulative-equity vector* and - throws everything but its last value, its peak-trough, and the sign-flip count - away. Exposing `r_t` (or retaining the equity vector as a recorded stream/sink - per C18) unlocks — with no engine change beyond a sink and O(T) reductions — - essentially the entire single-run scorecard: Sharpe, Sortino, volatility, CAGR, - Omega, skewness, excess kurtosis, downside deviation, historical & parametric - VaR/ES, tail ratio, Ulcer/Pain indices + their ratios, gain-to-pain, - common-sense ratio, the drawdown-episode table, and the PSR/DSR moment inputs. -2. **The derived position-event / trade table (C7/C10).** First difference of the - exposure state → `(event_ts, action, position_id, volume)`; pairing - entries/exits yields per-trade `(direction, entry_ts, exit_ts, pnl_pips, - mae_pips, mfe_pips, holding_cycles)`. Unlocks the trade-level family. **This - layer is explicitly "deferred, derived" in C10 — building it is a contract - realization, not a new contract.** -3. **Multi-run distributions (the World family-level reducer).** MC-over-seeds, - sweep-cell, WFA-fold collections. Unlocks the MC tail family, WFE / IS-OOS gap - / stitched-OOS curve, and the overfitting layer (DSR needs V[SR_n] across cells - + effective N; PBO/CPCV need the cell×time matrix M). Seed plumbing is live - ([#66](http://192.168.178.103:3000/Brummel/Aura/issues/66)); generators and - reducers are not. -4. **Metadata channels (beside the hot path, C4/C7).** A benchmark/market stream - at C3 (IR, Treynor, alpha/beta); a regime-label stream (recorded HMM, C6); an - ADV channel + realistic broker with cost params; per-observation label-span - `(t0,t1)` (CPCV purging); and `periods_per_year` from the clock. - -### Minimal additions with the most leverage - -- **(M0) Recovery Factor — zero new inputs.** `total_pips / max_drawdown`. -- **(M1) Retain/expose `r_t`** (a returns sink + moment/quantile/drawdown - reductions). One sink + pure O(T) reductions unlock ~20 single-run metrics. - Highest value per unit work; touches no C1/C2/C8 invariant. -- **(M2) The trade segmenter over the C7 position-event table.** Unlocks the whole - trade-level family. -- **(M3) The World family-level reducer over multi-run collections.** Unlocks MC - tail stats, WFA, and the DSR/PBO/CPCV overfitting layer — for which aura is - uniquely positioned (N = sweep cardinality, known and logged). - -**Registry note:** `metric_cmp` (`registry/src/lib.rs:83`) hard-codes the three -known names + their better-direction. Each new scalar metric needs an entry there -(name → kind, better-direction) to be optimizable/rankable — small but real (the -optimizer's objective vocabulary *is* this match arm). - -## 5. Recommended layering / sequencing - -| Tranche | Scope | Depends on | Substance | -|---|---|---|---| -| **1 — single-run scorecard** | `r_t` (M1) + Recovery Factor (M0); emit Sharpe, Sortino, volatility, CAGR, Calmar, MAR, Recovery Factor, Ulcer/Pain + ratios, Omega, skew, excess kurtosis, downside dev, VaR/ES, tail ratio, gain-to-pain, common-sense, drawdown-episode table | nothing unbuilt (reuses axis 1 as-is) | pure O(T) functions of a vector aura already computes and discards; they are the numerators/denominators every later layer reuses; immediately fixes the axis-2 objective problem — `optimize` can argmax a *defensible* yardstick instead of raw `total_pips` | -| **2 — trade-level** | win rate, profit factor, expectancy, payoff + PF↔win-rate identity, SQN, MAE/MFE, trade efficiency, holding period, true turnover, trade-based risk-of-ruin/Kelly | the C7 trade table (M2) | a genuinely new table; answers a *different* question (per-trade outcome distribution) | -| **3 — multi-run orchestration** | **MC first** (cheapest unbuilt axis; seed plumbing live) → MC tail family + MCPT null p-value + the assumption-free MaxDD-bias cure; then **WFA** (composes axes 1-2-4) | the World family-level reducer (M3) | builds the unbuilt axes 3 & 4 | -| **4 — overfitting / significance** | DSR, PBO, CPCV, MinBTL, Harvey-Liu-Zhu | tranche-3 orchestration (DSR needs V[SR_n]+effective N; PBO/CPCV need matrix M) | the capstone — but aura is *uniquely* positioned: C1 determinism makes N exact and every trial logged. **Rule: once it exists, never report an argmax winner without its deflated significance** | - -Cross-cutting metadata (benchmark stream, regime labels, ADV + realistic-broker -cost params, label spans for CPCV) seeps in when the consuming tool is built — -inputs to specific tranches, not a standalone phase. - -## Recommended next steps (tracker) - -1. **New issue: single-run scorecard (Tranche 1)** — `r_t` sink + risk-adjusted + - drawdown + tail metrics as O(T) reductions; extend `RunMetrics` + `metric_cmp`. - Depends on nothing unbuilt; startable immediately. -2. **New issue: position-event / trade table (C10 realization) + trade metrics - (Tranche 2).** -3. Extend existing [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68) - (MC family) / [#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) (WFA) - with the distribution reducers/metrics specified in the catalogue (Tranche 3). -4. **New issue: overfitting significance layer (DSR/PBO/CPCV) (Tranche 4)** — the - capstone; aura is uniquely positioned for it. - ---- - -*Provenance: internal recon (ledger C10/C12/C18–C23, `aura-engine`/`aura-registry` -code, open issues) + a multi-stage research workflow (10 topic agents → 23 -adversarially-verified formulas/thresholds → synthesis, all read-only with web -fetch). Sources include Bailey/López de Prado (PSR/DSR/PBO/CPCV), Pardo (WFA), -Magdon-Ismail (E[MaxDD]), Basel/FRTB (VaR/ES), Bacon (Pain/Ulcer). See -[Metric Verification Log](Metric-Verification-Log) for the corrections and -citations.* +- **Analysis axes:** + [Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) · + [Monte-Carlo method](https://en.wikipedia.org/wiki/Monte_Carlo_method) · + [Permutation test](https://en.wikipedia.org/wiki/Permutation_test) · + [Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis) +- **Overfitting & validation statistics:** + [Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) · + [Probability of Backtest Overfitting / CSCV](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) · + [Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) · + [Pseudo-mathematics and financial charlatanism (Bailey et al.)](https://scholarworks.wmich.edu/math_pubs/42/) · + [López de Prado, Advances in Financial ML](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml) +- **Risk & benchmark-relative:** + [Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) · + [Expected Shortfall](https://en.wikipedia.org/wiki/Expected_shortfall) · + [Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) · + [CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model)