Brummel/Aura
1
0
Fork 0
Code Issues 33 Pull Requests Actions Packages Projects Releases Wiki Activity

Make project-neutral (drop implementation state); add sourced references

Brummel
2026-06-15 13:55:07 +02:00
parent 04b077be32
commit 9cf1f13547
1 changed files with 121 additions and 186 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Home.md
+121 -186
@@ -1,209 +1,144 @@
# Strategy Analysis & Validation
> Design analysis of aura's strategy-analysis / validation layer: what analyses
> aura plans, what the wider quant-finance landscape offers, and how results are
> professionally classified. This page is the narrative; the formula-level
> reference lives in **[Metric Catalogue](Metric-Catalogue)** and the
> source-checking of every load-bearing number in **[Metric Verification Log](Metric-Verification-Log)**.
> A knowledge base on how trading strategies are analysed, validated, and graded
> in quantitative finance. Project-neutral and durable: it records the *methods,
> formulas, and benchmark thresholds*, not the state of any one implementation.
>
> The formula-level reference is in **[Metric Catalogue](Metric-Catalogue)**; the
> source-checking of every load-bearing number is in
> **[Metric Verification Log](Metric-Verification-Log)**.
**Status:** design groundwork (not an implementation order). Milestone *The World,
part II — orchestration families*.
## Why validation, not just backtesting
## Why this exists
Running a single backtest and reading off the equity curve is commodity — every
quant system does it. The hard, value-adding part is **validation**: deciding
whether an apparent edge is *real and robust* or an artefact of luck and
overfitting. A backtest produces *one* number; validation asks how that number
would hold up under different parameters, different random draws, unseen data, and
the statistics of having tried many variants. This page surveys the analysis
methods that answer that question and how their results are professionally
classified.
aura's differentiator is not the single backtest but the **World** (C21):
orchestrating *families* of harnesses. C12 names the four axes those families run
along — but *how the results of such a family are professionally classified*
(which metrics, which benchmarks, how computed) is not yet specified. Today
`RunMetrics` is exactly three scalars (`crates/aura-engine/src/report.rs:16`) and
`metric_cmp` (`crates/aura-registry/src/lib.rs:83`) knows exactly those three
names. There is **no** tracker issue for a metric catalogue, overfitting
statistics (PBO/DSR), CPCV, or sensitivity scoring. This page fills that gap.
### How to read the thresholds
Verification markers used throughout: **[L]** = law / exact, **[C]** = convention
(no authoritative standard), **[CORR]** = corrected during source verification
(corrected value in use). 23 load-bearing formulas/thresholds were adversarially
verified against primary sources; 9 were corrected — see
[Metric Verification Log](Metric-Verification-Log).
Two kinds of statement appear throughout, and the catalogue marks each:
## aura grounding (read from the working tree)
- **[L] law / exact** — a definition or identity (e.g. the Sharpe ratio formula,
the profit-factor↔win-rate identity). These do not vary.
- **[C] convention** — a practitioner rule-of-thumb (e.g. "Sharpe > 2 is
excellent"). These vary by source, asset class, and market regime; treat them as
relative-comparison guidelines, never as hard pass/fail gates without checking
the primary source.
- **[CORR]** marks a value that was corrected during source verification — see the
[Metric Verification Log](Metric-Verification-Log).
- `RunMetrics` today is exactly `{ total_pips: f64, max_drawdown: f64, exposure_sign_flips: u64 }` (`report.rs:16`).
- `summarize(equity, exposure) -> RunMetrics` (`report.rs:75`) consumes the
**cumulative pip-equity samples** and reduces them to those three scalars. The
per-period return series is `diff(equity)` — **present in the input, discarded
by the reduction.** This is the single most load-bearing observation for the
gap analysis below.
- `RunManifest` already carries `seed: u64` (live, C12 seed-as-input), plus
`commit`, `params`, `window`, `broker` — the full reproducibility manifest
exists today.
- Sweep (axis 1) is built: `GridSpace` + `sweep()` → odometer-ordered
`SweepFamily` (`crates/aura-engine/src/sweep.rs`). Optimize (axis 2) is partial:
`optimize(family, metric)` is a single-objective argmax keyed by one of the
three metric names. Registry (C18) is an append-only JSONL of
`RunReport = (manifest, metrics)`.
### Common inputs
## 1. What aura already plans — the four C12 orchestration axes
Most metrics are computed from one of a few generic inputs, defined once here and
referenced throughout the catalogue:
Each axis is a *family generator* over the atomic unit `(frozen topology +
param-set + data-window + RNG-seed) → deterministic run → metrics`. C1
determinism makes every cell reproducible from its `RunManifest`; C1 disjointness
makes the family embarrassingly parallel.
- **per-period return** `r` — the return of one clock period; the first difference
of the cumulative equity curve (`r_t = E_t − E_{t-1}`).
- **equity curve** `E` — the running cumulative P&L (or cumulative return).
- **trade list** — the set of closed trades, each with entry/exit, P&L, and
excursions.
- **benchmark** `b` — a market/benchmark return series, for relative metrics.
- **multi-run distribution** — the set of results across many resamples, seeds,
parameter cells, or folds.
| Axis | Status | Produces | Financial question |
## The four classic analysis axes
These are the standard ways to put a strategy through its paces. Each generates a
*family* of backtests from one strategy and reduces it to a verdict.
| Axis | What it is | Produces | Financial question |
|---|---|---|---|
| **1 — Param-sweep** (grid/random) | **built** (grid: `sweep()`→`SweepFamily`; random = [#52](http://192.168.178.103:3000/Brummel/Aura/issues/52)) | a *surface*: one `RunReport` per grid cell, indexed by param coordinate | "Over the tuning region, where does it make money — broad robust plateau or a single lucky spike?" |
| **2 — Optimization** (argmax metric) | **partial** (`optimize`/`rank_by`) | one winning `SweepPoint` (coordinate + `RunReport`) | "Which configuration is best by my chosen yardstick?" |
| **3 — Walk-forward** (rolling IS-optimize + OOS-test) | **unbuilt** ([#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) `WalkForwardResult`) | stitched OOS equity curve + Walk-Forward Efficiency + parameter-stability + Pardo verdict | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
| **4 — Monte-Carlo** (N seeded realizations) | **unbuilt but enabled** (seed plumbing landed in 0042; [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68) `McFamily`) | a *distribution* of every metric across N seeds: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, MCPT p-value | "Given that one backtest is one draw, what is the distribution, the tail risk, the chance the edge is luck?" |
| **Parameter sweep** (grid / random) | run the strategy across a grid (or random sample) of its tunable parameters | a *surface*: one result per parameter cell | "Where does it make money — a broad robust plateau, or a single lucky spike?" |
| **Optimization** (argmax of a metric) | select the parameter set that maximizes an objective | the winning configuration + its scorecard | "Which configuration is best by my chosen yardstick?" (and: is that yardstick robust, or does it reward luck?) |
| **Walk-forward analysis** | chain *(optimize on an in-sample window)* then *(test on the next out-of-sample window)*, marched forward; stitch the OOS segments | a stitched out-of-sample curve + Walk-Forward Efficiency + parameter stability | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
| **Monte-Carlo analysis** | re-run the strategy across N randomized realizations (resampled returns/trade order, permuted prices, or a synthetic generator) | a *distribution* of every metric: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, a permutation-test p-value | "Given that one backtest is just one draw, what is the distribution, the tail risk, and the chance the edge is luck?" |
**"Partial" at axis 2** means three things: (a) single-objective only (no
constraints like *argmax return s.t. ES ≤ cap*, no Pareto front); (b) argmax over
a pre-enumerated grid, not search over a continuous space (Bayesian/genetic are
explicitly *pluggable*, undesigned); (c) the objective vocabulary is the three
implemented metrics, **none risk-adjusted**. This is the load-bearing weakness:
`argmax total_pips` selects the cell most contaminated by luck and a single
fat-tail trade.
A recurring trap sits inside the optimization axis: maximizing raw net profit
selects the cell most contaminated by luck and a single fat-tail trade. The
professional move is to optimize a *risk-adjusted* objective and read robustness
from the *neighbourhood* of the optimum, not its peak — see §A and §B of the
[Metric Catalogue](Metric-Catalogue).
## 2. The broader analysis landscape (beyond the four axes)
## The broader analysis landscape
The four axes *generate* families; the following are the *reads* and *tests*
layered over them — none has a place in aura today.
The four axes *generate* families of backtests; the following are the *reads* and
*tests* layered over them — the part that separates a rigorous validation from a
pretty equity curve.
- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, MinBTL,
Harvey-Liu-Zhu haircut). A high in-sample Sharpe over N trials is *expected by
chance even at zero true skill* (False Strategy Theorem). These deflate the
observed Sharpe for sample length, non-normality, and the **number of trials** —
turning a sweep/optimize result into a significance verdict. **aura is unusually
well placed: N *is* the sweep cardinality — exactly known and logged per run.**
- **CPCV + purging + embargo** (López de Prado). Fixes the two leakage channels
that make plain k-fold lie on time series (serial correlation; overlapping
labels). Generates many purged/embargoed splits stitched into `φ(N,k)` fully-OOS
*paths* → a *distribution* of Sharpe feeding PBO. Structurally a C12 family
(combinatorial data-window assignment) composed with walk-forward.
- **Sensitivity / robustness.** Reads the sweep *surface* geometrically:
- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, Minimum Backtest
Length, the Harvey-Liu-Zhu haircut). A high in-sample Sharpe over many trials is
*expected by chance even at zero true skill* (the False Strategy Theorem). These
deflate the observed Sharpe for sample length, non-normality, and the **number
of trials**, turning a sweep/optimize result into a significance verdict.
- **Cross-validation for time series (CPCV, purging, embargo).** Plain k-fold
leaks information on serially-correlated, overlapping-label series. Purging and
embargo remove the leakage; Combinatorial Purged Cross-Validation builds many
fully-out-of-sample paths → a *distribution* of the Sharpe rather than one
number.
- **Sensitivity / robustness.** Read the sweep *surface* geometrically:
plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness,
parameter-perturbation dispersion. Robust edge = broad smooth plateau; overfit =
isolated island.
- **Stress / scenario.** Replays curated crisis windows (2008 / 2015-CHF / 2020)
and injects stylized shocks (vol ×3, spread blow-out). A *conditional*
worst-case profile ("if X then Y", not "X is likely").
- **Regime analysis.** Slices performance by market state (bull/bear,
high/low-vol, trend/range), optionally with a detector (HMM). "Where does the
money come from, and does the edge invert in some regime?"
- **Cost / capacity.** The frictionless sim-optimal broker hides execution
reality; sweeping a realistic broker's cost params yields the *break-even cost*;
the square-root impact law yields the *capacity ceiling*.
- **Benchmark-relative significance.** CAPM alpha/beta, information ratio, and the
t-stat of mean return test whether return is skill (alpha) or just market
exposure (beta). Needs a benchmark source ingested alongside the strategy
(C3 one-merge).
- **Null / random-entry sanity.** MCPT and a random-entry null strategy ask "does
it beat random?" — permute the input return series (keep the marginal, destroy
time-ordering) and re-run; the edge must sit in the right tail.
- **Live-vs-backtest divergence.** The one analysis *outside* the sim: diff
realized live performance against the backtest prediction (Sharpe haircut), and
reconcile a deterministic re-run against live fills to separate a code/data bug
from genuine alpha decay.
parameter-perturbation dispersion. A robust edge is a broad smooth plateau; an
overfit one is an isolated island.
- **Stress / scenario analysis.** Replay curated crisis windows (2008, 2015 CHF,
2020) and inject stylized shocks (vol ×3, spread blow-out). A *conditional*
worst-case profile — "if X then Y", not "X is likely".
- **Regime analysis.** Slice performance by market state (bull/bear,
high/low-vol, trend/range). "Where does the money come from, and does the edge
invert in some regime?"
- **Cost / capacity analysis.** A frictionless backtest hides execution reality.
Sweeping cost assumptions yields the *break-even cost*; the square-root impact
law yields the *capacity ceiling* (the AUM before the strategy's own market
impact erodes the edge).
- **Benchmark-relative significance.** CAPM alpha/beta, the information ratio, and
the t-statistic of the mean return test whether the return is skill (alpha) or
just market exposure (beta).
- **Null / random-entry sanity.** A permutation test or a random-entry null
strategy asks "does it beat random?" — permute the input return series (keep the
marginal, destroy the time-ordering an edge would exploit) and re-run; the real
result must sit in the right tail.
- **Live-vs-backtest divergence.** The one analysis *outside* the simulation: diff
realized live performance against the backtest prediction (the Sharpe haircut) to
separate genuine alpha decay from a code/data bug.
## 3. Per-tool result classification
## Where the detail lives
The formula-level reference — for each analysis tool and each metric family: exact
formula, what aura datum it is computed from, and the conventional benchmark
threshold — lives in **[Metric Catalogue](Metric-Catalogue)**. Throughout it:
**per-period return** `r_t = pips[t] - pips[t-1]` = first difference of the
sim-optimal broker's cumulative pip-equity; **trade list** = derived from the C7
position-event table. Pip returns are additive, so `total_pips = sum(r_t)`.
- **[Metric Catalogue](Metric-Catalogue)** — for each analysis tool and metric
family: exact formula, what input it is computed from, and the conventional
benchmark threshold.
- **[Metric Verification Log](Metric-Verification-Log)** — how the formulas and
thresholds were source-checked, which were corrected, and the citations.
## 4. Mapping to aura's data model & the gaps
*This knowledge base was compiled from the quant-finance literature and
adversarially source-checked. Primary references include Bailey & López de Prado
(PSR / DSR / PBO / CPCV), Pardo (walk-forward), Magdon-Ismail et al. (expected
maximum drawdown), the Basel/FRTB market-risk standards (VaR / ES), and Bacon
(drawdown metrics). Per-topic citations are listed on each page.*
### What aura emits today vs what the catalogue needs
## References
- **total_pips** = `sum(r_t)` = total return in pips, and the numerator of
Recovery Factor, Calmar, MAR, UPI, Pain ratio. **Already emitted.**
- **max_drawdown** = the denominator of Calmar, MAR, Recovery Factor. **Already
emitted** — Recovery Factor is literally `total_pips / max_drawdown`, computable
with **zero new inputs**.
- **exposure_sign_flips** = coarse turnover proxy; true turnover = `sum|Δexposure|`.
**Partially covered.**
Overview-level external references (each verified reachable and on-topic). The
full per-metric citation list is in the
[Metric Catalogue → References](Metric-Catalogue#references); corrected formulas
and thresholds are sourced in the [Metric Verification Log](Metric-Verification-Log).
### New inputs the rest of the catalogue requires (dependency order)
1. **The per-period return series `r_t = diff(E)`.** The single highest-leverage
gap. `summarize` *already receives the full cumulative-equity vector* and
throws everything but its last value, its peak-trough, and the sign-flip count
away. Exposing `r_t` (or retaining the equity vector as a recorded stream/sink
per C18) unlocks — with no engine change beyond a sink and O(T) reductions —
essentially the entire single-run scorecard: Sharpe, Sortino, volatility, CAGR,
Omega, skewness, excess kurtosis, downside deviation, historical & parametric
VaR/ES, tail ratio, Ulcer/Pain indices + their ratios, gain-to-pain,
common-sense ratio, the drawdown-episode table, and the PSR/DSR moment inputs.
2. **The derived position-event / trade table (C7/C10).** First difference of the
exposure state → `(event_ts, action, position_id, volume)`; pairing
entries/exits yields per-trade `(direction, entry_ts, exit_ts, pnl_pips,
mae_pips, mfe_pips, holding_cycles)`. Unlocks the trade-level family. **This
layer is explicitly "deferred, derived" in C10 — building it is a contract
realization, not a new contract.**
3. **Multi-run distributions (the World family-level reducer).** MC-over-seeds,
sweep-cell, WFA-fold collections. Unlocks the MC tail family, WFE / IS-OOS gap
/ stitched-OOS curve, and the overfitting layer (DSR needs V[SR_n] across cells
+ effective N; PBO/CPCV need the cell×time matrix M). Seed plumbing is live
([#66](http://192.168.178.103:3000/Brummel/Aura/issues/66)); generators and
reducers are not.
4. **Metadata channels (beside the hot path, C4/C7).** A benchmark/market stream
at C3 (IR, Treynor, alpha/beta); a regime-label stream (recorded HMM, C6); an
ADV channel + realistic broker with cost params; per-observation label-span
`(t0,t1)` (CPCV purging); and `periods_per_year` from the clock.
### Minimal additions with the most leverage
- **(M0) Recovery Factor — zero new inputs.** `total_pips / max_drawdown`.
- **(M1) Retain/expose `r_t`** (a returns sink + moment/quantile/drawdown
reductions). One sink + pure O(T) reductions unlock ~20 single-run metrics.
Highest value per unit work; touches no C1/C2/C8 invariant.
- **(M2) The trade segmenter over the C7 position-event table.** Unlocks the whole
trade-level family.
- **(M3) The World family-level reducer over multi-run collections.** Unlocks MC
tail stats, WFA, and the DSR/PBO/CPCV overfitting layer — for which aura is
uniquely positioned (N = sweep cardinality, known and logged).
**Registry note:** `metric_cmp` (`registry/src/lib.rs:83`) hard-codes the three
known names + their better-direction. Each new scalar metric needs an entry there
(name → kind, better-direction) to be optimizable/rankable — small but real (the
optimizer's objective vocabulary *is* this match arm).
## 5. Recommended layering / sequencing
| Tranche | Scope | Depends on | Substance |
|---|---|---|---|
| **1 — single-run scorecard** | `r_t` (M1) + Recovery Factor (M0); emit Sharpe, Sortino, volatility, CAGR, Calmar, MAR, Recovery Factor, Ulcer/Pain + ratios, Omega, skew, excess kurtosis, downside dev, VaR/ES, tail ratio, gain-to-pain, common-sense, drawdown-episode table | nothing unbuilt (reuses axis 1 as-is) | pure O(T) functions of a vector aura already computes and discards; they are the numerators/denominators every later layer reuses; immediately fixes the axis-2 objective problem — `optimize` can argmax a *defensible* yardstick instead of raw `total_pips` |
| **2 — trade-level** | win rate, profit factor, expectancy, payoff + PF↔win-rate identity, SQN, MAE/MFE, trade efficiency, holding period, true turnover, trade-based risk-of-ruin/Kelly | the C7 trade table (M2) | a genuinely new table; answers a *different* question (per-trade outcome distribution) |
| **3 — multi-run orchestration** | **MC first** (cheapest unbuilt axis; seed plumbing live) → MC tail family + MCPT null p-value + the assumption-free MaxDD-bias cure; then **WFA** (composes axes 1-2-4) | the World family-level reducer (M3) | builds the unbuilt axes 3 & 4 |
| **4 — overfitting / significance** | DSR, PBO, CPCV, MinBTL, Harvey-Liu-Zhu | tranche-3 orchestration (DSR needs V[SR_n]+effective N; PBO/CPCV need matrix M) | the capstone — but aura is *uniquely* positioned: C1 determinism makes N exact and every trial logged. **Rule: once it exists, never report an argmax winner without its deflated significance** |
Cross-cutting metadata (benchmark stream, regime labels, ADV + realistic-broker
cost params, label spans for CPCV) seeps in when the consuming tool is built —
inputs to specific tranches, not a standalone phase.
## Recommended next steps (tracker)
1. **New issue: single-run scorecard (Tranche 1)** — `r_t` sink + risk-adjusted +
drawdown + tail metrics as O(T) reductions; extend `RunMetrics` + `metric_cmp`.
Depends on nothing unbuilt; startable immediately.
2. **New issue: position-event / trade table (C10 realization) + trade metrics
(Tranche 2).**
3. Extend existing [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68)
(MC family) / [#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) (WFA)
with the distribution reducers/metrics specified in the catalogue (Tranche 3).
4. **New issue: overfitting significance layer (DSR/PBO/CPCV) (Tranche 4)** — the
capstone; aura is uniquely positioned for it.
---
*Provenance: internal recon (ledger C10/C12/C18–C23, `aura-engine`/`aura-registry`
code, open issues) + a multi-stage research workflow (10 topic agents → 23
adversarially-verified formulas/thresholds → synthesis, all read-only with web
fetch). Sources include Bailey/López de Prado (PSR/DSR/PBO/CPCV), Pardo (WFA),
Magdon-Ismail (E[MaxDD]), Basel/FRTB (VaR/ES), Bacon (Pain/Ulcer). See
[Metric Verification Log](Metric-Verification-Log) for the corrections and
citations.*
- **Analysis axes:**
[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) ·
[Monte-Carlo method](https://en.wikipedia.org/wiki/Monte_Carlo_method) ·
[Permutation test](https://en.wikipedia.org/wiki/Permutation_test) ·
[Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis)
- **Overfitting & validation statistics:**
[Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) ·
[Probability of Backtest Overfitting / CSCV](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) ·
[Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) ·
[Pseudo-mathematics and financial charlatanism (Bailey et al.)](https://scholarworks.wmich.edu/math_pubs/42/) ·
[López de Prado, Advances in Financial ML](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml)
- **Risk & benchmark-relative:**
[Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) ·
[Expected Shortfall](https://en.wikipedia.org/wiki/Expected_shortfall) ·
[Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) ·
[CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model)