Add strategy-analysis design analysis (overview)
+209
@@ -0,0 +1,209 @@
|
||||
# Strategy Analysis & Validation
|
||||
|
||||
> Design analysis of aura's strategy-analysis / validation layer: what analyses
|
||||
> aura plans, what the wider quant-finance landscape offers, and how results are
|
||||
> professionally classified. This page is the narrative; the formula-level
|
||||
> reference lives in **[Metric Catalogue](Metric-Catalogue)** and the
|
||||
> source-checking of every load-bearing number in **[Metric Verification Log](Metric-Verification-Log)**.
|
||||
|
||||
**Status:** design groundwork (not an implementation order). Milestone *The World,
|
||||
part II — orchestration families*.
|
||||
|
||||
## Why this exists
|
||||
|
||||
aura's differentiator is not the single backtest but the **World** (C21):
|
||||
orchestrating *families* of harnesses. C12 names the four axes those families run
|
||||
along — but *how the results of such a family are professionally classified*
|
||||
(which metrics, which benchmarks, how computed) is not yet specified. Today
|
||||
`RunMetrics` is exactly three scalars (`crates/aura-engine/src/report.rs:16`) and
|
||||
`metric_cmp` (`crates/aura-registry/src/lib.rs:83`) knows exactly those three
|
||||
names. There is **no** tracker issue for a metric catalogue, overfitting
|
||||
statistics (PBO/DSR), CPCV, or sensitivity scoring. This page fills that gap.
|
||||
|
||||
Verification markers used throughout: **[L]** = law / exact, **[C]** = convention
|
||||
(no authoritative standard), **[CORR]** = corrected during source verification
|
||||
(corrected value in use). 23 load-bearing formulas/thresholds were adversarially
|
||||
verified against primary sources; 9 were corrected — see
|
||||
[Metric Verification Log](Metric-Verification-Log).
|
||||
|
||||
## aura grounding (read from the working tree)
|
||||
|
||||
- `RunMetrics` today is exactly `{ total_pips: f64, max_drawdown: f64, exposure_sign_flips: u64 }` (`report.rs:16`).
|
||||
- `summarize(equity, exposure) -> RunMetrics` (`report.rs:75`) consumes the
|
||||
**cumulative pip-equity samples** and reduces them to those three scalars. The
|
||||
per-period return series is `diff(equity)` — **present in the input, discarded
|
||||
by the reduction.** This is the single most load-bearing observation for the
|
||||
gap analysis below.
|
||||
- `RunManifest` already carries `seed: u64` (live, C12 seed-as-input), plus
|
||||
`commit`, `params`, `window`, `broker` — the full reproducibility manifest
|
||||
exists today.
|
||||
- Sweep (axis 1) is built: `GridSpace` + `sweep()` → odometer-ordered
|
||||
`SweepFamily` (`crates/aura-engine/src/sweep.rs`). Optimize (axis 2) is partial:
|
||||
`optimize(family, metric)` is a single-objective argmax keyed by one of the
|
||||
three metric names. Registry (C18) is an append-only JSONL of
|
||||
`RunReport = (manifest, metrics)`.
|
||||
|
||||
## 1. What aura already plans — the four C12 orchestration axes
|
||||
|
||||
Each axis is a *family generator* over the atomic unit `(frozen topology +
|
||||
param-set + data-window + RNG-seed) → deterministic run → metrics`. C1
|
||||
determinism makes every cell reproducible from its `RunManifest`; C1 disjointness
|
||||
makes the family embarrassingly parallel.
|
||||
|
||||
| Axis | Status | Produces | Financial question |
|
||||
|---|---|---|---|
|
||||
| **1 — Param-sweep** (grid/random) | **built** (grid: `sweep()`→`SweepFamily`; random = [#52](http://192.168.178.103:3000/Brummel/Aura/issues/52)) | a *surface*: one `RunReport` per grid cell, indexed by param coordinate | "Over the tuning region, where does it make money — broad robust plateau or a single lucky spike?" |
|
||||
| **2 — Optimization** (argmax metric) | **partial** (`optimize`/`rank_by`) | one winning `SweepPoint` (coordinate + `RunReport`) | "Which configuration is best by my chosen yardstick?" |
|
||||
| **3 — Walk-forward** (rolling IS-optimize + OOS-test) | **unbuilt** ([#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) `WalkForwardResult`) | stitched OOS equity curve + Walk-Forward Efficiency + parameter-stability + Pardo verdict | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
|
||||
| **4 — Monte-Carlo** (N seeded realizations) | **unbuilt but enabled** (seed plumbing landed in 0042; [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68) `McFamily`) | a *distribution* of every metric across N seeds: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, MCPT p-value | "Given that one backtest is one draw, what is the distribution, the tail risk, the chance the edge is luck?" |
|
||||
|
||||
**"Partial" at axis 2** means three things: (a) single-objective only (no
|
||||
constraints like *argmax return s.t. ES ≤ cap*, no Pareto front); (b) argmax over
|
||||
a pre-enumerated grid, not search over a continuous space (Bayesian/genetic are
|
||||
explicitly *pluggable*, undesigned); (c) the objective vocabulary is the three
|
||||
implemented metrics, **none risk-adjusted**. This is the load-bearing weakness:
|
||||
`argmax total_pips` selects the cell most contaminated by luck and a single
|
||||
fat-tail trade.
|
||||
|
||||
## 2. The broader analysis landscape (beyond the four axes)
|
||||
|
||||
The four axes *generate* families; the following are the *reads* and *tests*
|
||||
layered over them — none has a place in aura today.
|
||||
|
||||
- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, MinBTL,
|
||||
Harvey-Liu-Zhu haircut). A high in-sample Sharpe over N trials is *expected by
|
||||
chance even at zero true skill* (False Strategy Theorem). These deflate the
|
||||
observed Sharpe for sample length, non-normality, and the **number of trials** —
|
||||
turning a sweep/optimize result into a significance verdict. **aura is unusually
|
||||
well placed: N *is* the sweep cardinality — exactly known and logged per run.**
|
||||
- **CPCV + purging + embargo** (López de Prado). Fixes the two leakage channels
|
||||
that make plain k-fold lie on time series (serial correlation; overlapping
|
||||
labels). Generates many purged/embargoed splits stitched into `φ(N,k)` fully-OOS
|
||||
*paths* → a *distribution* of Sharpe feeding PBO. Structurally a C12 family
|
||||
(combinatorial data-window assignment) composed with walk-forward.
|
||||
- **Sensitivity / robustness.** Reads the sweep *surface* geometrically:
|
||||
plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness,
|
||||
parameter-perturbation dispersion. Robust edge = broad smooth plateau; overfit =
|
||||
isolated island.
|
||||
- **Stress / scenario.** Replays curated crisis windows (2008 / 2015-CHF / 2020)
|
||||
and injects stylized shocks (vol ×3, spread blow-out). A *conditional*
|
||||
worst-case profile ("if X then Y", not "X is likely").
|
||||
- **Regime analysis.** Slices performance by market state (bull/bear,
|
||||
high/low-vol, trend/range), optionally with a detector (HMM). "Where does the
|
||||
money come from, and does the edge invert in some regime?"
|
||||
- **Cost / capacity.** The frictionless sim-optimal broker hides execution
|
||||
reality; sweeping a realistic broker's cost params yields the *break-even cost*;
|
||||
the square-root impact law yields the *capacity ceiling*.
|
||||
- **Benchmark-relative significance.** CAPM alpha/beta, information ratio, and the
|
||||
t-stat of mean return test whether return is skill (alpha) or just market
|
||||
exposure (beta). Needs a benchmark source ingested alongside the strategy
|
||||
(C3 one-merge).
|
||||
- **Null / random-entry sanity.** MCPT and a random-entry null strategy ask "does
|
||||
it beat random?" — permute the input return series (keep the marginal, destroy
|
||||
time-ordering) and re-run; the edge must sit in the right tail.
|
||||
- **Live-vs-backtest divergence.** The one analysis *outside* the sim: diff
|
||||
realized live performance against the backtest prediction (Sharpe haircut), and
|
||||
reconcile a deterministic re-run against live fills to separate a code/data bug
|
||||
from genuine alpha decay.
|
||||
|
||||
## 3. Per-tool result classification
|
||||
|
||||
The formula-level reference — for each analysis tool and each metric family: exact
|
||||
formula, what aura datum it is computed from, and the conventional benchmark
|
||||
threshold — lives in **[Metric Catalogue](Metric-Catalogue)**. Throughout it:
|
||||
**per-period return** `r_t = pips[t] - pips[t-1]` = first difference of the
|
||||
sim-optimal broker's cumulative pip-equity; **trade list** = derived from the C7
|
||||
position-event table. Pip returns are additive, so `total_pips = sum(r_t)`.
|
||||
|
||||
## 4. Mapping to aura's data model & the gaps
|
||||
|
||||
### What aura emits today vs what the catalogue needs
|
||||
|
||||
- **total_pips** = `sum(r_t)` = total return in pips, and the numerator of
|
||||
Recovery Factor, Calmar, MAR, UPI, Pain ratio. **Already emitted.**
|
||||
- **max_drawdown** = the denominator of Calmar, MAR, Recovery Factor. **Already
|
||||
emitted** — Recovery Factor is literally `total_pips / max_drawdown`, computable
|
||||
with **zero new inputs**.
|
||||
- **exposure_sign_flips** = coarse turnover proxy; true turnover = `sum|Δexposure|`.
|
||||
**Partially covered.**
|
||||
|
||||
### New inputs the rest of the catalogue requires (dependency order)
|
||||
|
||||
1. **The per-period return series `r_t = diff(E)`.** The single highest-leverage
|
||||
gap. `summarize` *already receives the full cumulative-equity vector* and
|
||||
throws everything but its last value, its peak-trough, and the sign-flip count
|
||||
away. Exposing `r_t` (or retaining the equity vector as a recorded stream/sink
|
||||
per C18) unlocks — with no engine change beyond a sink and O(T) reductions —
|
||||
essentially the entire single-run scorecard: Sharpe, Sortino, volatility, CAGR,
|
||||
Omega, skewness, excess kurtosis, downside deviation, historical & parametric
|
||||
VaR/ES, tail ratio, Ulcer/Pain indices + their ratios, gain-to-pain,
|
||||
common-sense ratio, the drawdown-episode table, and the PSR/DSR moment inputs.
|
||||
2. **The derived position-event / trade table (C7/C10).** First difference of the
|
||||
exposure state → `(event_ts, action, position_id, volume)`; pairing
|
||||
entries/exits yields per-trade `(direction, entry_ts, exit_ts, pnl_pips,
|
||||
mae_pips, mfe_pips, holding_cycles)`. Unlocks the trade-level family. **This
|
||||
layer is explicitly "deferred, derived" in C10 — building it is a contract
|
||||
realization, not a new contract.**
|
||||
3. **Multi-run distributions (the World family-level reducer).** MC-over-seeds,
|
||||
sweep-cell, WFA-fold collections. Unlocks the MC tail family, WFE / IS-OOS gap
|
||||
/ stitched-OOS curve, and the overfitting layer (DSR needs V[SR_n] across cells
|
||||
+ effective N; PBO/CPCV need the cell×time matrix M). Seed plumbing is live
|
||||
([#66](http://192.168.178.103:3000/Brummel/Aura/issues/66)); generators and
|
||||
reducers are not.
|
||||
4. **Metadata channels (beside the hot path, C4/C7).** A benchmark/market stream
|
||||
at C3 (IR, Treynor, alpha/beta); a regime-label stream (recorded HMM, C6); an
|
||||
ADV channel + realistic broker with cost params; per-observation label-span
|
||||
`(t0,t1)` (CPCV purging); and `periods_per_year` from the clock.
|
||||
|
||||
### Minimal additions with the most leverage
|
||||
|
||||
- **(M0) Recovery Factor — zero new inputs.** `total_pips / max_drawdown`.
|
||||
- **(M1) Retain/expose `r_t`** (a returns sink + moment/quantile/drawdown
|
||||
reductions). One sink + pure O(T) reductions unlock ~20 single-run metrics.
|
||||
Highest value per unit work; touches no C1/C2/C8 invariant.
|
||||
- **(M2) The trade segmenter over the C7 position-event table.** Unlocks the whole
|
||||
trade-level family.
|
||||
- **(M3) The World family-level reducer over multi-run collections.** Unlocks MC
|
||||
tail stats, WFA, and the DSR/PBO/CPCV overfitting layer — for which aura is
|
||||
uniquely positioned (N = sweep cardinality, known and logged).
|
||||
|
||||
**Registry note:** `metric_cmp` (`registry/src/lib.rs:83`) hard-codes the three
|
||||
known names + their better-direction. Each new scalar metric needs an entry there
|
||||
(name → kind, better-direction) to be optimizable/rankable — small but real (the
|
||||
optimizer's objective vocabulary *is* this match arm).
|
||||
|
||||
## 5. Recommended layering / sequencing
|
||||
|
||||
| Tranche | Scope | Depends on | Substance |
|
||||
|---|---|---|---|
|
||||
| **1 — single-run scorecard** | `r_t` (M1) + Recovery Factor (M0); emit Sharpe, Sortino, volatility, CAGR, Calmar, MAR, Recovery Factor, Ulcer/Pain + ratios, Omega, skew, excess kurtosis, downside dev, VaR/ES, tail ratio, gain-to-pain, common-sense, drawdown-episode table | nothing unbuilt (reuses axis 1 as-is) | pure O(T) functions of a vector aura already computes and discards; they are the numerators/denominators every later layer reuses; immediately fixes the axis-2 objective problem — `optimize` can argmax a *defensible* yardstick instead of raw `total_pips` |
|
||||
| **2 — trade-level** | win rate, profit factor, expectancy, payoff + PF↔win-rate identity, SQN, MAE/MFE, trade efficiency, holding period, true turnover, trade-based risk-of-ruin/Kelly | the C7 trade table (M2) | a genuinely new table; answers a *different* question (per-trade outcome distribution) |
|
||||
| **3 — multi-run orchestration** | **MC first** (cheapest unbuilt axis; seed plumbing live) → MC tail family + MCPT null p-value + the assumption-free MaxDD-bias cure; then **WFA** (composes axes 1-2-4) | the World family-level reducer (M3) | builds the unbuilt axes 3 & 4 |
|
||||
| **4 — overfitting / significance** | DSR, PBO, CPCV, MinBTL, Harvey-Liu-Zhu | tranche-3 orchestration (DSR needs V[SR_n]+effective N; PBO/CPCV need matrix M) | the capstone — but aura is *uniquely* positioned: C1 determinism makes N exact and every trial logged. **Rule: once it exists, never report an argmax winner without its deflated significance** |
|
||||
|
||||
Cross-cutting metadata (benchmark stream, regime labels, ADV + realistic-broker
|
||||
cost params, label spans for CPCV) seeps in when the consuming tool is built —
|
||||
inputs to specific tranches, not a standalone phase.
|
||||
|
||||
## Recommended next steps (tracker)
|
||||
|
||||
1. **New issue: single-run scorecard (Tranche 1)** — `r_t` sink + risk-adjusted +
|
||||
drawdown + tail metrics as O(T) reductions; extend `RunMetrics` + `metric_cmp`.
|
||||
Depends on nothing unbuilt; startable immediately.
|
||||
2. **New issue: position-event / trade table (C10 realization) + trade metrics
|
||||
(Tranche 2).**
|
||||
3. Extend existing [#68](http://192.168.178.103:3000/Brummel/Aura/issues/68)
|
||||
(MC family) / [#69](http://192.168.178.103:3000/Brummel/Aura/issues/69) (WFA)
|
||||
with the distribution reducers/metrics specified in the catalogue (Tranche 3).
|
||||
4. **New issue: overfitting significance layer (DSR/PBO/CPCV) (Tranche 4)** — the
|
||||
capstone; aura is uniquely positioned for it.
|
||||
|
||||
---
|
||||
|
||||
*Provenance: internal recon (ledger C10/C12/C18–C23, `aura-engine`/`aura-registry`
|
||||
code, open issues) + a multi-stage research workflow (10 topic agents → 23
|
||||
adversarially-verified formulas/thresholds → synthesis, all read-only with web
|
||||
fetch). Sources include Bailey/López de Prado (PSR/DSR/PBO/CPCV), Pardo (WFA),
|
||||
Magdon-Ismail (E[MaxDD]), Basel/FRTB (VaR/ES), Bacon (Pain/Ulcer). See
|
||||
[Metric Verification Log](Metric-Verification-Log) for the corrections and
|
||||
citations.*
|
||||
Reference in New Issue
Block a user