Brummel/Aura
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

Add Backtesting Engine Architecture knowledge area (9 pages) 

Distil the durable, project-neutral knowledge behind the design ledger's
architectural invariants into a new wiki cluster, stripped of all
implementation state (contract labels, identifiers, file anchors, status):

- Backtesting Engine Architecture (overview)
- Event-Driven Backtesting (data-driven clock, k-way merge, single timeline)
- Look-Ahead Bias and Causality (structural prevention)
- Determinism and Reproducibility (disjoint parallelism, seed-as-input,
  record-then-replay boundary)
- Reactive Streaming Dataflow (freshness gating, sample-and-hold, delay/register
  feedback, as-of vs. barrier joins)
- Columnar Data Layout (structure-of-arrays, cache/SIMD)
- Signal, Exposure, and Execution (target exposure vs. equity; frictionless vs.
  realistic; position management as a derived layer)
- Graph Compilation and Optimization (compile to a flat instance; CSE/DCE and
  loop-invariant code motion over a sweep, licensed by determinism)
- Authoring and Deployment Lifecycle (engine/app split; hot-reload vs. frozen
  reproducible artifact)

Every load-bearing claim is marked [L]/[C]/[CORR] and carries a validated
external source; each page has a grouped References section.

Restructure Home into a master landing for two knowledge areas (engine
architecture + strategy validation); move the validation narrative into its own
Strategy-Analysis-and-Validation overview; add a _Sidebar for navigation and
repoint the metric pages' narrative backlinks.
Brummel
2026-06-15 14:40:51 +02:00
parent 7736e47d6a
commit da887a26e9
14 changed files with 1905 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Authoring-and-Deployment-Lifecycle.md
+245
@@ -0,0 +1,245 @@
# Authoring and Deployment Lifecycle
This page describes the engineering lifecycle that carries a simulation or strategy from fast,
exploratory authoring to a frozen, reproducible deployed artifact. The lifecycle rests on a
structural split between a reusable **engine** (a software framework) and the separate
**applications** built on it, and on a deliberate inversion: authoring optimises for *fast feedback*
via dynamic reloading, while deployment optimises for *immutability and auditability* via static,
reproducible builds. A fourth pattern — treating tuning parameters as runtime data rather than
compiled-in constants — ties the two ends together by letting one compiled artifact be re-used across
many configurations without rebuilding.
The patterns here are general software-engineering practice, neutral of any one toolchain or language.
They are most visible in systems that run large families of deterministic simulations (see
[Backtesting Engine Architecture](Backtesting-Engine-Architecture) and
[Determinism and Reproducibility](Determinism-and-Reproducibility)), but the framework/application
split, hot reloading, and reproducible deployment apply to any extensible platform.
**Markers:** [L] law/exact, [C] convention, [CORR] corrected.
## The framework / application split
A **software framework** is reusable software that provides generic functionality which developers
extend to build complete solutions. The defining property of a framework — as opposed to a plain
library — is **inversion of control**: the framework dictates the overall structure of execution and
*calls user code* at predefined extension points, rather than user code calling it.
[L] A library, by contrast, is a collection of resources that a program *calls* on its own terms; the
caller controls the flow ([Inversion of control](https://en.wikipedia.org/wiki/Inversion_of_control)).
This is summarised as the Hollywood Principle, "Don't call us, we'll call you"
([Software framework](https://en.wikipedia.org/wiki/Software_framework)). [L]
A framework typically exhibits three further traits beyond inversion of control:
pre-implemented **default behaviour** usable as-is or customised within a predefined structure;
**structured extensibility** through hooks, callbacks, or APIs; and a **non-modifiable core** —
the framework's own logic is fixed, extended but not edited, reflecting the open–closed principle
([Software framework](https://en.wikipedia.org/wiki/Software_framework)). [L]
### The "game engine vs. the game" analogy
The cleanest illustration of this split is the **game engine**. A game engine is a software framework
that supplies the underlying technology — rendering, physics, audio, scheduling — reused unchanged
across many distinct games. [L] The historically decisive move was separating *game-specific rules and
data* from *basic engine concepts* such as collision detection and entity management, which let teams
specialise and let one engine power many titles
([Game engine](https://en.wikipedia.org/wiki/Game_engine)). [L] Reusable engines make building
sequels and variants faster, a recognised competitive advantage. [C]
By analogy, a simulation or trading-research framework is the "engine"; the individual strategies,
experiments, and models authored on top are the "games." The engine is the reusable, stable, broadly
tested substrate; each application is a separate, faster-moving artifact that depends on it. Keeping
the two as **separate units** — separate repositories, separate versioning, separate release cadence —
is what allows the engine to stay general while applications proliferate. [C]
### The reuse gradient
Components in such a system sit on a gradient from most-specific to most-universal, and *where* a
component lives encodes how widely it may be reused:
1. **Application-local components** — experimental nodes and blocks specific to one project, expected
to change often and not yet proven general. They live with the application, not the engine. [C]
2. **Shared reusable libraries** — components proven useful across several applications, factored out
into a library that multiple independent applications depend on. Code reuse is exactly this:
*"A library can be used by multiple, independent consumers"*, so each gains its value without
re-implementing it ([Library (computing)](https://en.wikipedia.org/wiki/Library_(computing))). [L]
3. **A universal standard library** — the building blocks so fundamental that they ship *with* the
engine itself. A standard library is, by definition, "the library made available across
implementations of a programming language" — the components every consumer can rely on being
present ([Standard library](https://en.wikipedia.org/wiki/Standard_library)). [L]
Promoting a component up this gradient (local → shared → standard) is a deliberate, reviewed decision,
not an accident of import path. [C] The general rule of thumb: a component should be promoted only
once it has demonstrated genuine cross-application reuse, because every promotion widens the surface
that the engine must keep stable. [C]
## Hot reloading for authoring
The authoring loop optimises for one thing: **the time between editing a definition and seeing its
effect**. The dominant technique is to compile the author's code into a **dynamically loaded library**
and reload it into the running host *live*, so the author observes the change without restarting the
process.
### Dynamic loading
**Dynamic loading** is the mechanism by which a running program loads a library into memory *at run
time*, resolves the addresses of its functions and variables, calls them, and can later unload the
library — as opposed to binding the library at program startup
([Dynamic loading](https://en.wikipedia.org/wiki/Dynamic_loading)). [L] Its two canonical use cases
are exactly the authoring scenario: implementing **plugin / extension systems**, and **selecting at
run time** which of several interchangeable implementations to use
([Dynamic loading](https://en.wikipedia.org/wiki/Dynamic_loading)). [L] An authored strategy or node
set is, in these terms, a plugin: a dynamically loaded module the host can swap without relinking
itself.
This is distinct from ordinary **dynamic linking**, where a shared library is resolved automatically
at load time before `main` runs. Dynamic *loading* is explicit and happens later, under program
control — which is what makes *reloading* possible ([Library (computing)](https://en.wikipedia.org/wiki/Library_(computing))). [L]
### Hot swapping and edit-and-continue
**Hot swapping**, in the software sense, is "the ability to alter the running code of a program without
needing to interrupt its execution" ([Hot swapping](https://en.wikipedia.org/wiki/Hot_swapping)). [L]
A closely related debugger feature is **edit and continue**, which lets a developer change source code
while the program is paused and apply the change on resume, without stopping, recompiling, and
restarting ([Edit and Continue](https://learn.microsoft.com/en-us/visualstudio/debugger/edit-and-continue)). [L]
Both share a goal: collapse the edit → rebuild → restart → re-navigate cycle.
The motivation is feedback latency, and the subtler cost it hides is **lost state**. When a feature
under test is several steps removed from a fresh start, every restart forces the author to replay the
path back to that point, "making the cycle multiple-seconds long." Keeping the process alive and
injecting only the edited code "this way, you don't lose any of your state"
([Hot reloading, React Native](https://reactnative.dev/blog/2016/03/24/introducing-hot-reloading)). [L]
For a simulation host this state is expensive: loaded historical data, warmed buffers, an open viewer
session. Reloading just the authored module preserves all of it. [C]
### Authoring-loop only
A load-bearing discipline: **dynamic reloading is an authoring-loop tool, not a deployment mechanism.**
The same dynamic-loading flexibility that accelerates iteration also undermines the property that a
deployed artifact most needs — a fixed, known correspondence between the running code and an exact
source version. The two ends of the lifecycle therefore use *opposite* linking strategies on purpose:
hot-reloadable dynamic modules during authoring, a frozen static build for deployment. [C] Treating a
hot-reloadable module as if it were a release artifact is an anti-pattern, because "what code is
actually running" becomes unanswerable.
## The frozen, reproducible deploy artifact
Deployment inverts every authoring priority. Where authoring wants mutability and speed, deployment
wants **immutability and auditability**: the running artifact must correspond to one exact source
version, and that correspondence must be verifiable after the fact.
### Static linking
The first half is achieved by **static linking**. A static (statically linked) library "contains
functions and data that can be included in a consuming computer program at build-time such that the
library does not need to be accessible in a separate file at run-time"
([Static library](https://en.wikipedia.org/wiki/Static_library)). [L] When every dependency is linked
this way, the result is a **stand-alone executable**, also called a **static build**, in which all
bindings are resolved at compile time ([Static build](https://en.wikipedia.org/wiki/Static_build)). [L]
The advantages align precisely with deployment needs:
- **Guaranteed availability.** "The application is guaranteed to have the library routines it requires
available at run-time, as the code to those routines is embedded in the executable file"
([Static library](https://en.wikipedia.org/wiki/Static_library)). [L]
- **No dependency drift.** Static linking "avoids DLL Hell or more generally dependency hell," so
behaviour does not change because some other version of a shared library happens to be present on
the host ([Static library](https://en.wikipedia.org/wiki/Static_library)). [L]
- **Predictable, portable behaviour.** A static build does "not rely on the particular version of
libraries available on the final system," giving "very predictable behavior"
([Static build](https://en.wikipedia.org/wiki/Static_build)). [L]
The trade-off is real but secondary for this use case: static builds are larger because library code
is copied in rather than shared ([Static build](https://en.wikipedia.org/wiki/Static_build)). [C] For
a deployed, audited artifact, self-containment outweighs binary size.
### Reproducible builds
Static linking pins *what is in* the artifact; **reproducible builds** pin the *correspondence between
that artifact and its source*. The canonical definition: "A build is reproducible if given the same
source code, build environment and build instructions, any party can recreate bit-by-bit identical
copies of all specified artifacts"
([Reproducible Builds definition](https://reproducible-builds.org/docs/definition/)). [L] Equivalently,
reproducible builds — also called deterministic compilation — ensure "the resulting binary code can be
reproduced" from the same source
([Reproducible builds](https://en.wikipedia.org/wiki/Reproducible_builds)). [L]
The payoff is a verifiable trust chain. Because anyone can rebuild and byte-compare, reproducible
builds "provide a strong countermeasure against attacks where binaries do not match their source code,
e.g., because an attacker has inserted malicious code into a binary"
([Reproducible builds](https://en.wikipedia.org/wiki/Reproducible_builds)). [L] Practically, achieving
bit-for-bit reproducibility requires building in a controlled environment with pinned toolchains and
dependencies, and normalising sources of nondeterminism such as embedded timestamps and build paths. [C]
### The artifact is a commit
Combining the two halves yields the deployment law for this lifecycle: **the deployed artifact
corresponds to one exact source version — the artifact *is* a specific commit.** It is built once,
versioned, frozen, and never hot-swapped. The audit statement "this running program is exactly this
reviewed source at this revision" is then both true by construction (static linking removes runtime
dependency variance) and independently checkable (reproducible builds let a third party rebuild and
compare). [L] This is the deliberate opposite of the authoring loop: there, code is reloaded freely;
here, code is immutable for the artifact's whole life. Compare the engine-side reproducibility
guarantee in [Determinism and Reproducibility](Determinism-and-Reproducibility): determinism makes a
*run* reproducible; reproducible builds make the *binary* reproducible. Both are required for an
end-to-end audit trail — the same source, built the same way, fed the same inputs, yields the same
results.
## Parameters as runtime data
A pattern that connects authoring speed to deployment economics: **tuning parameters are runtime
values, not compiled-in constants.** A parameter sweep — running the same model across a grid of
parameter settings — then needs *no rebuild at all*. The compiled module is loaded once; each
configuration is just a different set of values bound at run time.
This matters because the two obvious alternatives are both costly. Baking parameters in as constants
would require a recompile per configuration — multiplying build cost by the size of the sweep grid.
Re-loading the dynamic module per configuration would re-pay loading cost needlessly. Keeping
parameters as runtime data sidesteps both: one load, many cheap re-parameterizations. [C] This is the
same separation the framework/application split already encodes — the engine provides the parameterised
machinery; the values that configure it are *data*, supplied later — applied here to the inner loop of
an experiment.
The cheap-re-derivation claim has a structural precondition: parameters may *configure and size* the
computation but must not change its *topology* (a topology change is a different program, and would
need a different compiled module). The formal treatment of how a parameterised computation is compiled
once and then re-derived cheaply across a parameter family — sharing the parameter-invariant work and
recomputing only what each setting changes — belongs to
[Graph Compilation and Optimization](Graph-Compilation-and-Optimization). The results a sweep produces,
and how they are scored and compared across the grid, are covered by the
[Metric Catalogue](Metric-Catalogue); the discipline of checking those metric computations against
references is the [Metric Verification Log](Metric-Verification-Log).
## Lifecycle summary
The lifecycle is two opposed regimes joined by a shared, parameterised engine:
| Phase | Linking | Mutability | Optimised for | Mechanism |
|---|---|---|---|---|
| **Authoring** | dynamic loading | hot-reloadable | feedback latency, preserved state | plugin module reloaded live [L] |
| **Sweeping** | (one load) | re-parameterised | reuse without rebuild | parameters as runtime data [C] |
| **Deployment** | static linking | frozen / immutable | audit, reproducibility | stand-alone reproducible build [L] |
The engine that all three share is a framework: it owns the control flow and calls authored code at
its extension points, while authored applications remain separate, versioned units on a reuse gradient
from local to standard. [L]
## References
### The framework / application split
- [Software framework — Wikipedia](https://en.wikipedia.org/wiki/Software_framework)
- [Inversion of control — Wikipedia](https://en.wikipedia.org/wiki/Inversion_of_control)
- [Library (computing) — Wikipedia](https://en.wikipedia.org/wiki/Library_(computing))
- [Standard library — Wikipedia](https://en.wikipedia.org/wiki/Standard_library)
- [Game engine — Wikipedia](https://en.wikipedia.org/wiki/Game_engine)
### Hot reloading for authoring
- [Dynamic loading — Wikipedia](https://en.wikipedia.org/wiki/Dynamic_loading)
- [Hot swapping — Wikipedia](https://en.wikipedia.org/wiki/Hot_swapping)
- [Edit and Continue — Microsoft Learn](https://learn.microsoft.com/en-us/visualstudio/debugger/edit-and-continue)
- [Introducing Hot Reloading — React Native](https://reactnative.dev/blog/2016/03/24/introducing-hot-reloading)
### The frozen, reproducible deploy artifact
- [Static library — Wikipedia](https://en.wikipedia.org/wiki/Static_library)
- [Static build — Wikipedia](https://en.wikipedia.org/wiki/Static_build)
- [Reproducible builds — Wikipedia](https://en.wikipedia.org/wiki/Reproducible_builds)
- [Reproducible Builds — official definition](https://reproducible-builds.org/docs/definition/)
Backtesting-Engine-Architecture.md
+129
@@ -0,0 +1,129 @@
# Backtesting Engine Architecture
A modern backtesting engine is the deterministic core of a quantitative research stack: it replays historical (or recorded) market data through a trading strategy and produces a reproducible record of what that strategy would have done. The decisive architectural insight is that a *single* backtest — one strategy over one data window producing one result — is a commodity that every system provides; durable value comes from the layer *above* it that constructs and runs whole *families* of simulations to establish statistical confidence. This page is the navigational landing point for that architecture cluster: it lays out the two-layer view, the load-bearing commitments that make a single simulation trustworthy, and the orchestration and validation machinery built on top, linking each topic to its detail page.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
## The two-layer view
A serious engine separates cleanly into two layers with different correctness obligations.
The **single-simulation substrate** is the reproducible unit of work: one strategy, one parameter binding, one data window, in, one deterministic result out. [Backtesting](https://en.wikipedia.org/wiki/Backtesting) in finance "seeks to estimate the performance of a strategy or model if it had been employed during a past period" [L]. The substrate's obligation is *fidelity and reproducibility*: the same inputs must yield exactly the same run, and the run must contain no information that would not have been available at decision time.
The **orchestration layer** (sometimes called the experiment-orchestration layer or meta-layer) treats the substrate as a black-box function and invokes it many times under systematically varied conditions: parameter sweeps, optimization, walk-forward windows, Monte-Carlo resampling, and cross-strategy or cross-instrument comparison. Its obligation is *statistical*: to distinguish a strategy that is genuinely robust from one that merely fit the noise of a single history.
The central thesis follows directly. A lone backtest with an attractive equity curve is nearly worthless as evidence, because "high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations" and "the higher the number of configurations tried, the greater is the probability that the backtest is overfit" [L] ([Bailey, Borwein, López de Prado & Zhu](https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf)). Robustness and confidence are therefore properties established at the orchestration/validation layer, not at the substrate. The substrate must be *fast and exactly reproducible* precisely so the layer above can run thousands of variations and reason about the distribution of outcomes rather than a single lucky path.
## Why event-driven rather than vectorized
Two families of backtester exist, and the architectural commitments below assume the event-driven family.
A **vectorized** backtester applies array operations across the whole dataset at once. It "operates on arrays of data all at once, making it efficient for testing strategies over long periods or large datasets" and is correspondingly "faster," but it "simplifies the trading environment, focusing on end-of-period prices" and "typically assumes perfect execution" [C] ([marketcalls](https://www.marketcalls.in/system-trading/comparision-of-event-driven-backtesting-vs-vectorized-backtesting.html)). It is ideal for rapid factor screening where intra-bar dynamics are negligible.
An **event-driven** backtester instead "tries to mimic the live trading environment as closely as possible" by processing "market data as 'events' in a sequence," at the cost of being "slower" because it "processes each market event sequentially" [C]. Two structural advantages make it the foundation for a production engine. First, code reuse: "an event-driven backtester, by design, can be used for both historical backtesting and live trading with minimal switch-out of components" [L] ([QuantStart](https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I/)). Second, and most important, "with an event-driven backtester there is no lookahead bias as market data receipt is treated as an 'event'" [L]. A common practical compromise is to vectorize the screening stage and reserve event-driven simulation for final validation [C]. See [Event-Driven Backtesting](Event-Driven-Backtesting) for the event loop, the queue, and the producer/consumer components in depth.
## Load-bearing architectural commitments
The substrate earns its trustworthiness from a small set of commitments, each of which has its own detail page. They are listed here as an index; the rationale and the exact mechanics live downstream.
### A data-driven event clock
Time advances by consuming the next event from a chronologically ordered stream, not by an external wall clock or an imperative loop counter. This is the [event-driven architecture](https://en.wikipedia.org/wiki/Event-driven_architecture) pattern, in which an *event* is "a significant change in state" and components are *producers* that emit events and *consumers* (sinks) that "apply reactions when events arrive" [L]. Heterogeneous timestamped sources are merged into a single chronological stream at the ingestion boundary, so the strategy graph sees exactly one ordered sequence of cycles. See [Event-Driven Backtesting](Event-Driven-Backtesting).
### Look-ahead bias made structurally impossible
The most damaging silent error in backtesting is *look-ahead bias*: using information that would not have been available at the modeled decision instant. The robust design does not merely discourage it — it makes it unrepresentable, e.g. by exposing only read-only input windows that end at the cursor and by emitting a resampled bar only once it is complete. The event-driven discipline is itself the primary defense, since data is delivered strictly as it arrives in time: "with an event-driven backtester there is no lookahead bias as market data receipt is treated as an 'event'" [L] ([QuantStart](https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I/)). See [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality).
### Determinism and reproducibility
A backtest is a *deterministic algorithm*: "given a particular input, [it] will always produce the same output, with the underlying machine always passing through the same sequence of states" [L] ([deterministic algorithm](https://en.wikipedia.org/wiki/Deterministic_algorithm)). This is what [computational reproducibility](https://en.wikipedia.org/wiki/Reproducibility) requires — "obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis" [L]. Any stochastic element (resampling, randomized parameters) is pinned by an explicit seed so the pseudo-random sequence is reproducible run-to-run [L]. Critically, two backtests with disjoint state are independent and can run *concurrently without locking*; parallelism is exploited *across* independent simulations, never within a single one, which keeps each run trivially deterministic. See [Determinism and Reproducibility](Determinism-and-Reproducibility).
### Push-based reactive streaming dataflow on a DAG
The strategy is expressed as a graph of nodes through which data flows. [Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming) "models a program as a directed graph of the data flowing between operations," where operations "function like black boxes" with explicit inputs and outputs and "an operation runs as soon as all of its inputs become valid" [L] — a push, data-driven model. The graph is a *directed acyclic graph*: the only feedback path is an explicit delay/state node (the dataflow analogue of a register), so the topology stays acyclic and evaluation order is well-defined. Because dataflow operations "activate based on data readiness rather than sequential ordering," such designs "are inherently parallel" [L], which dovetails with the determinism commitment above. See [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow).
### A columnar (structure-of-arrays) hot path
The streamed numeric payload is laid out as a *Structure of Arrays* (SoA) — "separating elements of a record into one parallel array per field" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)) — rather than an Array of Structures. When a computation touches only some fields, "only those parts need to be iterated over, allowing more data to fit onto a single cache line," and "a single SIMD register can load homogeneous data" [L]. This is the same rationale that makes a [column-oriented DBMS](https://en.wikipedia.org/wiki/Column-oriented_DBMS) fast for analytics: storing "each column of the table one after the other" yields "fast access under a projection," "smaller compressed size," and the ability "to leverage SIMD instructions" [L]. Restricting the hot path to a few scalar base types streamed columnarly (composite streams such as OHLCV are bundles of base columns) keeps the inner loop cache-friendly and vectorizable; non-scalar metadata lives beside the hot path, never in it. See [Columnar Data Layout](Columnar-Data-Layout).
### Separating the strategy signal from the execution model
A robust design does not let a strategy emit an equity curve directly. Instead, the strategy's primary, backtestable output is a *target-exposure signal*: a signed, bounded desired position (conventionally a value in the closed interval [-1, +1]) per cycle [C]. Signal quality is then measured by an idealized, frictionless execution model that integrates exposure against forward returns into a synthetic equity stream, while a separate, derived position-event layer feeds realistic execution models that account for frictions (spread, slippage, financing). This is an application of [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns) — "a complex problem should be divided into distinct concerns ... that can be analyzed, addressed or managed individually" [L] — and it keeps the alpha question ("is the directional signal good?") cleanly separate from the execution question ("does it survive costs?"). Execution models are ordinary downstream consumer nodes, never part of the strategy. See [Signal, Exposure, and Execution](Signal-Exposure-and-Execution).
### Graph construction as a behaviour-preserving compilation
The graph the author writes is a *parameter-generic graph specification* (a template). Binding parameters, data, and a seed *compiles* it into a frozen, flat instance: buffers are sized, topology is fixed, and nodes are wired by raw index. Because the compiled form is just a dataflow graph, it is a legitimate target for classic behaviour-preserving optimizations — common-subexpression elimination, dead-code elimination — and, across a family of simulations, a sweep-invariant sub-graph can be computed once and shared. The correctness invariant is that optimization must not change the result of any single run. See [Graph Compilation and Optimization](Graph-Compilation-and-Optimization).
### An authoring/deployment lifecycle
Research and deployment have opposite priorities. During authoring, a fast hot-reload loop (a dynamically loaded library swapped in place) maximizes iteration speed. For deployment, the artifact is *frozen*: a statically linked, versioned binary that is never hot-swapped, so the audit trail can state unambiguously that this running bot equals this exact build. This builds on the event-driven advantage that one code path serves "both historical backtesting and live trading with minimal switch-out of components" [L] ([QuantStart](https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I/)); freezing the live path is then a deployment convention layered on top [C]. See [Authoring and Deployment Lifecycle](Authoring-and-Deployment-Lifecycle).
## The orchestration layer in detail
Above the substrate, the orchestration layer constructs and runs families of simulations. This is where confidence is manufactured.
### Parameter sweeps and optimization
Sweeping a strategy's tunable parameters across a grid (or via a search) produces a surface of outcomes. The orchestration layer's job is not merely to find the peak — that peak is exactly what overfits — but to characterize the *neighborhood*: a robust strategy sits on a broad plateau, a fragile one on a sharp spike. The compilation step above makes sweeps cheap by sharing the sweep-invariant sub-graph across the family.
### Walk-forward analysis
[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) is "a method used in finance to determine the optimal parameters for a trading strategy and to determine the robustness of the strategy" [L]. A strategy "is optimized with in-sample data for a time window," the immediately following out-of-sample window is tested, then "the in-sample time window is shifted forward by the period covered by the out of sample test, and the process repeated" [L]. It "reduces overfitting by testing each segment of data in a forward-looking manner, preventing the false confidence that can come from a single, potentially lucky validation period" [C] ([QuantInsti](https://blog.quantinsti.com/walk-forward-optimization-introduction/)), which makes it one of the stronger out-of-sample validation methods in common practitioner use [C] ([Wikipedia](https://en.wikipedia.org/wiki/Walk_forward_optimization)). Its main limitation is that, however many times the windows are rolled forward, it still exercises the strategy against only the one historical price path that actually occurred — a constraint that Monte-Carlo resampling (below) is designed to relax [C].
### Monte-Carlo resampling
Monte-Carlo methods attack the single-path limitation by generating many synthetic variations — resampling returns, shuffling trade order, perturbing fills — and re-running the substrate against each. The distribution of resulting metrics (rather than a point estimate) reveals how much of the headline performance is path-dependent luck. Because each run is deterministic given its seed, the whole ensemble is itself reproducible [L].
### Cross-strategy and cross-instrument comparison
Running the same experiment setup over multiple instruments, regimes, or competing strategy variants turns isolated results into comparable evidence and exposes strategies that work only on a single symbol or epoch.
### Validation and the multiple-testing problem
Every additional configuration the orchestration layer tries *inflates* the best observed result, because "the more trials, the higher the probability of overfitting" and "in-sample performance is a poor predictor of out-of-sample results" [L] ([Bailey et al.](https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf)). The honest orchestration layer therefore *records the number of trials* and discounts the apparent best accordingly (deflated performance statistics, probability-of-overfitting estimates). How the resulting return series are scored — Sharpe ratio, drawdown, and their multiple-testing-aware corrections — is catalogued in [Metric Catalogue](Metric-Catalogue); the per-metric correctness checks live in [Metric Verification Log](Metric-Verification-Log). For reference, the [Sharpe ratio](https://en.wikipedia.org/wiki/Sharpe_ratio) is "the difference between the returns of the investment and the risk-free return, divided by the standard deviation of the investment returns" [L].
## Headless core principle
The engine is *UI-agnostic*. Visualization is a downstream consumer of recorded streams, not a component of the engine. This applies the [headless software](https://en.wikipedia.org/wiki/Headless_software) discipline to a simulation core: headless software is "software capable of working on a device without a graphical user interface," receiving inputs and providing output through other interfaces [C] ([Wikipedia](https://en.wikipedia.org/wiki/Headless_software)). The engine has no presentation layer of its own; any viewer interacts with it only through its recorded outputs [C]. Concretely, only what a *sink* node records becomes displayable; a trace/visualization viewer reads those recorded streams (structure before a run, live values during, recorded traces after) and never reaches into engine internals. Two consequences follow: the engine can run fully headless at scale (no rendering on the critical path of an orchestration sweep), and any viewer is replaceable without touching the deterministic core. The viewer is an execution viewer and trace explorer, not a strategy editor — topology is authored in code, while runtime parameters can be tuned through the interface.
## How the pages fit together
- The substrate's mechanics: [Event-Driven Backtesting](Event-Driven-Backtesting), [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality), [Determinism and Reproducibility](Determinism-and-Reproducibility).
- The dataflow model and its data layout: [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow), [Columnar Data Layout](Columnar-Data-Layout).
- Strategy output vs execution, and how the graph is compiled and frozen: [Signal, Exposure, and Execution](Signal-Exposure-and-Execution), [Graph Compilation and Optimization](Graph-Compilation-and-Optimization), [Authoring and Deployment Lifecycle](Authoring-and-Deployment-Lifecycle).
- Scoring the output of the families the orchestration layer produces: [Metric Catalogue](Metric-Catalogue), [Metric Verification Log](Metric-Verification-Log).
## References
### The two-layer view and overfitting
- [Backtesting — Wikipedia](https://en.wikipedia.org/wiki/Backtesting)
- [Statistical Overfitting and Backtest Performance (Bailey, Borwein, López de Prado, Zhu) — LBL/SDM PDF](https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf)
### Event-driven vs vectorized
- [Comparison of Event-Driven vs Vectorized Backtesting — marketcalls](https://www.marketcalls.in/system-trading/comparision-of-event-driven-backtesting-vs-vectorized-backtesting.html)
- [Event-Driven Backtesting with Python, Part I — QuantStart](https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I/)
### Event clock and event-driven architecture
- [Event-driven architecture — Wikipedia](https://en.wikipedia.org/wiki/Event-driven_architecture)
### Determinism and reproducibility
- [Deterministic algorithm — Wikipedia](https://en.wikipedia.org/wiki/Deterministic_algorithm)
- [Reproducibility — Wikipedia](https://en.wikipedia.org/wiki/Reproducibility)
### Dataflow on a DAG
- [Dataflow programming — Wikipedia](https://en.wikipedia.org/wiki/Dataflow_programming)
### Columnar / structure-of-arrays hot path
- [AoS and SoA — Wikipedia](https://en.wikipedia.org/wiki/AoS_and_SoA)
- [Column-oriented DBMS — Wikipedia](https://en.wikipedia.org/wiki/Column-oriented_DBMS)
### Separation of concerns (signal/execution split)
- [Separation of concerns — Wikipedia](https://en.wikipedia.org/wiki/Separation_of_concerns)
### Headless core
- [Headless software — Wikipedia](https://en.wikipedia.org/wiki/Headless_software)
### Orchestration: walk-forward and metrics
- [Walk forward optimization — Wikipedia](https://en.wikipedia.org/wiki/Walk_forward_optimization)
- [Walk-Forward Optimization: Introduction — QuantInsti](https://blog.quantinsti.com/walk-forward-optimization-introduction/)
- [Sharpe ratio — Wikipedia](https://en.wikipedia.org/wiki/Sharpe_ratio)
Columnar-Data-Layout.md
+124
@@ -0,0 +1,124 @@
# Columnar Data Layout (Structure-of-Arrays)
A high-throughput numeric streaming hot path scans the same field across many records, computes over it, and moves on — it almost never wants one whole record at a time. The data-layout decision that follows from this access pattern is **columnar**: store each field as its own contiguous column, the arrangement known in systems programming as **Structure-of-Arrays (SoA)** and in databases as **column-oriented storage**. The two names describe the same physical idea — same-type values of one field laid out adjacent in memory — and the same payoff: better cache-line utilization on scans and amenability to SIMD vectorization. This page defines the layout, explains *why* it suits streaming numerics, relates it to columnar analytics and the [Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html) in-memory format, and states the trade-offs that decide when *not* to use it.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
See also: [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow), [Backtesting Engine Architecture](Backtesting-Engine-Architecture), [Determinism and Reproducibility](Determinism-and-Reproducibility), [Metric Catalogue](Metric-Catalogue), [Metric Verification Log](Metric-Verification-Log).
## AoS vs SoA: two ways to lay out a sequence of records
Given a sequence of records, each with several fields, there are two contrasting ways to arrange them in linear memory, differing in whether fields are *interleaved*.
**Array-of-Structures (AoS)** is the conventional layout "in which data for different fields is interleaved" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)). All fields of record 0 sit adjacent, then all fields of record 1, and so on. For a record of three coordinates `(x, y, z)`, memory reads:
```
x0 y0 z0 x1 y1 z1 x2 y2 z2 ...
```
This is the layout most programming languages give by default when an array holds a struct type — it is "more intuitive, and supported directly by most programming languages" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)).
**Structure-of-Arrays (SoA)** is "a layout separating elements of a record … into one parallel array per field" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)). Each field becomes its own contiguous column; the record at index *i* is reconstructed by reading index *i* of every column. The same three-coordinate data reads:
```
x0 x1 x2 ... (the x column)
y0 y1 y2 ... (the y column)
z0 z1 z2 ... (the z column)
```
A record is now a *logical* tuple — index *i* across the columns — not a contiguous physical object. The columnar / column-oriented database community describes precisely this layout from the storage side: a column store places "each column of the table … one after the other. In this orientation, values on the same column are close in space," whereas a row store places "each row … one after the other" so that "values in the same row are close in space" [L] ([Column-oriented DBMS](https://en.wikipedia.org/wiki/Column-oriented_DBMS)). SoA (the in-memory term) and column-oriented storage (the database term) are the same physical decision viewed from two communities.
A third, hybrid layout, **Array-of-Structures-of-Arrays (AoSoA)**, interleaves fields "using tiles or blocks with size equal to the SIMD vector size" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)). It "can achieve the memory throughput of the SoA approach, while being more friendly to the cache locality and load port architectures of modern processors" [C] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)). AoSoA is an advanced tuning option; the rest of this page treats the AoS/SoA dichotomy, which is the decision that matters for a streaming hot path.
## Why SoA for streaming numerics
The case for columnar layout in a numeric streaming hot path rests on two hardware facts — cache-line transfer and SIMD vector registers — and one workload fact: the hot path scans one field across many records.
### Cache lines and spatial locality
Memory does not move between RAM and the CPU one byte at a time. "Data is transferred between memory and cache in blocks of fixed size, called *cache lines* or *cache blocks*" [L] ([CPU cache](https://en.wikipedia.org/wiki/CPU_cache)). On widely deployed hardware a cache line is **64 bytes** [C] — e.g. the original Pentium 4 had "64-byte cache blocks" in its L1 data cache [L] ([CPU cache](https://en.wikipedia.org/wiki/CPU_cache)); the exact size is microarchitecture-dependent (32, 64, and 128-byte lines all appear historically), so 64 bytes is a convention, not a universal law.
Because a whole line is fetched on every miss, the layout that *fills each line with bytes the loop will actually use* wins. This is **spatial locality**: "if a particular storage location is referenced at a particular time, then it is likely that nearby memory locations will be referenced in the near future" [L] ([Locality of reference](https://en.wikipedia.org/wiki/Locality_of_reference)), and "data elements are brought into cache one cache line at a time … if one element is referenced, a few neighboring elements will also be brought into cache" [L] ([Locality of reference](https://en.wikipedia.org/wiki/Locality_of_reference)).
Now apply this to a scan of one field across many records:
- **SoA.** The field's values are contiguous, so every byte of every fetched cache line is a value the loop consumes. Reading an 8-byte field, a 64-byte line delivers 8 consecutive values — eight usable values per memory transaction. "If only a specific part of the record is needed, only those parts need to be iterated over, allowing more data to fit onto a single cache line" [L] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)).
- **AoS.** The field's values are separated by the other fields of each record. A fetched line is mostly the *other* fields, which the single-field scan discards. The concrete contrast: a column-scan over an 8-byte field "loads 8 … entries at once (8 \* 8 byte = 64 byte), whereas the AoS storage can only move 2 records (2 \* 32 byte = 64 byte) per cache line" — so on such a scan the AoS layout "wastes 75% of the memory bandwidth, achieving a goodput of only 16 bytes per 64 bytes loaded" [C] ([CedarDB — Optimizing Data Layouts](https://cedardb.com/blog/optimizing_data_layouts/)).
The cache-line *transfer mechanism* and *what a given layout puts on a line* are facts of the hardware and the layout [L]. The resulting *magnitude* of any speed-up ("4× faster", "wastes 75%") is workload-, field-width-, and machine-dependent [C] — quoted figures illustrate the mechanism, they are not portable constants.
### SIMD and auto-vectorization
The second reason is **SIMD** (Single Instruction, Multiple Data): hardware with "multiple processing elements that perform the same operation on multiple data points simultaneously," exploiting "data level parallelism" [L] ([SIMD](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data)). A SIMD *vector register* holds a fixed number of equal-width elements, called **lanes**; an instruction operates on every lane at once. Intel AVX-512 "SIMD instructions process 512 bits of data at once" [L] ([SIMD](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data)) — sixteen 32-bit floats, or eight 64-bit values, per instruction.
To fill a vector register in a single load, the elements must be **contiguous and same-type** — exactly what a SoA column is. **Automatic vectorization** is "a special case of automatic parallelization, where a computer program is converted from a scalar implementation, which processes a single pair of operands at a time, to a vector implementation, which processes one operation on multiple pairs of operands at once" [L] ([Automatic vectorization](https://en.wikipedia.org/wiki/Automatic_vectorization)). The canonical vectorizable kernel is a tight loop over contiguous arrays:
```c
for (i = 0; i < n; i++)
c[i] = a[i] + b[i]; // each of a, b, c is a contiguous SoA column
```
The compiler turns this into vector adds over batches of lanes. SoA "can greatly benefit of auto vectorization within the compiler" [C] ([CedarDB — Optimizing Data Layouts](https://cedardb.com/blog/optimizing_data_layouts/)). AoS works against the vectorizer: a single-field scan over an interleaved layout needs *gathers* (strided/scattered loads) to assemble a vector register, which is slower than a contiguous load and often blocks vectorization outright. The relationship is structural [L]: a contiguous same-type column *can* be loaded straight into a vector register; an interleaved field generally cannot. (Vectorization additionally requires the compiler to prove the loop carries no disqualifying dependence — a separate condition from layout — ([Automatic vectorization](https://en.wikipedia.org/wiki/Automatic_vectorization)).)
For a streaming engine, this matters because the per-tick work over a window is typically a tight reduction or map over one column — moving averages, sums, comparisons, exposure integration. Columnar layout is what lets that inner loop vectorize. See [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow) for how such per-field work is scheduled across nodes, and [Backtesting Engine Architecture](Backtesting-Engine-Architecture) for the overall hot-path shape.
## Columnar analytics: column stores and Apache Arrow
Columnar layout is the defining choice of analytical (OLAP) data systems, and the reasoning there mirrors the streaming case exactly.
**Selective column I/O.** An analytical query usually reads a few fields of a wide table. With columns stored separately, "a query that aggregates prices touches only the price column and skips everything else. On a table with 50 columns, a query that reads 3 of them does roughly 6% of the I/O that a row store would need" [C] ([ClickHouse — What is a columnar database](https://clickhouse.com/resources/engineering/what-is-columnar-database)). The same logic that keeps unused fields off a cache line at the CPU level keeps unused columns off the I/O path at the storage level.
**Vectorized execution.** Column stores "make analytical operations like filtering, grouping, aggregations and others more efficient thanks to memory locality," and enable "vectorization of the computations" because "most modern CPUs have SIMD instructions" [L] ([Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html)). The columnar execution model loads a batch of one column's values into a vector register and applies an operation to all lanes together.
**Compression.** Adjacency of same-type values is what makes columns compress well. "Because adjacent values in a column share the same data type — and often similar magnitudes — encodings like dictionary encoding, run-length encoding, and delta encoding achieve compression ratios of 5–10× on real-world columnar data" [C] ([ClickHouse — What is a columnar database](https://clickhouse.com/resources/engineering/what-is-columnar-database)). The classic illustration: a 128-row boolean column "requires 128 bytes in a row-oriented format … but 128 bits (16 bytes) in a column-oriented format (via a bitmap)" [L] ([Column-oriented DBMS](https://en.wikipedia.org/wiki/Column-oriented_DBMS)). The compression ratios are data-dependent [C]; the *reason* compression is possible — homogeneous adjacent values — is structural [L].
### Apache Arrow as a reference columnar format
[Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html) is a standardized, language-independent **in-memory** columnar format, and a useful reference for how a columnar layout is specified in practice. Its stated design goals read as a checklist of the benefits above: "Data adjacency for sequential access (scans)", "O(1) (constant-time) random access", "SIMD and vectorization-friendly", and arrays "Relocatable without 'pointer swizzling', allowing for true zero-copy access in shared memory" [L] ([Apache Arrow — Columnar Format](https://arrow.apache.org/docs/format/Columnar.html)).
Arrow also documents the alignment convention that makes columns vectorize cleanly: a recommended **64-byte alignment and padding**, because "the recommendation for 64 byte alignment comes from the Intel performance guide that recommends alignment of memory to match SIMD register width," and "the recommended padding of 64 bytes allows for using SIMD instructions consistently in loops without additional conditional checks" [C] ([Apache Arrow — Columnar Format](https://arrow.apache.org/docs/format/Columnar.html)). 64 bytes matches AVX-512's 512-bit register; it is a convention pinned to a specific (widely deployed) SIMD width, not a universal law.
Crucially, Arrow shows how **composite / nested data is represented as a bundle of plain columns plus structure metadata** rather than as interleaved records. Each column is an *Array* made of "one or more buffers"; a struct type "has one child array for each field," and those "child arrays are independent and need not be adjacent to each other in memory" [L] ([Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html)). A nested list keeps "all the values being stored consecutively in a values child array" with a separate offsets buffer [L] ([Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html)). This is the same principle a streaming engine applies to a composite stream such as OHLCV: it is not one interleaved record stream but a *bundle of base-typed columns* travelling together (see the closed-set section below).
## Trade-offs: choose by access pattern
Columnar layout is not universally faster — it is faster *for the access pattern it is built for*, and slower for the opposite one. The decision is governed by **how the data is accessed**, not by which layout is "modern".
**SoA / columnar wins when** the operation scans or transforms one (or a few) fields across many records: aggregations, filters, vectorized math, streaming reductions over a window. With a field's values laid out as a contiguous column, "if only a specific part of the record is needed, only those parts need to be iterated over, allowing more data to fit onto a single cache line," and the homogeneous adjacency enables "easier manipulation with packed SIMD instructions … since a single SIMD register can load homogeneous data" [C] ([AoS and SoA](https://en.wikipedia.org/wiki/AoS_and_SoA)). The cache-line and SIMD arguments above all apply.
**AoS / row layout wins when** the operation touches most fields of one record at a time — random whole-record access, per-record mutation, and transactional insert/update/delete. "If you usually touch most of the data of your structure when looping over it, it is better to do AoS" [C] ([CedarDB — Optimizing Data Layouts](https://cedardb.com/blog/optimizing_data_layouts/)), because all of a record's fields then share a cache line and one fetch serves the whole record. From the database side, columnar "provides analytical performance and data locality guarantees in exchange for comparatively more expensive mutation operations" [L] ([Apache Arrow — Columnar Format](https://arrow.apache.org/docs/format/Columnar.html)), while row stores offer "fast insertion of a new row" and suit OLTP workloads where complete records are written and read [C] ([Column-oriented DBMS](https://en.wikipedia.org/wiki/Column-oriented_DBMS)). The same split appears system-wide: "most traditional database systems focus on transactional processing … and implement AoS, whereas modern analytical database systems … usually prefer the SoA data orientation" [C] ([CedarDB — Optimizing Data Layouts](https://cedardb.com/blog/optimizing_data_layouts/)).
The practical rule [C]: **profile the real access pattern before restructuring**, because the wrong layout for a given pattern can cost a large multiple in cache misses, and the best layout depends on both hardware and access pattern. A deterministic, replayable streaming backtest is squarely in the column-scan regime — it appends and replays, it does not randomly mutate individual past records — which is exactly why columnar is the natural fit. The reproducibility consequences of an append-only, replay-oriented column stream are discussed in [Determinism and Reproducibility](Determinism-and-Reproducibility).
## Tie-in: a closed set of scalar machine types, streamed as columns
Columnar layout pays off most when the columns are **uniform and machine-vectorizable**, which argues for streaming only a small, fixed set of scalar machine types as columns:
- **integers** (a fixed-width signed integer),
- **floating-point** (a fixed-width float),
- **booleans** (naturally a packed bitmap column — the 128-bit-for-128-rows case above), and
- **timestamps** (a fixed-width integer column under the hood).
Each is fixed-width and same-type down the column, so each fills cache lines fully and loads straight into vector lanes — the two properties this whole page rests on [L]. Keeping the set *closed* keeps every stream uniform, so a single vectorized scan kernel applies to any column without per-field special-casing.
**Composite streams are bundles of base columns.** A structured stream such as OHLCV is not an interleaved record stream; it is several base-typed columns (open, high, low, close as float columns; volume as an integer column) carried together — exactly the Arrow model where a struct is "one child array for each field" with independent child buffers [L] ([Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html)). Any node consuming one field reads one contiguous column; the composite is an authoring-level convenience, not a runtime interleaving.
**Non-scalar data lives beside the hot path, not in it.** Strings, lookup tables, calendars, and other variable-width or structured values do not belong in a fixed-width, vectorizable column stream — they would break uniformity, defeat SIMD, and bloat cache lines. They are kept as **metadata** alongside the stream and referenced from it (e.g. by an integer id or dictionary index, the same indirection a column store uses for dictionary encoding), so the hot path stays a set of clean numeric columns. This is the in-memory analogue of Arrow keeping variable-length data in a separate child buffer addressed by offsets [L] ([Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html)). The metrics computed downstream over these columns are catalogued in [Metric Catalogue](Metric-Catalogue), and the practice of verifying those numeric results against independent references is covered in [Metric Verification Log](Metric-Verification-Log).
## References
### AoS vs SoA: definitions and layout
- [AoS and SoA — Wikipedia](https://en.wikipedia.org/wiki/AoS_and_SoA) — definitions of Array-of-Structures, Structure-of-Arrays, and the AoSoA hybrid; SIMD and cache-line rationale for SoA; access-pattern trade-offs.
### Why SoA for streaming numerics
- [CPU cache — Wikipedia](https://en.wikipedia.org/wiki/CPU_cache) — cache lines / blocks as the fixed-size unit of memory transfer; 64-byte (and 32/128-byte) line examples.
- [Locality of reference — Wikipedia](https://en.wikipedia.org/wiki/Locality_of_reference) — spatial vs temporal locality; data brought into cache one line at a time.
- [Single instruction, multiple data — Wikipedia](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data) — SIMD definition, vector registers and lanes, AVX-512 = 512 bits.
- [Automatic vectorization — Wikipedia](https://en.wikipedia.org/wiki/Automatic_vectorization) — converting a scalar loop into vector operations; the contiguous-array loop kernel; dependence conditions.
- [Optimizing Data Layouts — CedarDB](https://cedardb.com/blog/optimizing_data_layouts/) — cache-line utilization of SoA vs AoS field scans; auto-vectorization of SoA; OLAP-vs-OLTP layout split.
### Columnar analytics and Apache Arrow
- [Column-oriented DBMS — Wikipedia](https://en.wikipedia.org/wiki/Column-oriented_DBMS) — column- vs row-oriented storage; analytical-scan and compression benefits; boolean bitmap example; OLTP strengths of row stores.
- [Apache Arrow — Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) — design goals (sequential-scan adjacency, O(1) random access, SIMD-friendly, zero-copy); 64-byte alignment/padding rationale; analytics-vs-mutation trade-off.
- [Apache Arrow — Introduction](https://arrow.apache.org/docs/format/Intro.html) — columnar vs row layout and its benefits; nested/struct types as bundles of independent child buffers.
- [What is a columnar database — ClickHouse](https://clickhouse.com/resources/engineering/what-is-columnar-database) — selective per-column I/O; columnar compression ratios via dictionary/RLE/delta encoding; vectorized execution lineage.
Determinism-and-Reproducibility.md
+139
@@ -0,0 +1,139 @@
# Determinism and Reproducibility
Determinism is the property that fixing every input to a computation fixes its output exactly — the same inputs always yield the same result, ideally bit-for-bit. For a backtesting engine, where conclusions can move real capital, determinism is not a performance detail but the foundation of trust: it is what makes a result *reproducible*, *auditable*, *debuggable*, and *parallelizable without coordination*. This page defines determinism precisely, derives the practical consequences that follow from it (reproduce-from-manifest, embarrassingly-parallel runs, seed-as-input randomness, and a record-then-replay boundary around anything nondeterministic), and marks which statements are exact laws versus practitioner conventions.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
See also the sibling pages on [Backtesting Engine Architecture](Backtesting-Engine-Architecture), [Event-Driven Backtesting](Event-Driven-Backtesting), [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow), [Graph Compilation and Optimization](Graph-Compilation-and-Optimization), the [Metric Catalogue](Metric-Catalogue), and the [Metric Verification Log](Metric-Verification-Log).
## What determinism means
A **deterministic algorithm** is one that, given a particular input, always produces the same output, with the underlying machine always passing through the same sequence of states. [L] It computes a mathematical function — and a function assigns a unique output to each input in its domain — so the same input can only ever map to the same output. [L]
The state-machine framing makes the "same sequence of states" precise: a state describes what the machine is doing at a particular instant, the machine passes in discrete steps from one state to the next, and in a deterministic system the current state alone determines the next state, so the entire trajectory through the state space is predetermined by the input. [L] This is exactly the discipline an [event-driven backtest](Event-Driven-Backtesting) imposes: a single, synchronous event loop that reaches one well-defined state after each input tick.
Nondeterminism enters when the computed result depends on something other than the declared inputs. The standard sources are: hidden **external state** (user input, global variables, a hardware timer, an unrecorded random draw, or data read from disk); **timing sensitivity** under concurrency (e.g. several threads writing a shared location, where the interleaving order changes the result); and **hardware faults** that flip state unexpectedly. [L] A practical corollary: a computation that reads the wall clock, an environment variable, an un-seeded random generator, or a network response mid-run is nondeterministic *by construction*, regardless of intent. Eliminating these reads — or pushing them to a controlled boundary — is the engineering work of making a system deterministic. [C]
### Bit-for-bit determinism and the build itself
The strongest form of determinism is **bit-for-bit** (byte-for-byte) identity of outputs, not merely equality up to rounding. The analogous discipline for compilation is the **reproducible build**: a build is reproducible if, given the same source code, build environment, and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts. [L] The same enemies recur at the build level — embedded timestamps, file-ordering, locale, build paths, and randomized hash seeds or address-space layout all leak environment into output and must be normalized away. [L]
Reproducible builds matter to a backtesting engine for the audit chain: a deterministic compilation lets a signed source tree be tied to the exact binary it produces, so a deployed artifact can be independently verified to derive from published source rather than from tampered binaries. [L] This is the build-time half of the audit trail; the run-time half (below) ties a *result* to an exact code version plus inputs. The deploy artifact of a validated strategy should therefore be a versioned, frozen binary — *this result came from this commit* — rather than a hot-swapped or rebuilt-on-the-fly target. [C]
## Why determinism matters when money rides on the result
Four properties follow directly from determinism, and each is load-bearing when a backtest informs a capital allocation.
- **Reproducibility.** If output is a function of input, anyone with the same inputs recomputes the same output. A reported Sharpe ratio or drawdown is then a checkable claim, not an anecdote. The replication crisis in empirical science — large fractions of published results failing to reproduce — is the cautionary backdrop: a [2016 *Nature* survey](https://en.wikipedia.org/wiki/Replication_crisis) of 1,576 researchers found that more than 70% had tried and failed to reproduce another scientist's experiments. [C] A nondeterministic backtest is a private experiment of exactly this fragile kind.
- **Auditability / provenance.** Determinism makes a result *traceable*: it can be attributed to an exact code version plus an exact set of inputs. **Provenance** — the formal record of what happened during an experiment — is precisely the variable set that computational reproducibility depends on, and capturing it is what lets a third party (a risk desk, a regulator, a future maintainer) re-derive and audit a number. [L]
- **Debuggability.** A bug that reproduces on demand is a bug that can be fixed; an intermittent one often cannot. Determinism converts "it sometimes misbehaves" into "it misbehaves every time on this input," which is the precondition for the RED-first debugging loop and for the [metric verification log](Metric-Verification-Log) to pin a number against a hand-checked oracle.
- **Trust.** The composite of the three above. A strategy whose results cannot be reproduced, traced, or debugged cannot be trusted with capital, however attractive its headline metrics.
## Reproduce-from-manifest: store the inputs, re-derive the result
Because a deterministic run is *entirely* a function of its inputs, the result need not be stored as a primary artifact — only the inputs need be. The minimal record (call it a manifest) is the set that fixes the run:
- the **code version** (commit hash) of the engine and strategy;
- the **parameters** bound into the run;
- the **data window** (instrument(s), date range, source identity/version);
- the **random seed**(s); and
- any remaining **configuration** that the run reads.
Given that manifest, the full result — every intermediate stream, every metric — is *re-derivable on demand*. [L, from the deterministic-function definition: identical inputs ⇒ identical output.] This is the same principle as **event sourcing**, where application state is not stored as the source of truth but is rebuilt by replaying a log of events: one can discard the materialized state entirely and reconstruct it by re-running the events from the log against an empty start. [L] A reproducible backtest treats its result the way event sourcing treats state — as a derived view of an authoritative, compact input log.
Two practical consequences follow. First, **storage is cheap and honest**: persisting kilobytes of manifest instead of gigabytes of result rows both saves space and removes the risk that a stored result silently diverges from what the current code would now produce. Second, **a stored result is a cache, not a record of truth** — if the manifest re-derives a different number, the cache was stale and the manifest wins. [C] Storing results alongside manifests is still useful as a regression tripwire (re-derive and compare), but the manifest is the authority.
## Disjoint runs are embarrassingly parallel
A workload is **embarrassingly parallel** when little or no effort is needed to split it into parallel tasks, because there is minimal or no dependency — no communication — between those tasks or their results. [L] Independent backtest runs are the textbook case: two runs over disjoint inputs share no mutable state, so neither can observe or perturb the other, and they can execute concurrently with no locks, no message passing, and no ordering constraints. Such problems run well even on loosely-coupled server farms and volunteer-computing pools precisely because they need no constant data exchange. [L]
This yields the single most important design rule for a deterministic engine's performance:
> **Parallelism is *across* runs, never *within* a single run.** [C]
Each individual run stays strictly sequential and deterministic — a single-threaded event loop reaching a unique state after each tick — and *all* the speedup comes from executing many disjoint runs at once. Introducing threads *inside* one run reintroduces exactly the timing-sensitivity nondeterminism defined above (interleaved writes whose order changes the result), forfeiting bit-for-bit reproducibility for a local speedup that the across-runs axis supplies far more safely.
This is not merely a backtesting convention; it is the architecture behind **deterministic simulation testing** of distributed systems, most associated with FoundationDB — whose Flow framework dates to 2009 for C++. There, an entire cluster is simulated inside a single-threaded process so that the run is perfectly repeatable: cooperative scheduling means only one actor runs at a time, so "single-threaded execution means no race conditions — same seed, same event ordering, exact same execution path." [L] The same property that makes a distributed-systems bug reproducible after a trillion simulated operations makes a backtest reproducible after millions of ticks: the run is sequential and seed-determined, and scale comes from running *many* such runs in parallel. The bootstrap/compilation step that flattens a [graph template into a runnable form](Graph-Compilation-and-Optimization) is what produces these disjoint, independently-schedulable units.
## Seed-as-explicit-input: making randomness reproducible
Randomness and determinism are reconciled by making the source of randomness an *explicit input*. A **pseudorandom number generator (PRNG)** is an algorithm that produces a sequence whose statistical properties approximate those of a random sequence, but the sequence "is not truly random, because it is completely determined by an initial value, called the PRNG's seed." [L] A **random seed** is that initializing value, and the defining reproducibility property is exact: "A pseudorandom number generator's number sequence is completely determined by the seed: thus, if a pseudorandom number generator is later reinitialized with the same seed, it will produce the same sequence of numbers." [L] PRNGs are valued precisely for their speed *and* their reproducibility, the latter being a direct consequence of this determinism. [L]
Threading an explicit seed through a run therefore folds all stochastic behavior back under the deterministic-function umbrella: the seed is just another declared input in the manifest, and fixing it fixes the run.
### Monte-Carlo is "a sweep over seeds"
The **Monte-Carlo method** is a broad class of algorithms based on repeated random sampling to obtain numerical results — using randomness to solve (or estimate) deterministic quantities. [L] Its validity rests on the **law of large numbers**: an expectation can be approximated by the empirical mean of independent samples, and a sufficiently large number of samples produces an estimate arbitrarily close to the true value. [L] The samples must be independent of one another for the estimate to be sound. [L]
Under the seed-as-input lens, a Monte-Carlo study decomposes cleanly: it is **a sweep over seeds**, where each realization is a single deterministic run fixed by its own seed, and the study aggregates a statistic (mean, quantiles, a confidence band) over those realizations. Each realization is individually reproducible (re-run its seed, get the identical path), and the study as a whole is reproducible (store the seed set). This makes a Monte-Carlo validation just another embarrassingly-parallel family of disjoint runs — the realizations share no state and parallelize trivially. How the resulting distribution is *interpreted* (robustness, overfitting detection, confidence intervals) belongs to the validation cluster; see the [Metric Catalogue](Metric-Catalogue) and [Metric Verification Log](Metric-Verification-Log).
### Seed pitfalls — two corrections
A seed reproduces a sequence only *within the same PRNG implementation*. The same named algorithm (e.g. a Mersenne Twister or PCG variant), seeded identically, can emit different sequences across different language libraries and framework versions, because seeding details and internal layout differ between implementations. [CORR] The popular shorthand "the same seed always reproduces the same numbers" is therefore true only with the implementation (and its version) held fixed — record it in the manifest alongside the seed value.
A second, costlier correction concerns **parallel** seeding. A widespread idiom is to derive each worker's seed by adding its worker index to a root seed (`worker_seed = root_seed + worker_id`). Authoritative guidance flags this as **unsafe**: "It is quite likely that multiple invocations of the program with different seeds will get overlapping sets of worker seeds … subsets of the workers will return identical results, causing a bias in the overall ensemble of results." [CORR] The correct construction spawns *independent* streams from a single root via a seed-sequence mechanism whose hashing has strong avalanche properties — flipping one input bit flips about half the output bits — so that even adjacent seeds map to far-apart states, with collision probability for ~2²⁰ streams on the order of 2⁻⁸⁸ (negligible). [L] Relatedly, generators that mutate a *global* shared state are fragile under reproducibility: any imported code can reset that global and silently change results, so the recommended practice is to instantiate a local generator object with an explicit seed and thread it through, rather than relying on a process-global seed. [C]
The net rule for a deterministic engine: seeds are inputs, one independent stream per disjoint run, derived by a proper seed-sequence — never by arithmetic on a root seed, never from un-seeded global state.
## The record-then-replay determinism boundary
Some inputs are intrinsically nondeterministic, external, or slow: a wall clock, a network fetch, a web scrape, the output of an LLM or other agent. These cannot be made deterministic in place — but they can be moved to a **boundary** and *recorded* once, so that the deterministic core only ever reads a fixed log.
The pattern is **record-then-replay** (record-and-replay):
- **Record mode** runs the nondeterministic source live, *once*, capturing its outputs into a timestamped stream on disk. Nondeterministic and external inputs are logged exactly as they occurred, so the captured run can later be "replayed again and again, and debugged exactly as it happened." [L]
- **Replay mode** is the backtest: the deterministic core consumes the recorded stream as ordinary input and makes *no* live external call. Because every nondeterministic input has been frozen into the log, the replayed execution is reproducible and can be re-run identically on demand.
This is the technique behind record-and-replay debuggers, which are "particularly useful for debugging intermittent and non-deterministic defects, which can be difficult to reproduce" — including heisenbugs that vanish under observation. [L] It is also exactly the discipline of deterministic simulation testing, which feeds the system entropy in a controlled way "such that the system and workload can still appear to have random behavior, while at the same time being perfectly reproducible." [L]
The crucial invariant for a backtesting engine:
> The deterministic core **never makes a live external call mid-replay.** [C]
A live run (as opposed to a replay) computes fresh against the real external source *and records that computation* into the same timestamped log, so that today's live run becomes tomorrow's reproducible replay input. This cleanly separates the two regimes: live runs sit at the recording edge and may incur external latency, consent requirements, and nondeterminism; replays are pure functions of recorded inputs. (External-service calls — for instance to a hosted LLM acting as a data source — therefore belong at the recording/live edge, under whatever per-session consent policy governs them, and never inside a backtest.)
Conceptually this is event sourcing applied to a research pipeline: the recorded, timestamped stream is the authoritative event log, and every backtest result is a *derived* view that can be discarded and rebuilt from the log. [L] The recording boundary is also where the [single ingestion-time merge](Reactive-Streaming-Dataflow) of heterogeneous timestamped sources naturally lives — by the time data crosses into the deterministic graph, it is one ordered, replayable stream.
## How the pieces compose
The five ideas reinforce one another into a single discipline:
1. **Determinism** (same input ⇒ same output, ideally bit-for-bit) is the root property.
2. It makes a run **reproducible-from-manifest** — store the inputs, re-derive the result — which gives storage economy and an audit trail.
3. Disjoint deterministic runs are **embarrassingly parallel**, so throughput comes from running many at once: parallel *across* runs, sequential *within* a run.
4. **Seeds are explicit inputs**, so randomness is reproducible and a Monte-Carlo study is just a parallel sweep over seeds.
5. A **record-then-replay boundary** quarantines everything that cannot be made deterministic in place, so the core stays a pure, replayable function of recorded inputs.
Together these are what let a backtest be trusted with money: a reported number is not a one-off observation but a checkable, re-derivable, attributable claim. See [Backtesting Engine Architecture](Backtesting-Engine-Architecture) for how the engine is structured around these invariants, and [Event-Driven Backtesting](Event-Driven-Backtesting) for the single-threaded event loop at the core.
## References
### What determinism means
- [Deterministic algorithm — Wikipedia](https://en.wikipedia.org/wiki/Deterministic_algorithm)
- [Reproducible builds — reproducible-builds.org definition](https://reproducible-builds.org/docs/definition/)
- [Reproducible builds — Wikipedia](https://en.wikipedia.org/wiki/Reproducible_builds)
### Why determinism matters
- [Replication crisis — Wikipedia (Baker 2016 *Nature* survey of 1,576 researchers, Nature 533, 452-454)](https://en.wikipedia.org/wiki/Replication_crisis)
- [Reproducibility / replication and provenance (computational reproducibility survey, IEEE Data Engineering Bulletin)](http://sites.computer.org/debull/A18mar/p15.pdf)
### Reproduce-from-manifest
- [Event Sourcing — Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html)
- [Deterministic algorithm — Wikipedia](https://en.wikipedia.org/wiki/Deterministic_algorithm)
### Embarrassingly-parallel runs
- [Embarrassingly parallel — Wikipedia](https://en.wikipedia.org/wiki/Embarrassingly_parallel)
- [Diving into FoundationDB's Simulation Framework — Pierre Zemb](https://pierrezemb.fr/posts/diving-into-foundationdb-simulation/)
- [Deterministic simulation testing — Antithesis docs](https://antithesis.com/docs/resources/deterministic_simulation_testing/)
### Seed-as-explicit-input and Monte-Carlo
- [Random seed — Wikipedia](https://en.wikipedia.org/wiki/Random_seed)
- [Pseudorandom number generator — Wikipedia](https://en.wikipedia.org/wiki/Pseudorandom_number_generator)
- [Monte Carlo method — Wikipedia](https://en.wikipedia.org/wiki/Monte_Carlo_method)
- [Parallel random number generation — NumPy manual](https://numpy.org/doc/stable/reference/random/parallel.html)
- [Stop using NumPy's global random seed — Built In](https://builtin.com/data-science/numpy-random-seed)
### Record-then-replay boundary
- [Record and replay debugging — Wikipedia](https://en.wikipedia.org/wiki/Record_and_replay_debugging)
- [Deterministic simulation testing — Antithesis docs](https://antithesis.com/docs/resources/deterministic_simulation_testing/)
- [Event Sourcing — Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html)
Event-Driven-Backtesting.md
+353
@@ -0,0 +1,353 @@
# Event-Driven Backtesting
> Backtesting can be built two structurally different ways. The **vectorized**
> model applies whole-series array operations and is fast but easy to leak the
> future. The **event-driven** model processes one timestamped market event at a
> time in strict chronological order — slower, but realistic and reusable for
> live trading because the very same loop also consumes a live feed. This page
> contrasts the two models, describes the data-driven clock that drives an
> event loop, explains how heterogeneous sources are merged once at ingestion,
> and states the backtest/live symmetry that the event model buys. Narrative
> context on the engine side lives on
> **[Backtesting Engine Architecture](Backtesting-Engine-Architecture)**; the
> causal guarantees on
> **[Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)**.
**Markers:** [L] law/exact, [C] convention, [CORR] corrected.
A backtest, generically, is the *testing of a predictive model on historical
data*: it "seeks to estimate the performance of a strategy or model if it had
been employed during a past period" ([Backtesting][bt-wiki]) **[L]**. The two
models below differ only in *how* that estimate is computed — not in what it
estimates. The choice has consequences for speed, for how easily look-ahead bias
creeps in, for how faithfully path-dependent execution is modelled, and for
whether the same code can be redeployed live.
Related pages: **[Reactive Streaming Dataflow](Reactive-Streaming-Dataflow)**
(the node/stream model an event loop drives),
**[Determinism and Reproducibility](Determinism-and-Reproducibility)**
(why the merge and the loop must be deterministic),
**[Metric Catalogue](Metric-Catalogue)** and
**[Metric Verification Log](Metric-Verification-Log)** (what is computed from the
resulting equity curve).
---
## 1. Two models of backtesting
### 1.1 The vectorized model
A vectorized (also called array-based or, in its simplest form, *for-loop*)
backtester treats the whole price history as columns and applies array
operations across the entire series at once: compute an indicator over the full
column, compare two full columns to derive a signal column, multiply the signal
column by a forward-return column to get a P&L column. The attraction is raw
speed and brevity — "For-Loop backtesters are straightforward to implement in
nearly any programming language and are very fast to execute"
([QuantStart: Should You Build Your Own Backtester?][qs-build]) **[C]**. That
speed makes vectorized code the natural tool for early research and for sweeping
many parameter combinations cheaply.
The model has two structural weaknesses, both of which are *easy to introduce
and hard to see*:
- **Look-ahead leakage through indexing.** Because the whole series is in scope
at once, a single off-by-one — "should you have used `i`, `i+1` or `i-1`?" —
silently lets a signal at time *t* read data from *t+1*
([QuantStart: Should You Build Your Own Backtester?][qs-build]) **[C]**. The
computation has no built-in notion of "now"; nothing structurally prevents a
cell from referencing a future cell. See
**[Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)** for why
this is the dominant failure mode of array backtests.
- **Coarse execution.** Vectorized P&L typically assumes a fill at the next
bar's open or close and ignores intra-bar path detail (slippage, partial
fills, the bid–ask spread, order-book state). Path-dependent execution —
stops, trailing exits, position-dependent sizing — does not express cleanly as
a whole-column operation, because each step's action depends on the realized
outcome of the previous step.
These are *conventions about practice*, not laws: a disciplined vectorized
backtester can avoid both with careful shifting and explicit lag columns. The
point is that the model gives no *structural* help — correctness rests on the
author's vigilance.
### 1.2 The event-driven model
An event-driven backtester instead simulates the passage of time as a *queue of
discrete events* consumed one at a time. Each incoming market record (a tick, a
completed bar, a news item) is "treated as an 'event' that must be acted upon"
([QuantStart: Event-Driven Backtesting with Python][qs-event]) **[L]**. A loop
pulls the next event, lets the strategy react to it, lets any resulting orders be
handled, and only then advances to the following event. QuantStart describes the
canonical two-loop shape: an outer loop that "give[s] the backtester a heartbeat"
and an inner loop that "actually handles the Events from the events Queue object"
([QuantStart: Event-Driven Backtesting with Python][qs-event]) **[C]**.
The defining advantage is that, *by construction*, the strategy can only see
events that have already arrived. "With an event-driven backtester there is no
lookahead bias as market data receipt is treated as an 'event'"
([QuantStart][qs-event]) **[L for the mechanism; the practical guarantee is
[C]]** — more precisely, "by virtue of its message-passing design, Event-Driven
systems are usually free from Look-Ahead Bias, at least at the trading level"
([QuantStart: Should You Build Your Own Backtester?][qs-build]) **[C]**. Because
the engine is "drip fed" one event at a time, the future is not in scope; the
strategy literally cannot index into it.
The same one-event-at-a-time discipline is what permits realistic execution:
"event-driven backtesters allow significant customisation over how orders are
executed and transaction costs are incurred"
([QuantStart][qs-event]) **[C]**. Path-dependent logic is natural here, because
each event handler runs after the previous one's effects are fully realized.
### 1.3 The trade-off, stated plainly
The cost is speed. Event-driven systems "are slower to execute compared to a
vectorised system" because "optimal vectorised operations are unable to be
utilised when carrying out mathematical calculations"
([QuantStart][qs-event]) **[C]**. The two models therefore occupy opposite ends
of a spectrum: vectorized for fast, exploratory research where a leaked bar is a
tolerable risk; event-driven where realism, path-dependence, and live-reuse
matter ([QuantStart: Should You Build Your Own Backtester?][qs-build]) **[C]**.
A common and reasonable workflow is to *prototype vectorized, validate
event-driven* **[C]**: use array operations to scan the idea space cheaply, then
re-run the survivors through an event loop that cannot cheat and that models
execution faithfully. This is practitioner guidance, not a rule — some teams
build event-driven from the start to keep a single codebase.
| dimension | vectorized | event-driven |
|---|---|---|
| unit of work | whole series / column | one event at a time |
| speed | fast (array ops) **[C]** | slower (per-event loop) **[C]** |
| look-ahead risk | high — no structural guard **[C]** | low — future not in scope **[L mechanism]** |
| execution realism | coarse (next-bar fill) **[C]** | fine (custom fills, frictions) **[C]** |
| path-dependence | awkward | natural |
| live reuse | not directly **[C]** | same loop consumes a live feed **[L]** |
---
## 2. The data-driven clock
### 2.1 Markets are an irregular event stream, not a grid
Real market activity is "an irregular sequence of timestamped events": ticks
arrive when trades happen, bars complete on their own cadence, news lands
whenever it lands. There is no global metronome. An event-driven engine therefore
takes its clock *from the data*: it advances **one input record = one logical
step**, in global timestamp order. This is exactly the next-event model of
discrete-event simulation, in which a system is modelled as "a discrete sequence
of events in time" and "each event occurs at a particular instant in time and
marks a change of state in the system" ([Discrete-event simulation][des-wiki])
**[L]**.
### 2.2 Next-event time advance vs a fixed grid
The simulation clock does **not** tick at a fixed rate. In next-event time
progression, "between consecutive events, no change in the system is assumed to
occur; thus the simulation time can directly jump to the occurrence time of the
next event" ([Discrete-event simulation][des-wiki]) **[L]**. The alternative —
*incremental* or fixed-increment time advance — divides time into uniform slices
and processes each one; this "approach typically runs slower than next-event
simulation because many time slices require processing"
([Discrete-event simulation][des-wiki]) **[L]**.
For a backtest this distinction is not academic:
- **A fixed grid that is too coarse clumps events.** Several records that fall in
the same slice get processed together, collapsing their order and destroying
exactly the per-event causality the model exists to protect.
- **A fixed grid that is too fine wastes empty steps.** Most slices contain no
event; the loop spends its time advancing a clock past nothing. The
next-event model "skip[s] periods where nothing happens"
([Discrete-event simulation][des-wiki]) **[L]**, which is both more efficient
and a closer match to how markets actually behave.
Advancing one record at a time, in timestamp order, sidesteps both failure
modes: every event is processed exactly once, in its true position, and no clock
cycles are spent on emptiness. See
**[Determinism and Reproducibility](Determinism-and-Reproducibility)** for why
this stepping must also be reproducible, and
**[Reactive Streaming Dataflow](Reactive-Streaming-Dataflow)** for how each step
propagates through a graph of nodes.
---
## 3. Merging heterogeneous sources
### 3.1 The problem: many cadences, one timeline
A realistic strategy reads from several streams at once — high-rate ticks,
lower-rate completed bars, and intermittent, irregular news or sentiment events.
Each stream is individually sorted by time, but they have wildly different
cadences. For the event loop to honour causality, these must be presented as a
*single chronological sequence*: at every step, the engine must process the
globally earliest unprocessed record, whichever stream it came from. Otherwise a
slower stream could deliver an event "late" relative to wall-clock processing and
let a later strategy step see something it should not have, or miss something it
should have.
### 3.2 The k-way merge
Combining several already-sorted sequences into one sorted sequence is exactly
the **k-way merge** (multiway merge) problem. K-way merge algorithms "are a
specific type of sequence merge algorithms that specialize in taking in k sorted
lists and merging them into a single sorted list"
([K-way merge algorithm][kway-wiki]) **[L]**. More generally, merge algorithms
"take multiple sorted lists as input and produce a single list as output,
containing all the elements of the inputs lists in sorted order"
([Merge algorithm][merge-wiki]) **[L]**. The k-way case is the generalization of
the binary merge step at the heart of merge sort: "a K-way merge method, a
generalization of binary merge, in which k sorted sequences are merged"
([Merge sort][msort-wiki]) **[L]**.
The efficient implementation uses a **priority queue / min-heap** keyed by the
front element of each stream. One maintains "a min-heap *h* of the *k* lists,
using the first element as the key" ([Merge algorithm][merge-wiki]) **[L]**; the
heap repeatedly yields the globally smallest front element, that record is
emitted, and its stream is advanced by one and re-inserted. A priority queue is
the right abstraction here: "each element has an associated priority, which
determines its order of service [and the queue] serves highest priority items
first" ([Priority queue][pq-wiki]) **[L]** — with the ordering chosen so that the
*earliest timestamp* has highest priority.
**Complexity [L].** Merging *k* sorted sequences with a total of *n* elements via
a binary heap runs in **O(n log k)** time, versus **O(k·n)** for the naive
scan-all-fronts approach ([K-way merge algorithm][kway-wiki],
[Merge algorithm][merge-wiki]). The heap holds at most *k* entries, so the extra
space is **O(k)**. A binary heap gives "O(log n) performance for inserts and
removals" ([Priority queue][pq-wiki]) **[L]**, which is what yields the `log k`
factor when the heap holds *k* stream fronts.
> **[CORR] Heap vs tournament tree.** A frequent claim is that the min-heap is
> the *fastest* practical k-way merger. The k-way-merge reference corrects this:
> "a tournament tree is faster in practice. A heap uses approximately 2·log(k)
> comparisons in each step ... [a] tournament tree only needs log(k)
> comparisons" ([K-way merge algorithm][kway-wiki]). Both are O(n log k); the
> constant factor differs. Treat "use a min-heap" as the *standard, correct, and
> simplest* choice **[C]**, not as the comparison-optimal one.
### 3.3 The tie-break for equal timestamps
Two records from different streams can carry the *same* timestamp. The merge must
then apply a **deterministic tie-break**, because the relative order of
equal-keyed records is not fixed by the key alone. The plain merge routine "is
not stable: if equal items are separated ... they will become interleaved"
([Merge algorithm][merge-wiki]) **[L]** — i.e. without an explicit rule the order
of simultaneous events is unspecified. A *stable* merge, by contrast, preserves a
defined order of equal elements: most merge-sort implementations "are stable,
which means that the relative order of equal elements is the same between the
input and output" ([Merge sort][msort-wiki]) **[L]**.
For a backtest the practical rule **[C]** is to make the tie-break total and
fixed — for example, break ties by a stable per-source priority (a stream rank),
then by per-source arrival order — so that equal-timestamp events are always
processed in the *same* order on every run. Which order is chosen matters less
than that it is deterministic; ambiguity at equal timestamps is a classic source
of irreproducible results (see
**[Determinism and Reproducibility](Determinism-and-Reproducibility)**).
### 3.4 Why merge once, at the ingestion boundary
A load-bearing architectural choice is to perform this merge **exactly once, at
the ingestion edge**, producing a single ordered timeline that everything
downstream consumes. The reasons are causal and semantic:
- **One ordered timeline is what makes heterogeneous-rate sources combinable
without leaking the future.** Once every record sits in one global order, "see
only the past" reduces to "see only records earlier in this one sequence." A
fast and a slow stream are reconciled by position in the merged order, not by
ad-hoc rate matching. The next-event clock of §2 then simply walks this
sequence.
- **Merging only at the boundary keeps the computation semantics simple.** If
alignment of differently-timed streams were instead done *inside* the
computation — scattered as-of joins or "most-recent value as of *t*" lookups at
many nodes — every such site would be a fresh opportunity to misalign and a
fresh place where a future value could leak in. Concentrating all
cross-stream ordering at one boundary means the downstream graph never
performs a temporal join at all; it only ever reacts to the next event handed
to it. This is the dataflow discipline described on
**[Reactive Streaming Dataflow](Reactive-Streaming-Dataflow)** and is closely
tied to the structural look-ahead guarantees on
**[Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)**.
In discrete-event terms, the merged stream *is* the event list of the
simulation, and the loop is next-event time advance over it
([Discrete-event simulation][des-wiki]) **[L]**.
---
## 4. Backtest / live symmetry
The deepest payoff of the event-driven model is that **backtest and live trading
share identical event semantics; only the origin of the events differs.** In a
backtest the events are replayed from historical records; live, they arrive from
a market feed. The handling loop — react, order, advance — is the same. An
event-driven backtester "by design, can be used for both historical backtesting
and live trading with minimal switch-out of components"
([QuantStart: Event-Driven Backtesting with Python][qs-event]) **[L]**, and such
systems "are much more akin to live-trading infrastructure implementations"
([QuantStart: Should You Build Your Own Backtester?][qs-build]) **[C]**. In
practice only the data origin and the execution endpoint are swapped: replace the
historical event source with a live feed and the simulated fill model with a real
broker connection, and the strategy logic is untouched.
This symmetry is structurally identical to event-driven architecture in
software at large, where an event is "a significant change in state"
([Event-driven architecture][eda-wiki]) **[L]**, emitters "detect, gather, and
transfer events" without knowing their consumers, and consumers (sinks) apply "a
reaction as soon as an event is presented"
([Event-driven architecture][eda-wiki]) **[L]**. A backtest engine and a live
engine are two emitters feeding the same consumer graph. Because the strategy
sees only the event interface — never the origin — a strategy validated in the
loop is the *same artifact* that runs live, eliminating the
research-to-production rewrite that plagues vectorized pipelines.
Two consequences follow **[C]**:
- **Less code to trust.** A single event-handling path is exercised by both
backtests and live trading, so bugs in that path surface during research
rather than in production.
- **Fidelity is bounded by the source and the execution model, not the loop.**
Since the loop is shared, any backtest-vs-live divergence is localized to the
two swapped components — the event origin (was the recorded data faithful?) and
the fill model (does the simulated execution match the broker?). The realistic
execution model is discussed on
**[Backtesting Engine Architecture](Backtesting-Engine-Architecture)**.
A caveat **[C]**: symmetry of *semantics* does not by itself guarantee symmetry
of *results*. Live feeds carry latency, gaps, and revisions that a clean
historical replay omits; the recorded source must be captured faithfully (and
deterministically — see
**[Determinism and Reproducibility](Determinism-and-Reproducibility)**) for the
backtest to be a fair proxy for the live system.
---
## References
### Backtesting and the two models (§1, §4)
- [Backtesting — Wikipedia][bt-wiki]
- [Event-Driven Backtesting with Python, Part I — QuantStart][qs-event]
- [Should You Build Your Own Backtester? — QuantStart][qs-build]
### The data-driven clock (§2)
- [Discrete-event simulation — Wikipedia][des-wiki]
### Merging heterogeneous sources (§3)
- [K-way merge algorithm — Wikipedia][kway-wiki]
- [Merge algorithm — Wikipedia][merge-wiki]
- [Merge sort — Wikipedia][msort-wiki]
- [Priority queue — Wikipedia][pq-wiki]
### Event-driven architecture (§4)
- [Event-driven architecture — Wikipedia][eda-wiki]
[bt-wiki]: https://en.wikipedia.org/wiki/Backtesting
[qs-event]: https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-I/
[qs-build]: https://www.quantstart.com/articles/Should-You-Build-Your-Own-Backtester/
[des-wiki]: https://en.wikipedia.org/wiki/Discrete-event_simulation
[kway-wiki]: https://en.wikipedia.org/wiki/K-way_merge_algorithm
[merge-wiki]: https://en.wikipedia.org/wiki/Merge_algorithm
[msort-wiki]: https://en.wikipedia.org/wiki/Merge_sort
[pq-wiki]: https://en.wikipedia.org/wiki/Priority_queue
[eda-wiki]: https://en.wikipedia.org/wiki/Event-driven_architecture
Graph-Compilation-and-Optimization.md
+130
@@ -0,0 +1,130 @@
# Graph Compilation and Behaviour-Preserving Optimization
Constructing a runnable computation graph from a higher-level description is profitably viewed as a **compilation**: a parameter-generic graph specification is lowered, in well-defined stages, into a concrete, flat, executable instance. Once a dataflow graph is treated as inspectable data rather than an opaque black box, the same family of transformations that an [optimizing compiler](https://en.wikipedia.org/wiki/Optimizing_compiler) applies to instruction streams — common-subexpression elimination, dead-code elimination, loop-invariant code motion — applies to the graph, and for the same reason. This page describes the compilation pipeline, the correctness criterion that licenses every rewrite (behaviour preservation, made decidable by [determinism](Determinism-and-Reproducibility)), the classical optimizations and how they map onto a dataflow graph, and why a flat, graph-as-data intermediate representation is the precondition for any of it.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
## Graph-as-data to flat instance: the compilation
A **parameter-generic graph specification** (a graph template) describes a topology with free numeric parameters (window lengths, thresholds, buffer depths) and free input roles (which streams feed which nodes), but is not yet runnable: buffers are unsized, the input bindings are symbolic, and nested sub-graph abstractions are still folded as named groupings. Producing something executable from it is a translation between two representations — the textbook definition of compilation, where a source representation is lowered into a target representation conducive to execution. The target here is a concrete, runnable instance.
The lowering binds three things and fixes the rest:
- **Parameters** — every free numeric parameter receives a concrete value, which both *configures* node behaviour and *sizes* node state. Buffer and window sizes are derived once parameters are known; this is a static, graph-level property — in synchronous dataflow, where "the number of tokens read and written by each process is known ahead of time," a static schedule can be found such that "channels have bounded FIFOs," i.e. the channel buffer sizes are fixed by analysis of the graph rather than discovered at run time [[synchronous dataflow](https://en.wikipedia.org/wiki/Synchronous_Data_Flow)]. [C]
- **Data and seed** — the concrete input streams are bound to the symbolic input roles, and any stochastic element is pinned by a seed so the instance is reproducible (see [Determinism and Reproducibility](Determinism-and-Reproducibility)). [C]
- **Topology** — fixed: the set of nodes and edges no longer changes. A parameter never adds or removes a node; a change of topology is a *different* template, not a different parameter set. [C]
Two further lowering steps make the result analysable:
- **Inlining of nested sub-graphs.** A composite — a named sub-graph used as a reusable building block — is an authoring-level convenience, not a runtime object. Compilation **inlines** it: the composite boundary is dissolved and its internal nodes are spliced directly into the enclosing graph. This mirrors [inline expansion](https://en.wikipedia.org/wiki/Inline_expansion) in a compiler, which "replaces a function call site with the body of the called function." Crucially, inlining is an *enabling* transformation: once a body is expanded in the context of its use site, "it may be able to do a variety of transformations that were not possible before" — dead-code elimination, loop-invariant code motion, constant propagation [[inline expansion](https://en.wikipedia.org/wiki/Inline_expansion)]. [L]
- **Lowering to a flat, index-wired representation.** The final instance is a single flat graph wired by raw integer index rather than by name. Names survive, if at all, only as non-load-bearing debug symbols. This flat form is the **intermediate representation** that optimization passes operate on (see the last section). [C]
### "No recompile" is two different statements
A persistent source of confusion: re-deriving an instance for a new parameter set is **not** a source or native rebuild.
- **Code recompilation** rebuilds the native artifact from source text — slow, and unnecessary when only parameters change. [L]
- **Graph recompilation** re-runs the cheap graph-lowering above to produce a fresh instance from the *already-built* template. This is the inner loop of a parameter sweep and costs a graph traversal, not a toolchain invocation. [C]
Stating "no recompile per parameter set" should always be read in the first sense (no native rebuild); a sweep still performs the second kind of (cheap) re-compilation per point. Conflating the two overstates the cost of exploring a parameter space. [C]
## Determinism as the licence to rewrite
Every transformation in this pipeline is a **program transformation**: "any operation that takes a computer program and generates another program" [[program transformation](https://en.wikipedia.org/wiki/Program_transformation)]. A transformation is **valid** — permitted — exactly when it is *behaviour-preserving*: "the transformed program is required to be semantically equivalent to the original, relative to a particular formal semantics" [[program transformation](https://en.wikipedia.org/wiki/Program_transformation)]. [L] An optimizing compiler is constrained the same way: its transformations "produce semantically equivalent code optimized for some aspect" [[optimizing compiler](https://en.wikipedia.org/wiki/Optimizing_compiler)]. [L]
The sharp formal version is **observational equivalence**: two program phrases are observationally equivalent if, in any context, they give rise to the same observations — one cannot distinguish them no matter what context they are placed in. Semantic preservation is "the standard semantic preservation property required in source-to-source program rewriting, optimizing compilation, equivalence verification" [[semantics-preserving transformations](https://www.emergentmind.com/topics/semantically-preserving-transformations)]. [L]
The catch is that "same observable output" must be **decidable** for a rewrite to be applied safely, and this is where determinism earns its keep:
- **[Referential transparency](https://en.wikipedia.org/wiki/Referential_transparency)** is the property that "replacing a subexpression with another one that denotes the same value does not change the value of the expression" [[referential transparency](https://en.wikipedia.org/wiki/Referential_transparency)]. [L] It "allow[s] the programmer and the compiler to reason about program behavior as a rewrite system" and is precisely what enables "memoization, common subexpression elimination, lazy evaluation, constant folding, parallelization" [[referential transparency](https://en.wikipedia.org/wiki/Referential_transparency)]. [L]
- A **deterministic** node is referentially transparent by construction: same inputs and same bound parameters produce the same output, every time. Therefore "is the result the same?" reduces to "are the operator, parameters, and inputs the same?" — a *structural* check on the graph, not a semantic theorem to prove per case. [L]
- Two computations with no shared mutable state are independent and may be reordered or run in parallel without changing the result — the same property that makes disjoint simulations concurrently executable underlies the safety of these rewrites. See [Determinism and Reproducibility](Determinism-and-Reproducibility). [L]
Without determinism, "same result" is undecidable in general, and the rewrites below would be unsound. Determinism is therefore not merely a reproducibility nicety; it is the *enabling precondition* for behaviour-preserving optimization of the graph. [C]
## Classical optimizations on a dataflow graph
The three workhorse optimizations of classical compilers transfer directly to a dataflow graph. Each is defined below from compiler theory, then mapped onto graph nodes and edges.
### Common-subexpression elimination (CSE)
**Definition.** CSE "searches for instances of identical expressions (i.e., they all evaluate to the same value), and analyzes whether it is worthwhile replacing them with a single variable holding the computed value" [[CSE](https://en.wikipedia.org/wiki/Common_subexpression_elimination)]. [L] An expression is *available* at a point if every path to that point already evaluates it with no intervening change to its operands [[CSE](https://en.wikipedia.org/wiki/Common_subexpression_elimination)]. [L] CSE comes in **local** (within a single basic block) and **global** (whole-procedure) variants [[CSE](https://en.wikipedia.org/wiki/Common_subexpression_elimination)]. [L]
**On the graph.** Two sub-computations are *identical* when they have the **same operator, the same bound parameters, and the same input edges**. Such duplicate sub-graphs are merged into one node whose single output **fans out** to all former consumers. The redundant computation is performed once and shared, so reusing a building block (a moving average, a normalization) many times costs no redundant compute. [C] Because dataflow nodes are deterministic, syntactic identity (same operator + params + inputs) already implies value identity — the availability condition is satisfied structurally. [L]
**Value-equivalence beyond syntax.** Lexical CSE matches only textually identical expressions; **[global value numbering](https://en.wikipedia.org/wiki/Value_numbering) (GVN)** instead "tries to determine an underlying equivalence" by assigning the same value number to expressions that compute the same value, and so "sometimes helps eliminate redundant code that common subexpression elimination (CSE) does not" [[value numbering](https://en.wikipedia.org/wiki/Value_numbering)]. [L] In production compilers GVN "eliminate[s] fully and partially redundant instructions" and also performs redundant load elimination [[LLVM passes](https://llvm.org/docs/Passes.html)]. [L] On a graph, GVN-style reasoning can unify nodes that are not byte-identical but provably compute the same stream (e.g. two differently-ordered but commutative reductions). [C]
### Dead-code / dead-node elimination (DCE)
**Definition.** DCE removes "code that does not affect the program results" [[DCE](https://en.wikipedia.org/wiki/Dead-code_elimination)]. [L] It covers two distinct categories: **unreachable code** (can never execute) and **dead stores** — "code that only affects dead variables (written to, but never read again)" [[DCE](https://en.wikipedia.org/wiki/Dead-code_elimination)]. [L] The two are found by different analyses: unreachable code by control-flow reachability, dead computations by **live-variable / data-flow analysis** [[DCE](https://en.wikipedia.org/wiki/Dead-code_elimination)]. [L] Iterative variants re-check the inputs of just-removed instructions to see if they have become newly dead [[LLVM passes](https://llvm.org/docs/Passes.html)]. [L]
**On the graph.** A node is **dead** if its output reaches **no observable output** — that is, no [sink](Reactive-Streaming-Dataflow). Liveness propagates backward from sinks: mark every sink live, then mark live any node feeding a live node; everything left unmarked is dead and is dropped. This is exactly backward live-variable propagation over the DAG. [C] The reduction can cascade: removing one dead node may render its sole upstream producer dead in turn — the iterative re-check above. [L]
**The side-effect caveat — never eliminate a sink.** DCE is sound only because it removes code whose *results* are unused; code with **observable side effects must be preserved**. Eliminating a binding is safe only when the compiler can establish it has no side effect; the conservative rule is to preserve any "code with observable behavior" — code that "[m]ust [be] preserve[d] despite no apparent data usage," including I/O and other externally visible operations — so semantics are not changed by the removal [[DCE side effects](https://app.studyraid.com/en/read/15449/536640/dead-code-elimination-techniques)]. [C] A [sink](Reactive-Streaming-Dataflow) **is** the observable effect — it records output into the result registry — so a sink is *never* dead, even if nothing downstream reads it, and is **never eliminated**. Liveness is defined *relative to the sinks*; the sinks are the roots of the live set, not candidates for removal. [C] Eliminating a sink because "nothing consumes its output" is the classic DCE bug: confusing a value-producing node with an effect-producing one.
### Loop-invariant code motion over a parameter sweep
**Definition.** [Loop-invariant code motion](https://en.wikipedia.org/wiki/Loop-invariant_code_motion) (LICM, "hoisting") moves a computation out of a loop when it "produces identical results regardless of which iteration executes it"; the condition is that "all reaching definitions for the operands ... are outside of the loop," in which case "the expression can be moved out of the loop" to before the loop entry [[LICM](https://en.wikipedia.org/wiki/Loop-invariant_code_motion)]. [L] The payoff: hoisted code "is executed less often" [[LICM](https://en.wikipedia.org/wiki/Loop-invariant_code_motion)]. [L] In a production compiler the LICM pass "attempt[s] to remove as much code from the body of a loop as possible ... by hoisting code into the preheader block" [[LLVM passes](https://llvm.org/docs/Passes.html)]. [L]
**The "loop" in a sweep.** A **parameter sweep** runs the same graph template many times, once per value of a swept parameter — the loop body is one graph evaluation, the loop variable is the swept parameter. The graph then partitions in two:
- The **sweep-invariant frontier**: every node that does **not** depend on the swept parameter *and* whose inputs are all themselves sweep-invariant. By the LICM condition (all reaching definitions outside the loop), this sub-graph computes the same streams at every sweep point. [L]
- The **parameter-dependent suffix**: the swept node and everything transitively downstream of it.
The invariant sub-graph is computed **once** and its result is shared **read-only** across all sweep points; only the dependent suffix re-runs per point. [C] This is LICM with the sweep as the loop: the shared prefix is "hoisted" out of the per-point work. It composes with CSE across the *family* of instances — the invariant frontier is a common sub-expression shared by every member of the sweep, computed once and fanned out (cross-link [Backtesting Engine Architecture](Backtesting-Engine-Architecture) for how a sweep family is orchestrated). [C]
A subtlety inherited from LICM: hoisting is only sound when the moved work is genuinely invariant. Mis-classifying a parameter-dependent node as invariant would share a wrong result across points — the analogue of LICM's safety conditions, where a computation must be proven invariant (and side-effect-free / safe to speculate) before it is moved [[LICM](https://en.wikipedia.org/wiki/Loop-invariant_code_motion)]. [L]
### How the optimizations stack
These passes reinforce one another, which is why compilers run them as a pipeline:
- **Inlining first** dissolves composite boundaries so that duplicate building blocks inside different composites become visible to CSE [[inline expansion](https://en.wikipedia.org/wiki/Inline_expansion)]. [L]
- **CSE / GVN** then merges the now-visible duplicates. [C]
- **DCE** prunes whatever no longer reaches a sink (including nodes orphaned by a merge). [C]
- **LICM over the sweep** hoists the surviving invariant frontier out of the per-point loop. [C]
SSA-style intermediate forms make this stacking cheap: "most optimizations can be adapted to preserve [SSA] form, so that one optimization can be performed after another with no additional analysis" [[SSA](https://en.wikipedia.org/wiki/Static_single-assignment_form)]. [L] A dataflow DAG with single-output nodes and explicit input edges is already close to SSA in spirit — each node defines one value, used by its consumers — which is part of why these passes port over so directly. [C]
## Why a flat, inspectable representation is the precondition
None of the above is possible against an opaque sub-engine. Every one of these analyses is a **whole-graph** query:
- CSE/GVN must find identical sub-computations **across** former composite boundaries — it has to see inside them.
- DCE must trace liveness **backward from every sink** through the entire graph.
- LICM over a sweep must identify the **invariant frontier**, which requires knowing the full dependency structure and which nodes touch the swept parameter.
All three are cross-boundary analyses, and all three demand that the graph be available as inspectable data. This is the role of an **[intermediate representation](https://en.wikipedia.org/wiki/Intermediate_representation)**: "the data structure or code used internally by a compiler ... designed to be conducive to further processing, such as optimization and translation," and required to represent the program "without loss of information" [[IR](https://en.wikipedia.org/wiki/Intermediate_representation)]. [L] A graph-based IR exists precisely so that flow analysis and re-arrangement can be performed before execution [[IR](https://en.wikipedia.org/wiki/Intermediate_representation)]. [L]
The streaming-systems analogue is the split between a **logical graph** and a **physical graph**: the logical graph "[r]epresents the high-level structure defined by the user's code or query," and an optimizer/planner compiles it into the physical graph — "[t]he detailed execution plan generated by the system's optimizer and planner" — a step that, as the source puts it, "might involve fusing operators, reordering operations, choosing optimal join strategies, or determining appropriate state backends" [[dataflow graph](https://risingwave.com/glossary/dataflow-graph/)]. [C] The parameter-generic template is the logical graph; the flat, index-wired instance is the physical graph; the optimizer in between is exactly the pass pipeline above. [C]
The design consequence is decisive:
- A **flat, graph-as-data instance** exposes the whole topology to analysis, so CSE, DCE, and sweep-level LICM are all available. [C]
- An **opaque sub-engine boundary** — a node implemented as a black box that the host graph cannot see into — **forecloses** every cross-boundary optimization: duplicates inside it cannot be merged with duplicates outside it, its internal dead nodes cannot be pruned by the host, and it cannot participate in the invariant-frontier computation. The boundary is a wall to analysis. [C]
This is why inlining composites down to a single flat representation is not cosmetic: it is the step that *creates the analysable IR*. The flat graph is the object the optimizer rewrites; behaviour preservation (grounded in [determinism](Determinism-and-Reproducibility)) is the correctness invariant the rewrites must respect; and the [columnar data layout](Columnar-Data-Layout) of the streams flowing along its edges is what makes the surviving, optimized computation fast to execute. The outputs that any of this is allowed to change are exactly the sink-recorded streams — see [Metric Catalogue](Metric-Catalogue) and [Metric Verification Log](Metric-Verification-Log) for what those observable outputs are and how their invariance under optimization is checked.
## References
### Compilation, IR, and lowering
- [Intermediate representation — Wikipedia](https://en.wikipedia.org/wiki/Intermediate_representation)
- [Static single-assignment form — Wikipedia](https://en.wikipedia.org/wiki/Static_single-assignment_form)
- [Inline expansion — Wikipedia](https://en.wikipedia.org/wiki/Inline_expansion)
- [Synchronous Data Flow (bounded FIFOs / channel buffer sizes fixed by static schedule) — Wikipedia](https://en.wikipedia.org/wiki/Synchronous_Data_Flow)
- [Dataflow Graph: logical vs physical graph and the optimizer — RisingWave glossary](https://risingwave.com/glossary/dataflow-graph/)
### Behaviour preservation as the correctness criterion
- [Program transformation (semantics-preserving definition) — Wikipedia](https://en.wikipedia.org/wiki/Program_transformation)
- [Optimizing compiler (semantically equivalent transformations) — Wikipedia](https://en.wikipedia.org/wiki/Optimizing_compiler)
- [Referential transparency — Wikipedia](https://en.wikipedia.org/wiki/Referential_transparency)
- [Semantically-preserving transformations / observational equivalence — Emergent Mind](https://www.emergentmind.com/topics/semantically-preserving-transformations)
### Classical optimizations
- [Common subexpression elimination — Wikipedia](https://en.wikipedia.org/wiki/Common_subexpression_elimination)
- [Value numbering / global value numbering — Wikipedia](https://en.wikipedia.org/wiki/Value_numbering)
- [Dead-code elimination — Wikipedia](https://en.wikipedia.org/wiki/Dead-code_elimination)
- [Dead-code elimination must preserve side effects / I/O — studyraid](https://app.studyraid.com/en/read/15449/536640/dead-code-elimination-techniques)
- [Loop-invariant code motion — Wikipedia](https://en.wikipedia.org/wiki/Loop-invariant_code_motion)
- [LLVM pass descriptions (LICM, GVN, DCE/ADCE) — LLVM docs](https://llvm.org/docs/Passes.html)
- [Compiler optimization (overview of standard passes) — Wikipedia](https://en.wikipedia.org/wiki/Optimizing_compiler)
Home.md
+69 -130
@@ -1,144 +1,83 @@
# Strategy Analysis & Validation
# Trading-System Engineering & Strategy Validation
> A knowledge base on how trading strategies are analysed, validated, and graded
> in quantitative finance. Project-neutral and durable: it records the *methods,
> formulas, and benchmark thresholds*, not the state of any one implementation.
>
> The formula-level reference is in **[Metric Catalogue](Metric-Catalogue)**; the
> source-checking of every load-bearing number is in
> **[Metric Verification Log](Metric-Verification-Log)**.
> A durable, **project-neutral** knowledge base for building and validating
> systematic trading systems. It records what is *true about the domain, the
> methods, and the tools* — the architecture of a deterministic backtesting
> engine, and the statistics of grading a strategy — not the state of any one
> codebase. Anything that would drift when an implementation changes (field names,
> file anchors, build status, internal contracts) is deliberately kept out.
## Why validation, not just backtesting
The wiki covers two complementary subject areas. The first is about the **machine**
that runs a strategy; the second about **judging the numbers** that machine
produces.
Running a single backtest and reading off the equity curve is commodity — every
quant system does it. The hard, value-adding part is **validation**: deciding
whether an apparent edge is *real and robust* or an artefact of luck and
overfitting. A backtest produces *one* number; validation asks how that number
would hold up under different parameters, different random draws, unseen data, and
the statistics of having tried many variants. This page surveys the analysis
methods that answer that question and how their results are professionally
classified.
## Area I — [Backtesting Engine Architecture](Backtesting-Engine-Architecture)
### How to read the thresholds
How a modern, deterministic, event-driven backtesting and trading engine is built:
the architectural commitments that make a backtest fast, reproducible, and free of
the future-leaking bugs that silently invalidate results. Start at the
**[architecture overview](Backtesting-Engine-Architecture)**, then the detail
pages:
Two kinds of statement appear throughout, and the catalogue marks each:
- **[Event-Driven Backtesting](Event-Driven-Backtesting)** — vectorized vs.
event-driven; the data-driven irregular clock; merging heterogeneous timestamped
sources into one chronological timeline; backtest/live symmetry.
- **[Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)** — the cardinal
backtesting bug, and how to make it *structurally impossible* rather than merely
discouraged.
- **[Determinism and Reproducibility](Determinism-and-Reproducibility)** — same
input → identical run; reproduce-from-manifest; embarrassingly-parallel runs;
seed-as-input; the record-then-replay boundary for nondeterministic sources.
- **[Reactive Streaming Dataflow](Reactive-Streaming-Dataflow)** — push-based
synchronous reactive dataflow on an acyclic graph; freshness-gated recompute and
sample-and-hold; feedback via an explicit delay/register; as-of vs. barrier joins.
- **[Columnar Data Layout](Columnar-Data-Layout)** — structure-of-arrays vs.
array-of-structures; why columnar layout is cache- and SIMD-friendly for a
streaming numeric hot path.
- **[Signal, Exposure, and Execution](Signal-Exposure-and-Execution)** — separating
the alpha signal (a target-exposure signal) from the execution model;
frictionless signal-quality measurement vs. realistic frictions; position
management as a derived layer.
- **[Graph Compilation and Optimization](Graph-Compilation-and-Optimization)** — a
parameter-generic graph compiled to a flat runnable instance; determinism as the
licence for behaviour-preserving rewrites (CSE, dead-code elimination,
loop-invariant code motion over a sweep).
- **[Authoring and Deployment Lifecycle](Authoring-and-Deployment-Lifecycle)** — the
engine/application split; fast hot-reload iteration vs. a frozen, reproducible
deploy artifact.
- **[L] law / exact** — a definition or identity (e.g. the Sharpe ratio formula,
the profit-factor↔win-rate identity). These do not vary.
- **[C] convention** — a practitioner rule-of-thumb (e.g. "Sharpe > 2 is
excellent"). These vary by source, asset class, and market regime; treat them as
relative-comparison guidelines, never as hard pass/fail gates without checking
the primary source.
- **[CORR]** marks a value that was corrected during source verification — see the
[Metric Verification Log](Metric-Verification-Log).
## Area II — [Strategy Analysis & Validation](Strategy-Analysis-and-Validation)
### Common inputs
Most metrics are computed from one of a few generic inputs, defined once here and
referenced throughout the catalogue:
- **per-period return** `r` — the return of one clock period; the first difference
of the cumulative equity curve (`r_t = E_t − E_{t-1}`).
- **equity curve** `E` — the running cumulative P&L (or cumulative return).
- **trade list** — the set of closed trades, each with entry/exit, P&L, and
excursions.
- **benchmark** `b` — a market/benchmark return series, for relative metrics.
- **multi-run distribution** — the set of results across many resamples, seeds,
parameter cells, or folds.
## The four classic analysis axes
These are the standard ways to put a strategy through its paces. Each generates a
*family* of backtests from one strategy and reduces it to a verdict.
| Axis | What it is | Produces | Financial question |
|---|---|---|---|
| **Parameter sweep** (grid / random) | run the strategy across a grid (or random sample) of its tunable parameters | a *surface*: one result per parameter cell | "Where does it make money — a broad robust plateau, or a single lucky spike?" |
| **Optimization** (argmax of a metric) | select the parameter set that maximizes an objective | the winning configuration + its scorecard | "Which configuration is best by my chosen yardstick?" (and: is that yardstick robust, or does it reward luck?) |
| **Walk-forward analysis** | chain *(optimize on an in-sample window)* then *(test on the next out-of-sample window)*, marched forward; stitch the OOS segments | a stitched out-of-sample curve + Walk-Forward Efficiency + parameter stability | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
| **Monte-Carlo analysis** | re-run the strategy across N randomized realizations (resampled returns/trade order, permuted prices, or a synthetic generator) | a *distribution* of every metric: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, a permutation-test p-value | "Given that one backtest is just one draw, what is the distribution, the tail risk, and the chance the edge is luck?" |
A recurring trap sits inside the optimization axis: maximizing raw net profit
selects the cell most contaminated by luck and a single fat-tail trade. The
professional move is to optimize a *risk-adjusted* objective and read robustness
from the *neighbourhood* of the optimum, not its peak — see §A and §B of the
[Metric Catalogue](Metric-Catalogue).
## The broader analysis landscape
The four axes *generate* families of backtests; the following are the *reads* and
*tests* layered over them — the part that separates a rigorous validation from a
pretty equity curve.
- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, Minimum Backtest
Length, the Harvey-Liu-Zhu haircut). A high in-sample Sharpe over many trials is
*expected by chance even at zero true skill* (the False Strategy Theorem). These
deflate the observed Sharpe for sample length, non-normality, and the **number
of trials**, turning a sweep/optimize result into a significance verdict.
- **Cross-validation for time series (CPCV, purging, embargo).** Plain k-fold
leaks information on serially-correlated, overlapping-label series. Purging and
embargo remove the leakage; Combinatorial Purged Cross-Validation builds many
fully-out-of-sample paths → a *distribution* of the Sharpe rather than one
number.
- **Sensitivity / robustness.** Read the sweep *surface* geometrically:
plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness,
parameter-perturbation dispersion. A robust edge is a broad smooth plateau; an
overfit one is an isolated island.
- **Stress / scenario analysis.** Replay curated crisis windows (2008, 2015 CHF,
2020) and inject stylized shocks (vol ×3, spread blow-out). A *conditional*
worst-case profile — "if X then Y", not "X is likely".
- **Regime analysis.** Slice performance by market state (bull/bear,
high/low-vol, trend/range). "Where does the money come from, and does the edge
invert in some regime?"
- **Cost / capacity analysis.** A frictionless backtest hides execution reality.
Sweeping cost assumptions yields the *break-even cost*; the square-root impact
law yields the *capacity ceiling* (the AUM before the strategy's own market
impact erodes the edge).
- **Benchmark-relative significance.** CAPM alpha/beta, the information ratio, and
the t-statistic of the mean return test whether the return is skill (alpha) or
just market exposure (beta).
- **Null / random-entry sanity.** A permutation test or a random-entry null
strategy asks "does it beat random?" — permute the input return series (keep the
marginal, destroy the time-ordering an edge would exploit) and re-run; the real
result must sit in the right tail.
- **Live-vs-backtest divergence.** The one analysis *outside* the simulation: diff
realized live performance against the backtest prediction (the Sharpe haircut) to
separate genuine alpha decay from a code/data bug.
## Where the detail lives
How the equity and return series a backtest produces are analysed, validated, and
graded: deciding whether an apparent edge is *real and robust* or an artefact of
luck and overfitting. Start at the
**[validation overview](Strategy-Analysis-and-Validation)**, then:
- **[Metric Catalogue](Metric-Catalogue)** — for each analysis tool and metric
family: exact formula, what input it is computed from, and the conventional
family: the exact formula, the input it is computed from, and the conventional
benchmark threshold.
- **[Metric Verification Log](Metric-Verification-Log)** — how the formulas and
thresholds were source-checked, which were corrected, and the citations.
- **[Metric Verification Log](Metric-Verification-Log)** — how every load-bearing
formula and threshold was adversarially source-checked, which were corrected, and
the citations.
*This knowledge base was compiled from the quant-finance literature and
adversarially source-checked. Primary references include Bailey & López de Prado
(PSR / DSR / PBO / CPCV), Pardo (walk-forward), Magdon-Ismail et al. (expected
maximum drawdown), the Basel/FRTB market-risk standards (VaR / ES), and Bacon
(drawdown metrics). Per-topic citations are listed on each page.*
## How to read this wiki
## References
Every load-bearing claim is **marked** so a definition is never confused with a
rule-of-thumb, and **sourced** with a validated external link (each page carries a
grouped `## References` section):
Overview-level external references (each verified reachable and on-topic). The
full per-metric citation list is in the
[Metric Catalogue → References](Metric-Catalogue#references); corrected formulas
and thresholds are sourced in the [Metric Verification Log](Metric-Verification-Log).
- **[L] law / exact** — a definition, identity, or derivation that does not vary.
- **[C] convention** — a practitioner rule-of-thumb that varies by context (asset
class, regime, source); never a hard pass/fail gate without checking the primary
source.
- **[CORR] corrected** — a value wrong in a common source and fixed here, with the
correction cited.
- **Analysis axes:**
[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) ·
[Monte-Carlo method](https://en.wikipedia.org/wiki/Monte_Carlo_method) ·
[Permutation test](https://en.wikipedia.org/wiki/Permutation_test) ·
[Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis)
- **Overfitting & validation statistics:**
[Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) ·
[Probability of Backtest Overfitting / CSCV](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) ·
[Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) ·
[Pseudo-mathematics and financial charlatanism (Bailey et al.)](https://scholarworks.wmich.edu/math_pubs/42/) ·
[López de Prado, Advances in Financial ML](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml)
- **Risk & benchmark-relative:**
[Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) ·
[Expected Shortfall](https://en.wikipedia.org/wiki/Expected_shortfall) ·
[Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) ·
[CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model)
The two areas meet where the engine *produces* the series the metrics *grade*: the
four classic analysis axes (sweep / optimize / walk-forward / Monte-Carlo) on
[Strategy Analysis & Validation](Strategy-Analysis-and-Validation) are validation
methods, while
[Determinism and Reproducibility](Determinism-and-Reproducibility) and
[Graph Compilation and Optimization](Graph-Compilation-and-Optimization) describe
how an engine constructs and runs those families of backtests efficiently and
reproducibly.
Look-Ahead-Bias-and-Causality.md
+123
@@ -0,0 +1,123 @@
# Look-Ahead Bias and Causality
Look-ahead bias is the cardinal error of historical simulation: at a decision time *t*, a computation reads information that was not yet available at *t*. It silently inflates simulated performance, and because the leaked edge evaporates the instant a strategy trades live, a fast backtester that leaks the future is worse than no backtester at all. This page treats look-ahead bias as the dual of **causality** — the signal-processing property that a system's output depends only on present and past inputs — and argues that a backtesting engine should make non-causality *structurally impossible* rather than merely discouraged. Related biases (survivorship, data-snooping, fill optimism) are surveyed briefly for context, with their own sources; the focus stays on look-ahead and causality.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
See also: [Event-Driven Backtesting](Event-Driven-Backtesting), [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow), [Backtesting Engine Architecture](Backtesting-Engine-Architecture), [Determinism and Reproducibility](Determinism-and-Reproducibility), [Metric Catalogue](Metric-Catalogue), [Metric Verification Log](Metric-Verification-Log).
## What look-ahead bias is
[L] **Definition.** Look-ahead bias is the use, in a study or simulation, of data or information that was not yet available or known during the time period being modelled. Equivalently for a trading rule: it occurs when a strategy or model uses information that would not have been available at the time of the trade decision. The defining property is temporal: the computation at instant *t* reads a value whose true availability time is strictly after *t*.
[L] **It biases results toward the desired outcome, not the true one.** Incorporating information unavailable at the time "delivers biased results that are often close to the desired outcome but not to the real outcome." In a trading context the simulation shows "unreasonably positive results," while live application "is likely to deliver dramatically different results."
[C] **It is among the most damaging backtesting errors — arguably the worst.** A widely held practitioner view holds look-ahead bias to be the most severe bias because *the results are simply wrong*, and because the error "is immediately revealed in real-world execution, whereas the other form of biases are revealed after a regime change where the strategy becomes unprofitable." This severity ranking is a rule of thumb, not a theorem: some authoritative surveys list look-ahead alongside survivorship bias and overfitting as co-equal "sins" without crowning any one as uniquely worst (see [CORR] below). What is *not* in dispute is the direction of the error — look-ahead always flatters the backtest.
[CORR] **"The worst bias" is a convention, not a law.** A common framing asserts look-ahead bias is categorically the single most dangerous backtesting error. The academic treatment in Palomar's *Portfolio Optimization* ("The Seven Sins of Quantitative Investing") describes look-ahead bias as "a very common bias in backtesting" but explicitly does **not** rank it above data-snooping/overfitting or survivorship bias. Treat the "worst bias" claim as a practitioner heuristic about *detectability and immediacy of failure*, not as an established ordering of harm.
### Canonical sources of leakage
[C] The leak is rarely a deliberate `future[t+1]` read; it hides in data alignment and aggregation. Recurring sources:
- **Misaligned time series.** Price, indicator, or event data not matched to the time it actually became public. This is one of the most common causes — improperly aligned series silently grant the backtest access to data a live system would not have.
- **End-of-period values consumed too early.** Assuming the day's or quarter's *closing* value is tradable at or before the close. "A trader cannot execute a trade at past price levels"; entering "at the previous day's close" is impossible in real life yet trivial to code by accident.
- **Reporting lag on fundamentals.** A classic example assumes a company's quarterly earnings are available on the quarter's closing date, when in reality "quarterly earnings reports become available only one month after the end of the quarter."
- **Restated / backfilled data.** Using a figure as later revised, rather than as first reported, so the backtest "trades" on numbers nobody held at the time (see *Point-in-time data* below).
- **A not-yet-complete bar.** Reading a forming (partial) aggregate as if it were final — the streaming analogue of consuming an end-of-period value early (see *Resamplers and the forming-bar trap*).
[C] **Smell tests, not proofs.** Practitioners watch for suspiciously smooth equity curves and implausibly strong summary statistics (annualized return well into double digits, Sharpe ratio above ~1.5) as *symptoms* of leakage. These thresholds are heuristics — useful alarms, never confirmation; a clean-looking curve does not prove the absence of look-ahead, and a noisy one does not prove its presence. For how such summary statistics are defined and validated, see [Metric Catalogue](Metric-Catalogue) and [Metric Verification Log](Metric-Verification-Log).
## Causality: the signal-processing dual
The precise, project-neutral way to state "no look-ahead" is to borrow the definition of a **causal system** from signal processing. A backtest that is free of look-ahead bias is exactly a backtest in which every node is a causal function of its inputs.
[L] **Causal system.** A system is causal when its output depends only on past and present (current) inputs, never on future inputs. Formally, the output *y*(*t₀*) depends only on the input *x*(*t*) for *t* ≤ *t₀*.
[L] **Causal filter.** A causal filter is a linear time-invariant causal system; "the word causal indicates that the filter output depends only on past and present inputs." In impulse-response terms the system is causal **if and only if** its impulse response satisfies *h*(*t*) = 0 for all *t* < 0 (discrete-time: *h*[*n*] = 0 for all *n* < 0). A nonzero response at negative lag *is* a future dependence.
[L] **Non-causal and anti-causal.** A system whose output depends partly or fully on future inputs is **non-causal** (also *acausal*); one whose output depends *solely* on future inputs is **anti-causal**. Look-ahead bias is precisely the (usually unintended) introduction of a non-causal component into what is supposed to be a causal trading rule.
[L] **Real-time realizability requires causality.** "Systems (including filters) that are realizable (i.e., that operate in real time) must be causal because such systems cannot act on a future input." This is the bridge from theory to backtesting: a live trading system is a real-time system, so it is necessarily causal; therefore any backtest that hopes to predict live behaviour must itself be causal. Non-causal filters are perfectly legitimate in *offline* signal processing: zero-phase filtering, for instance, "uses the information in the signal at points before and after the current point, in essence 'looking into the future,'" which is possible only "given all the samples of the sequence" — i.e. when the whole signal is stored, so future samples are available. That is exactly the trap, because a backtest *is* offline post-processing over a stored history, which makes accidentally reading the future physically possible even though it is operationally forbidden.
[C] **Causality is the invariant; look-ahead is its violation.** Framing the requirement as "every node must be a causal function of the cursor" turns a vague injunction ("don't peek at the future") into a checkable structural property. The remainder of this page is about enforcing that property by construction.
## Structural prevention versus discipline
[C] **Discipline does not scale.** "Be careful not to use future data" is an instruction to the author, re-imposed by hand at every node and every join. Each manual alignment is a place the bias can re-enter, and — as noted above — look-ahead is notoriously hard to spot in a passing backtest. The stronger design goal is to make the future *physically absent* from what a computation is able to read, so that a non-causal node is not merely frowned upon but *unexpressible*.
The following mechanisms are the standard structural defences. They are conventions of robust engine design, not mathematical laws — but each one converts a class of look-ahead bug from "possible and easy" into "impossible to express."
### Past-only data windows (the cursor ends the window)
[C] Expose to each node a **read-only view of its input that ends at the current cursor**. A node can index into the past — any offset *≤ 0* relative to the cursor — but there is no addressable slot for a future sample, so `input[t+1]` is not a value the node can name. This is the engine-level encoding of the causality condition *h*(*t*) = 0 for *t* < 0: the data structure itself has no negative-lag (future) entry to multiply against. Read-only is load-bearing — a node that could *write* ahead of the cursor would reintroduce a channel for leakage. See [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow) for how such bounded, append-only windows are modelled as streams, and [Event-Driven Backtesting](Event-Driven-Backtesting) for the cursor-advancing event loop that defines "now."
### Resamplers and the forming-bar trap
[C] **Emit a coarser bar only once it is complete.** A resampler/aggregator (e.g. ticks → 1-minute bars, or 1-minute → hourly) must publish an aggregate *only after the interval has closed*, never while it is still forming. A partial bar's high, low, close, and volume are all still changing; consuming a forming bar's "close" is reading a value that will not exist until the interval ends — a future read in disguise. The structural fix is that the resampler holds the forming bar private and releases it as a single immutable event on completion, so downstream nodes can only ever see finished bars. This is the streaming counterpart of "do not consume an end-of-period value before the period ends." Because the boundary is enforced by the resampler and not by each consumer, every downstream node inherits causality for free.
### Point-in-time data (as-it-was-known)
[C] **Replay history as it was known at each date, including restatements and reporting lags.** Point-in-time (PIT) data answers two questions for every value: *when was it known* and *what was known at that time*. PIT storage keeps the originally reported figure timestamped at its true publication moment and keeps later revisions as *new* facts with *their own* later timestamps, rather than overwriting the past. A backtest stepping its cursor through PIT data therefore sees first-reported numbers when a contemporaneous trader would have, and only sees a restatement once it was actually published. This neutralizes the fundamentals-lag and restatement leaks at the data layer rather than the node layer. Using restated figures in place of first-reported ones is a textbook look-ahead leak: financial statements are often restated and macroeconomic data revised, and — absent point-in-time storage — "the revised data often replaces the old data, and an analyst will, as such, use information that was unavailable to them when carrying out a backtest," erasing what was actually knowable at the time.
[C] **Recording boundary for non-deterministic / external inputs.** Any input that is external, slow, or non-deterministic (live feeds, model-generated signals, web data) should be *materialized into a recorded, timestamped stream before it enters the simulation*, so replay reads a fixed past rather than making a live call mid-run. This both preserves determinism (see [Determinism and Reproducibility](Determinism-and-Reproducibility)) and prevents a subtle look-ahead variant: an input whose timestamp is the moment it was *requested during replay* rather than the moment it was *truly available* in history.
### One merge, at ingestion only
[C] **Do the chronological merge once, at the ingestion boundary, not inside the graph.** When heterogeneous timestamped sources are k-way merged into a single ordered event stream *before* the graph runs, every node downstream sees one monotonic clock and there is no in-graph as-of/temporal join to get wrong. In-graph joins are a frequent home for alignment bugs (matching a value to the wrong "now"); removing them removes the bug class. See [Backtesting Engine Architecture](Backtesting-Engine-Architecture) for where this ingestion merge sits relative to the rest of the pipeline.
[L] **Why "structural" is stronger than "careful."** A structural guarantee is a property of the engine that holds for *every* node an author can write, including nodes not yet written; discipline is a property re-established by hand per node and lost the first time attention lapses. The two are not interchangeable — only the structural form makes a non-causal computation *unrepresentable*, which is the difference between "no known leak" and "no possible leak."
## Related biases (brief, for context)
Look-ahead bias is one of several ways a backtest can flatter a strategy. The others are surveyed here only to delineate the boundary; each has its own literature and its own page-cluster cross-links.
### Survivorship bias
[L] **Definition.** Survivorship bias is "a statistical error that results from concentrating on entities that passed a selection process while overlooking those that did not," typically because the excluded entities are no longer visible in the data. In finance, "a mutual fund company's selection of funds today will include only those that are successful now"; failed or delisted instruments drop out of the database, so a backtest run over *today's* surviving universe systematically overstates historical returns. It is distinct from look-ahead bias: survivorship is a *universe-selection* error (which series exist), look-ahead is a *temporal* error (when a value is read).
### Data-snooping / multiple-testing bias
[L] **Definition.** Data dredging (data snooping, p-hacking) is "the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives." The mechanism is the multiple-comparisons problem: run enough tests and some pass by chance — "5% of randomly chosen hypotheses might be (erroneously) reported to be statistically significant at the 5% significance level."
[L] **In backtesting: overfitting from configuration search.** Bailey, Borwein, López de Prado, and Zhu show that "high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations," and that "the higher the number of configurations tried, the greater is the probability that the backtest is overfit." Worse, under realistic conditions backtest overfitting "leads to negative expected returns out-of-sample, rather than zero performance." Because the number of configurations tried is rarely reported, an observer cannot gauge the overfit from the backtest alone.
[C] **The remedies live in the validation cluster.** Deflated significance (e.g. the Deflated Sharpe Ratio), combinatorially symmetric cross-validation, and walk-forward / out-of-sample protocols correct for selection over many trials. These belong to the validation discussion rather than this page; this section exists only to mark the boundary between *temporal* leakage (look-ahead) and *statistical* leakage (snooping). See [Metric Verification Log](Metric-Verification-Log) for how individual metrics are pinned and checked.
### Fill / execution optimism
[C] **Definition.** Fill or execution optimism is the assumption of unrealistically favourable trade execution — filling at the close that triggered the signal, ignoring slippage, spread, partial fills, or assuming infinite liquidity. It overlaps look-ahead at one edge: "assuming the closing price as the trading price, disregarding that it is impossible to know the closing price in advance" is *both* a fill assumption and a future read. In general, though, execution optimism concerns the realism of the *cost/fill model* once an order exists, whereas look-ahead concerns whether the *information* driving the order was knowable. A frictionless, idealized execution model is a legitimate analysis tool for isolating signal quality, but it is not a substitute for a realistic execution model (with frictions) when assessing deployability — and neither rescues a strategy whose signal leaked the future.
## Practical checklist
[C] A causality audit for an engine or a strategy, distilled from the structural defences above:
1. Can any node address an input slot *ahead* of the cursor? If yes, the engine permits non-causal nodes by construction — fix the data-view shape, not the node.
2. Does every resampler/aggregator emit only *completed* intervals? Probe with a deliberately partial final bar and confirm it is not visible downstream.
3. Is fundamental / external data stored point-in-time, with restatements as new timestamped facts rather than overwrites?
4. Is the chronological merge performed once at ingestion, with no as-of join inside the graph?
5. Are external/non-deterministic inputs recorded to a timestamped stream *before* replay, never fetched live mid-run?
6. Do summary statistics look "too clean"? Treat as a *prompt to investigate* alignment and bar-completion, not as a verdict either way.
[L] Items 1–5 are structural — when satisfied, the corresponding leak is *unexpressible*. Item 6 is a heuristic symptom check and can neither confirm nor refute leakage on its own.
## References
### What look-ahead bias is
- [Look-Ahead Bias — Definition and Practical Example (Corporate Finance Institute)](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/look-ahead-bias/) — definition; biased toward desired outcome; the quarterly-earnings reporting-lag example.
- [Look-Ahead Bias In Backtests And How To Detect It — Michael Harris](https://mikeharrisny.medium.com/look-ahead-bias-in-backtests-and-how-to-detect-it-ad5e42d97879) — "using information that would not normally be available… when [the signal] occurs"; the "worst bias / revealed immediately in live execution" argument; close-as-entry example; smell-test thresholds.
- [The Seven Sins of Quantitative Investing — D. P. Palomar, *Portfolio Optimization*](https://portfoliooptimizationbook.com/book/8.2-seven-sins.html) — academic definitions of look-ahead, survivorship, and data-snooping biases; basis for the [CORR] that look-ahead is *not* ranked uniquely worst.
### Causality (signal-processing dual)
- [Causal system — Wikipedia](https://en.wikipedia.org/wiki/Causal_system) — output depends only on past/present inputs; *y*(*t₀*) depends only on *x*(*t*) for *t* ≤ *t₀*; acausal/anti-causal; real-time systems must be causal.
- [Causal filter — Wikipedia](https://en.wikipedia.org/wiki/Causal_filter) — causal filter as LTI causal system; *h*(*t*) = 0 for *t* < 0; non-causal/anti-causal; realizable real-time systems must be causal.
- [Anti-Causal, Zero-Phase Filter Implementation — MathWorks](https://www.mathworks.com/help/signal/ug/anti-causal-zero-phase-filter-implementation.html) — the offline / non-real-time exception: zero-phase filtering "uses the information in the signal at points before and after the current point, in essence 'looking into the future,'" possible only "given all the samples of the sequence."
### Structural prevention — point-in-time data
- [Look-Ahead Bias — Corporate Finance Institute](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/look-ahead-bias/) — reporting-lag mechanics underpinning the point-in-time defence.
- [Problems in Backtesting and Biases in Data (AnalystPrep, CFA Level 2)](https://analystprep.com/study-notes/cfa-level-2/problems-in-backtesting/) — restated/revised data as a look-ahead source: "the revised data often replaces the old data, and an analyst will, as such, use information that was unavailable to them when carrying out a backtest"; point-in-time data as the remedy.
### Related biases
- [Survivorship bias — Wikipedia](https://en.wikipedia.org/wiki/Survivorship_bias) — "concentrating on entities that passed a selection process while overlooking those that did not"; surviving-funds inflation of performance.
- [Data dredging — Wikipedia](https://en.wikipedia.org/wiki/Data_dredging) — data snooping / p-hacking definition; multiple-comparisons problem and the 5%-of-tests-by-chance illustration.
- [Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance — Bailey, Borwein, López de Prado, Zhu](https://scholarworks.wmich.edu/math_pubs/40/) — high in-sample performance easily achieved over a small configuration search; more configurations → higher overfit probability; overfitting → negative out-of-sample expected return.
Metric-Catalogue.md
+2 -1
@@ -2,7 +2,8 @@
> Formula-level reference for trading-strategy analysis. For each analysis tool
> and metric family: exact formula, what input it is computed from, and the
> conventional benchmark threshold. Narrative on **[Home](Home)**;
> conventional benchmark threshold. Narrative on
> **[Strategy Analysis & Validation](Strategy-Analysis-and-Validation)**;
> source-checking on **[Metric Verification Log](Metric-Verification-Log)**.
**Conventions.** **Per-period return** `r` (`r_t = E_t − E_{t-1}`) = first
Metric-Verification-Log.md
+2 -1
@@ -1,7 +1,8 @@
# Metric Verification Log
> Source-checking of the load-bearing formulas and thresholds in the
> **[Metric Catalogue](Metric-Catalogue)**. Narrative on **[Home](Home)**.
> **[Metric Catalogue](Metric-Catalogue)**. Narrative on
> **[Strategy Analysis & Validation](Strategy-Analysis-and-Validation)**.
## Method
Reactive-Streaming-Dataflow.md
+115
@@ -0,0 +1,115 @@
# Reactive Streaming Dataflow
A **reactive streaming dataflow** model expresses a computation as a directed graph of nodes through which data is *pushed* from sources, through transforms, to sinks, with changes propagating automatically along the edges. When the graph is driven by a discrete logical clock — every node observing its inputs and emitting outputs at each tick — the model becomes a *synchronous* reactive dataflow, the same family of semantics that underpins synchronous dataflow languages and synchronous digital hardware. This page describes that computational model and the engineering properties it gives a streaming backtest engine: deterministic per-tick evaluation, freshness-gated recompute, acyclic structure with explicit feedback, and the join semantics needed to combine streams that tick at different rates.
This is a companion to [Event-Driven Backtesting](Event-Driven-Backtesting) and [Backtesting Engine Architecture](Backtesting-Engine-Architecture); the causality guarantees it depends on are detailed in [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality), the on-the-wire data representation in [Columnar Data Layout](Columnar-Data-Layout), and how the graph is lowered for execution in [Graph Compilation and Optimization](Graph-Compilation-and-Optimization).
Markers: [L] law/exact, [C] convention, [CORR] corrected.
## Dataflow and reactive programming
**Dataflow programming** is a programming paradigm that models a program as a directed graph of the data flowing between operations. [L] Operations are connected by explicitly defined inputs and outputs and behave like black boxes; the language runtime, not the programmer, manages the movement of data between them ([Wikipedia: Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming)). Nodes are typically classified by their role in the graph: **sources** introduce data, **sinks** consume it (write to a file, database, stream, or display), and intermediate **processing nodes** transform it ([Devopedia: Dataflow Programming](https://devopedia.org/dataflow-programming)).
The defining execution rule distinguishes dataflow from control-flow programming: [L] an operation runs as soon as all of its inputs become valid, rather than when a program counter reaches it ([Wikipedia: Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming)). Because a node's only coupling to the rest of the program is through its declared ports, [C] dataflow graphs are described as inherently parallel: two nodes with no path between them share no hidden state and can in principle execute concurrently ([Wikipedia: Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming)). A backtest engine exploits exactly this property *across* independent simulation runs (see [Backtesting Engine Architecture](Backtesting-Engine-Architecture)), while keeping each individual run a single deterministic sequence.
**Reactive programming** is the closely related declarative paradigm "concerned with data streams and the propagation of change" ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)). [L] Its central distinction from ordinary assignment is *persistence of the dependency*: where the imperative statement `a := b + c` computes `a` once, in a reactive setting the relationship is maintained, so that `a` is recomputed whenever `b` or `c` changes ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)). The set of such relationships forms a dependency graph; a change at a source propagates to every transitive dependent.
### Push versus pull
How a change reaches its dependents distinguishes two evaluation strategies. [L] In a **push-based** ("data-driven") model the producer hands a value to its consumers as soon as the value becomes available; in a **pull-based** ("demand-driven") model the consumer queries its sources when it wants a value ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)). A streaming engine that drives a deterministic event loop is fundamentally push-based: each arriving tick is pushed from the source into the graph, and the engine walks the affected nodes forward. [C] A push-pull hybrid is common in practice — a lightweight change *notification* is pushed, and the actual (possibly large) value is pulled on demand — and is preferred when data volumes make eager value delivery wasteful ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)).
### Glitches and propagation order
A naive propagation can produce a **glitch**: a transient, momentarily incorrect output caused by evaluating a node before all of its inputs have been brought up to date. [L] The canonical example is `t = seconds + 1; g = (t > seconds)`: the invariant `g` is always true, but if `g` is recomputed using the new `seconds` and the stale `t`, it transiently reads false — a glitch ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)). [L] The standard cure is to propagate in **topological order**: sort the dependency graph so that every node is evaluated only after all the nodes it depends on, guaranteeing each node sees consistent inputs and recomputes at most once per propagation ([Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming); [Acolyer, *A Survey on Reactive Programming*](https://blog.acolyer.org/2015/12/08/a-survey-on-reactive-programming/)). A glitch-free schedule is one of the concrete payoffs of insisting the graph be acyclic — a topological order exists iff the graph is a DAG (see below, and [Graph Compilation and Optimization](Graph-Compilation-and-Optimization)).
## Synchronous reactive dataflow
When the propagation above is organized around a single discrete clock, the model becomes a **synchronous** reactive system. [L] A synchronous programming language targets reactive systems and structures execution as a sequence of *logical ticks*; within each tick, computation is assumed instantaneous — "a synchronous program reacts to its environment in a sequence of ticks, and computations within a tick are assumed to be instantaneous" ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)). This is the **synchronous hypothesis** (or synchronous abstraction): outputs are produced synchronously with inputs, with no observable delay, exactly as a synchronous digital circuit is reasoned about as if signals propagated infinitely fast ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)).
The payoff of the synchronous hypothesis is **determinism**. [L] By collapsing each reaction into one logical instant, the abstraction "eliminates the non-determinism resulting from the interleaving of concurrent behaviors," giving a well-defined, reproducible state after every tick ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)). This is precisely the determinism a backtest requires: the same input stream must produce the same run, every time (see [Event-Driven Backtesting](Event-Driven-Backtesting)). A synchronous dataflow graph reaches a single, unique state after each input tick, and that property is what makes two independent runs safely concurrent.
### The classical synchronous languages
Three synchronous languages, developed in France in the 1980s, established the paradigm ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)):
- **Lustre** — a declarative, dataflow synchronous language. [L] Every variable is a *flow*: a stream of values paired with a clock, indexed by the native notion of logical time, and programs are built from dataflow *nodes* that map input flows to output flows ([Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language))). Lustre is the closest classical relative of a streaming backtest graph.
- **Esterel** — an imperative, control-oriented synchronous language. [L] It is a "deterministic concurrent programming language" for reactive systems whose threads "march in step with a global clock"; determinacy is ensured by the *signal coherence rule*, which requires every signal to hold a stable value throughout one reaction ([ScienceDirect: *The Esterel synchronous programming language*](https://www.sciencedirect.com/science/article/pii/016764239290005V)).
- **SIGNAL** — [L] "a dataflow-oriented synchronous language enabling multi-clock specifications," i.e. it natively expresses streams that tick on *different* clocks within one program ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)). Multi-rate composition (below) is its central concern, and it is the cleanest classical antecedent for combining a tick stream with a slower bar stream.
A streaming engine for backtesting does not adopt any of these languages, but it inherits their semantics: a logically clocked, deterministic, per-tick dataflow whose nodes are flows of values.
## Freshness-gated recompute and sample-and-hold
A literal reading of the synchronous hypothesis — *every* node recomputes *every* tick — does not scale when a graph mixes many sparse and high-rate streams. [C] In a purely push-based model "you perform potentially wasteful recomputations every time input sources change," and values are recomputed even when the inputs that feed them did not change ([arXiv: *Reactive Programming without Functions*](https://arxiv.org/pdf/2403.02296)). With dozens of instruments each emitting at irregular times, a total per-tick sweep over the whole graph spends most of its work re-deriving outputs that are bit-for-bit identical to the previous tick.
**Freshness gating** is the standard remedy. [L] The propagation is restricted so that "computations dependent on values that did not change are not scheduled for execution" — only nodes with at least one genuinely changed input this tick are recomputed ([Acolyer, *A Survey on Reactive Programming*](https://blog.acolyer.org/2015/12/08/a-survey-on-reactive-programming/); [Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming)). A node whose inputs are all unchanged simply **holds** its previous output.
The behavior of a *stale* input — one that did not tick this cycle — is the dual of recompute gating. The convention is **sample-and-hold**: a stale input contributes its last produced value rather than a missing one. [L] The term is borrowed exactly from the analog circuit, which "samples (captures, takes) the voltage of a continuously varying analog signal and holds (locks, freezes) its value at a constant level" until the next sample ([Wikipedia: Sample and hold](https://en.wikipedia.org/wiki/Sample_and_hold)). In the streaming graph the analogue is: between two ticks of a slow stream, every downstream node sees that stream's last emitted value held constant, so a fast input ticking against a slow one always reads a well-defined slow value.
[C] Freshness gating and topological ordering are complementary, not alternatives: topological order makes a propagation *glitch-free*, and freshness gating makes it *cheap* by pruning unchanged sub-graphs from that ordered sweep. Many practical reactive runtimes combine data- and demand-driven evaluation precisely so that "values are recomputed only when necessary" while reactions stay near-instantaneous ([arXiv: *Reactive Programming without Functions*](https://arxiv.org/pdf/2403.02296)). Note that freshness gating must never let a node observe a *future* value of a held input — the hold always carries the last value at or before the current cursor, preserving causality (see [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)).
## Acyclicity and explicit feedback
A reactive dataflow graph is required to be a **directed acyclic graph (DAG)**. The reason is semantic, not merely tidiness: an implicit cycle is a *combinational loop* — a node whose value, within a single instant, depends on itself — and is ill-defined under the synchronous hypothesis, because there is no fixed evaluation order that lets the instant complete. [L] The classical synchronous languages reject such loops outright: Lustre forbids zero-delay (combinational) loops, and its static analysis includes a cyclic-dependency check that rejects programs not reducible to a bounded-space, acyclic schedule ([arXiv: *Secure Information Flow Typing in LUSTRE*](https://arxiv.org/pdf/2201.00184)). Acyclicity is also what guarantees the topological order used for glitch-free propagation actually exists.
Legitimate feedback — a value at tick *t* that depends on a value from tick *t−1* — is not forbidden; it is made **explicit** by routing it through a unit-delay / register / state node. The same device appears under several names across disciplines, and the equivalence is exact:
- **Synchronous-language delay operators.** [L] Lustre's `pre` operator "returns the value of `p` in the previous cycle," and the `->` ("followed-by") operator supplies an initial value for the first cycle ([Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language))). [L] The combined unit-delay `fby` ("followed-by") is defined by `(x fby y)_i = x_0` when `i = 0` and `y_{i−1}` otherwise ([arXiv: *A Type System for the Automatic Distribution of Higher-order Synchronous Dataflow Programs*](https://arxiv.org/pdf/1211.2776)). Feedback in Lustre is expressed by passing a flow through `pre`, and the rule is that every backward path carries exactly one unit delay.
- **The DSP unit delay / z⁻¹.** [L] In discrete-time signal processing a unit-delay element produces output `x[n−1]` from input `x[n]`; under the z-transform a one-sample delay is exactly multiplication by `z⁻¹` (the time-shifting property `Z{x[n−k]} = z⁻ᵏ X(z)`), so `z⁻¹` *is* the unit-delay operator ([Wikipedia: Z-transform](https://en.wikipedia.org/wiki/Z-transform); [MathWorks: Unit Delay](https://www.mathworks.com/help/simulink/slref/unitdelay.html)).
- **The hardware register at register-transfer level.** [L] RTL "models a synchronous digital circuit in terms of the flow of digital signals (data) between hardware registers, and the logical operations performed on those signals"; registers "are the only elements in the circuit that have memory properties" and synchronize the circuit to the clock edges ([Wikipedia: Register-transfer level](https://en.wikipedia.org/wiki/Register-transfer_level)). A register is precisely a one-cycle state element separating combinational logic on its input side from its output side.
The engineering consequence is uniform across all three: forcing every feedback path through a *visible* one-tick delay keeps per-tick evaluation well-defined. Each instant is computed strictly from this tick's fresh inputs plus the *frozen* outputs of delay nodes captured at the end of the previous tick, so the within-tick graph remains acyclic and topologically orderable. Stateful operators that "remember" — moving averages, running counters, edge detectors (`E = X and not pre X` in Lustre) — are all built from this single primitive ([Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language))). The research workflow's own "cycle" (sweep, re-run, refine) is an outer-loop activity and is not a dataflow cycle in this sense.
## Stream join semantics: combining different rates
The hardest modeling problem in a multi-rate streaming graph is a node with several inputs that tick at *different* rates — a fast tick stream and a slow bar stream, two instruments quoting asynchronously, a price stream and a periodic indicator. Two distinct join semantics answer "when does the node fire, and what does it read for the inputs that did not just tick?", and a real graph needs both.
### As-of / latest join
An **as-of** (latest-value) join fires when *any* input arrives, and reads each *other* input at its most recent value at or before the current time — i.e. the held value of sample-and-hold, applied at join time.
- [L] In **kdb+/q**, `aj` (as-of join) aligns "the prevailing value from the value table with each record in the source table": for each left-table record, it appends the items of the *last* matching right-table record whose time key is ≤ the left record's time ([kdb+/q reference: aj](https://code.kx.com/q/ref/aj/)). The motivating use is attaching, to each trade, the bid/ask that prevailed at the trade's instant.
- [L] In **pandas**, `merge_asof` is "a left-join except that we match on nearest key rather than equal keys." The default `direction='backward'` "selects the last row in the right DataFrame whose 'on' key is less than or equal to the left's key"; `'forward'` and `'nearest'` are also available, and both inputs must be sorted on the key ([pandas: merge_asof](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.merge_asof.html)). [CORR] Only the **backward** direction is causal for streaming: a `'forward'` or `'nearest'` as-of match can read a value that occurs *after* the join instant, which is a look-ahead leak — a streaming/backtest join must restrict to backward (≤ cursor) matching ([pandas: merge_asof](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.merge_asof.html); see [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality)).
The as-of join is the natural join for a tick-driven graph: it never blocks waiting, it fires on every arrival, and the held value it supplies for stale inputs is exactly the sample-and-hold value described above.
### Synchronizing-barrier join
A **barrier** (synchronizing) join does the opposite: it *waits* until every input that belongs to the current logical step has arrived before firing, so the node observes a fully complete set of inputs for that step. The stream-processing realization of this wait is the **watermark**. [L] In Apache Flink a watermark "declares that event time has reached time `t` in that stream, meaning that there should be no more elements from the stream with a timestamp `t' ≤ t`" — an assertion of *completeness* up to `t` that lets a time-based (e.g. windowed) operator safely fire and emit its result ([Apache Flink: Timely Stream Processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/)). The barrier semantics matter when correctness requires *all* contributions for a step (a completed bar, an aggregate over every instrument in a basket) rather than a snapshot of whatever has arrived so far.
### Both, often in one node
[C] These are complementary, not competing: a single node frequently combines an *as-of* input (a reference price it reads at its prevailing value) with a *barrier* group (a set of streams it must wait on before emitting). Choosing the wrong one is a modeling error: a barrier join where an as-of join was meant stalls on a sparse input that may never tick in lockstep; an as-of join where a barrier was meant fires on a partial step and silently uses held, possibly-stale values for the inputs still pending. The synchronous-language framing makes the distinction precise — an as-of input is sampled on the *consumer's* clock, a barrier group forces the consumer's clock to be the (slowest) shared clock of its inputs, which is exactly the multi-clock composition SIGNAL was designed to express ([Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language)).
## Summary
A backtest engine built as a synchronous reactive streaming dataflow gets four properties from this model, each traceable to a named, established source. Push-based propagation in topological order over a DAG gives glitch-free, deterministic per-tick state (reactive programming + the synchronous hypothesis). Freshness gating plus sample-and-hold keeps that determinism affordable when many streams tick at very different rates. Explicit unit-delay nodes — the same device as Lustre's `pre`, the DSP `z⁻¹`, and the hardware register — make feedback well-defined without admitting ill-defined combinational loops. And the as-of / barrier join distinction gives a precise, causal vocabulary for combining streams of different rates. The downstream consequences for run reproducibility and causality are developed in [Event-Driven Backtesting](Event-Driven-Backtesting) and [Look-Ahead Bias and Causality](Look-Ahead-Bias-and-Causality); how the graph is lowered and optimized for execution is covered in [Graph Compilation and Optimization](Graph-Compilation-and-Optimization).
## References
### Dataflow and reactive programming
- [Wikipedia: Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming) — program as a directed graph; black-box operations; run-when-inputs-valid; inherent parallelism.
- [Devopedia: Dataflow Programming](https://devopedia.org/dataflow-programming) — sources, sinks, and processing nodes; ports and edges.
- [Wikipedia: Reactive programming](https://en.wikipedia.org/wiki/Reactive_programming) — propagation of change; persistence of dependencies; push vs pull; glitches and topological-order resolution.
- [Acolyer, *A Survey on Reactive Programming*](https://blog.acolyer.org/2015/12/08/a-survey-on-reactive-programming/) — unchanged inputs are not rescheduled; topological-order propagation.
### Synchronous reactive dataflow
- [Wikipedia: Synchronous programming language](https://en.wikipedia.org/wiki/Synchronous_programming_language) — synchronous hypothesis, logical ticks, determinism; Lustre, Esterel, SIGNAL (multi-clock).
- [Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language)) — flows indexed by a clock; the `pre` and `->` operators; edge-detection example.
- [ScienceDirect: *The Esterel synchronous programming language*](https://www.sciencedirect.com/science/article/pii/016764239290005V) — deterministic concurrency; global clock; signal coherence rule.
### Freshness-gated recompute and sample-and-hold
- [arXiv: *Reactive Programming without Functions*](https://arxiv.org/pdf/2403.02296) — wasteful recomputation in pure push; recompute-only-when-necessary hybrids.
- [Wikipedia: Sample and hold](https://en.wikipedia.org/wiki/Sample_and_hold) — capture a value and hold it constant until the next sample.
### Acyclicity and explicit feedback
- [arXiv: *Secure Information Flow Typing in LUSTRE*](https://arxiv.org/pdf/2201.00184) — Lustre forbids zero-delay loops; cyclic-dependency static check.
- [arXiv: *A Type System for the Automatic Distribution of Higher-order Synchronous Dataflow Programs*](https://arxiv.org/pdf/1211.2776) — formal definition of the `fby` unit delay.
- [Wikipedia: Z-transform](https://en.wikipedia.org/wiki/Z-transform) — time-shifting property; `z⁻¹` as the unit-delay operator.
- [MathWorks: Unit Delay](https://www.mathworks.com/help/simulink/slref/unitdelay.html) — the unit-delay block as the `z⁻¹` discrete-time operator.
- [Wikipedia: Register-transfer level](https://en.wikipedia.org/wiki/Register-transfer_level) — signal flow between registers; registers as the only memory elements, clocked.
### Stream join semantics
- [kdb+/q reference: aj](https://code.kx.com/q/ref/aj/) — as-of join: prevailing/last value where time key ≤ left key.
- [pandas: merge_asof](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.merge_asof.html) — match nearest key; backward/forward/nearest directions; sorted-input requirement.
- [Apache Flink: Timely Stream Processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/) — watermark as a completeness assertion that triggers time-based operators (the barrier-join mechanism).
Signal-Exposure-and-Execution.md
+304
@@ -0,0 +1,304 @@
# Signal, Exposure, and Execution
A recurring design pattern in systematic trading separates *what a strategy
believes* from *how that belief is traded and what it costs*. The strategy's
primary, backtestable output is best modelled not as an equity curve and not as
a list of orders, but as a **target-exposure signal** — a signed, bounded
fraction of capital to hold per instrument per time step. Two distinct execution
models then consume that signal: a **frictionless, idealized model** that scores
the *signal's* quality on a clean yardstick, and a **realistic model with
frictions** that estimates a deployable currency profit-and-loss. The actual
order and position events are not the strategy's output at all — they are a
*derived* layer, the first difference of the exposure path. This page lays out
the pattern, the causal convention that keeps it free of look-ahead, and how the
resulting return series feeds performance metrics.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
## The separation of concerns
The pattern rests on a clean ontological split: an **alpha signal** (a directional
or sizing belief about future returns) is one object; an **execution model** (the
machinery that turns a desired position into fills, subject to cost and
constraint) is a different object. Keeping them separate lets a single signal be
scored on a frictionless basis, traded through many execution models, and combined
with other signals — all without re-authoring the strategy.
This is the same modularity that
[backtesting-engine architectures](Backtesting-Engine-Architecture) enforce
structurally: brokers and cost models are ordinary downstream consumers of a
strategy's output, never entangled with the signal logic itself. The benefit is
comparability — two signals are only comparable if they are scored under the
*same* execution assumptions, which is impossible if each signal carries its own
bespoke order logic.
Four layers compose the pattern:
1. **Signal -> exposure.** The strategy emits a target exposure per step.
2. **Frictionless scoring.** Exposure times return, integrated under an idealized
fill model, yields a synthetic equity curve that measures *signal quality*.
3. **Derived position events.** The first difference of the exposure path yields
a broker-independent table of buy/sell/close events.
4. **Realistic execution.** That event table feeds a cost-aware execution model
producing a deployable currency P&L.
The sections below treat each layer in turn.
## Target exposure as the primary output
**Exposure** is the proportion of capital committed to a position, expressed as a
fraction (or percentage) of the portfolio. A portfolio with $100,000 of which
$20,000 sits in a given sector has 20% exposure to it [L]
([Market exposure](https://en.wikipedia.org/wiki/Market_exposure)). The natural,
backtestable primitive for a strategy is therefore a **target-exposure signal**:
a number per instrument per time step saying *what fraction of capital the
strategy wants to hold*, rather than an order or a realized P&L.
### Why signed and bounded
Exposure is **signed**: a positive value is a long position (the holder owns a
positive amount and profits if the price rises), a negative value is a short
position (profits if the price falls), and zero is flat/neutral [L]
([Long (finance)](https://en.wikipedia.org/wiki/Long_(finance))). The signed
convention mirrors the portfolio notion of **net exposure** — the sum of *signed*
position sizes, long counted positive and short negative, so that
`net = long% − short%` and a perfectly balanced book has net exposure zero
(market-neutral) [L]
([Net exposure](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/net-exposure/)).
Its companion, **gross exposure**, is the sum of *absolute* position sizes
(`gross = long% + short%`); a gross above 100% signals leverage [L]
(same source).
A natural bound for a single instrument's signed exposure is the interval
`[-1, +1]` — fully short to fully long, with the magnitude capped at the available
capital [C]. The bound is a convention, not a law: a leveraged or multi-asset
book deliberately exceeds it (gross exposure > 100% is precisely leverage [L]),
and the exact cap is a risk-budget choice. What is load-bearing is that the
primary output is a *bounded, signed fraction of capital* rather than an unbounded
order size or a path-dependent equity figure.
### Signal score -> sizing -> target exposure
The exposure is rarely the raw strategy score. A typical pipeline is
> raw signal score -> position sizing / risk shaping -> target exposure.
The sizing step converts a dimensionless score (a forecast, a rank, a z-score)
into a fraction of capital, and it is where risk is shaped. Common conventions:
- **Fixed-fractional sizing.** Allocate a predetermined fraction of total capital
(or risk a fixed fraction of equity) per position, scaling the size with account
equity so risk-per-trade stays roughly constant [C]
([Position sizing](https://blog.quantinsti.com/position-sizing/)).
- **Volatility targeting / volatility-based sizing.** Scale exposure *inversely*
to recent realized volatility so the position delivers a roughly constant risk
contribution — levering up when volatility is low, deleveraging when it is high
[C] (same source). The canonical academic statement is that scaling a factor's
weight by the inverse of its past realized variance — taking *less* risk when
volatility is high — raises Sharpe ratios, because changes in volatility are not
offset by proportional changes in expected return [C]
([Moreira & Muir, *Volatility Managed Portfolios*](https://www.nber.org/papers/w22208)).
- **Kelly sizing.** The Kelly criterion gives the fraction of capital that
maximizes the long-run expected logarithm of wealth; for a simple bet
`f* = p − q/b` (with win probability `p`, loss probability `q = 1−p`, and payoff
odds `b`) [L]
([Kelly criterion](https://en.wikipedia.org/wiki/Kelly_criterion)). In practice
many size *below* full Kelly — **fractional Kelly** (half-Kelly, quarter-Kelly)
— to cut volatility and hedge estimation error [C] (same source).
All three are sizing conventions, interchangeable mappings from score to
exposure; none changes the *shape* of the primary output, which remains a signed,
bounded target-exposure signal.
## Measuring signal quality with a frictionless model
To judge a *signal* — independent of any broker, currency, or cost regime — the
target exposure is integrated against the instrument's return under an **idealized
execution model**: no spread, no commission, no slippage, perfect fills at the
reference price. The result is a synthetic, currency-free equity curve whose
units are conveniently expressed in return terms (or pips), serving as a level
yardstick for comparing and combining signals.
### The integral and its causal convention
For an exposure path `e_t` and per-period instrument return `r_t`, the synthetic
strategy return for a period is
> `strategy_return_t = e_{t-1} · r_t`
— the exposure **held into** period `t`, decided at the *prior* step `t−1`, earns
period `t`'s return. This one-step lag is the load-bearing causal convention [L]:
the position is established immediately after the decision at `t−1` and held over
the interval `[t−1, t]`, so the realized return uses the *next* period's return,
and the signal is formed strictly before the return is realized — making the
strategy implementable and free of look-ahead. Practitioners enforce this by
*shifting the signal/weight by one period* — generating each step's signal from
the *prior* bar's data, not the current bar's — before multiplying by returns;
failing to do so is a classic look-ahead bug that inflates backtest results [C]
([Backtesting pitfalls — lag the signal with `shift(1)`](https://coriva.eu.org/en/backtesting-pitfalls/)).
The structural side of this guarantee — read-only past-only windows, resamplers
that emit only completed bars — is treated on the
[look-ahead and causality](Look-Ahead-Bias-and-Causality) page; here the point is
that the *integration timing* (`e_{t-1} · r_t`, not `e_t · r_t`) is itself a
causality contract, not a stylistic choice. See also
[determinism and reproducibility](Determinism-and-Reproducibility): the same
exposure and return inputs must yield the same synthetic curve, every run.
### What it does and does not measure
The frictionless curve measures the **signal's** quality — its ability to be on
the right side of returns at the right size — and nothing about tradeability. It
is deliberately *not* a realistic P&L: by zeroing all costs and assuming perfect
fills it isolates alpha from execution. This is exactly what makes it a fair basis
for **comparing** two candidate signals or **combining** them (e.g. averaging or
risk-weighting their exposures), since both are scored under identical, cost-free
assumptions. A signal that looks strong frictionlessly but evaporates under
realistic costs is a known and important outcome — which is the job of the next
layer to reveal.
## Realistic execution with frictions
A **realistic execution model** re-scores the same strategy under real-world
costs and constraints, producing a currency profit-and-loss used for viability
assessment and deployment. Where the idealized model assumes a perfect fill at
the reference price, the realistic model charges for the gap between intention and
outcome. Its principal frictions:
- **Spread.** The difference between the quoted ask (immediate-purchase) and bid
(immediate-sale) price — a measure of the transaction cost and the cost of
trading without delay [L]
([Bid–ask spread](https://en.wikipedia.org/wiki/Bid%E2%80%93ask_spread)).
- **Commission / fees.** An explicit charge by the venue or broker for executing
an order — a broker is paid a commission when the deal is executed [L]
([Broker](https://en.wikipedia.org/wiki/Broker)).
- **Slippage.** The difference between the execution price a trader *expected*
(typically the price shown when the decision was made) and the price at which
the trade *actually* fills [L]
([Slippage (finance)](https://en.wikipedia.org/wiki/Slippage_(finance))).
Slippage arises from market impact, liquidity, and frictional costs, and cannot
be fully eliminated even with optimized execution (same source).
- **Market impact.** The price moving *against* the trader because of their own
order — up when buying, down when selling — a transaction cost that grows with
order size relative to turnover and shrinks with liquidity [L]
([Market impact](https://en.wikipedia.org/wiki/Market_impact)).
- **Lot, margin, and capital constraints.** Minimum lot sizes, margin
requirements, and available capital can force the realized position to differ
from the target; the model **may reject or modify** an order rather than fill it
as requested.
All of these are species of **transaction cost** — the costs incurred in making a
trade beyond the asset price itself, classically including search, bargaining, and
enforcement costs, and in markets the brokerage, spread, and impact components [L]
([Transaction cost](https://en.wikipedia.org/wiki/Transaction_cost)). A realistic
backtest applies these on each trade — typically charged on the *turnover* (the
change in positions) at each rebalance — and deducts them from the equity curve
[L]/[C]
([Backtest engine — transaction costs on position changes](https://www.mathworks.com/help/finance/backtestengine.runbacktest.html)).
Because spread, slippage, and impact are *hidden and recurring*, over many trades
they frequently dominate the explicit commission [C] — which is why a signal must
be scored under realistic execution before any claim of deployability.
## Position management as a derived layer
A subtle but central point: the actual **order and position events** —
buy/sell/close with sizes — are *not* the strategy's per-step output. They are a
**computed, broker-independent table derived from the exposure path**: a change in
desired exposure implies a trade, so the event stream is the **first difference**
of the target-exposure path.
> `trade_t ≈ e_t − e_{t-1}` (a non-zero difference implies an order)
An **order** is an instruction to buy or sell on a trading venue [L]
([Order (exchange)](https://en.wikipedia.org/wiki/Order_(exchange))); a buy or
sell order can *enter* or *exit* a position depending on the prior state. A
target-exposure path encodes the desired position at every instant, and
differencing it yields the order events needed to move from one desired state to
the next.
### Why it cannot be a per-step emission
A single decision instant may imply **more than one** event. The canonical case is
a *stop-and-reverse*: an exposure that flips from `+1` to `−1` in one step implies
both a *close* of the long and an *open* of the short — two position events from
one decision. Because one decision step can yield several events, the event
stream cannot be a single per-step output field; it must be a *derived table*
computed from the exposure path after the fact. (More generally, partial
scaling-in and scaling-out produce variable numbers of events per step.) This is
why position management sits *downstream* of the signal, as a transformation of
its output, rather than inside it.
### One table, many execution models
The derived event table is **broker-independent**: it states *what trades the
exposure path requires*, not *how any particular venue fills them*. The same table
can therefore feed many different realistic execution models — different spread,
commission, and slippage assumptions, or different venues — yielding directly
**comparable** P&L results for the identical underlying decisions. This closes the
loop on the separation-of-concerns goal: one signal -> one exposure path -> one
derived event table -> N execution models, all comparable because the decision
layer is shared and only the cost/fill layer varies.
## Connecting to scoring
Both execution models terminate in a **return series**: the frictionless model in
a synthetic, currency-free curve; the realistic model in a currency P&L net of
costs. That return series — *not* the raw exposure or the order table — is the
input to the performance metrics. Risk-adjusted ratios (Sharpe, Sortino),
path statistics (maximum drawdown), and trade statistics (profit factor, win rate)
are all computed *on* the return series produced by whichever execution model is
in use. See the [Metric Catalogue](Metric-Catalogue) for the definitions and the
[Metric Verification Log](Metric-Verification-Log) for their corrected formulas
and edge cases.
The practical consequence: a signal is typically scored **twice** — once
frictionlessly (does the alpha exist?) and once realistically (does it survive
costs?) — and the *same* metric definitions are applied to both return series.
A large gap between the two scores is the diagnostic signature of a signal whose
edge is too small to overcome its execution costs.
## Summary
| Layer | Object | Units | Purpose |
|---|---|---|---|
| Signal -> exposure | signed, bounded fraction of capital per step | dimensionless `∈ [-1,+1]` | the strategy's primary, backtestable output |
| Frictionless scoring | `e_{t-1} · r_t` integrated | returns / pips | measure *signal* quality, comparable across signals |
| Derived events | first difference of exposure path | orders (buy/sell/close, size) | broker-independent trade table |
| Realistic execution | event table + spread/commission/slippage/impact + constraints | currency P&L | viability and deployment |
| Scoring | return series of either model | ratios, drawdown, profit factor | performance metrics |
The throughline is that **alpha is separated from execution**: the strategy owns
the exposure, execution models own the cost, and the order events are a derived
consequence of the exposure path — a widely-practiced pattern, not a single
product's choice.
## References
### The separation of concerns
- [Market exposure — Wikipedia](https://en.wikipedia.org/wiki/Market_exposure)
### Target exposure as the primary output
- [Market exposure — Wikipedia](https://en.wikipedia.org/wiki/Market_exposure)
- [Long (finance) — Wikipedia](https://en.wikipedia.org/wiki/Long_(finance))
- [Net exposure — Corporate Finance Institute](https://corporatefinanceinstitute.com/resources/career-map/sell-side/capital-markets/net-exposure/)
- [Position sizing in trading — QuantInsti](https://blog.quantinsti.com/position-sizing/)
- [Volatility Managed Portfolios — Moreira & Muir (NBER w22208)](https://www.nber.org/papers/w22208)
- [Kelly criterion — Wikipedia](https://en.wikipedia.org/wiki/Kelly_criterion)
### Measuring signal quality with a frictionless model
- [Backtesting pitfalls — lag the signal with `shift(1)`](https://coriva.eu.org/en/backtesting-pitfalls/)
### Realistic execution with frictions
- [Bid–ask spread — Wikipedia](https://en.wikipedia.org/wiki/Bid%E2%80%93ask_spread)
- [Broker — Wikipedia](https://en.wikipedia.org/wiki/Broker)
- [Slippage (finance) — Wikipedia](https://en.wikipedia.org/wiki/Slippage_(finance))
- [Market impact — Wikipedia](https://en.wikipedia.org/wiki/Market_impact)
- [Transaction cost — Wikipedia](https://en.wikipedia.org/wiki/Transaction_cost)
- [runBacktest (transaction costs on position changes) — MathWorks documentation](https://www.mathworks.com/help/finance/backtestengine.runbacktest.html)
### Position management as a derived layer
- [Order (exchange) — Wikipedia](https://en.wikipedia.org/wiki/Order_(exchange))
### Connecting to scoring
- [Metric Catalogue](Metric-Catalogue)
- [Metric Verification Log](Metric-Verification-Log)
Strategy-Analysis-and-Validation.md
+153
@@ -0,0 +1,153 @@
# Strategy Analysis & Validation
> A knowledge base on how trading strategies are analysed, validated, and graded
> in quantitative finance. Project-neutral and durable: it records the *methods,
> formulas, and benchmark thresholds*, not the state of any one implementation.
>
> Part of the **[Trading-System Engineering & Strategy Validation](Home)** wiki;
> the sibling subject area is **[Backtesting Engine Architecture](Backtesting-Engine-Architecture)**
> (how the engine that *produces* these results is built). The formula-level
> reference is in **[Metric Catalogue](Metric-Catalogue)**; the source-checking of
> every load-bearing number is in **[Metric Verification Log](Metric-Verification-Log)**.
## Why validation, not just backtesting
Running a single backtest and reading off the equity curve is commodity — every
quant system does it. The hard, value-adding part is **validation**: deciding
whether an apparent edge is *real and robust* or an artefact of luck and
overfitting. A backtest produces *one* number; validation asks how that number
would hold up under different parameters, different random draws, unseen data, and
the statistics of having tried many variants. This page surveys the analysis
methods that answer that question and how their results are professionally
classified.
### How to read the thresholds
Two kinds of statement appear throughout, and the catalogue marks each:
- **[L] law / exact** — a definition or identity (e.g. the Sharpe ratio formula,
the profit-factor↔win-rate identity). These do not vary.
- **[C] convention** — a practitioner rule-of-thumb (e.g. "Sharpe > 2 is
excellent"). These vary by source, asset class, and market regime; treat them as
relative-comparison guidelines, never as hard pass/fail gates without checking
the primary source.
- **[CORR]** marks a value that was corrected during source verification — see the
[Metric Verification Log](Metric-Verification-Log).
### Common inputs
Most metrics are computed from one of a few generic inputs, defined once here and
referenced throughout the catalogue:
- **per-period return** `r` — the return of one clock period; the first difference
of the cumulative equity curve (`r_t = E_t − E_{t-1}`).
- **equity curve** `E` — the running cumulative P&L (or cumulative return).
- **trade list** — the set of closed trades, each with entry/exit, P&L, and
excursions.
- **benchmark** `b` — a market/benchmark return series, for relative metrics.
- **multi-run distribution** — the set of results across many resamples, seeds,
parameter cells, or folds.
## The four classic analysis axes
These are the standard ways to put a strategy through its paces. Each generates a
*family* of backtests from one strategy and reduces it to a verdict. The
engine-side counterpart — how a system constructs and runs these families
efficiently — is in
[Determinism and Reproducibility](Determinism-and-Reproducibility) (Monte-Carlo as
a sweep over seeds) and
[Graph Compilation and Optimization](Graph-Compilation-and-Optimization)
(computing a sweep-invariant sub-graph once).
| Axis | What it is | Produces | Financial question |
|---|---|---|---|
| **Parameter sweep** (grid / random) | run the strategy across a grid (or random sample) of its tunable parameters | a *surface*: one result per parameter cell | "Where does it make money — a broad robust plateau, or a single lucky spike?" |
| **Optimization** (argmax of a metric) | select the parameter set that maximizes an objective | the winning configuration + its scorecard | "Which configuration is best by my chosen yardstick?" (and: is that yardstick robust, or does it reward luck?) |
| **Walk-forward analysis** | chain *(optimize on an in-sample window)* then *(test on the next out-of-sample window)*, marched forward; stitch the OOS segments | a stitched out-of-sample curve + Walk-Forward Efficiency + parameter stability | "Do parameters chosen on the past generalize to the unseen future, or did optimization curve-fit noise?" |
| **Monte-Carlo analysis** | re-run the strategy across N randomized realizations (resampled returns/trade order, permuted prices, or a synthetic generator) | a *distribution* of every metric: percentile fans, stress drawdown, empirical VaR/ES, ruin probability, a permutation-test p-value | "Given that one backtest is just one draw, what is the distribution, the tail risk, and the chance the edge is luck?" |
A recurring trap sits inside the optimization axis: maximizing raw net profit
selects the cell most contaminated by luck and a single fat-tail trade. The
professional move is to optimize a *risk-adjusted* objective and read robustness
from the *neighbourhood* of the optimum, not its peak — see §A and §B of the
[Metric Catalogue](Metric-Catalogue).
## The broader analysis landscape
The four axes *generate* families of backtests; the following are the *reads* and
*tests* layered over them — the part that separates a rigorous validation from a
pretty equity curve.
- **Overfitting / selection-bias statistics** (PSR → DSR, PBO, Minimum Backtest
Length, the Harvey-Liu-Zhu haircut). A high in-sample Sharpe over many trials is
*expected by chance even at zero true skill* (the False Strategy Theorem). These
deflate the observed Sharpe for sample length, non-normality, and the **number
of trials**, turning a sweep/optimize result into a significance verdict.
- **Cross-validation for time series (CPCV, purging, embargo).** Plain k-fold
leaks information on serially-correlated, overlapping-label series. Purging and
embargo remove the leakage; Combinatorial Purged Cross-Validation builds many
fully-out-of-sample paths → a *distribution* of the Sharpe rather than one
number.
- **Sensitivity / robustness.** Read the sweep *surface* geometrically:
plateau-vs-spike, plateau-to-peak ratio, neighbourhood-averaged fitness,
parameter-perturbation dispersion. A robust edge is a broad smooth plateau; an
overfit one is an isolated island.
- **Stress / scenario analysis.** Replay curated crisis windows (2008, 2015 CHF,
2020) and inject stylized shocks (vol ×3, spread blow-out). A *conditional*
worst-case profile — "if X then Y", not "X is likely".
- **Regime analysis.** Slice performance by market state (bull/bear,
high/low-vol, trend/range). "Where does the money come from, and does the edge
invert in some regime?"
- **Cost / capacity analysis.** A frictionless backtest hides execution reality.
Sweeping cost assumptions yields the *break-even cost*; the square-root impact
law yields the *capacity ceiling* (the AUM before the strategy's own market
impact erodes the edge). The frictionless-vs-realistic distinction is detailed in
[Signal, Exposure, and Execution](Signal-Exposure-and-Execution).
- **Benchmark-relative significance.** CAPM alpha/beta, the information ratio, and
the t-statistic of the mean return test whether the return is skill (alpha) or
just market exposure (beta).
- **Null / random-entry sanity.** A permutation test or a random-entry null
strategy asks "does it beat random?" — permute the input return series (keep the
marginal, destroy the time-ordering an edge would exploit) and re-run; the real
result must sit in the right tail.
- **Live-vs-backtest divergence.** The one analysis *outside* the simulation: diff
realized live performance against the backtest prediction (the Sharpe haircut) to
separate genuine alpha decay from a code/data bug.
## Where the detail lives
- **[Metric Catalogue](Metric-Catalogue)** — for each analysis tool and metric
family: exact formula, what input it is computed from, and the conventional
benchmark threshold.
- **[Metric Verification Log](Metric-Verification-Log)** — how the formulas and
thresholds were source-checked, which were corrected, and the citations.
*This knowledge base was compiled from the quant-finance literature and
adversarially source-checked. Primary references include Bailey & López de Prado
(PSR / DSR / PBO / CPCV), Pardo (walk-forward), Magdon-Ismail et al. (expected
maximum drawdown), the Basel/FRTB market-risk standards (VaR / ES), and Bacon
(drawdown metrics). Per-topic citations are listed on each page.*
## References
Overview-level external references (each verified reachable and on-topic). The
full per-metric citation list is in the
[Metric Catalogue → References](Metric-Catalogue#references); corrected formulas
and thresholds are sourced in the [Metric Verification Log](Metric-Verification-Log).
- **Analysis axes:**
[Walk-forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization) ·
[Monte-Carlo method](https://en.wikipedia.org/wiki/Monte_Carlo_method) ·
[Permutation test](https://en.wikipedia.org/wiki/Permutation_test) ·
[Sensitivity analysis](https://en.wikipedia.org/wiki/Sensitivity_analysis)
- **Overfitting & validation statistics:**
[Deflated Sharpe Ratio & False Strategy Theorem](https://en.wikipedia.org/wiki/Deflated_Sharpe_ratio) ·
[Probability of Backtest Overfitting / CSCV](https://cran.r-project.org/web/packages/pbo/vignettes/pbo.html) ·
[Purged cross-validation (CPCV, purging, embargo)](https://en.wikipedia.org/wiki/Purged_cross-validation) ·
[Pseudo-mathematics and financial charlatanism (Bailey et al.)](https://scholarworks.wmich.edu/math_pubs/42/) ·
[López de Prado, Advances in Financial ML](https://www.oreilly.com/library/view/advances-in-financial/9781119482086/c07.xhtml)
- **Risk & benchmark-relative:**
[Value-at-Risk](https://en.wikipedia.org/wiki/Value_at_risk) ·
[Expected Shortfall](https://en.wikipedia.org/wiki/Expected_shortfall) ·
[Stress testing (financial)](https://en.wikipedia.org/wiki/Stress_test_(financial)) ·
[CAPM (alpha/beta)](https://en.wikipedia.org/wiki/Capital_asset_pricing_model)
_Sidebar.md
+17
@@ -0,0 +1,17 @@
**[Home](Home)**
**Backtesting Engine Architecture**
- [Architecture overview](Backtesting-Engine-Architecture)
- [Event-Driven Backtesting](Event-Driven-Backtesting)
- [Look-Ahead Bias & Causality](Look-Ahead-Bias-and-Causality)
- [Determinism & Reproducibility](Determinism-and-Reproducibility)
- [Reactive Streaming Dataflow](Reactive-Streaming-Dataflow)
- [Columnar Data Layout](Columnar-Data-Layout)
- [Signal, Exposure & Execution](Signal-Exposure-and-Execution)
- [Graph Compilation & Optimization](Graph-Compilation-and-Optimization)
- [Authoring & Deployment Lifecycle](Authoring-and-Deployment-Lifecycle)
**Strategy Analysis & Validation**
- [Validation overview](Strategy-Analysis-and-Validation)
- [Metric Catalogue](Metric-Catalogue)
- [Metric Verification Log](Metric-Verification-Log)