Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity
Page: Determinism and Reproducibility
Pages
Adaptive Search Authoring and Deployment Lifecycle Backtesting Engine Architecture Columnar Data Layout Determinism and Reproducibility Evaluation Dependency and Parallelism Event Driven Backtesting Graph Compilation and Optimization Grid Search Home Look Ahead Bias and Causality Metric Catalogue Metric Verification Log Parameter Search Methods Random Search Reactive Streaming Dataflow Signal Exposure and Execution Strategy Analysis and Validation
Clone
2
Determinism and Reproducibility
Brummel edited this page 2026-06-15 14:58:46 +02:00
Table of Contents

Determinism and Reproducibility

Determinism is the property that fixing every input to a computation fixes its output exactly — the same inputs always yield the same result, ideally bit-for-bit. For a backtesting engine, where conclusions can move real capital, determinism is not a performance detail but the foundation of trust: it is what makes a result reproducible, auditable, debuggable, and parallelizable without coordination. This page defines determinism precisely, derives the practical consequences that follow from it (reproduce-from-manifest, embarrassingly-parallel runs, seed-as-input randomness, and a record-then-replay boundary around anything nondeterministic), and marks which statements are exact laws versus practitioner conventions.

Markers: [L] law/exact, [C] convention, [CORR] corrected.

See also the sibling pages on Backtesting Engine Architecture, Event-Driven Backtesting, Reactive Streaming Dataflow, Graph Compilation and Optimization, the Metric Catalogue, and the Metric Verification Log.

What determinism means

A deterministic algorithm is one that, given a particular input, always produces the same output, with the underlying machine always passing through the same sequence of states. [L] It computes a mathematical function — and a function assigns a unique output to each input in its domain — so the same input can only ever map to the same output. [L]

The state-machine framing makes the "same sequence of states" precise: a state describes what the machine is doing at a particular instant, the machine passes in discrete steps from one state to the next, and in a deterministic system the current state alone determines the next state, so the entire trajectory through the state space is predetermined by the input. [L] This is exactly the discipline an event-driven backtest imposes: a single, synchronous event loop that reaches one well-defined state after each input tick.

Nondeterminism enters when the computed result depends on something other than the declared inputs. The standard sources are: hidden external state (user input, global variables, a hardware timer, an unrecorded random draw, or data read from disk); timing sensitivity under concurrency (e.g. several threads writing a shared location, where the interleaving order changes the result); and hardware faults that flip state unexpectedly. [L] A practical corollary: a computation that reads the wall clock, an environment variable, an un-seeded random generator, or a network response mid-run is nondeterministic by construction, regardless of intent. Eliminating these reads — or pushing them to a controlled boundary — is the engineering work of making a system deterministic. [C]

Bit-for-bit determinism and the build itself

The strongest form of determinism is bit-for-bit (byte-for-byte) identity of outputs, not merely equality up to rounding. The analogous discipline for compilation is the reproducible build: a build is reproducible if, given the same source code, build environment, and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts. [L] The same enemies recur at the build level — embedded timestamps, file-ordering, locale, build paths, and randomized hash seeds or address-space layout all leak environment into output and must be normalized away. [L]

Reproducible builds matter to a backtesting engine for the audit chain: a deterministic compilation lets a signed source tree be tied to the exact binary it produces, so a deployed artifact can be independently verified to derive from published source rather than from tampered binaries. [L] This is the build-time half of the audit trail; the run-time half (below) ties a result to an exact code version plus inputs. The deploy artifact of a validated strategy should therefore be a versioned, frozen binary — this result came from this commit — rather than a hot-swapped or rebuilt-on-the-fly target. [C]

Why determinism matters when money rides on the result

Four properties follow directly from determinism, and each is load-bearing when a backtest informs a capital allocation.

  • Reproducibility. If output is a function of input, anyone with the same inputs recomputes the same output. A reported Sharpe ratio or drawdown is then a checkable claim, not an anecdote. The replication crisis in empirical science — large fractions of published results failing to reproduce — is the cautionary backdrop: a 2016 Nature survey of 1,576 researchers found that more than 70% had tried and failed to reproduce another scientist's experiments. [C] A nondeterministic backtest is a private experiment of exactly this fragile kind.
  • Auditability / provenance. Determinism makes a result traceable: it can be attributed to an exact code version plus an exact set of inputs. Provenance — the formal record of what happened during an experiment — is precisely the variable set that computational reproducibility depends on, and capturing it is what lets a third party (a risk desk, a regulator, a future maintainer) re-derive and audit a number. [L]
  • Debuggability. A bug that reproduces on demand is a bug that can be fixed; an intermittent one often cannot. Determinism converts "it sometimes misbehaves" into "it misbehaves every time on this input," which is the precondition for the RED-first debugging loop and for the metric verification log to pin a number against a hand-checked oracle.
  • Trust. The composite of the three above. A strategy whose results cannot be reproduced, traced, or debugged cannot be trusted with capital, however attractive its headline metrics.

Reproduce-from-manifest: store the inputs, re-derive the result

Because a deterministic run is entirely a function of its inputs, the result need not be stored as a primary artifact — only the inputs need be. The minimal record (call it a manifest) is the set that fixes the run:

  • the code version (commit hash) of the engine and strategy;
  • the parameters bound into the run;
  • the data window (instrument(s), date range, source identity/version);
  • the random seed(s); and
  • any remaining configuration that the run reads.

Given that manifest, the full result — every intermediate stream, every metric — is re-derivable on demand. [L, from the deterministic-function definition: identical inputs ⇒ identical output.] This is the same principle as event sourcing, where application state is not stored as the source of truth but is rebuilt by replaying a log of events: one can discard the materialized state entirely and reconstruct it by re-running the events from the log against an empty start. [L] A reproducible backtest treats its result the way event sourcing treats state — as a derived view of an authoritative, compact input log.

Two practical consequences follow. First, storage is cheap and honest: persisting kilobytes of manifest instead of gigabytes of result rows both saves space and removes the risk that a stored result silently diverges from what the current code would now produce. Second, a stored result is a cache, not a record of truth — if the manifest re-derives a different number, the cache was stale and the manifest wins. [C] Storing results alongside manifests is still useful as a regression tripwire (re-derive and compare), but the manifest is the authority.

Disjoint runs are embarrassingly parallel

A workload is embarrassingly parallel when little or no effort is needed to split it into parallel tasks, because there is minimal or no dependency — no communication — between those tasks or their results. [L] Independent backtest runs are the textbook case: two runs over disjoint inputs share no mutable state, so neither can observe or perturb the other, and they can execute concurrently with no locks, no message passing, and no ordering constraints. Such problems run well even on loosely-coupled server farms and volunteer-computing pools precisely because they need no constant data exchange. [L]

This yields the single most important design rule for a deterministic engine's performance:

Parallelism is across runs, never within a single run. [C]

Each individual run stays strictly sequential and deterministic — a single-threaded event loop reaching a unique state after each tick — and all the speedup comes from executing many disjoint runs at once. Introducing threads inside one run reintroduces exactly the timing-sensitivity nondeterminism defined above (interleaved writes whose order changes the result), forfeiting bit-for-bit reproducibility for a local speedup that the across-runs axis supplies far more safely.

This is not merely a backtesting convention; it is the architecture behind deterministic simulation testing of distributed systems, most associated with FoundationDB — whose Flow framework dates to 2009 for C++. There, an entire cluster is simulated inside a single-threaded process so that the run is perfectly repeatable: cooperative scheduling means only one actor runs at a time, so "single-threaded execution means no race conditions — same seed, same event ordering, exact same execution path." [L] The same property that makes a distributed-systems bug reproducible after a trillion simulated operations makes a backtest reproducible after millions of ticks: the run is sequential and seed-determined, and scale comes from running many such runs in parallel. The bootstrap/compilation step that flattens a graph template into a runnable form is what produces these disjoint, independently-schedulable units.

Seed-as-explicit-input: making randomness reproducible

Randomness and determinism are reconciled by making the source of randomness an explicit input. A pseudorandom number generator (PRNG) is an algorithm that produces a sequence whose statistical properties approximate those of a random sequence, but the sequence "is not truly random, because it is completely determined by an initial value, called the PRNG's seed." [L] A random seed is that initializing value, and the defining reproducibility property is exact: "A pseudorandom number generator's number sequence is completely determined by the seed: thus, if a pseudorandom number generator is later reinitialized with the same seed, it will produce the same sequence of numbers." [L] PRNGs are valued precisely for their speed and their reproducibility, the latter being a direct consequence of this determinism. [L]

Threading an explicit seed through a run therefore folds all stochastic behavior back under the deterministic-function umbrella: the seed is just another declared input in the manifest, and fixing it fixes the run.

Monte-Carlo is "a sweep over seeds"

The Monte-Carlo method is a broad class of algorithms based on repeated random sampling to obtain numerical results — using randomness to solve (or estimate) deterministic quantities. [L] Its validity rests on the law of large numbers: an expectation can be approximated by the empirical mean of independent samples, and a sufficiently large number of samples produces an estimate arbitrarily close to the true value. [L] The samples must be independent of one another for the estimate to be sound. [L]

Under the seed-as-input lens, a Monte-Carlo study decomposes cleanly: it is a sweep over seeds, where each realization is a single deterministic run fixed by its own seed, and the study aggregates a statistic (mean, quantiles, a confidence band) over those realizations. Each realization is individually reproducible (re-run its seed, get the identical path), and the study as a whole is reproducible (store the seed set). This makes a Monte-Carlo validation just another embarrassingly-parallel family of disjoint runs — the realizations share no state and parallelize trivially. How the resulting distribution is interpreted (robustness, overfitting detection, confidence intervals) belongs to the validation cluster; see the Metric Catalogue and Metric Verification Log.

Seed pitfalls — two corrections

A seed reproduces a sequence only within the same PRNG implementation. The same named algorithm (e.g. a Mersenne Twister or PCG variant), seeded identically, can emit different sequences across different language libraries and framework versions, because seeding details and internal layout differ between implementations. [CORR] The popular shorthand "the same seed always reproduces the same numbers" is therefore true only with the implementation (and its version) held fixed — record it in the manifest alongside the seed value.

A second, costlier correction concerns parallel seeding. A widespread idiom is to derive each worker's seed by adding its worker index to a root seed (worker_seed = root_seed + worker_id). Authoritative guidance flags this as unsafe: "It is quite likely that multiple invocations of the program with different seeds will get overlapping sets of worker seeds … subsets of the workers will return identical results, causing a bias in the overall ensemble of results." [CORR] The correct construction spawns independent streams from a single root via a seed-sequence mechanism whose hashing has strong avalanche properties — flipping one input bit flips about half the output bits — so that even adjacent seeds map to far-apart states, with collision probability for ~2²⁰ streams on the order of 2⁻⁸⁸ (negligible). [L] Relatedly, generators that mutate a global shared state are fragile under reproducibility: any imported code can reset that global and silently change results, so the recommended practice is to instantiate a local generator object with an explicit seed and thread it through, rather than relying on a process-global seed. [C]

The net rule for a deterministic engine: seeds are inputs, one independent stream per disjoint run, derived by a proper seed-sequence — never by arithmetic on a root seed, never from un-seeded global state.

The record-then-replay determinism boundary

Some inputs are intrinsically nondeterministic, external, or slow: a wall clock, a network fetch, a web scrape, the output of an LLM or other agent. These cannot be made deterministic in place — but they can be moved to a boundary and recorded once, so that the deterministic core only ever reads a fixed log.

Recording quarantines nondeterminism before the boundary; the deterministic core only ever replays a recorded stream and never makes a live external call mid-replay.

flowchart LR
    ext["external or slow source, web, hosted model"] --> rec["record"]
    feed["market data feed"] --> rec
    feed --> fresh["compute fresh"]
    fresh --> rec
    rec --> store[("recorded timestamped stream")]
    subgraph core["deterministic core"]
        replay["replay reads recording"] --> run["deterministic run"]
    end
    store --> replay

The pattern is record-then-replay (record-and-replay):

  • Record mode runs the nondeterministic source live, once, capturing its outputs into a timestamped stream on disk. Nondeterministic and external inputs are logged exactly as they occurred, so the captured run can later be "replayed again and again, and debugged exactly as it happened." [L]
  • Replay mode is the backtest: the deterministic core consumes the recorded stream as ordinary input and makes no live external call. Because every nondeterministic input has been frozen into the log, the replayed execution is reproducible and can be re-run identically on demand.

This is the technique behind record-and-replay debuggers, which are "particularly useful for debugging intermittent and non-deterministic defects, which can be difficult to reproduce" — including heisenbugs that vanish under observation. [L] It is also exactly the discipline of deterministic simulation testing, which feeds the system entropy in a controlled way "such that the system and workload can still appear to have random behavior, while at the same time being perfectly reproducible." [L]

The crucial invariant for a backtesting engine:

The deterministic core never makes a live external call mid-replay. [C]

A live run (as opposed to a replay) computes fresh against the real external source and records that computation into the same timestamped log, so that today's live run becomes tomorrow's reproducible replay input. This cleanly separates the two regimes: live runs sit at the recording edge and may incur external latency, consent requirements, and nondeterminism; replays are pure functions of recorded inputs. (External-service calls — for instance to a hosted LLM acting as a data source — therefore belong at the recording/live edge, under whatever per-session consent policy governs them, and never inside a backtest.)

Conceptually this is event sourcing applied to a research pipeline: the recorded, timestamped stream is the authoritative event log, and every backtest result is a derived view that can be discarded and rebuilt from the log. [L] The recording boundary is also where the single ingestion-time merge of heterogeneous timestamped sources naturally lives — by the time data crosses into the deterministic graph, it is one ordered, replayable stream.

How the pieces compose

The five ideas reinforce one another into a single discipline:

  1. Determinism (same input ⇒ same output, ideally bit-for-bit) is the root property.
  2. It makes a run reproducible-from-manifest — store the inputs, re-derive the result — which gives storage economy and an audit trail.
  3. Disjoint deterministic runs are embarrassingly parallel, so throughput comes from running many at once: parallel across runs, sequential within a run.
  4. Seeds are explicit inputs, so randomness is reproducible and a Monte-Carlo study is just a parallel sweep over seeds.
  5. A record-then-replay boundary quarantines everything that cannot be made deterministic in place, so the core stays a pure, replayable function of recorded inputs.

Together these are what let a backtest be trusted with money: a reported number is not a one-off observation but a checkable, re-derivable, attributable claim. See Backtesting Engine Architecture for how the engine is structured around these invariants, and Event-Driven Backtesting for the single-threaded event loop at the core.

References

What determinism means

Why determinism matters

Reproduce-from-manifest

Embarrassingly-parallel runs

Seed-as-explicit-input and Monte-Carlo

Record-then-replay boundary

Home

Backtesting Engine Architecture

Strategy Analysis & Validation