Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity
Page: Reactive Streaming Dataflow
Pages
Adaptive Search Authoring and Deployment Lifecycle Backtesting Engine Architecture Columnar Data Layout Determinism and Reproducibility Evaluation Dependency and Parallelism Event Driven Backtesting Graph Compilation and Optimization Grid Search Home Look Ahead Bias and Causality Metric Catalogue Metric Verification Log Parameter Search Methods Random Search Reactive Streaming Dataflow Signal Exposure and Execution Strategy Analysis and Validation
Clone
2
Reactive Streaming Dataflow
Brummel edited this page 2026-06-15 14:58:46 +02:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Reactive Streaming Dataflow

A reactive streaming dataflow model expresses a computation as a directed graph of nodes through which data is pushed from sources, through transforms, to sinks, with changes propagating automatically along the edges. When the graph is driven by a discrete logical clock — every node observing its inputs and emitting outputs at each tick — the model becomes a synchronous reactive dataflow, the same family of semantics that underpins synchronous dataflow languages and synchronous digital hardware. This page describes that computational model and the engineering properties it gives a streaming backtest engine: deterministic per-tick evaluation, freshness-gated recompute, acyclic structure with explicit feedback, and the join semantics needed to combine streams that tick at different rates.

This is a companion to Event-Driven Backtesting and Backtesting Engine Architecture; the causality guarantees it depends on are detailed in Look-Ahead Bias and Causality, the on-the-wire data representation in Columnar Data Layout, and how the graph is lowered for execution in Graph Compilation and Optimization.

Markers: [L] law/exact, [C] convention, [CORR] corrected.

Dataflow and reactive programming

Dataflow programming is a programming paradigm that models a program as a directed graph of the data flowing between operations. [L] Operations are connected by explicitly defined inputs and outputs and behave like black boxes; the language runtime, not the programmer, manages the movement of data between them (Wikipedia: Dataflow programming). Nodes are typically classified by their role in the graph: sources introduce data, sinks consume it (write to a file, database, stream, or display), and intermediate processing nodes transform it (Devopedia: Dataflow Programming).

The defining execution rule distinguishes dataflow from control-flow programming: [L] an operation runs as soon as all of its inputs become valid, rather than when a program counter reaches it (Wikipedia: Dataflow programming). Because a node's only coupling to the rest of the program is through its declared ports, [C] dataflow graphs are described as inherently parallel: two nodes with no path between them share no hidden state and can in principle execute concurrently (Wikipedia: Dataflow programming). A backtest engine exploits exactly this property across independent simulation runs (see Backtesting Engine Architecture), while keeping each individual run a single deterministic sequence.

Reactive programming is the closely related declarative paradigm "concerned with data streams and the propagation of change" (Wikipedia: Reactive programming). [L] Its central distinction from ordinary assignment is persistence of the dependency: where the imperative statement a := b + c computes a once, in a reactive setting the relationship is maintained, so that a is recomputed whenever b or c changes (Wikipedia: Reactive programming). The set of such relationships forms a dependency graph; a change at a source propagates to every transitive dependent.

Push versus pull

How a change reaches its dependents distinguishes two evaluation strategies. [L] In a push-based ("data-driven") model the producer hands a value to its consumers as soon as the value becomes available; in a pull-based ("demand-driven") model the consumer queries its sources when it wants a value (Wikipedia: Reactive programming). A streaming engine that drives a deterministic event loop is fundamentally push-based: each arriving tick is pushed from the source into the graph, and the engine walks the affected nodes forward. [C] A push-pull hybrid is common in practice — a lightweight change notification is pushed, and the actual (possibly large) value is pulled on demand — and is preferred when data volumes make eager value delivery wasteful (Wikipedia: Reactive programming).

Glitches and propagation order

A naive propagation can produce a glitch: a transient, momentarily incorrect output caused by evaluating a node before all of its inputs have been brought up to date. [L] The canonical example is t = seconds + 1; g = (t > seconds): the invariant g is always true, but if g is recomputed using the new seconds and the stale t, it transiently reads false — a glitch (Wikipedia: Reactive programming). [L] The standard cure is to propagate in topological order: sort the dependency graph so that every node is evaluated only after all the nodes it depends on, guaranteeing each node sees consistent inputs and recomputes at most once per propagation (Wikipedia: Reactive programming; Acolyer, A Survey on Reactive Programming). A glitch-free schedule is one of the concrete payoffs of insisting the graph be acyclic — a topological order exists iff the graph is a DAG (see below, and Graph Compilation and Optimization).

Synchronous reactive dataflow

When the propagation above is organized around a single discrete clock, the model becomes a synchronous reactive system. [L] A synchronous programming language targets reactive systems and structures execution as a sequence of logical ticks; within each tick, computation is assumed instantaneous — "a synchronous program reacts to its environment in a sequence of ticks, and computations within a tick are assumed to be instantaneous" (Wikipedia: Synchronous programming language). This is the synchronous hypothesis (or synchronous abstraction): outputs are produced synchronously with inputs, with no observable delay, exactly as a synchronous digital circuit is reasoned about as if signals propagated infinitely fast (Wikipedia: Synchronous programming language).

The payoff of the synchronous hypothesis is determinism. [L] By collapsing each reaction into one logical instant, the abstraction "eliminates the non-determinism resulting from the interleaving of concurrent behaviors," giving a well-defined, reproducible state after every tick (Wikipedia: Synchronous programming language). This is precisely the determinism a backtest requires: the same input stream must produce the same run, every time (see Event-Driven Backtesting). A synchronous dataflow graph reaches a single, unique state after each input tick, and that property is what makes two independent runs safely concurrent.

The classical synchronous languages

Three synchronous languages, developed in France in the 1980s, established the paradigm (Wikipedia: Synchronous programming language):

  • Lustre — a declarative, dataflow synchronous language. [L] Every variable is a flow: a stream of values paired with a clock, indexed by the native notion of logical time, and programs are built from dataflow nodes that map input flows to output flows (Wikipedia: Lustre (programming language)). Lustre is the closest classical relative of a streaming backtest graph.
  • Esterel — an imperative, control-oriented synchronous language. [L] It is a "deterministic concurrent programming language" for reactive systems whose threads "march in step with a global clock"; determinacy is ensured by the signal coherence rule, which requires every signal to hold a stable value throughout one reaction (ScienceDirect: The Esterel synchronous programming language).
  • SIGNAL — [L] "a dataflow-oriented synchronous language enabling multi-clock specifications," i.e. it natively expresses streams that tick on different clocks within one program (Wikipedia: Synchronous programming language). Multi-rate composition (below) is its central concern, and it is the cleanest classical antecedent for combining a tick stream with a slower bar stream.

A streaming engine for backtesting does not adopt any of these languages, but it inherits their semantics: a logically clocked, deterministic, per-tick dataflow whose nodes are flows of values.

Freshness-gated recompute and sample-and-hold

A literal reading of the synchronous hypothesis — every node recomputes every tick — does not scale when a graph mixes many sparse and high-rate streams. [C] In a purely push-based model "you perform potentially wasteful recomputations every time input sources change," and values are recomputed even when the inputs that feed them did not change (arXiv: Reactive Programming without Functions). With dozens of instruments each emitting at irregular times, a total per-tick sweep over the whole graph spends most of its work re-deriving outputs that are bit-for-bit identical to the previous tick.

Freshness gating is the standard remedy. [L] The propagation is restricted so that "computations dependent on values that did not change are not scheduled for execution" — only nodes with at least one genuinely changed input this tick are recomputed (Acolyer, A Survey on Reactive Programming; Wikipedia: Reactive programming). A node whose inputs are all unchanged simply holds its previous output.

The behavior of a stale input — one that did not tick this cycle — is the dual of recompute gating. The convention is sample-and-hold: a stale input contributes its last produced value rather than a missing one. [L] The term is borrowed exactly from the analog circuit, which "samples (captures, takes) the voltage of a continuously varying analog signal and holds (locks, freezes) its value at a constant level" until the next sample (Wikipedia: Sample and hold). In the streaming graph the analogue is: between two ticks of a slow stream, every downstream node sees that stream's last emitted value held constant, so a fast input ticking against a slow one always reads a well-defined slow value.

[C] Freshness gating and topological ordering are complementary, not alternatives: topological order makes a propagation glitch-free, and freshness gating makes it cheap by pruning unchanged sub-graphs from that ordered sweep. Many practical reactive runtimes combine data- and demand-driven evaluation precisely so that "values are recomputed only when necessary" while reactions stay near-instantaneous (arXiv: Reactive Programming without Functions). Note that freshness gating must never let a node observe a future value of a held input — the hold always carries the last value at or before the current cursor, preserving causality (see Look-Ahead Bias and Causality).

Acyclicity and explicit feedback

A reactive dataflow graph is required to be a directed acyclic graph (DAG). The reason is semantic, not merely tidiness: an implicit cycle is a combinational loop — a node whose value, within a single instant, depends on itself — and is ill-defined under the synchronous hypothesis, because there is no fixed evaluation order that lets the instant complete. [L] The classical synchronous languages reject such loops outright: Lustre forbids zero-delay (combinational) loops, and its static analysis includes a cyclic-dependency check that rejects programs not reducible to a bounded-space, acyclic schedule (arXiv: Secure Information Flow Typing in LUSTRE). Acyclicity is also what guarantees the topological order used for glitch-free propagation actually exists.

The only legal feedback path: a direct edge from node B back to node A would be an illegal combinational loop, so it is routed through an explicit unit-delay register, making this tick read the previous tick's value.

flowchart LR
  src["source"] --> a["node A"]
  a --> b["node B"]
  b --> sink["sink"]
  b --> reg[("unit delay register")]
  reg -.->|"last tick value"| a

Legitimate feedback — a value at tick t that depends on a value from tick t−1 — is not forbidden; it is made explicit by routing it through a unit-delay / register / state node. The same device appears under several names across disciplines, and the equivalence is exact:

  • Synchronous-language delay operators. [L] Lustre's pre operator "returns the value of p in the previous cycle," and the -> ("followed-by") operator supplies an initial value for the first cycle (Wikipedia: Lustre (programming language)). [L] The combined unit-delay fby ("followed-by") is defined by (x fby y)_i = x_0 when i = 0 and y_{i−1} otherwise (arXiv: A Type System for the Automatic Distribution of Higher-order Synchronous Dataflow Programs). Feedback in Lustre is expressed by passing a flow through pre, and the rule is that every backward path carries exactly one unit delay.
  • The DSP unit delay / z⁻¹. [L] In discrete-time signal processing a unit-delay element produces output x[n−1] from input x[n]; under the z-transform a one-sample delay is exactly multiplication by z⁻¹ (the time-shifting property Z{x[n−k]} = z⁻ᵏ X(z)), so z⁻¹ is the unit-delay operator (Wikipedia: Z-transform; MathWorks: Unit Delay).
  • The hardware register at register-transfer level. [L] RTL "models a synchronous digital circuit in terms of the flow of digital signals (data) between hardware registers, and the logical operations performed on those signals"; registers "are the only elements in the circuit that have memory properties" and synchronize the circuit to the clock edges (Wikipedia: Register-transfer level). A register is precisely a one-cycle state element separating combinational logic on its input side from its output side.

The engineering consequence is uniform across all three: forcing every feedback path through a visible one-tick delay keeps per-tick evaluation well-defined. Each instant is computed strictly from this tick's fresh inputs plus the frozen outputs of delay nodes captured at the end of the previous tick, so the within-tick graph remains acyclic and topologically orderable. Stateful operators that "remember" — moving averages, running counters, edge detectors (E = X and not pre X in Lustre) — are all built from this single primitive (Wikipedia: Lustre (programming language)). The research workflow's own "cycle" (sweep, re-run, refine) is an outer-loop activity and is not a dataflow cycle in this sense.

Stream join semantics: combining different rates

The hardest modeling problem in a multi-rate streaming graph is a node with several inputs that tick at different rates — a fast tick stream and a slow bar stream, two instruments quoting asynchronously, a price stream and a periodic indicator. Two distinct join semantics answer "when does the node fire, and what does it read for the inputs that did not just tick?", and a real graph needs both.

As-of / latest join

An as-of (latest-value) join fires when any input arrives, and reads each other input at its most recent value at or before the current time — i.e. the held value of sample-and-hold, applied at join time.

  • [L] In kdb+/q, aj (as-of join) aligns "the prevailing value from the value table with each record in the source table": for each left-table record, it appends the items of the last matching right-table record whose time key is ≤ the left record's time (kdb+/q reference: aj). The motivating use is attaching, to each trade, the bid/ask that prevailed at the trade's instant.
  • [L] In pandas, merge_asof is "a left-join except that we match on nearest key rather than equal keys." The default direction='backward' "selects the last row in the right DataFrame whose 'on' key is less than or equal to the left's key"; 'forward' and 'nearest' are also available, and both inputs must be sorted on the key (pandas: merge_asof). [CORR] Only the backward direction is causal for streaming: a 'forward' or 'nearest' as-of match can read a value that occurs after the join instant, which is a look-ahead leak — a streaming/backtest join must restrict to backward (≤ cursor) matching (pandas: merge_asof; see Look-Ahead Bias and Causality).

The as-of join is the natural join for a tick-driven graph: it never blocks waiting, it fires on every arrival, and the held value it supplies for stale inputs is exactly the sample-and-hold value described above.

Synchronizing-barrier join

A barrier (synchronizing) join does the opposite: it waits until every input that belongs to the current logical step has arrived before firing, so the node observes a fully complete set of inputs for that step. The stream-processing realization of this wait is the watermark. [L] In Apache Flink a watermark "declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t' ≤ t" — an assertion of completeness up to t that lets a time-based (e.g. windowed) operator safely fire and emit its result (Apache Flink: Timely Stream Processing). The barrier semantics matter when correctness requires all contributions for a step (a completed bar, an aggregate over every instrument in a basket) rather than a snapshot of whatever has arrived so far.

Both, often in one node

[C] These are complementary, not competing: a single node frequently combines an as-of input (a reference price it reads at its prevailing value) with a barrier group (a set of streams it must wait on before emitting). Choosing the wrong one is a modeling error: a barrier join where an as-of join was meant stalls on a sparse input that may never tick in lockstep; an as-of join where a barrier was meant fires on a partial step and silently uses held, possibly-stale values for the inputs still pending. The synchronous-language framing makes the distinction precise — an as-of input is sampled on the consumer's clock, a barrier group forces the consumer's clock to be the (slowest) shared clock of its inputs, which is exactly the multi-clock composition SIGNAL was designed to express (Wikipedia: Synchronous programming language).

Summary

A backtest engine built as a synchronous reactive streaming dataflow gets four properties from this model, each traceable to a named, established source. Push-based propagation in topological order over a DAG gives glitch-free, deterministic per-tick state (reactive programming + the synchronous hypothesis). Freshness gating plus sample-and-hold keeps that determinism affordable when many streams tick at very different rates. Explicit unit-delay nodes — the same device as Lustre's pre, the DSP z⁻¹, and the hardware register — make feedback well-defined without admitting ill-defined combinational loops. And the as-of / barrier join distinction gives a precise, causal vocabulary for combining streams of different rates. The downstream consequences for run reproducibility and causality are developed in Event-Driven Backtesting and Look-Ahead Bias and Causality; how the graph is lowered and optimized for execution is covered in Graph Compilation and Optimization.

References

Dataflow and reactive programming

Synchronous reactive dataflow

Freshness-gated recompute and sample-and-hold

Acyclicity and explicit feedback

Stream join semantics

Home

Backtesting Engine Architecture

Strategy Analysis & Validation