Add Mermaid diagrams to seven engine-architecture pages
Insert one structure-bearing, project-neutral Mermaid diagram into each of the densest / most topological pages, where a picture carries what prose conveys poorly: - Backtesting Engine Architecture: the two-layer view (deterministic substrate vs. orchestration layer building families of runs) - Event-Driven Backtesting: k-way merge of heterogeneous-cadence sources into one chronological timeline - Reactive Streaming Dataflow: the only legal feedback path (routed through an explicit unit-delay register) - Graph Compilation and Optimization: spec -> inline -> flat -> behaviour- preserving passes -> frozen instance - Signal, Exposure, and Execution: the fork into a frictionless model on the exposure signal vs. a realistic model on the derived position-event table - Authoring and Deployment Lifecycle: the hot-reload authoring loop vs. the one-way freeze to a reproducible artifact - Determinism and Reproducibility: the record-then-replay boundary Each diagram uses conservative, theme-neutral syntax (quoted labels, no custom colours) and was validated by real render (mermaid-cli + headless Chrome). Columnar Data Layout and Look-Ahead Bias are left without a flowchart on purpose — a memory-layout sketch and a cursor/window timeline fit those better than a node graph.
Brummel
@@ -213,6 +213,27 @@ references is the [Metric Verification Log](Metric-Verification-Log).
|
||||
|
||||
The lifecycle is two opposed regimes joined by a shared, parameterised engine:
|
||||
|
||||
_The fast hot-reload authoring loop versus the one-way freeze to a reproducible deployed artifact._
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph loop["Authoring loop"]
|
||||
edit["edit code"]
|
||||
rebuild["rebuild dynamic library"]
|
||||
reload["hot-reload into host"]
|
||||
inspect["run and inspect"]
|
||||
edit --> rebuild
|
||||
rebuild --> reload
|
||||
reload --> inspect
|
||||
end
|
||||
inspect --> validated{"validated?"}
|
||||
validated -->|"no"| edit
|
||||
validated -->|"yes"| freeze["freeze"]
|
||||
freeze --> artifact(["statically linked frozen artifact, equals a commit"])
|
||||
artifact --> deploy["deploy"]
|
||||
```
|
||||
|
||||
|
||||
| Phase | Linking | Mutability | Optimised for | Mechanism |
|
||||
|---|---|---|---|---|
|
||||
| **Authoring** | dynamic loading | hot-reloadable | feedback latency, preserved state | plugin module reloaded live [L] |
|
||||
|
||||
@@ -8,6 +8,35 @@ Markers: [L] law/exact, [C] convention, [CORR] corrected.
|
||||
|
||||
A serious engine separates cleanly into two layers with different correctness obligations.
|
||||
|
||||
_The substrate runs one reproducible simulation; the orchestration layer builds and runs many._
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Orchestration["Orchestration layer"]
|
||||
sweep["parameter sweep"]
|
||||
opt["optimization"]
|
||||
wf["walk-forward"]
|
||||
mc["Monte-Carlo"]
|
||||
cmp["comparison"]
|
||||
fam["build and run a family of runs"]
|
||||
agg["aggregate and compare"]
|
||||
sweep --> fam
|
||||
opt --> fam
|
||||
wf --> fam
|
||||
mc --> fam
|
||||
cmp --> fam
|
||||
end
|
||||
subgraph Substrate["Deterministic substrate"]
|
||||
inp["data window, params, seed"]
|
||||
run["deterministic run"]
|
||||
res["result and metrics"]
|
||||
inp --> run --> res
|
||||
end
|
||||
fam -->|"constructs many"| inp
|
||||
res -.->|"feeds back"| agg
|
||||
```
|
||||
|
||||
|
||||
The **single-simulation substrate** is the reproducible unit of work: one strategy, one parameter binding, one data window, in, one deterministic result out. [Backtesting](https://en.wikipedia.org/wiki/Backtesting) in finance "seeks to estimate the performance of a strategy or model if it had been employed during a past period" [L]. The substrate's obligation is *fidelity and reproducibility*: the same inputs must yield exactly the same run, and the run must contain no information that would not have been available at decision time.
|
||||
|
||||
The **orchestration layer** (sometimes called the experiment-orchestration layer or meta-layer) treats the substrate as a black-box function and invokes it many times under systematically varied conditions: parameter sweeps, optimization, walk-forward windows, Monte-Carlo resampling, and cross-strategy or cross-instrument comparison. Its obligation is *statistical*: to distinguish a strategy that is genuinely robust from one that merely fit the noise of a single history.
|
||||
|
||||
@@ -79,6 +79,22 @@ The net rule for a deterministic engine: seeds are inputs, one independent strea
|
||||
|
||||
Some inputs are intrinsically nondeterministic, external, or slow: a wall clock, a network fetch, a web scrape, the output of an LLM or other agent. These cannot be made deterministic in place — but they can be moved to a **boundary** and *recorded* once, so that the deterministic core only ever reads a fixed log.
|
||||
|
||||
_Recording quarantines nondeterminism before the boundary; the deterministic core only ever replays a recorded stream and never makes a live external call mid-replay._
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
ext["external or slow source, web, hosted model"] --> rec["record"]
|
||||
feed["market data feed"] --> rec
|
||||
feed --> fresh["compute fresh"]
|
||||
fresh --> rec
|
||||
rec --> store[("recorded timestamped stream")]
|
||||
subgraph core["deterministic core"]
|
||||
replay["replay reads recording"] --> run["deterministic run"]
|
||||
end
|
||||
store --> replay
|
||||
```
|
||||
|
||||
|
||||
The pattern is **record-then-replay** (record-and-replay):
|
||||
|
||||
- **Record mode** runs the nondeterministic source live, *once*, capturing its outputs into a timestamped stream on disk. Nondeterministic and external inputs are logged exactly as they occurred, so the captured run can later be "replayed again and again, and debugged exactly as it happened." [L]
|
||||
|
||||
@@ -200,6 +200,24 @@ the binary merge step at the heart of merge sort: "a K-way merge method, a
|
||||
generalization of binary merge, in which k sorted sequences are merged"
|
||||
([Merge sort][msort-wiki]) **[L]**.
|
||||
|
||||
_Heterogeneous-cadence sources are merged once, by timestamp, into a single ordered event stream._
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
tick["tick stream, irregular"]
|
||||
bars["one-minute bars"]
|
||||
news["daily news bias"]
|
||||
merge["k-way merge by timestamp, min-heap"]
|
||||
timeline["one chronological event stream"]
|
||||
strat["strategy graph"]
|
||||
tick --> merge
|
||||
bars --> merge
|
||||
news --> merge
|
||||
merge -->|"ties break by source order"| timeline
|
||||
timeline --> strat
|
||||
```
|
||||
|
||||
|
||||
The efficient implementation uses a **priority queue / min-heap** keyed by the
|
||||
front element of each stream. One maintains "a min-heap *h* of the *k* lists,
|
||||
using the first element as the key" ([Merge algorithm][merge-wiki]) **[L]**; the
|
||||
|
||||
@@ -8,6 +8,22 @@ Markers: [L] law/exact, [C] convention, [CORR] corrected.
|
||||
|
||||
A **parameter-generic graph specification** (a graph template) describes a topology with free numeric parameters (window lengths, thresholds, buffer depths) and free input roles (which streams feed which nodes), but is not yet runnable: buffers are unsized, the input bindings are symbolic, and nested sub-graph abstractions are still folded as named groupings. Producing something executable from it is a translation between two representations — the textbook definition of compilation, where a source representation is lowered into a target representation conducive to execution. The target here is a concrete, runnable instance.
|
||||
|
||||
_A spec is compiled, not interpreted, into a flat instance that is then the target of behaviour-preserving optimization._
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
spec["parameter-generic graph spec"] -->|"bind params, data, seed"| inline["inline nested composites"]
|
||||
inline --> flat["flat index-wired graph"]
|
||||
flat --> passes
|
||||
subgraph passes["behaviour-preserving passes"]
|
||||
cse["common-subexpression elimination"]
|
||||
dce["dead-node elimination"]
|
||||
hoist["sweep-invariant hoisting"]
|
||||
end
|
||||
passes --> frozen[("frozen runnable instance")]
|
||||
```
|
||||
|
||||
|
||||
The lowering binds three things and fixes the rest:
|
||||
|
||||
- **Parameters** — every free numeric parameter receives a concrete value, which both *configures* node behaviour and *sizes* node state. Buffer and window sizes are derived once parameters are known; this is a static, graph-level property — in synchronous dataflow, where "the number of tokens read and written by each process is known ahead of time," a static schedule can be found such that "channels have bounded FIFOs," i.e. the channel buffer sizes are fixed by analysis of the graph rather than discovered at run time [[synchronous dataflow](https://en.wikipedia.org/wiki/Synchronous_Data_Flow)]. [C]
|
||||
|
||||
@@ -52,6 +52,17 @@ The behavior of a *stale* input — one that did not tick this cycle — is the
|
||||
|
||||
A reactive dataflow graph is required to be a **directed acyclic graph (DAG)**. The reason is semantic, not merely tidiness: an implicit cycle is a *combinational loop* — a node whose value, within a single instant, depends on itself — and is ill-defined under the synchronous hypothesis, because there is no fixed evaluation order that lets the instant complete. [L] The classical synchronous languages reject such loops outright: Lustre forbids zero-delay (combinational) loops, and its static analysis includes a cyclic-dependency check that rejects programs not reducible to a bounded-space, acyclic schedule ([arXiv: *Secure Information Flow Typing in LUSTRE*](https://arxiv.org/pdf/2201.00184)). Acyclicity is also what guarantees the topological order used for glitch-free propagation actually exists.
|
||||
|
||||
_The only legal feedback path: a direct edge from node B back to node A would be an illegal combinational loop, so it is routed through an explicit unit-delay register, making this tick read the previous tick's value._
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
src["source"] --> a["node A"]
|
||||
a --> b["node B"]
|
||||
b --> sink["sink"]
|
||||
b --> reg[("unit delay register")]
|
||||
reg -.->|"last tick value"| a
|
||||
```
|
||||
|
||||
Legitimate feedback — a value at tick *t* that depends on a value from tick *t−1* — is not forbidden; it is made **explicit** by routing it through a unit-delay / register / state node. The same device appears under several names across disciplines, and the equivalence is exact:
|
||||
|
||||
- **Synchronous-language delay operators.** [L] Lustre's `pre` operator "returns the value of `p` in the previous cycle," and the `->` ("followed-by") operator supplies an initial value for the first cycle ([Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language))). [L] The combined unit-delay `fby` ("followed-by") is defined by `(x fby y)_i = x_0` when `i = 0` and `y_{i−1}` otherwise ([arXiv: *A Type System for the Automatic Distribution of Higher-order Synchronous Dataflow Programs*](https://arxiv.org/pdf/1211.2776)). Feedback in Lustre is expressed by passing a flow through `pre`, and the rule is that every backward path carries exactly one unit delay.
|
||||
|
||||
@@ -24,6 +24,25 @@ constraint) is a different object. Keeping them separate lets a single signal be
|
||||
scored on a frictionless basis, traded through many execution models, and combined
|
||||
with other signals — all without re-authoring the strategy.
|
||||
|
||||
_The signal forks into two evaluation paths: a frictionless model on the exposure signal directly, and a realistic model on the derived position-event table._
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
scores["signal scores"]
|
||||
sizing["sizing and decision node"]
|
||||
exposure[("target-exposure signal")]
|
||||
frictionless["frictionless model"]
|
||||
sigeq["signal-quality equity, pips"]
|
||||
events["position-event table, derived"]
|
||||
realistic["realistic model, with frictions"]
|
||||
curreq["currency equity, viability"]
|
||||
|
||||
scores --> sizing --> exposure
|
||||
exposure --> frictionless --> sigeq
|
||||
exposure -.->|"first difference"| events --> realistic --> curreq
|
||||
```
|
||||
|
||||
|
||||
This is the same modularity that
|
||||
[backtesting-engine architectures](Backtesting-Engine-Architecture) enforce
|
||||
structurally: brokers and cost models are ordinary downstream consumers of a
|
||||
|
||||
Reference in New Issue
Block a user