diff --git a/Authoring-and-Deployment-Lifecycle.md b/Authoring-and-Deployment-Lifecycle.md index 3fa35e9..60494d2 100644 --- a/Authoring-and-Deployment-Lifecycle.md +++ b/Authoring-and-Deployment-Lifecycle.md @@ -213,6 +213,27 @@ references is the [Metric Verification Log](Metric-Verification-Log). The lifecycle is two opposed regimes joined by a shared, parameterised engine: +_The fast hot-reload authoring loop versus the one-way freeze to a reproducible deployed artifact._ + +```mermaid +flowchart TD + subgraph loop["Authoring loop"] + edit["edit code"] + rebuild["rebuild dynamic library"] + reload["hot-reload into host"] + inspect["run and inspect"] + edit --> rebuild + rebuild --> reload + reload --> inspect + end + inspect --> validated{"validated?"} + validated -->|"no"| edit + validated -->|"yes"| freeze["freeze"] + freeze --> artifact(["statically linked frozen artifact, equals a commit"]) + artifact --> deploy["deploy"] +``` + + | Phase | Linking | Mutability | Optimised for | Mechanism | |---|---|---|---|---| | **Authoring** | dynamic loading | hot-reloadable | feedback latency, preserved state | plugin module reloaded live [L] | diff --git a/Backtesting-Engine-Architecture.md b/Backtesting-Engine-Architecture.md index 9d82c70..3d17a21 100644 --- a/Backtesting-Engine-Architecture.md +++ b/Backtesting-Engine-Architecture.md @@ -8,6 +8,35 @@ Markers: [L] law/exact, [C] convention, [CORR] corrected. A serious engine separates cleanly into two layers with different correctness obligations. +_The substrate runs one reproducible simulation; the orchestration layer builds and runs many._ + +```mermaid +flowchart TD + subgraph Orchestration["Orchestration layer"] + sweep["parameter sweep"] + opt["optimization"] + wf["walk-forward"] + mc["Monte-Carlo"] + cmp["comparison"] + fam["build and run a family of runs"] + agg["aggregate and compare"] + sweep --> fam + opt --> fam + wf --> fam + mc --> fam + cmp --> fam + end + subgraph Substrate["Deterministic substrate"] + inp["data window, params, seed"] + run["deterministic run"] + res["result and metrics"] + inp --> run --> res + end + fam -->|"constructs many"| inp + res -.->|"feeds back"| agg +``` + + The **single-simulation substrate** is the reproducible unit of work: one strategy, one parameter binding, one data window, in, one deterministic result out. [Backtesting](https://en.wikipedia.org/wiki/Backtesting) in finance "seeks to estimate the performance of a strategy or model if it had been employed during a past period" [L]. The substrate's obligation is *fidelity and reproducibility*: the same inputs must yield exactly the same run, and the run must contain no information that would not have been available at decision time. The **orchestration layer** (sometimes called the experiment-orchestration layer or meta-layer) treats the substrate as a black-box function and invokes it many times under systematically varied conditions: parameter sweeps, optimization, walk-forward windows, Monte-Carlo resampling, and cross-strategy or cross-instrument comparison. Its obligation is *statistical*: to distinguish a strategy that is genuinely robust from one that merely fit the noise of a single history. diff --git a/Determinism-and-Reproducibility.md b/Determinism-and-Reproducibility.md index 690a634..a298bf1 100644 --- a/Determinism-and-Reproducibility.md +++ b/Determinism-and-Reproducibility.md @@ -79,6 +79,22 @@ The net rule for a deterministic engine: seeds are inputs, one independent strea Some inputs are intrinsically nondeterministic, external, or slow: a wall clock, a network fetch, a web scrape, the output of an LLM or other agent. These cannot be made deterministic in place — but they can be moved to a **boundary** and *recorded* once, so that the deterministic core only ever reads a fixed log. +_Recording quarantines nondeterminism before the boundary; the deterministic core only ever replays a recorded stream and never makes a live external call mid-replay._ + +```mermaid +flowchart LR + ext["external or slow source, web, hosted model"] --> rec["record"] + feed["market data feed"] --> rec + feed --> fresh["compute fresh"] + fresh --> rec + rec --> store[("recorded timestamped stream")] + subgraph core["deterministic core"] + replay["replay reads recording"] --> run["deterministic run"] + end + store --> replay +``` + + The pattern is **record-then-replay** (record-and-replay): - **Record mode** runs the nondeterministic source live, *once*, capturing its outputs into a timestamped stream on disk. Nondeterministic and external inputs are logged exactly as they occurred, so the captured run can later be "replayed again and again, and debugged exactly as it happened." [L] diff --git a/Event-Driven-Backtesting.md b/Event-Driven-Backtesting.md index 61b2e56..dc5aae4 100644 --- a/Event-Driven-Backtesting.md +++ b/Event-Driven-Backtesting.md @@ -200,6 +200,24 @@ the binary merge step at the heart of merge sort: "a K-way merge method, a generalization of binary merge, in which k sorted sequences are merged" ([Merge sort][msort-wiki]) **[L]**. +_Heterogeneous-cadence sources are merged once, by timestamp, into a single ordered event stream._ + +```mermaid +flowchart LR + tick["tick stream, irregular"] + bars["one-minute bars"] + news["daily news bias"] + merge["k-way merge by timestamp, min-heap"] + timeline["one chronological event stream"] + strat["strategy graph"] + tick --> merge + bars --> merge + news --> merge + merge -->|"ties break by source order"| timeline + timeline --> strat +``` + + The efficient implementation uses a **priority queue / min-heap** keyed by the front element of each stream. One maintains "a min-heap *h* of the *k* lists, using the first element as the key" ([Merge algorithm][merge-wiki]) **[L]**; the diff --git a/Graph-Compilation-and-Optimization.md b/Graph-Compilation-and-Optimization.md index d685975..bf3a0f5 100644 --- a/Graph-Compilation-and-Optimization.md +++ b/Graph-Compilation-and-Optimization.md @@ -8,6 +8,22 @@ Markers: [L] law/exact, [C] convention, [CORR] corrected. A **parameter-generic graph specification** (a graph template) describes a topology with free numeric parameters (window lengths, thresholds, buffer depths) and free input roles (which streams feed which nodes), but is not yet runnable: buffers are unsized, the input bindings are symbolic, and nested sub-graph abstractions are still folded as named groupings. Producing something executable from it is a translation between two representations — the textbook definition of compilation, where a source representation is lowered into a target representation conducive to execution. The target here is a concrete, runnable instance. +_A spec is compiled, not interpreted, into a flat instance that is then the target of behaviour-preserving optimization._ + +```mermaid +flowchart LR + spec["parameter-generic graph spec"] -->|"bind params, data, seed"| inline["inline nested composites"] + inline --> flat["flat index-wired graph"] + flat --> passes + subgraph passes["behaviour-preserving passes"] + cse["common-subexpression elimination"] + dce["dead-node elimination"] + hoist["sweep-invariant hoisting"] + end + passes --> frozen[("frozen runnable instance")] +``` + + The lowering binds three things and fixes the rest: - **Parameters** — every free numeric parameter receives a concrete value, which both *configures* node behaviour and *sizes* node state. Buffer and window sizes are derived once parameters are known; this is a static, graph-level property — in synchronous dataflow, where "the number of tokens read and written by each process is known ahead of time," a static schedule can be found such that "channels have bounded FIFOs," i.e. the channel buffer sizes are fixed by analysis of the graph rather than discovered at run time [[synchronous dataflow](https://en.wikipedia.org/wiki/Synchronous_Data_Flow)]. [C] diff --git a/Reactive-Streaming-Dataflow.md b/Reactive-Streaming-Dataflow.md index 1833778..2ab027c 100644 --- a/Reactive-Streaming-Dataflow.md +++ b/Reactive-Streaming-Dataflow.md @@ -52,6 +52,17 @@ The behavior of a *stale* input — one that did not tick this cycle — is the A reactive dataflow graph is required to be a **directed acyclic graph (DAG)**. The reason is semantic, not merely tidiness: an implicit cycle is a *combinational loop* — a node whose value, within a single instant, depends on itself — and is ill-defined under the synchronous hypothesis, because there is no fixed evaluation order that lets the instant complete. [L] The classical synchronous languages reject such loops outright: Lustre forbids zero-delay (combinational) loops, and its static analysis includes a cyclic-dependency check that rejects programs not reducible to a bounded-space, acyclic schedule ([arXiv: *Secure Information Flow Typing in LUSTRE*](https://arxiv.org/pdf/2201.00184)). Acyclicity is also what guarantees the topological order used for glitch-free propagation actually exists. +_The only legal feedback path: a direct edge from node B back to node A would be an illegal combinational loop, so it is routed through an explicit unit-delay register, making this tick read the previous tick's value._ + +```mermaid +flowchart LR + src["source"] --> a["node A"] + a --> b["node B"] + b --> sink["sink"] + b --> reg[("unit delay register")] + reg -.->|"last tick value"| a +``` + Legitimate feedback — a value at tick *t* that depends on a value from tick *t−1* — is not forbidden; it is made **explicit** by routing it through a unit-delay / register / state node. The same device appears under several names across disciplines, and the equivalence is exact: - **Synchronous-language delay operators.** [L] Lustre's `pre` operator "returns the value of `p` in the previous cycle," and the `->` ("followed-by") operator supplies an initial value for the first cycle ([Wikipedia: Lustre (programming language)](https://en.wikipedia.org/wiki/Lustre_(programming_language))). [L] The combined unit-delay `fby` ("followed-by") is defined by `(x fby y)_i = x_0` when `i = 0` and `y_{i−1}` otherwise ([arXiv: *A Type System for the Automatic Distribution of Higher-order Synchronous Dataflow Programs*](https://arxiv.org/pdf/1211.2776)). Feedback in Lustre is expressed by passing a flow through `pre`, and the rule is that every backward path carries exactly one unit delay. diff --git a/Signal-Exposure-and-Execution.md b/Signal-Exposure-and-Execution.md index 2bf2a0a..aad3d6b 100644 --- a/Signal-Exposure-and-Execution.md +++ b/Signal-Exposure-and-Execution.md @@ -24,6 +24,25 @@ constraint) is a different object. Keeping them separate lets a single signal be scored on a frictionless basis, traded through many execution models, and combined with other signals — all without re-authoring the strategy. +_The signal forks into two evaluation paths: a frictionless model on the exposure signal directly, and a realistic model on the derived position-event table._ + +```mermaid +flowchart LR + scores["signal scores"] + sizing["sizing and decision node"] + exposure[("target-exposure signal")] + frictionless["frictionless model"] + sigeq["signal-quality equity, pips"] + events["position-event table, derived"] + realistic["realistic model, with frictions"] + curreq["currency equity, viability"] + + scores --> sizing --> exposure + exposure --> frictionless --> sigeq + exposure -.->|"first difference"| events --> realistic --> curreq +``` + + This is the same modularity that [backtesting-engine architectures](Backtesting-Engine-Architecture) enforce structurally: brokers and cost models are ordinary downstream consumers of a