Authoring and Deployment Lifecycle
This page describes the engineering lifecycle that carries a simulation or strategy from fast, exploratory authoring to a frozen, reproducible deployed artifact. The lifecycle rests on a structural split between a reusable engine (a software framework) and the separate applications built on it, and on a deliberate inversion: authoring optimises for fast feedback via dynamic reloading, while deployment optimises for immutability and auditability via static, reproducible builds. A fourth pattern — treating tuning parameters as runtime data rather than compiled-in constants — ties the two ends together by letting one compiled artifact be re-used across many configurations without rebuilding.
The patterns here are general software-engineering practice, neutral of any one toolchain or language. They are most visible in systems that run large families of deterministic simulations (see Backtesting Engine Architecture and Determinism and Reproducibility), but the framework/application split, hot reloading, and reproducible deployment apply to any extensible platform.
Markers: [L] law/exact, [C] convention, [CORR] corrected.
The framework / application split
A software framework is reusable software that provides generic functionality which developers extend to build complete solutions. The defining property of a framework — as opposed to a plain library — is inversion of control: the framework dictates the overall structure of execution and calls user code at predefined extension points, rather than user code calling it. [L] A library, by contrast, is a collection of resources that a program calls on its own terms; the caller controls the flow (Inversion of control). This is summarised as the Hollywood Principle, "Don't call us, we'll call you" (Software framework). [L]
A framework typically exhibits three further traits beyond inversion of control: pre-implemented default behaviour usable as-is or customised within a predefined structure; structured extensibility through hooks, callbacks, or APIs; and a non-modifiable core — the framework's own logic is fixed, extended but not edited, reflecting the open–closed principle (Software framework). [L]
The "game engine vs. the game" analogy
The cleanest illustration of this split is the game engine. A game engine is a software framework that supplies the underlying technology — rendering, physics, audio, scheduling — reused unchanged across many distinct games. [L] The historically decisive move was separating game-specific rules and data from basic engine concepts such as collision detection and entity management, which let teams specialise and let one engine power many titles (Game engine). [L] Reusable engines make building sequels and variants faster, a recognised competitive advantage. [C]
By analogy, a simulation or trading-research framework is the "engine"; the individual strategies, experiments, and models authored on top are the "games." The engine is the reusable, stable, broadly tested substrate; each application is a separate, faster-moving artifact that depends on it. Keeping the two as separate units — separate repositories, separate versioning, separate release cadence — is what allows the engine to stay general while applications proliferate. [C]
The reuse gradient
Components in such a system sit on a gradient from most-specific to most-universal, and where a component lives encodes how widely it may be reused:
- Application-local components — experimental nodes and blocks specific to one project, expected to change often and not yet proven general. They live with the application, not the engine. [C]
- Shared reusable libraries — components proven useful across several applications, factored out into a library that multiple independent applications depend on. Code reuse is exactly this: "A library can be used by multiple, independent consumers", so each gains its value without re-implementing it (Library (computing)). [L]
- A universal standard library — the building blocks so fundamental that they ship with the engine itself. A standard library is, by definition, "the library made available across implementations of a programming language" — the components every consumer can rely on being present (Standard library). [L]
Promoting a component up this gradient (local → shared → standard) is a deliberate, reviewed decision, not an accident of import path. [C] The general rule of thumb: a component should be promoted only once it has demonstrated genuine cross-application reuse, because every promotion widens the surface that the engine must keep stable. [C]
Hot reloading for authoring
The authoring loop optimises for one thing: the time between editing a definition and seeing its effect. The dominant technique is to compile the author's code into a dynamically loaded library and reload it into the running host live, so the author observes the change without restarting the process.
Dynamic loading
Dynamic loading is the mechanism by which a running program loads a library into memory at run time, resolves the addresses of its functions and variables, calls them, and can later unload the library — as opposed to binding the library at program startup (Dynamic loading). [L] Its two canonical use cases are exactly the authoring scenario: implementing plugin / extension systems, and selecting at run time which of several interchangeable implementations to use (Dynamic loading). [L] An authored strategy or node set is, in these terms, a plugin: a dynamically loaded module the host can swap without relinking itself.
This is distinct from ordinary dynamic linking, where a shared library is resolved automatically
at load time before main runs. Dynamic loading is explicit and happens later, under program
control — which is what makes reloading possible (Library (computing)). [L]
Hot swapping and edit-and-continue
Hot swapping, in the software sense, is "the ability to alter the running code of a program without needing to interrupt its execution" (Hot swapping). [L] A closely related debugger feature is edit and continue, which lets a developer change source code while the program is paused and apply the change on resume, without stopping, recompiling, and restarting (Edit and Continue). [L] Both share a goal: collapse the edit → rebuild → restart → re-navigate cycle.
The motivation is feedback latency, and the subtler cost it hides is lost state. When a feature under test is several steps removed from a fresh start, every restart forces the author to replay the path back to that point, "making the cycle multiple-seconds long." Keeping the process alive and injecting only the edited code "this way, you don't lose any of your state" (Hot reloading, React Native). [L] For a simulation host this state is expensive: loaded historical data, warmed buffers, an open viewer session. Reloading just the authored module preserves all of it. [C]
Authoring-loop only
A load-bearing discipline: dynamic reloading is an authoring-loop tool, not a deployment mechanism. The same dynamic-loading flexibility that accelerates iteration also undermines the property that a deployed artifact most needs — a fixed, known correspondence between the running code and an exact source version. The two ends of the lifecycle therefore use opposite linking strategies on purpose: hot-reloadable dynamic modules during authoring, a frozen static build for deployment. [C] Treating a hot-reloadable module as if it were a release artifact is an anti-pattern, because "what code is actually running" becomes unanswerable.
The frozen, reproducible deploy artifact
Deployment inverts every authoring priority. Where authoring wants mutability and speed, deployment wants immutability and auditability: the running artifact must correspond to one exact source version, and that correspondence must be verifiable after the fact.
Static linking
The first half is achieved by static linking. A static (statically linked) library "contains functions and data that can be included in a consuming computer program at build-time such that the library does not need to be accessible in a separate file at run-time" (Static library). [L] When every dependency is linked this way, the result is a stand-alone executable, also called a static build, in which all bindings are resolved at compile time (Static build). [L]
The advantages align precisely with deployment needs:
- Guaranteed availability. "The application is guaranteed to have the library routines it requires available at run-time, as the code to those routines is embedded in the executable file" (Static library). [L]
- No dependency drift. Static linking "avoids DLL Hell or more generally dependency hell," so behaviour does not change because some other version of a shared library happens to be present on the host (Static library). [L]
- Predictable, portable behaviour. A static build does "not rely on the particular version of libraries available on the final system," giving "very predictable behavior" (Static build). [L]
The trade-off is real but secondary for this use case: static builds are larger because library code is copied in rather than shared (Static build). [C] For a deployed, audited artifact, self-containment outweighs binary size.
Reproducible builds
Static linking pins what is in the artifact; reproducible builds pin the correspondence between that artifact and its source. The canonical definition: "A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts" (Reproducible Builds definition). [L] Equivalently, reproducible builds — also called deterministic compilation — ensure "the resulting binary code can be reproduced" from the same source (Reproducible builds). [L]
The payoff is a verifiable trust chain. Because anyone can rebuild and byte-compare, reproducible builds "provide a strong countermeasure against attacks where binaries do not match their source code, e.g., because an attacker has inserted malicious code into a binary" (Reproducible builds). [L] Practically, achieving bit-for-bit reproducibility requires building in a controlled environment with pinned toolchains and dependencies, and normalising sources of nondeterminism such as embedded timestamps and build paths. [C]
The artifact is a commit
Combining the two halves yields the deployment law for this lifecycle: the deployed artifact corresponds to one exact source version — the artifact is a specific commit. It is built once, versioned, frozen, and never hot-swapped. The audit statement "this running program is exactly this reviewed source at this revision" is then both true by construction (static linking removes runtime dependency variance) and independently checkable (reproducible builds let a third party rebuild and compare). [L] This is the deliberate opposite of the authoring loop: there, code is reloaded freely; here, code is immutable for the artifact's whole life. Compare the engine-side reproducibility guarantee in Determinism and Reproducibility: determinism makes a run reproducible; reproducible builds make the binary reproducible. Both are required for an end-to-end audit trail — the same source, built the same way, fed the same inputs, yields the same results.
Parameters as runtime data
A pattern that connects authoring speed to deployment economics: tuning parameters are runtime values, not compiled-in constants. A parameter sweep — running the same model across a grid of parameter settings — then needs no rebuild at all. The compiled module is loaded once; each configuration is just a different set of values bound at run time.
This matters because the two obvious alternatives are both costly. Baking parameters in as constants would require a recompile per configuration — multiplying build cost by the size of the sweep grid. Re-loading the dynamic module per configuration would re-pay loading cost needlessly. Keeping parameters as runtime data sidesteps both: one load, many cheap re-parameterizations. [C] This is the same separation the framework/application split already encodes — the engine provides the parameterised machinery; the values that configure it are data, supplied later — applied here to the inner loop of an experiment.
The cheap-re-derivation claim has a structural precondition: parameters may configure and size the computation but must not change its topology (a topology change is a different program, and would need a different compiled module). The formal treatment of how a parameterised computation is compiled once and then re-derived cheaply across a parameter family — sharing the parameter-invariant work and recomputing only what each setting changes — belongs to Graph Compilation and Optimization. The results a sweep produces, and how they are scored and compared across the grid, are covered by the Metric Catalogue; the discipline of checking those metric computations against references is the Metric Verification Log.
Lifecycle summary
The lifecycle is two opposed regimes joined by a shared, parameterised engine:
The fast hot-reload authoring loop versus the one-way freeze to a reproducible deployed artifact.
flowchart TD
subgraph loop["Authoring loop"]
edit["edit code"]
rebuild["rebuild dynamic library"]
reload["hot-reload into host"]
inspect["run and inspect"]
edit --> rebuild
rebuild --> reload
reload --> inspect
end
inspect --> validated{"validated?"}
validated -->|"no"| edit
validated -->|"yes"| freeze["freeze"]
freeze --> artifact(["statically linked frozen artifact, equals a commit"])
artifact --> deploy["deploy"]
| Phase | Linking | Mutability | Optimised for | Mechanism |
|---|---|---|---|---|
| Authoring | dynamic loading | hot-reloadable | feedback latency, preserved state | plugin module reloaded live [L] |
| Sweeping | (one load) | re-parameterised | reuse without rebuild | parameters as runtime data [C] |
| Deployment | static linking | frozen / immutable | audit, reproducibility | stand-alone reproducible build [L] |
The engine that all three share is a framework: it owns the control flow and calls authored code at its extension points, while authored applications remain separate, versioned units on a reuse gradient from local to standard. [L]
References
The framework / application split
- Software framework — Wikipedia
- Inversion of control — Wikipedia
- Library (computing) — Wikipedia
- Standard library — Wikipedia
- Game engine — Wikipedia
Hot reloading for authoring
- Dynamic loading — Wikipedia
- Hot swapping — Wikipedia
- Edit and Continue — Microsoft Learn
- Introducing Hot Reloading — React Native
The frozen, reproducible deploy artifact
Backtesting Engine Architecture
- Architecture overview
- Event-Driven Backtesting
- Look-Ahead Bias & Causality
- Determinism & Reproducibility
- Reactive Streaming Dataflow
- Columnar Data Layout
- Signal, Exposure & Execution
- Graph Compilation & Optimization
- Authoring & Deployment Lifecycle
Strategy Analysis & Validation