63ea7eb3b1
Architect drift review (f68258b..HEAD) found no code drift — the firewall is
real at the dependency-graph level (aura-core/std/engine/cli carry no external
dep; the chrono/regex/zip tree is confined to aura-ingest), and C3/C7/C1 are
faithfully realized. Four ledger-level items, all resolved here:
- [ledger-drift] The zero-external-dependency commitment — the load-bearing
rationale for making aura-ingest a crate rather than a feature-gate — lived
only in spec 0011. Recorded in the ledger: C16 now states the engine workspace
is zero-external-dependency by commitment, names aura-ingest as the ingestion
edge / external-dependency firewall, and forbids an external dep in any engine
crate other than aura-ingest.
- [ledger-drift] C16's reuse taxonomy did not place aura-ingest. C16 now lists
the non-node engine crates (aura-engine, aura-cli, aura-ingest).
- [ledger-debt] C12's eager-materialization-now / Arc<[T]>-sharing-later choice
was only in the spec. C12 now carries a "Status (cycle 0011)" note recording
the deliberate gap.
- [debt] aura-engine/Cargo.toml carried a stale comment ("data-server enters
when the ingestion task starts") that contradicted the firewall it should
protect. Replaced with the dependency-pure / firewall-in-aura-ingest note.
The External components data-server bullet is updated to reality (pulled in by
aura-ingest; workspace now resolves from a populated cargo cache, not fully
offline).
Regression gate: no-op (commands.regression empty); architect is the sole gate.
Cycle 0011 is drift-clean. (Drift-clean, not a milestone close — the
Walking-skeleton milestone close additionally needs its end-to-end milestone
fieldtest, a separate deliberate act.)
612 lines
40 KiB
Markdown
612 lines
40 KiB
Markdown
# aura design ledger — INDEX
|
||
|
||
The ledger records the load-bearing design contracts and their rationale. Each
|
||
contract states what it **guarantees**, what it **forbids**, and **why**. A
|
||
change that breaks a contract is a design decision (amend the contract here with
|
||
its new rationale), never a silent refactor.
|
||
|
||
Provenance: contracts C1–C18 were settled in the initial rough-sketch design
|
||
interview (2026-06-03), walking the design tree root-to-leaf (C16–C18 and the
|
||
C10 refinement to a broker-independent position table came in follow-up turns;
|
||
C10 was later reframed in cycle 0007 — the intent/exposure stream is the primary
|
||
output, the position table a derived layer — see C10).
|
||
The `CLAUDE.md` **Domain invariants** section is the compressed, always-loaded
|
||
summary of the
|
||
subset that agents must never violate; this file is the fuller form with
|
||
rationale.
|
||
|
||
Vocabulary: a *contract* is one ledger entry. A *cycle* is one pipeline round; a
|
||
*milestone* is a tracker container spanning many cycles (the first milestone is
|
||
the **walking skeleton**: ingest → one signal → exposure/intent → sim-optimal
|
||
broker → synthetic pip-equity signal-quality metric).
|
||
|
||
---
|
||
|
||
## Foundation — what aura is
|
||
|
||
aura is a framework **and** a playground for traders. A human and (primarily)
|
||
LLMs author trading **nodes** directly in Rust; the engine backtests them
|
||
deterministically and massively in parallel, composes them fractally, validates
|
||
them (sweep / Monte-Carlo / walk-forward), and freezes a validated strategy into
|
||
a standalone bot with a broker connection.
|
||
|
||
The predecessor RustAst (`myc`) tried this as a custom DSL and failed: too slow,
|
||
too buggy, and LLMs author far better in Rust than in an unfamiliar DSL. aura
|
||
inverts it — engine in Rust, strategies in Rust — but keeps RustAst's *concepts*
|
||
(synchronous reactive streams, bounded-lookback series, run-counting, SoA).
|
||
RustAst is a conceptual reference, not a dependency. The one reused component is
|
||
`data-server` (the first data source).
|
||
|
||
What sets aura apart is **not** the single backtest — every quant system does
|
||
that — but the **World**: the meta-level where harnesses are dynamically
|
||
constructed and *families* of them orchestrated and explored (walk-forward,
|
||
sweeps, Monte-Carlo, comparison). The deterministic single-harness engine is the
|
||
*substrate*; the World is the *product* (C20–C22).
|
||
|
||
---
|
||
|
||
## External components
|
||
|
||
Two sibling projects live outside this repo and are named throughout the
|
||
contracts. Their concrete location is recorded here so the repo is the single
|
||
source of truth — not session memory.
|
||
|
||
- **`data-server`** — `~/dev/libs/data-server`, Gitea `Brummel/data-server`
|
||
(`http://192.168.178.103:3000/Brummel/data-server.git`). aura's **first data
|
||
source** and the one reused component (Foundation; C3, C11, C12). A standalone
|
||
leaf crate (deps: `chrono`, `regex`, `zip`) that loads Pepperstone **M1/tick**
|
||
binary files and shares them lock-free as `Arc<[T]>` chunks
|
||
(`CHUNK_SIZE = 1024`) via `SymbolChunkIter::next_chunk` (an inherent method
|
||
driven by `while let`, not the `Iterator` trait); `stream_*_windowed(from_ms,
|
||
to_ms)` provides C12's data-window with inclusive Unix-ms bounds. Records are
|
||
**AoS** (`M1Parsed`/`TickParsed`, `time_ms: i64` Unix-ms). The ingestion
|
||
boundary (C3) therefore **transposes** these AoS records into aura's SoA
|
||
columns (C7) and normalizes `time_ms` → canonical epoch-ns at that one
|
||
boundary. Pulled in as a cargo git dependency by the **`aura-ingest`** crate
|
||
(cycle 0011) — the ingestion edge and the workspace's **external-dependency
|
||
firewall**: `aura-ingest` is the *only* crate that links `data-server` (hence
|
||
its transitive `chrono`/`regex`/`zip`), so the engine crates (`aura-core` /
|
||
`aura-std` / `aura-engine` / `aura-cli`) and the frozen deploy artifact (C13)
|
||
stay external-dependency-free. Consequence: `cargo build/test --workspace` now
|
||
resolves from a populated cargo cache (one Gitea fetch) rather than fully
|
||
offline.
|
||
- **RustAst (`myc`)** — `~/dev/RustAst`, Gitea `Brummel/RustAst`
|
||
(`http://192.168.178.103:3000/Brummel/RustAst.git`). The predecessor DSL attempt
|
||
(Foundation): **a conceptual reference, never a dependency** — its DSL authoring
|
||
surface and `Value`/HM-inference machinery are exactly what aura rejects (C17).
|
||
But its **`src/ast/rtl/` layer is a working reference implementation of the very
|
||
streaming substrate aura rebuilds**, and is worth reading before authoring
|
||
`aura-core` — these are not just concepts, they exist as code:
|
||
- `rtl/series/data.rs` — `RingBuffer<T>` with financial-style indexing (index 0
|
||
= newest), `total_count` (the **run-count** of C5) and `lookback_limit` (C8's
|
||
pre-sized window); the `ScalarValue` marker trait ("flat scalars only, no
|
||
String/Record" = C7's closed scalar set); `ScalarSeries<f64|i64|bool>` and the
|
||
**SoA** `RecordSeries` for composites (C7's "OHLCV = a bundle of base columns").
|
||
- `rtl/series/mod.rs` — `create_typed_series`, dispatching element type → storage
|
||
backend (the type-specialized **factory** of C19).
|
||
- `rtl/streams/mod.rs` — `Signal { cycle_id, value }` (C4's cycle clock) and the
|
||
`Stream` / `Observer` / `ObservableStream` push traits (the reactive model of
|
||
C4/C5).
|
||
- `rtl/streams/register.rs` — the RTL **"register" / delay node** (the one
|
||
explicit feedback path of C5/C9) plus a seeded, reproducible OHLC generator
|
||
(C12's seed-as-input).
|
||
|
||
aura reimplements these natively and **sharpens** them: types are monomorphized
|
||
and edges type-erased to the four scalar kinds with direct dispatch (C7) instead
|
||
of carrying boxed `Value`s, and input history is shared zero-copy as `Arc<[T]>`
|
||
(C12) instead of a `VecDeque<Value>`. RustAst shows the *shape*; aura makes it
|
||
fast and deterministic.
|
||
|
||
---
|
||
|
||
## Contracts
|
||
|
||
### C1 — Determinism and disjoint parallelism
|
||
**Guarantee.** A backtest is a deterministic, synchronous, non-concurrent event
|
||
loop that reaches a unique state after each input tick. Same input (incl. seed)
|
||
→ bit-identical run. Two backtests are fully disjoint and run concurrently
|
||
without locking.
|
||
**Forbids.** Concurrency *within* a single sim; any nondeterministic input that
|
||
is not captured as an explicit input (see C11, C12).
|
||
**Why.** Real money rides on backtest results; reproducibility and an audit
|
||
trail are non-negotiable. Speed comes from parallelism *across* sims, which
|
||
disjointness makes lock-free.
|
||
|
||
### C2 — Causality / no look-ahead
|
||
**Guarantee.** A node sees only the past. Input history is a read-only window
|
||
that ends at the current cursor; a resampler emits a bar only once it is
|
||
complete.
|
||
**Forbids.** Any node access to data with `timestamp > now`; emitting a partial
|
||
/ still-forming bar.
|
||
**Why.** Look-ahead is the cardinal backtester bug — a fast backtester that
|
||
leaks the future is worse than none. Making the future *physically absent* from
|
||
what a node receives beats merely discouraging it.
|
||
|
||
### C3 — One merge, at ingestion only
|
||
**Guarantee.** Heterogeneous timestamped sources are k-way-merged by timestamp
|
||
into one chronological cycle stream at the ingestion boundary; source-native
|
||
time units (e.g. data-server's Unix-`time_ms`) are normalized there to the
|
||
canonical epoch-ns `timestamp` of C7.
|
||
**Forbids.** Any merge / as-of join *inside* the graph.
|
||
**Why.** A single ordered timeline is the mechanism that makes heterogeneous-rate
|
||
sources (news daily-bias + M5 + ticks) causally combinable without leaking the
|
||
future. Keeping the merge at one boundary keeps the graph semantics simple.
|
||
|
||
### C4 — Cycle granularity
|
||
**Guarantee.** The clock is data-driven: one input record = one cycle, advanced
|
||
in global timestamp order, with a monotonic `cycle_id`. Ties (same timestamp,
|
||
multiple sources) break by source declaration order.
|
||
**Forbids.** A fixed time-grid clock; nondeterministic tie ordering.
|
||
**Why.** The market *is* an irregular event sequence; a grid is arbitrary and
|
||
either wastes empty cycles or clumps ticks. Backtest and live differ only in the
|
||
origin of records, not the cycle semantics. Tie determinism preserves C1.
|
||
|
||
### C5 — Freshness-gated recompute and sample-and-hold
|
||
**Guarantee.** The `cycle_id` advances everywhere (a cheap counter), but a node
|
||
re-evaluates only when ≥1 of its own inputs is fresh this cycle (detected by
|
||
run-count); otherwise it holds its last output. Stale inputs contribute their
|
||
last (held) value.
|
||
**Forbids.** Recomputing every node every cycle ("push all" is true for the
|
||
*clock*, not for *recompute"); treating a held value as missing.
|
||
**Why.** Total recompute does not scale to many sparse high-frequency sources;
|
||
freshness-gating is the performance discipline that keeps the synchronous model
|
||
fast.
|
||
|
||
### C6 — Firing policy A and B, per input group
|
||
**Guarantee.** A node declares, per input group, one of two firing policies:
|
||
**A** fire-on-any-fresh + hold (latest / as-of join — e.g. tick × held
|
||
daily-bias); **B** all-fresh barrier (synchronizing join — e.g. O/H/L/C from
|
||
four separate 15m sources: the candle is complete only when all four are fresh).
|
||
A single node may mix an A input and a B group.
|
||
**Forbids.** A single global firing mode; forcing per-node-only granularity.
|
||
**Why.** Both are genuinely needed; RustAst implemented only B. Per-input-group
|
||
granularity is required by the OHLC-plus-bias case where one node needs both.
|
||
**Realization (cycle 0004).** Firing is tagged per input — `Firing::{Any,
|
||
Barrier(u8)}` on `InputSpec` — and inputs sharing a `Barrier` id form a group; a
|
||
mode-A input is its own trivial group, so "per input group" and the per-input tag
|
||
coincide. The barrier's synchronization token is the cycle **timestamp**, not the
|
||
`cycle_id`: under C4 four same-timestamp sources are four distinct cycles, so
|
||
RustAst's `cycle_id`-equality barrier could never fire across them. A group fires
|
||
when every member's last push carries the current cycle's timestamp, guarded by
|
||
"≥1 member fresh this cycle" (so a group completed earlier does not re-fire). This
|
||
fires both the multi-source bar and the within-source diamond rejoin (every push
|
||
in a cycle carries that cycle's timestamp).
|
||
|
||
### C7 — Four scalar base types, streamed as SoA
|
||
**Guarantee.** Only `i64`, `f64`, `bool`, `timestamp` (newtype over i64,
|
||
epoch-ns UTC) are streamed, as columnar Structure-of-Arrays. Composite streams
|
||
(OHLCV) are bundles of base columns — this is the **node-output model** too: a
|
||
node emits a record of 1..K base columns (C8), each forwarded field-wise to a
|
||
consumer slot; the bundle is structural, never a fifth scalar type. Edges are
|
||
type-erased to these four kinds;
|
||
the type check is paid once at wiring/sim-start, then the topology is frozen per
|
||
sim → direct dispatch, no per-event allocation.
|
||
**Forbids.** Streaming non-scalars (String, Records, tables, calendars) — those
|
||
live as metadata beside the hot path; `dyn Any` payloads; per-event heap
|
||
allocation; topology mutation mid-sim.
|
||
**Why.** Maximal streaming performance (SIMD/cache) needs a tiny closed scalar
|
||
set and SoA. The open set is composites (schemas of columns), not scalar types.
|
||
Type-erasure at the edge is also forced by the cdylib boundary (C13).
|
||
|
||
### C8 — The node contract
|
||
**Guarantee.** A node implements `schema()` (declares each input's scalar type,
|
||
required lookback depth, and firing group, **and the node's own tunable
|
||
parameters — typed, with ranges**, which aggregate into the blueprint's
|
||
param-space the optimizer sweeps, C12/C19/C20) + `eval(ctx) -> Option<&[Scalar]>`. The
|
||
engine provides read-only, zero-copy windows into each input's SoA ring buffer
|
||
(`ctx.f64_in(x)[k]`, sized at wiring); a node may *additionally* keep its own
|
||
mutable series for derived/intermediate state. `None`/Void return = filter /
|
||
not-yet-warmed-up. A node is a **producer, a consumer, or both**: a
|
||
producer/transformer exposes **one output port**, whose payload is a **record of
|
||
1..K base-scalar columns** (a scalar is the degenerate K=1 record; an `eval`
|
||
returns a borrowed row, one value per column); a **pure consumer (sink)** — chart,
|
||
equity, logger — has **no** output. Sources are pure producers; sinks are pure
|
||
consumers.
|
||
**Forbids.** A node sizing/growing its input lookback at runtime; more than one
|
||
output **port** per node; a fifth scalar type or a heterogeneous output payload
|
||
(a record is a bundle of base columns, C7); copy-on-read of input history.
|
||
**Why.** Engine-provided windows mean LLM-authored code cannot mis-manage
|
||
lookback bookkeeping, and history passes through zero-copy. Fixed, pre-sized
|
||
buffers suit deterministic, pre-dimensioned sims (no realloc in the hot loop).
|
||
**Realization (cycle 0005).** `NodeSchema.output` is a `Vec<FieldSpec>` (named base
|
||
columns; length 1 = scalar). Binding is **field-wise only**: `Edge::from_field`
|
||
selects one producer column per edge; consuming a whole record is N edges (no
|
||
"bind whole record" mechanism). The K fields of one record are **co-fresh by
|
||
construction** (one `eval`, one timestamp), so C6 is untouched. `eval` returns
|
||
`Option<&[Scalar]>` — a borrowed row into a node-owned buffer — so the forward
|
||
path allocates nothing per cycle (C7).
|
||
**Realization (cycle 0006).** The pure-consumer (sink) half of this contract is
|
||
now realized at the substrate: **recording is a node role, not a type.** A
|
||
recording node reads its typed input windows + `ctx.now()` in `eval` and pushes
|
||
the record to a destination it holds as a field (a channel, a chart handle) — an
|
||
**out-of-graph side effect**. There is no `Sink` type, trait, or engine flag: a
|
||
node that only records returns `None` (pure consumer), and a node may record
|
||
**and** return an output the engine forwards in the same `eval` (the "both"
|
||
case). **Encoding & return contract.** A pure consumer declares `output: vec![]`
|
||
— the empty record *is* the sink declaration; there is no separate type, trait,
|
||
or marker. Its `eval` returns `None` or a zero-width `Some(&[])`, and the run
|
||
loop debug-asserts the returned row's width equals the declared output width
|
||
(`row.len() == schema.output.len()`). Field-wise wiring resolves
|
||
`Edge::from_field` against the producer's `output` at bootstrap, so no edge can
|
||
bind a field of a zero-output node — it fails with `BadIndex` — making a sink
|
||
structurally unwireable as an in-graph producer; its only output is the
|
||
out-of-graph side effect. In-graph routing stays engine-owned data (the edge table); the escape out
|
||
of the graph is the node's own side effect — and that boundary is the
|
||
determinism / graph-as-data boundary (C1/C7).
|
||
|
||
### C9 — Fractal, acyclic composition
|
||
**Guarantee.** A composite is itself a `Node` that wires a sub-graph and exposes
|
||
one output; signal, combined signal, and (with execution) strategy are all the
|
||
same abstraction, nestable arbitrarily. The dataflow graph is a DAG; the only
|
||
feedback path is an explicit delay/state node (the RTL "register"). Wiring is
|
||
written in Rust (builder API); the built graph is introspectable runtime data.
|
||
**Forbids.** Implicit dataflow cycles (combinational loops); special-casing
|
||
"signal-of-signals" as separate mechanics.
|
||
**Why.** Self-application of one contract gives unlimited composition with no
|
||
adapter zoo. Acyclicity keeps the synchronous reactive model well-defined;
|
||
forcing feedback through a visible delay node keeps the per-cycle determinism
|
||
intact and the one legitimate feedback path explicit. Graph-as-data enables
|
||
visualization, freezing, and re-parameterization for sweeps.
|
||
|
||
### C10 — Strategy output is an intent/exposure stream; position management is a decoupled derived layer; brokers are downstream nodes
|
||
**Guarantee.** A strategy's primary, backtestable output is **not** an equity
|
||
curve, **nor a position-event table**, but an **intent / exposure stream**: the
|
||
DAG expresses exactly one *state* at time t (C8 — a node emits at most one record
|
||
per `eval`), so a strategy emits one **signed, bounded exposure** `f64 ∈
|
||
[-1, +1]` per cycle (per instrument; the portfolio is multi-instrument). Exposure
|
||
is the desired fractional position; **position sizing and risk live in the
|
||
decision/sizing node** that shapes a raw signal score into exposure. The chain is
|
||
`signals (scores) → decision/sizing node → exposure stream`.
|
||
|
||
**Signal quality** is evaluated by the **(a) sim-optimal broker** — a downstream
|
||
consumer node (C8/C9) that consumes the **exposure stream** plus the relevant
|
||
price stream and integrates `exposure(t-1) · (price(t) − price(t-1))` into a
|
||
**synthetic equity stream in pips**: deterministic, frictionless, perfect-fill
|
||
(no currency, no real-broker constraints). The exposure held *into* a cycle
|
||
(decided at t-1) earns that cycle's return, so the integration is causal (C2 — no
|
||
look-ahead). Pip PnL uses per-instrument pip metadata (reference data beside the
|
||
hot path, C7/C15). This pip-equity curve measures the **signal's quality**, not
|
||
an execution-modelled P&L; it is the primary research loop — backtest one signal,
|
||
combine it with another, backtest the combination.
|
||
|
||
**Position management** — turning the exposure stream into a broker-independent,
|
||
time-ordered **table of position events** (pure scalar columns, C7: `event_ts:
|
||
timestamp`, `action: i64` buy / sell / close, `position_id: i64`,
|
||
`instrument_id: i64`, `volume: f64` unsigned — direction is the `action`; a
|
||
position's open time is its opening event's `event_ts`, no separate `open_ts`; a
|
||
`close` references a `position_id` and may be partial via its own `volume`) — is a
|
||
**decoupled, derivable, downstream** layer. The events are the **first difference
|
||
of the exposure state**, materialized as a *computed table* (where multiple
|
||
events may share one `event_ts` — e.g. a reversal's close + open), **never as a
|
||
per-`eval` node output** (which C8 caps at one record per cycle). This layer feeds
|
||
**(b) realistic broker nodes** (Pepperstone, …): downstream consumer nodes that
|
||
consume the position-event table plus prices, apply real spread / commission /
|
||
slippage / lot / margin, may reject or modify positions, and produce a
|
||
**currency** equity stream for viability and deployment. The position-event table
|
||
stays broker-independent — one table feeds many realistic brokers, giving
|
||
directly comparable currency curves. Live: a realistic broker node consumes the
|
||
events in real time and routes orders as a side effect; reconciliation with the
|
||
real account is an external adapter.
|
||
|
||
So **the two broker classes attach at two different points**: the sim-optimal
|
||
broker on the **exposure stream** (signal quality, pips, now), realistic brokers
|
||
on the **derived position-event table** (execution viability, currency, later).
|
||
Both are ordinary downstream nodes (C8/C9); several can attach at once for
|
||
directly comparable curves.
|
||
**Forbids.** Treating an equity curve as the strategy's output; making the
|
||
**position-event table the strategy's direct DAG output** (it is derived, not
|
||
emitted per `eval` — a decision instant may need >1 event, which C8 forbids) or
|
||
the measure of signal quality; baking a broker into the strategy; a special
|
||
external broker subsystem (a broker is an ordinary node); storing `open_ts`
|
||
(derive it from the opening event); a signed-volume direction trick in the event
|
||
table (use `action`); broker-specific assumptions leaking into the strategy logic.
|
||
**Why.** The DAG is a synchronous reactive graph: at time t it holds exactly one
|
||
state, and a node emits at most one record per `eval` (C8). A *sequence of
|
||
position events* — where one decision instant (a stop-and-reverse) needs a close
|
||
**and** an open at the same `event_ts` — cannot be the DAG's per-cycle output
|
||
without violating C8. The state the DAG *can* express faithfully is the **desired
|
||
exposure** (one value per cycle); the position events are its first difference, a
|
||
derived consequence. Decoupling them resolves the C8↔C10 impedance and matches
|
||
how signal research actually proceeds: you first measure a signal's *quality*
|
||
(does this exposure, held over time, make pips?) independently of any execution
|
||
model; only later do you model position management and real-broker frictions for
|
||
deployment. The sim-optimal pip curve is a level, currency-free playing field for
|
||
comparing and combining signals; the derived position-event table + realistic
|
||
broker nodes then test real-world viability. Modelling brokers as nodes (not a
|
||
bespoke subsystem) keeps them within the one Node/graph abstraction (C9). This
|
||
supersedes the earlier "the strategy's output is the position-event table"
|
||
framing (cycle 0007 reframe).
|
||
**Realization (cycle 0007).** The signal-quality half of this contract is now
|
||
realized at the substrate, as two `aura-std` nodes composed on the unchanged
|
||
engine (the engine stays domain-free — it routes only `f64` records, never
|
||
"exposure" or "equity"). The **exposure stream** is realized as `Exposure { scale
|
||
}`: the decision/sizing node, `clamp(signal / scale, -1, +1)`, one `f64` per fired
|
||
cycle (`None` until warmed up). The **sim-optimal broker** is realized as
|
||
`SimBroker { pip_size }`: a two-input node (exposure, price) that accumulates
|
||
`prev_exposure · (price − prev_price) / pip_size` and emits cumulative pip equity
|
||
— the exposure held *into* a cycle (decided at t-1) earns that cycle's return, so
|
||
the integration is causal (C2), and `pip_size` is held reference metadata
|
||
(C7/C15), never streamed. An end-to-end harness (SMA-cross → `Exposure` →
|
||
`SimBroker` → recording sink) produces a recorded pip-equity curve, bit-identical
|
||
across runs (C1). The **position-management half** — deriving the position-event
|
||
table (buy/sell/close, `position_id`, partial closes) from the exposure history,
|
||
and the realistic broker nodes that consume it — is deliberately **not** built
|
||
this cycle; it remains the decoupled, derived, deferred layer described above.
|
||
|
||
### C11 — Generalized sources; record-then-replay determinism boundary
|
||
**Guarantee.** A source is anything that produces timestamped scalar streams —
|
||
market data (`data-server`) and non-financial sources (e.g. a news-agent node
|
||
emitting a bias) are treated identically. Anything nondeterministic, external,
|
||
or slow (LLM/news/web) is materialized into a recorded, timestamped stream
|
||
*before* it enters the engine; backtest replays the recording, live computes
|
||
fresh in real time and records it for future backtests. A bias enters as a value
|
||
held until the next event (firing policy A).
|
||
**Forbids.** Any live external call *inside* a backtest replay.
|
||
**Why.** It is the only model compatible with reproducible backtests — LLM calls
|
||
are nondeterministic and far too slow per-cycle. Per `~/.claude/CLAUDE.md`,
|
||
external LLM (IONOS) calls happen only at the recording/live-source edge, with
|
||
explicit per-session consent, never inside a sim.
|
||
|
||
### C12 — The atomic sim unit and the four orchestration axes
|
||
**Guarantee.** The atomic unit is `(frozen topology + param-set + data-window +
|
||
RNG-seed) → deterministic run → metrics`. Parameters are typed, ranged, runtime
|
||
values injected at graph build (no recompile per param-set; the optimizer sees a
|
||
generic vector of typed ranges). Raw data is shared read-only across sims via
|
||
`Arc<[T]>` (data-server is built for this). Four axes orchestrate the atomic
|
||
unit: (1) param-sweep (grid/random), (2) optimization (argmax metric),
|
||
(3) walk-forward (rolling in-sample optimize + out-of-sample test),
|
||
(4) Monte-Carlo (N seeded realizations perturbing input). **MC = sweep over
|
||
seeds**; each realization is itself deterministic given its seed.
|
||
*Status (cycle 0011).* The first ingestion (`aura-ingest`) **eagerly
|
||
materializes** a data-server window into owned source streams rather than
|
||
sharing `data-server`'s `Arc<[T]>` chunks zero-copy across sims — the cross-sim
|
||
`Arc<[T]>` sharing has no consumer until the orchestration axes above are built.
|
||
That sharing remains the target for the orchestration cycle that introduces
|
||
them; the transpose logic is unchanged by it. A known, deliberate gap, not drift.
|
||
**Forbids.** Baking a specific search strategy (Bayesian/genetic) into the
|
||
primitive — those are pluggable policies atop the atomic unit; recompiling on a
|
||
param change.
|
||
**Why.** A stable primitive + orchestration axes keeps "wahnsinnig schnell"
|
||
(embarrassingly parallel across the unit) cleanly separated from search policy.
|
||
Seed-as-input reconciles Monte-Carlo with C1. The "frozen topology" of the
|
||
atomic unit is one harness instance, selected by the harness's **structural
|
||
axes** (C20); the structural experiment matrix is the outer orchestration over
|
||
this dimension, the tuning sweep the inner (C19/C20).
|
||
|
||
### C13 — Hot-reload is authoring-only; deploy is frozen
|
||
**Guarantee.** A node/strategy is authored as a native Rust `cdylib`,
|
||
hot-reloaded during the authoring loop (Rust-ABI; host and node built with the
|
||
same toolchain). The live/deploy bot is a statically-linked, versioned, frozen
|
||
artifact.
|
||
**Forbids.** Hot-swapping a running live bot; loading third-party / foreign-
|
||
toolchain plugins.
|
||
**Why.** Hot-reload makes the research loop fast; a live artifact must be frozen
|
||
and reproducible (audit trail: this bot = this commit). A sweep pays no
|
||
hot-reload tax — params are runtime data (C12), so the cdylib loads once.
|
||
|
||
### C14 — Headless core, two faces
|
||
**Guarantee.** The engine is a UI-agnostic library. Two faces sit on it: a
|
||
**programmatic/CLI** face (the primary surface for the LLM and automation —
|
||
author a node, run a sim/sweep, emit structured metrics) and a **visual** face
|
||
for human exploration. Visualization is only a downstream consumer node on the
|
||
streams.
|
||
**Forbids.** Any UI/pixel knowledge inside the engine.
|
||
**Why.** The LLM drives programmatically, the human visually; a headless core
|
||
serves both. The visual face is the **playground** (C22) — egui-native,
|
||
in-process zero-copy from the SoA columns; it is staged after the runnable
|
||
substrate but is core to aura's identity, not optional.
|
||
|
||
### C15 — Resampling-as-node; sessions/calendars
|
||
**Guarantee.** A resampler is a node (finer stream → coarser bar stream),
|
||
clock-sensitive, emitting a completed bar only at the boundary (C2). Calendars
|
||
and instrument specs are metadata (non-scalar, beside the hot path); session
|
||
*context* is exposed as scalar streams via a `SessionNode` (`bars_since_open:
|
||
i64`, `in_session: bool`, `session_open_ts: timestamp`). "3rd 15m candle after
|
||
session open" is then a plain node checking `bars_since_open == 3`.
|
||
**Forbids.** Streaming the calendar; special-casing session logic outside the
|
||
stream model.
|
||
**Why.** Keeps the line consistent — everything a signal needs arrives as a
|
||
stream; reference data feeds source/session nodes from beside the hot path.
|
||
|
||
### C16 — Engine / project separation; three-tier node reuse
|
||
**Guarantee.** aura is the reusable **engine**; each research project is a
|
||
separate external repo that depends on aura via cargo (the game-engine / game
|
||
split). Node reuse is cargo-native, in three tiers: **`aura-std`** (universal
|
||
blocks, ship with the engine) / **shared node crates** (cross-project-reusable,
|
||
their own repos, pulled as cargo git deps) / **project-local `nodes/`**
|
||
(experimental, project-specific). A reusable node is an `rlib` dependency; the
|
||
hot-reload unit stays the project-side `cdylib` that composes it (consistent
|
||
with C13). Concretely a project is a **Rust crate** — a cdylib library of node /
|
||
strategy / experiment blueprints — plus a static `Aura.toml` (project context:
|
||
data paths, instrument/pip metadata, default broker & window, runs dir). During
|
||
research the `aura` host loads and runs it (C13 hot-reload); for deploy the
|
||
chosen strategy + broker freeze into a standalone binary. **A project is
|
||
therefore always a Rust program built on the engine.** Beyond the node-reuse
|
||
tiers, the engine workspace also carries non-node crates — `aura-engine` (the
|
||
run loop), `aura-cli` (the `aura` binary), and **`aura-ingest`** (the
|
||
data-source ingestion edge, cycle 0011). `aura-ingest` is the
|
||
**external-dependency firewall**: it alone links external crates (via
|
||
`data-server`), keeping the other engine crates and the frozen deploy artifact
|
||
(C13) dependency-pure (see External components). The engine workspace is
|
||
**zero-external-dependency** by commitment — the hand-rolled JSON of C14/C18 and
|
||
the from-scratch `aura-core` substrate exist precisely to honour it; the one
|
||
sanctioned external tree enters at the ingestion edge and is firewalled there.
|
||
**Forbids.** Project-specific signals in the aura repo (it keeps at most
|
||
example/fixture nodes under `examples/` for its own tests); a multi-project
|
||
manager inside aura; a bespoke node registry/marketplace (cargo + Gitea *is* the
|
||
package mechanism); an external (crates.io / git) dependency in any engine crate
|
||
other than the ingestion-edge crate `aura-ingest`.
|
||
**Why.** The engine/game split keeps the engine sharp and reusable while each
|
||
project versions its own research with its own forward-queue. Promotion
|
||
(local → shared → std) is the ordinary Rust reuse gradient, no new mechanism.
|
||
|
||
### C17 — Authoring surface
|
||
**Guarantee.** All *logic* — nodes, strategies, **and experiments/harnesses** —
|
||
is authored in native Rust through **Claude Code + the skills pipeline**: the
|
||
human describes, Claude writes the Rust, builds it, runs it via the `aura` CLI,
|
||
and reports metrics. Declarative config (`Aura.toml`) carries only **static
|
||
project context** (data paths, instrument/pip metadata, defaults, runs dir),
|
||
never logic. aura ships **no embedded coding-LLM**. IONOS LLMs are used only as a
|
||
*runtime data source* (news-agent bias, C11), gated by per-session consent,
|
||
never in the code path.
|
||
**Forbids.** An in-app LLM chat that generates node code inside aura; using
|
||
IONOS (weaker models) as the authoring brain.
|
||
**Why.** LLMs author Rust well in Claude Code — that is the fix to RustAst's
|
||
failure; making weaker models the coding brain reintroduces the very problem.
|
||
Keeps aura's scope an engine + playground, not an LLM-IDE.
|
||
|
||
### C18 — Project management: one repo = one project, plus a run registry
|
||
**Guarantee.** Management has two planes. (1) **Code & forward-queue:** git
|
||
(commit = identity; the frozen bot *is* a commit) + Gitea (ideas/hypotheses as
|
||
the forward-queue, a research thrust = a milestone, the
|
||
`idea → experimental → validated → deployed` label gradient). (2) **Experiments
|
||
& results:** an Aura-native **run registry** — one record per run = a *manifest*
|
||
(node-commit + params + data-window + seed + broker profile) + *metrics*,
|
||
queryable, with *lineage* (composite ← signals; run ← inputs). Determinism
|
||
(C1/C12) makes a run reproducible from its tiny manifest, so the registry stores
|
||
manifests + metrics and re-derives full results on demand. Depth: **structured**
|
||
(promotion/status, lineage, run-diff).
|
||
**Forbids.** Storing results not reproducible from a recorded manifest;
|
||
duplicating git/Gitea inside aura; a multi-project workspace manager.
|
||
**Why.** Comparing experiments over time is the heart of the research loop and
|
||
has no home in git/Gitea; determinism makes a structured registry cheap.
|
||
Sequencing: the walking skeleton emits a manifest + metrics per run from day
|
||
one; the registry/index is a later milestone over manifests that already exist.
|
||
|
||
### C19 — Bootstrap: blueprint → instance (recursive)
|
||
**Guarantee.** Construction is a distinct phase, recursive at every level. Each
|
||
node type has a **factory** `params → sized concrete node` (e.g. `SMA(length)`
|
||
sizes its ring buffer). A **blueprint** is the param-generic, input-role-generic
|
||
graph-as-data produced by running a Rust builder (C9); it carries *free* numeric
|
||
params (declared ranges) and *free* input roles. The **bootstrap** binds
|
||
`(blueprint + param-set + data bindings + seed)` into a concrete, **frozen
|
||
instance** — buffers sized, topology fixed. This is precisely the "wiring /
|
||
graph build" that C7 ("sized at wiring", "topology frozen per sim") and C12
|
||
("params injected at graph build") already reference. The same machinery applies
|
||
recursively up to the harness (C20). A sweep builds many instances from one
|
||
blueprint; instances are disjoint (C1).
|
||
**Forbids.** Params that change topology (a topology change is a *different*
|
||
blueprint — Fork A, C7 "frozen"); resizing buffers after bootstrap; running a
|
||
sim against an un-bootstrapped blueprint.
|
||
**Why.** Separating the param-generic blueprint from the param-bound instance is
|
||
what makes one strategy reusable across a whole sweep and lets the optimizer
|
||
mutate "the 20" by *rebuilding* an instance (cheap; no recompile, C12) instead
|
||
of rewriting code. Naming the build phase makes the implicit "wiring" of C7/C12
|
||
explicit.
|
||
|
||
### C20 — Strategy ↔ harness; the harness is the root sim graph
|
||
**Guarantee.** A **strategy** is a reusable composite-node blueprint (C9):
|
||
broker-, data-, and viz-independent, with inputs declared as named **roles**
|
||
(symbol-agnostic where possible) and the intent/exposure stream (C10) as output.
|
||
A **harness** (the experimental setup) is the **root sim graph** — sources bound
|
||
to the strategy's input roles + the strategy + attached broker node(s) + sinks —
|
||
and is itself produced by the bootstrap (C19). A harness *instance* is C1's
|
||
disjoint unit (RustAst's "root scope"). The harness has **two kinds of
|
||
parameterization**: **structural axes** (which strategy, which instrument(s),
|
||
which broker(s), which window) whose variation selects *different* instances —
|
||
the **experiment matrix**; and **tuning params** (the strategy's numeric params)
|
||
swept *within* a fixed structure (Fork A). The same strategy blueprint is reused
|
||
across backtest, sweep, visual workspaces, and the frozen live bot — each a
|
||
different harness. **Both strategy and harness/experiment are authored in Rust**
|
||
via builder APIs (C17); the experiment matrix is ordinary Rust control flow
|
||
(loops/conditionals), not a config schema. Ontologically, a node (incl. a
|
||
strategy composite) is an **open** fragment — free input roles + ≤1 output
|
||
(C8/C9) — that does not run alone; a **harness is the closed root graph**: a
|
||
strategy with its input roles bound to sources and its output terminated in
|
||
broker/sink nodes, under a clock. A harness is therefore **not a node** (no free
|
||
inputs, no output; it does not fit `eval`) — it is the *closure that runs*, C1's
|
||
disjoint unit / the root scope. Harnesses do not nest as nodes; the World (C21)
|
||
orchestrates them as objects.
|
||
**Forbids.** Embedding data sources / brokers / sinks inside a strategy; a
|
||
declarative experiment mini-DSL (logic is Rust — C17); modelling the harness as
|
||
a node (it is the closed root scope, not an open composable node).
|
||
**Why.** Reusability needs the strategy to be a context-free blueprint that many
|
||
harnesses embed. Modelling the harness as a root graph keeps it within the one
|
||
Node/graph abstraction (C9) and makes "10 strategies in one environment" and
|
||
"one strategy × N instruments" plain nested loops over the structural axes. Rust
|
||
authoring (not config) preserves full programmatic power — conditional/adaptive
|
||
matrices, generated axes, custom wiring — and avoids re-introducing the DSL trap
|
||
C17 rejects.
|
||
|
||
### C21 — The World: the meta-level is the product
|
||
**Guarantee.** Above the harness sits the **World** — the project's program /
|
||
"game" (C16: a Rust crate). Within it, **harnesses are dynamically constructible,
|
||
first-class objects**: meta-programs (walk-forward, sweep, optimize,
|
||
Monte-Carlo — C12's axes) construct harness instances at runtime via the
|
||
bootstrap (C19), run them disjointly in parallel (C1), aggregate / compare their
|
||
results, and discard the transient instances. A walk-forward rolls windows →
|
||
bootstraps a harness per window → stitches out-of-sample equity + parameter
|
||
stability into one meta-result. Orchestrating *families* of harnesses is
|
||
**first-class**, not a headless afterthought; the run registry (C18) is the
|
||
World's memory.
|
||
**Forbids.** Relegating multi-harness orchestration (walk-forward / sweep /
|
||
comparison) to second-class headless-only status; treating the single backtest
|
||
as the product.
|
||
**Why.** What happens *within* one harness — backtest a strategy → equity — is
|
||
commodity; every quant system covers it. aura exists for the meta-level: a
|
||
programmable space where harnesses are dynamically built and families of them
|
||
orchestrated and explored. The deterministic single-harness engine (C1–C20) is
|
||
the **substrate**; the World is the **product**.
|
||
|
||
### C22 — The playground is a trace explorer; sinks are the recording mechanism
|
||
**Guarantee.** The World is a *program*: nothing is displayable until it runs,
|
||
and its harnesses are transient machinery (built, run, discarded — C21). The only
|
||
durable, displayable substance is the **recorded trace**, captured by **sinks** —
|
||
pure consumer nodes (C8) that persist a stream (equity, position events, a node's
|
||
output) into the run registry (C18). **Displayable = exactly what a sink
|
||
recorded**; with no sink, only input params + summary metrics remain. The
|
||
**playground** is therefore an **execution viewer / trace explorer**, and it
|
||
plays *any* harness (it is the harness *player*, not a harness, never bound to a
|
||
default one): before a run it shows the program *structure* (graph-as-data, C9) +
|
||
param knobs; during a run, live sink streams; after a run, recorded traces +
|
||
metrics from the registry — including meta-views (stitched walk-forward, sweep
|
||
surfaces, multi-strategy / instrument comparison). Observability is **explicit
|
||
and selective** — you instrument what you want to see; the choice of sinks is
|
||
part of the experiment. The engine ships sample harnesses *with* sinks so a
|
||
newcomer sees a populated trace immediately; a fresh project is empty until
|
||
built, run, and instrumented.
|
||
**Forbids.** A scene-editor model that assumes a persistent populated world;
|
||
constructing or wiring topology in the UI (topology is Rust + hot-reload,
|
||
C9/C17 — the UI reflects the live graph-as-data and tunes runtime params via
|
||
sliders, C12); retaining un-sinked data; binding the playground to a single /
|
||
default harness.
|
||
**Why.** A program has no scene-at-rest to inspect; what persists is what it
|
||
records. Making sinks the one recording-and-observability mechanism keeps "what
|
||
can I see?" answerable by "what did I instrument?", and keeps the engine
|
||
UI-agnostic (C14). Live param tuning (runtime values, no topology change —
|
||
C12 / C19 Fork A) gives the interactive feel without a wiring DSL.
|
||
**Realization (cycle 0006).** Sinks-as-recording-mechanism is realized at the
|
||
substrate level: a recorded trace is exactly what a recording node pushed out of
|
||
the graph (no engine recording registry; the constructing World holds each
|
||
recording node's destination). The engine's single `observe: usize` affordance is
|
||
removed — `Harness::run` returns `()` and recording is a node-side concern, so one
|
||
run records *many* streams (one per recording node) instead of exactly one row.
|
||
Recorded streams are sparse and timestamped (a record per fired cycle, tagged
|
||
`ctx.now()`), matching a trace of timestamped events (C18). No new contract; the
|
||
`Harness` API change (observe removed, `run -> ()`) is recorded here.
|
||
|
||
---
|
||
|
||
## Open architectural threads not yet resolved
|
||
|
||
- **Playground & World UI surface** — the playground is core (C22), egui-native;
|
||
open is the concrete UI of the meta-views (walk-forward, sweep surfaces,
|
||
comparison) and the run/replay clock controls.
|
||
- **Parameter-space search strategies** (Bayesian/genetic) — pluggable policies
|
||
atop the atomic sim unit (C12), not yet designed.
|
||
- **`aura new` scaffolder, the experiment-builder API, and `Aura.toml`'s
|
||
static-context schema** — `aura new` scaffolds a Rust project *crate* (node /
|
||
strategy / experiment blueprints) against the engine (C16/C20); the
|
||
experiment-builder API surface (harness wiring, structural axes, sweep
|
||
combinators) and `Aura.toml`'s schema (data paths, instrument/pip metadata,
|
||
default broker & window, runs dir) are not yet designed.
|
||
- **`aura-std` contents** — the crate exists (doc-only); which universal blocks
|
||
land first follows the walking-skeleton's needs.
|
||
- **`strategies/` split** — a later split, *inside a project*, of top-level
|
||
strategies from reusable building blocks in `nodes/`; not a day-1 cut.
|
||
- **Sequencing** — the runnable single-harness *substrate* comes first (walking
|
||
skeleton: a closed harness that runs deterministically and records via a sink);
|
||
the World/meta layer (C21) and the playground trace-explorer (C22) are the
|
||
differentiating layers that follow — you orchestrate and visualize a thing that
|
||
must first run once.
|