docs/specs and docs/plans were retired (prior commit); the source/test
comments that cited them ("spec 0050 §4.1", "spec §Testing N", "per spec",
"the spec's ...") now point at nothing. Strip every such pointer while
preserving the technical substance, the design-ledger contract refs
(C1/C11/C20/C34/C12.1/...), and the Gitea issue refs (#41).
Comment/doc edits only across 20 files — no logic change; full workspace
suite green, clippy clean.
The residency probe was an inherent method on the concrete M1FieldSource, so a Box<dyn Source> consumer could not read it without downcasting (the adjacent friction noted in #95). Hoist it onto the Source trait as resident_records(&self) -> Option<usize> with a default of None, mirroring bounds()'s Option-for-unknown idiom: None = "does not report" (the eager VecSource, which has no streaming ring), distinct from Some(0) = "reports, holds none" (e.g. exhausted).
M1FieldSource overrides it to return Some(chunk len). Callers updated (streaming_seam, the two deep-dive fieldtest bins). Tests: a hermetic VecSource default-is-None check, and a &dyn Source probe in the gated residency test proving M1FieldSource residency is readable without a downcast.
refs #95
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes#95
data-server does no timezone math (delphi_to_unix_ms is pure arithmetic), so the
archive carries whatever the raw Pepperstone files contain, labelled as UTC. The
Session node's UTC->Berlin DST conversion is unit-tested, but the assumption that
the raw timestamps are themselves UTC was unverified — a broker-local archive
would shift the whole session and the error would jump across the DST boundary.
Verified empirically via the DAX cash-open volume spike (GER40): it lands at
Berlin 09:00 in BOTH seasons, at 07:00 UTC (summer, CEST, UTC+2) and 08:00 UTC
(winter, CET, UTC+1) — a clean +1h DST shift, the genuine-UTC signature. This
gated test pins it so a future archive/convention drift cannot silently corrupt
every session strategy's bar alignment.
The shipped Composite blueprint (ger40_breakout_blueprint) is now driven by the
World orchestration families on real GER40, with NO re-authoring — the #94
friction is gone:
- examples/ger40_breakout_sweep.rs — sweep the {entry_bar, exit_bar} grid built
directly from param_space() (best entry=3,exit=4 -> 214.0 over 2024).
- examples/ger40_breakout_walkforward.rs — non-degenerate walk_forward: a real
non-empty space optimized per window, chosen_params populated everywhere (#97).
- examples/ger40_breakout_compare.rs — one blueprint across two structural axes:
instrument (GER40 vs FRA40) and bar period (15m vs 30m — different strategies,
C34), no re-authoring.
- tests/ger40_breakout_world.rs — gated: sweep returns the ranked 9-point grid
over the two named params; walk_forward yields non-empty windows each with a
non-empty chosen_params (executable #97 proof).
No blueprint reshape needed — the factory returns a fresh (Composite, Taps) per
call, so the World families re-instantiate per grid point / window (the SMA-cross
pattern); baked recorders do not block multi-bootstrap.
closes#94, closes#96, closes#97
Author the shipped session-breakout as a reusable Composite blueprint via
GraphBuilder — the World-consumable form (param_space + bootstrap) — alongside
the hand-wired FlatGraph it reproduces. Applies the spec-0051 decisions:
- bar_period_minutes is a construction arg binding BOTH Resample.period_minutes
and Session's period, locked equal, so a sweep cannot desync them (#96);
- delay.lag is bound out (a structural constant, not a tuning knob — C34);
- the two EqConst targets (named entry_bar / exit_bar) are the ONLY params, so
param_space() == {entry_bar.target, exit_bar.target} — the genuine tuning
knobs, which makes walk_forward non-degenerate (#97).
The ger40_breakout_real example now bootstraps from the blueprint (still prints
-45.0 for 2024-09). New gated test ger40_breakout_blueprint.rs proves: (a) the
param_space is exactly the two targets (no lag, no period); (b) C23 — the
blueprint reproduces the FlatGraph bit-identically over real GER40 2024-09;
(c) determinism.
First increment of the milestone; the sweep / walk_forward / compare World-family
demos follow.
refs #94, #96, #97
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Hand-wires all seven new aura-std nodes (Resample, Delay, Gt, Session, EqConst
x2, And, Latch) + SimBroker into a raw-index FlatGraph over one synthetic
Frankfurt session, pinning the whole vocabulary composed end-to-end:
- held = [0,0,1,1,0] over the 5 bar emissions: flat, latched bar3-close (the
strict breakout landing on session bar 3) through bar4, exit at bar5-close;
- equity = [0,0,0,3,8] pips (SimBroker lagged-exposure integration, pip=1.0);
- a no-entry control (bar3 close == bar2 high) pins the strict-> semantic;
- two disjoint runs byte-identical (C1).
Both spec traps exercised (bar6 rollover closes bar5; bars 1-2 warm Delay[1]).
Phase-1 C9 deliverable; build-step 8 (capstone) of milestone 'Strategy node
vocabulary I'.
closes#91
Maps ctx.now() (epoch-ns UTC) to a Frankfurt session bar index: emits
bars_since_open:i64 = (local wall-clock minutes past the 09:00 open) /
period_minutes, in tz-aware DST-correct local time. So local 09:45 reads 3 in
both CEST (UTC+2) and CET (UTC+1) — the close-instant convention (spec 0050
§4.1). Trigger is an f64 Any input (value ignored, wired from close15) so the
node fires once per completed bar. Open/tz/period are baked structural config,
not scalar params (a timezone is not a scalar, C11). Admits chrono/chrono-tz to
aura-std — vetted DST math, never hand-rolled (per-case dep policy). Last node;
build-step 7 of milestone 'Strategy node vocabulary I'.
closes#90
Aggregates a fine 4-field OHLC stream into period_minutes buckets (open=first,
high=max, low=min, close=last), emitting a completed bar ONLY on bucket rollover
(ctx.now() crossing into the next bucket) — C2: a bar is actionable only once
complete, partials are never emitted and the last partial bar is dropped (no EOF
flush). Four Barrier(0) f64 inputs, a 4-field f64 record output (the first
multi-field-output node in aura-std, mirroring the Ohlcv engine fixture). The bar
carries no timestamp; the engine stamps the close-instant. Build-step 6 of
milestone 'Strategy node vocabulary I'.
closes#89
Level-sensitive SR latch: held -> 1.0 on set, -> 0.0 on reset (reset-dominant),
sample-and-held across quiet cycles; None while both legs are cold. Emits f64
directly (1.0 long / 0.0 flat) so it feeds SimBroker's exposure slot with no
cast — the resolution of the bool->f64 seam (no separate Exposure node in the
strategy DAG). State persisted in the struct across cycles, like SimBroker.
Build-step 5 of milestone 'Strategy node vocabulary I'.
closes#88
Emits the value its series input carried 'lag' fired cycles ago, held in a
node-owned ring (lookbacks()=[1], the past in node state not the input column).
Warm-up is skip-emit (None for the first 'lag' cycles — never a fabricated
sentinel). Output is a pure function of pre-cycle state, the precondition for a
future engine to close a feedback loop (the RTL register C5 names). prevHigh15 =
Delay[1] on high15. Build-step 4 of milestone 'Strategy node vocabulary I'.
closes#87
Stateless bool x bool -> bool, out = (a && b). Computes entry = breakout &&
isBar3 in the session-breakout strategy. The bool-input twin of Gt; seed of the
logic family (Or/Not are separate node types, the operator is topology, never a
swept param). Build-step 3 of milestone 'Strategy node vocabulary I'.
closes#86
Stateless f64 x f64 -> bool, out = (a > b), STRICT: a == b emits false (a close
exactly equal to the previous bar's high is not a breakout). Computes
breakout = close15 > prevHigh15 in the session-breakout strategy. First
bool-emitting f64 comparator; the operator is topology (relational siblings are
separate types, never a swept param). Build-step 2 of milestone 'Strategy node
vocabulary I'.
closes#85
Stateless gate, out = (value == target): the i64->bool comparator the
session-breakout strategy needs to turn Session's bars_since_open into
isBar3/isBar5Close (two instances bound to 3 and 5). The operator is topology
(a concrete node type), only target is a knob (C11/C12). Build-step 1 of
milestone 'Strategy node vocabulary I'.
closes#84
The cycle-0047 typed-Scalar params migration (86746e3) changed each
RunManifest param value on the C18 flat runs store from a bare JSON
number (2.0) to an externally-tagged Scalar ({"F64":2.0}), but did not
migrate the persisted runs.jsonl. Registry::load could no longer
deserialize any pre-0047 line, so `aura runs list` / `runs rank` errored
out on a researcher's existing run history (bug F5, surfaced by the
"The World, part II" milestone fieldtest).
Fix: a localized read-side back-compat mirror (src/compat.rs). load()
now reads each line via RunReportRead, whose ScalarRead accepts BOTH the
tagged form AND a bare JSON number (coerced to Scalar::f64), then
converts into the canonical RunReport. The forward write path (append /
RunReport::to_json) is untouched — it still emits the tagged form. This
is a one-directional read widening, not a format change.
The secondary clause of #83 (parse error exits 0) was triaged as
not-a-bug: runs list/rank already exit 2 via load_runs_or_exit
(main.rs:742); the "(exit 0)" in the fieldtest transcript was a pipe
measurement artifact ($? read head's exit code, not the binary's).
RED test (committed 20511d5) now green; full workspace suite + clippy
green.
closes#83
RED test pinning the real defect of bug F5 (surfaced by the milestone
"The World, part II" fieldtest). Registry::load cannot deserialize a
runs.jsonl line written before the cycle-0047 typed-Scalar params
migration, where each param value is a bare JSON float (e.g.
["sma_cross.fast",2.0]) rather than the tagged {"F64":2.0}. The test
feeds a minimal autonomous legacy line and asserts it loads — the bare
float read back via the documented f64 coercion — instead of erroring at
the first param.
Fails RED at the diagnosed cause (Parse "expected value" on the bare
float). GREEN side (back-compat read, no change to the typed-Scalar
forward format) follows.
refs #83
Add the typed/named convenience view on `WalkForwardResult`, closing the
symmetry cycle 0048 left open: it shares the `(space, tag-free Vec<Cell>)`
idiom with `SweepFamily` but had no named-params accessor, so a consumer
hand-wrote `zip_params(&result.space, &result.windows[i].run.chosen_params)`.
`named_params(window)` pairs each param-space name with that window's chosen
value in slot order, delegating to `zip_params` — the direct mirror of
`SweepFamily::named_params`. A derived view, no new stored state.
closes#76
Add `RandomBinder`, the by-name sibling to `SweepBinder`, so a random
sweep (`RandomSpace`, C12.1) can be built by name against `param_space()`
instead of by positional `Vec<ParamRange>` slot order. By-name resolution
makes a same-kind transposition (e.g. swapping the I64 ranges for
`fast.length` and `slow.length`) structurally impossible — the failure
class `RandomSpace::new`'s positional validation passes silently.
Direct structural mirror of `SweepBinder`: `Composite::range` opens the
binder, `.range(name, ParamRange)` accumulates, `.sweep(count, seed, run)`
resolves the named ranges via the shared `resolve_into` (new `resolve_ranges`
caller) and runs the disjoint sweep. New `BindError::EmptyRange` mirrors
`EmptyAxis`; `ParamRange::is_empty` homes the non-empty invariant the named
layer pre-checks so `RandomSpace::new` cannot fail.
closes#79
The SweepError type-level rustdoc still read "A structural fault constructing a
`GridSpace`" though the enum now also gates `RandomSpace::new` (the three
random-only variants are each documented individually, but the type summary was
stale). Broaden it to name both spaces and group the grid vs. random faults.
Doc-only, behaviour-preserving — the same low-grade doc-debt class the 0049
audit fixed inline for the module doc. Surfaced by the cycle-0049 fieldtest.
refs #52
Architect drift review over 3de00e2..HEAD (the 0049 spec/plan/feat plus the
two intervening refactors that had not been audited: aura-ingest M1 transpose
6390093, aura-std SMA perf 67c1f51). Verdict: feature clean — C1 determinism
preserved (RandomSpace points seed-determined before any run; grid path
behaviour-preserving via trait-forwards-to-inherent), the #52/#71 source-seam
firewall honoured in code (sweep stays &S: Space + Fn(&[Cell]) -> RunReport, no
Source type enters the sweep layer; SplitMix64 a code-path-disjoint instance),
and both intervening refactors behaviour-preserving (pinned by
incremental_matches_full_resum_within_tolerance and
chunked_accumulation_equals_single_transpose).
Two low-grade doc-debt items found and fixed inline (doc-only, behaviour-
preserving):
- sweep.rs module doc named only the grid axis; refreshed to name RandomSpace
+ the Space trait the module now also owns.
- C12 ledger had no realization note for axis-1 (param-sweep); added one
recording the grid (0028) + random (0049) landing and the Space-trait
unification.
Regression gate: the project configures no regression scripts, so the test
suite + lint are the gate. Independently verified: cargo test --workspace green
(incl. the 17 new engine tests + 6 public-API E2E), cargo clippy --workspace
--all-targets -- -D warnings clean.
Drift-clean, not a milestone close (no milestone fieldtest run this cycle).
Ship the random half of the C12.1 param-sweep axis (grid landed in 0028).
A new `Space` trait (`points`/`param_specs`) generalizes `sweep`/`sweep_with_threads`
from `&GridSpace` to `&S: Space`; `GridSpace`'s trait impl forwards to its existing
inherent methods, so the grid path is behaviour-preserving (C1, every pre-existing
grid sweep test stays green). `RandomSpace` is the second `Space` producer: N seeded
uniform points over per-slot typed `ParamRange { lo, hi }` ranges (I64 inclusive
[lo,hi], F64 half-open [lo,hi)), validated in `new` against the param-space — a
non-numeric slot, a range-kind mismatch, and an empty range are the three new typed
`SweepError` variants, all caught before any run. Sampling reuses the existing
bit-stable `SplitMix64` (promoted to `pub(crate)`) as a code-path-disjoint instance
from the data-edge seed RNG — the #52/#71 World-II source-seam firewall: they share
only the `u64` type, never a path. The sweep-layer signature stays param-only
(`Fn(&[Cell]) -> RunReport`); no `Source`/stream type enters it.
Deviation from the spec's verbatim sampler, accepted as a correctness hardening:
the I64 draw guards the full-width span ([i64::MIN, i64::MAX] computes a span of
2^64 that wraps a u64 to 0) and uses `lo.wrapping_add(draw as i64)` instead of a
plain `+`. This avoids both the `% 0` divide-by-zero at the full domain and the
signed-overflow panic the literal form hits on wide ranges in debug builds, while
preserving uniform-over-[lo,hi] draws and determinism (an extra RED test,
random_points_full_range_i64_does_not_panic, pins it). Modulo bias for spans not
dividing 2^64 remains the spec's documented, accepted simplification (param search
needs no cryptographic uniformity).
Includes a public-API E2E suite (tests/random_sweep_e2e.rs) exercising the feature
exactly as a downstream researcher would — reproducibility at the report boundary,
seed-sensitivity, in-range draws, the typed NonNumericRange gate, a well-formed
empty (count==0) family, and Grid/Random interchangeability through `Space`.
Verified: cargo test --workspace green (incl. 17 new engine tests + 6 E2E);
cargo clippy --workspace --all-targets -- -D warnings clean.
closes#52
Sma::eval re-summed the whole window every tick — O(length)/tick, multiplied
across millions of bars and every sweep point. Replace with the industry-standard
incremental running sum (ta-lib shape): the node owns the window ring and keeps a
running sum, adding the new sample and subtracting the evicted one each cycle, so
the input column drops to depth 1 (lookbacks() = [1], like Ema). The running sum
carries Kahan/Neumaier compensation (the fix pandas' rolling mean adopted) so it
does not drift over long runs; Ema needs none — its recurrence is contractive.
Determinism (C1) holds: same input -> same run, reproducibly. The float output
differs from the full re-sum in the last ULPs (a new, deterministic baseline); the
suite needed only one arity-assumption update (Sma lookback 3 -> 1), no equity
golden changed.
closes#39
load_m1_window collected every bar into an intermediate Vec<M1Parsed> (a full
AoS materialization, un-presized so it grew by reallocation) before transpose_m1
copied it again into the SoA columns — ~2x peak load memory plus realloc churn.
Transpose each chunk straight into the columns instead, via a new shared
M1Columns::extend_from_bars (reserves per chunk for amortized growth). One fewer
full copy, halved peak. Behaviour-preserving: the resulting M1Columns is
byte-identical and transpose_m1's / load_m1_window's signatures are unchanged.
Pinned by a new parity test that needs no live DataServer.
closes#78
Make the walk-forward param plane C7-clean and symmetric with the sweep param plane. WalkForwardResult gains `space: Vec<ParamSpec>` (the kinds, once, mirroring SweepFamily.space) and WindowRun.chosen_params changes Vec<Scalar> -> Vec<Cell> (the tag-free coordinate the inner sweep already produces). param_stability now resolves its numeric coercion once per slot against the schema (result.space) into a Vec<fn(Cell)->f64>, then runs a type-blind reduction loop — replacing the old per-window-value scalar_as_f64 with its per-value unreachable!. scalar_as_f64 is deleted.
This overturns cycle 0047's "Fork A" (chosen_params stays Vec<Scalar>), which was justified by Scalar's new serializability. That defense was moot: WalkForwardResult and WindowRun carry no serde derives — they are never serialized; only the embedded oos_report (a RunReport, typed via the 0047 lift) is. So chosen_params is a pure in-memory substrate for param_stability, and Vec<Scalar> made it C7-impure (carrying the kind N×M times at the value, where the kind is per-slot constant). The substantive case — kind at the schema, SweepFamily symmetry, per-value unreachable! eliminated — wins, the same argument-form that retired the {kind,cell} Scalar struct in 0047.
The seam ([A]): walk_forward / walk_forward_with_threads take `space: Vec<ParamSpec>` by value, a sibling of the run-window closure; the Fn(WindowBounds)->WindowRun+Sync bound is unchanged and run_indexed is untouched (space is never captured by the Sync closure, moved onto the result at the end). A debug_assert in walk_forward_with_threads guards the arity coupling (every window's chosen_params.len() == space.len(), guaranteed by the same blueprint). walkforward_family computes the space once before the roll and passes the tag-free winner (chosen_params: best.params) directly, dropping the from_cell+zip reconstruction.
[D]: a Bool param slot coerces to 0/1 (mean = fraction "on"), keeping the coercer total over the knob kinds; a Timestamp slot is structurally impossible (C20) so the schema pass carries one unreachable! (once per slot, never per value). param_stability keeps an `if result.windows.is_empty() { return Vec::new(); }` guard to preserve its documented empty-on-no-windows contract. WalkForwardResult and SweepFamily share the (space + tag-free Cell points + zip_params) idiom but stay distinct types — different axis (Sweep enumerates input coordinates; WF rolls time and chosen_params is the optimizer's output), and WF additionally carries bounds + oos_equity + stitched_oos_equity.
Behaviour-preserving (C1): the three migrated walk-forward tests stay green with identical MetricStats (param_stability_reduces_chosen_params_per_slot still pins mean 2.5 / p50 2.5 / mean 1.5); the i64->f64 and f64 coercions are value-identical to the deleted scalar_as_f64. The one genuinely new behaviour (Bool 0/1) is unexercised by built-in grids and pinned by the new param_stability_reduces_bool_slot_to_fraction_true (mean 0.75, 3-of-4 true).
Gates: build --workspace --all-targets, test --workspace, clippy --workspace --all-targets -D warnings, doc --workspace --no-deps — all clean.
closes#75
Scalar was `struct { kind: ScalarKind, cell: Cell }` — "a Cell wearing a kind hat." The recorded reason for that shape was migration ease, which is not a design rationale (CLAUDE.md: rationale != effort), so the struct had no substantive defense. Redefine it as the native tagged union it conceptually is:
enum Scalar { I64(i64), F64(f64), Bool(bool), Timestamp(Timestamp) }
This is substantively better on four axes: kind/bits skew becomes unrepresentable (illegal states gone); accessors panic loudly on the wrong variant instead of a release-mode silent bit reinterpret; PartialEq derives the documented IEEE-754 / cross-kind value semantics (the hand-roll, needed only because the struct inherited Cell's bitwise compare, is gone); and serde is a plain derive emitting the externally-tagged wire form ({"I64":10}/{"F64":2.5}) — the private ScalarRepr shadow enum that motivated this was never needed. It is also C7-honest: erased-on-the-hot-path (Cell) and self-describing-at-the-edge (Scalar) are two disjoint types bridged by explicit conversion.
The whole public API is preserved (Scalar's fields were private), so call sites do not churn: the enum change is contained to scalar.rs, where cell() now encodes and from_cell() decodes per kind. Cell and ScalarKind are untouched.
With Scalar serializable, lift RunManifest.params from Vec<(String, f64)> to Vec<(String, Scalar)>: the param's kind (an i64 length vs an f64 scale) now survives into the C18 record (runs.jsonl) and the CLI JSON instead of collapsing to f64. scalar_as_param_f64 is deleted; sim_optimal_manifest passes typed params through. This is a deliberate wire-shape change — params now render as tagged scalars; the JSON-asserting tests are updated to the new shape on purpose.
Hand-authored manifest params across the CLI's single-run/mc/macd sites use honest kinds (lengths -> i64, scales -> f64) so they match the sweep path (which already derives correct kinds via zip_params); their in-binary JSON assertions are re-tagged accordingly.
Walk-forward fork resolves with no code change: WindowRun.chosen_params stays Vec<Scalar> (the serializable record carrier), reduced to f64 only at the param_stability statistic boundary (scalar_as_f64 retained). Glossary 'cell' entry updated to describe Scalar as the disjoint tagged union, not 'a cell plus its kind tag'.
Gates: build --all-targets, test --workspace (incl. new scalar_serde_round_trips), clippy -D warnings, doc --no-deps — all clean.
Cell becomes the carrier of the construction path; Scalar narrows to the author/render boundaries. The validated/enumerated param point carries no redundant kind (it lives once, in the declared param-space); at the author edge the kind is a checksum — two independent sources, the typed value vs. the slot — so the self-describing Scalar stays there. The name->slot binding is dynamic (C23), so the check is necessarily a runtime one and the value must self-describe for it.
Base/frontend split (the AnyColumn push/push_cell pattern one level up): compile_with_cells / bootstrap_with_cells are the kind-check-free base; compile_with_params / bootstrap_with_params are the frontend that adds only the per-value kind checksum, strips to cells (new Scalar::cell(), the partner of Scalar::from_cell), and delegates. lower_items loses its per-primitive kind-check; PrimitiveBuilder::build and the std node builders (sma/ema/exposure/lincomb) read cells.
Boundary: construction -> Cell (PrimitiveBuilder::build, lower_items, GridSpace.axes/points, SweepPoint.params, the sweep closure). Author edges stay Scalar (GridSpace::new, bind, compile_with_params/bootstrap_with_params). walkforward chosen_params stays a self-describing Scalar report record (Option A — WalkForwardResult carries no space); the cell winner is reconstructed once at the WindowRun site via from_cell.
injective-check moved ahead of the arity-check in the frontend to preserve the pre-split error order (DuplicateParamPath before ParamArity). The lossy i64->f64 param projection (scalar_as_f64 / scalar_as_param_f64) is deliberately untouched — a separate follow-up.
Behaviour-preserving (C1): build --all-targets / test / clippy -D warnings / doc all clean across the workspace.
Motivation
----------
The C7 realization note (cd3d1ca / 049f22a) deferred this step: with
Scalar split into { kind, cell }, the tag-free Cell could become the
hot-path carrier, but eval and the edges still flowed the fatter
self-describing Scalar. This lands the swap.
Change
------
- Node::eval now returns Option<&[Cell]>; every node out-buffer is
[Cell; N]. The inter-node forward copies the row into a Vec<Cell> and
pushes each field via a new branch-free, infallible AnyColumn::push_cell
— the edge from_field->slot kind match is verified once at bootstrap
(C7), so the runtime push needs no per-value kind check. It only removes
a Result the forward path already .expect()ed.
- Scalar stays on the self-describing dynamic boundaries: the param plane
(build / bind / compile_with_params / sweep points / RunManifest),
AnyColumn::get (the type-erased read for sinks / serde), and source
ingestion (Source::next, the heterogeneous C3 merge). The fallible
AnyColumn::push is retained for ingestion + the kind-guard tests.
Trade
-----
The per-value runtime kind check on node OUTPUT is removed: a node that
builds a wrong-kind cell pushes raw bits with no panic. This is the same
authoring-bug class C8 already leaves to a debug_assert (output width);
node-output-kind correctness is the node's declared FieldSpec contract,
caught by each node's own eval test. Cross-node kind agreement still rests
on the bootstrap kind-check (bootstrap_rejects_*_kind_mismatch, green).
Tests
-----
Behaviour-preserving (C1): 285 green, 0 red. The only test-expectation
edits are carrier re-spellings (Scalar::f64(x) -> Cell::from_f64(x)) of
identical values; test column writes and out-of-graph channel rows stay
Scalar (the spec's three-role rule). Adds an AnyColumn::push_cell
round-trip test plus a cross-kind end-to-end forward test (a mixed
f64/i64 record whose bit patterns diverge across kinds — pins the i64
forward path the f64-only tests cannot catch). Ledger C7/C8 notes updated.
Gates: cargo build / test / clippy --all-targets -D warnings / doc
--workspace all clean.
closes#74
Motivation
----------
`Scalar` was a tagged enum (I64/F64/Bool/Ts), so every scalar value
physically carried its own kind tag. But the kind is already known from
the schema/port/column the value flows through (C7: the type is a
property of the column, not of the value — the hot path is already
columnar `Column<T>`, and `AnyColumn::get` *reconstructs* the tag from
the column on the way out). The per-value tag was therefore redundant
with the kind the surrounding context already holds.
That redundancy had three costs:
* It baked an implicit `match` (a branch) into every function that read
a Scalar payload — even where the caller statically knew the type.
The tag could never be exploited away.
* Size: a tagged enum is tag + payload = 16 bytes (f64/i64 alignment),
twice the 8 bytes the value needs. A `Column<Scalar>` would be double
the memory and half the cache utilisation.
* It is the shared root of several downstream papercuts we keep hitting
— the lossy f64 manifest field, the `unreachable!` panic on a
non-numeric param, the serde-tag question — all symptoms of "the type
is baked into the value".
Change
------
Introduce `Cell`: a type-erased 64-bit word (`struct Cell(u64)`) that is
not readable without external type context. It is constructed per base
type (`from_i64/from_f64/from_bool/from_ts`) and read only by naming the
type at the call site (`i64()/f64()/bool()/ts()`) — each a branch-free
bit-cast. The hot path resolves the kind once at the boundary (from the
schema) and then reads natively, with no per-value branch. `Cell` knows
nothing of `Scalar` or `ScalarKind`; the dependency is strictly one-way,
and it lives in its own `cell.rs` (more is planned on top of it).
`Scalar` becomes `struct { kind: ScalarKind, cell: Cell }` — the
self-describing form for the dynamic boundaries (builder binding,
serialization, rendering), built on top of `Cell`. Its `as_*` accessors
now `debug_assert` the kind and return the native value (free in
release); calling the wrong accessor is a caller bug, not a checked
`Option`. The variant constructors `Scalar::I64(..)` become associated
fns `Scalar::i64(..)`.
`PartialEq` is hand-written (not derived) to preserve the former enum's
value semantics: kinds must match, then native payloads compare, so f64
keeps IEEE-754 behaviour (`NaN != NaN`, `+0.0 == -0.0`) and a kind
mismatch is never equal even when the raw words coincide. A fixture
(`scalar_eq_is_value_not_bitwise`) pins exactly the cases where bit- and
value-equality diverge, so it can't silently regress. `Cell`'s own
`Eq`/`Hash` stay bitwise — correct for a raw word.
The change is behaviour-preserving: Scalar's observable behaviour is
identical to the pre-Cell enum (the value-equality fixture proves it);
only the internal representation changed. The ~440 call sites across the
workspace are a mechanical constructor rename plus ~12 destructuring
sites (match-arms / `let`-patterns) rewritten to `kind()` + `as_*`.
Verified: cargo build --workspace --all-targets, cargo clippy --workspace
--all-targets -- -D warnings, cargo test --workspace — all green.
Thread a derived named view of a sweep point so consumers stop hand-zipping
param_space() names onto the bare &[Scalar]. The free function
zip_params(space, point) -> Vec<(String, Scalar)> in aura-core is the one
shared projection; GridSpace retains the ParamSpec list it already receives in
new() (it was validated then discarded); SweepFamily carries it (stamped once
in sweep_with_threads) and exposes named_params(i). The three CLI hand-zip
sites collapse to one zip_params call each.
The run-closure signature `Fn(&[Scalar]) -> RunReport` is byte-unchanged — the
named view is a derived projection (C23: slot is identity, name is derived),
not a closure-currency change — so #52's RandomSpace plugs into the same
sweep() execution layer with zero signature reconciliation (#71 firewall held).
sim_optimal_manifest now takes typed Vec<(String, Scalar)> and does the lossy
f64 collapse internally (one place; the manifest's f64-precursor field owns its
own lossiness — the typed-manifest upgrade stays the deferred typed-param-space
item, a non-goal here). All eight call sites migrated: three sweep closures
pass zip_params, five hand-listing callers (run_sample, run_sample_real,
mc_family, run_macd, run_sample_seeded) pass Scalar::F64 literals.
Behaviour-preserving: the collapse output is identical to the old per-site zip,
so aura sweep / walkforward / run output is byte-identical; SweepFamily.space
threaded into the aura-registry optimize test literal.
Verified by the orchestrator: cargo test --workspace green (4 new tests:
zip_params x2, sweep_family_carries_param_space, family_named_params_round_trips),
cargo clippy --workspace --all-targets -D warnings clean. (rust-analyzer
emitted stale mid-edit diagnostics; the real cargo build/test is clean.)
refs #57
Behaviour-preserving normalisation of the lineage record. The fused
`family_id: String` ("{name}-{counter}") is split into its two factors —
`family: String` + `run: usize` — and the user-facing handle becomes a derived
`family_id()` method ("{family}-{run}"), never stored or parsed.
Why: the fused field forced the counter logic to generate-and-check candidate
strings (to stay robust to '-' inside a name) rather than read it; with `run` a
plain int, `next_run` is a clean numeric max+1 over the field. The split is more
normalised (one fact per field) and the only property the fused token bought — a
single paste-able CLI handle — is recovered by deriving it. Storage and all
user-visible output stay byte-identical: append_family still returns
"{name}-{run}", Family.id is still "{name}-{run}", and the CLI prints the same
"family_id" lines (the json! key + the returned String, both unchanged).
Internal-only: FamilyRunRecord and group_families (now keyed on the (family,
run) tuple, first-seen order preserved) change; lib.rs re-exports, the
extractors, and all of aura-cli are untouched (everything downstream goes
through Family.id + append_family's returned String). The on-disk
families.jsonl key changes ({"family","run"} vs {"family_id"}) — a gitignored
runtime artifact with no committed fixture, and tests round-trip within a temp
dir, so no migration is owed. Doc reconciliation: the lineage/lib module headers
and the C18 ledger realization note now describe the split shape.
Verification: a workflow ran the refactor in an implementer agent, then three
adversarial lenses (behaviour-preservation, new-logic correctness,
completeness/doc-lag) tried to refute it — the first two found nothing, the
third flagged the four doc-lag sites now fixed here. Self-run gates:
cargo test --workspace green (registry 11, incl. a strengthened round-trip test
pinning the split fields + derived handle), cargo clippy --workspace
--all-targets -D warnings clean, cargo doc clean.
Iteration 2 of spec 0045 — the CLI surface for orchestration families.
window_of migration: every manifest-window scan (run_sample, sweep_family,
walkforward_family, sweep_over, run_oos, run_macd) now reads the window from
Source::bounds() via window_of(&sources) instead of prices.first()/.last().
Behaviour-preserving (the walk-forward roller boundaries are bar-aligned, so
the producer-supplied window equals the old first/last) — pinned by the
unchanged run_sample (1,7) and walk-forward determinism tests.
aura mc: new built-in Monte-Carlo family on the CLI (monte_carlo over a
built-in seed set + the sample harness re-seeded per draw), persisted via
append_family and rendered as one member line per seed (carrying the
family_id) plus an mc_aggregate line. McAggregate is not Serialize, so the
aggregate line is built from its three MetricStats blocks.
family-aware runs: aura sweep / walkforward / mc now persist to the family
store via append_family with an optional --name (per-kind default), printing
the assigned family_id per member. aura runs families lists family headers
(id, kind, member count); aura runs family <id> [rank <metric>] lists/ranks
one family's members as a unit. An unknown family id is an empty family
(exit 0); an unknown metric is a usage error (exit 2).
Two pre-existing cli_run.rs E2E tests were adapted to the new family
contract (a required consequence of sweep/walkforward switching from the
flat store to the family store — a plan gap caught at implement time): the
sweep-output test now asserts the family-wrapper, and the runs list/rank
flow became the family list / per-family-rank flow (which also exercises the
per-name counter sweep-0/sweep-1 end-to-end). A new E2E test drives aura mc
through the binary.
Known cosmetic detail: the family-wrapped sweep/mc/walkforward stdout nests
the report via serde_json::json! (to_value -> alphabetical keys, leading
"broker"), while runs family / runs list print via RunReport::to_json()
(declaration order, leading "commit"). Both valid JSON; key order is not
load-bearing and the stored families.jsonl uses to_string (declaration
order), so the round-trip is unaffected.
Verification: cargo test --workspace green; cargo clippy --workspace
--all-targets -D warnings clean.
refs #70
Iteration 1 of spec 0045 — the engine + registry core for storing
orchestration families as linked records.
aura-engine: add Source::bounds() — the inclusive (from, to) data extent a
producer will stream, known without materialization (#71 firewall) — and a
free window_of() that folds the union extent across a run's sources. This is
the producer-supplied replacement for the prices.first()/.last() Vec scans
at the CLI call sites (those migrate in iteration 2); a lazy producer now
reports its window so stored lineage stays byte-identical whether a source
is eager or streamed.
aura-ingest: M1FieldSource stores its requested [from_ms, to_ms] window at
open and reports it via bounds() (normalized to epoch-ns; None when
open-ended — the archive-extent query is a deferred non-goal).
aura-registry: new lineage module — FamilyKind / FamilyRunRecord (a
RunReport stamped with family_id + kind + ordinal) / Family, plus
Registry::append_family (assigns family_id = "{name}-{counter}" via a
per-name generate-and-check counter, writes members to a sibling
families.jsonl) and load_family_members; group_families re-derives the
families (the round-trip), and the three *_member_reports extractors pull
the per-kind member reports. The flat runs.jsonl store and its
append/load/rank_by/optimize API are byte-for-byte unchanged (a test pins
the two stores' disjointness). C9 preserved: aura-engine gains no registry
dependency. New dep: serde derive on aura-registry (per-case policy, same
basis as serde_json already is).
Verification: cargo build --workspace; cargo test --workspace (228 green,
incl. 3 new engine bounds/window_of, 1 data-gated ingest bounds run against
real archive data, 5 new registry lineage round-trip/counter/disjointness
tests); cargo clippy --workspace --all-targets -D warnings clean. The CLI
surface (aura mc, family-aware aura runs, the window_of migration) is
iteration 2.
refs #70
The third C12 orchestration axis: walk-forward varies the data window. A
WindowRoller is a pure iterator of WindowBounds { is, oos } — bounds only, never
tick data (#71 firewall). walk_forward runs a caller closure per split disjointly
over the shared run_indexed core (C1), and stitches the OOS pip-equity segments
into one continuous curve (each segment offset by the running sum of prior
segments' finals; an empty segment contributes 0.0, mirroring summarize). C2
no-look-ahead is structural: the roller only ever emits oos.0 == is.1 + 1, pinned
by a zero-tick bounds test. The in-sample optimize (axis 2) is closure-supplied,
not called by the engine: aura-engine has no aura-registry dependency (C9), and
C12 forbids baking search policy into the primitive — the CLI bridges both crates.
Param-stability is on-demand, not stored (the R2 decision, recorded with
provenance on #69): WalkForwardResult stores only the raw per-window outcomes +
the stitched curve, and param_stability(&result) -> Vec<MetricStats> is a public
reduction over the retained per-window chosen params. A stored summary would be
recomputable from the windows (redundant) and would force one statistic canonical
when "stability" admits several; this mirrors SweepFamily (raw points + on-demand
optimize), not McFamily (stored aggregate). MetricStats::from_values is extracted
from McAggregate::from_draws (behaviour-preserving — the 7 mc tests stay green)
and gains serde so the CLI summary renders it and #70 lineage can persist it.
`aura walkforward` runs a built-in rolling walk-forward over a synthetic windowed
source: per window it sweeps the built-in grid in-sample, optimizes by total_pips,
runs the chosen params out-of-sample, persists each OOS RunReport (C18), and prints
a stitched summary line.
Two plan deviations, both compiler-forced and consistent with siblings: added
#[derive(Debug)] to WindowRoller (the RED tests' unwrap_err needs Ok: Debug; the
sibling types already derive Debug); removed a now-redundant test-module
SyntheticSpec import after Step 2 hoisted it to the top-level use.
Verification: cargo test --workspace green (aura-engine 152 incl. walkforward 9 +
mc 9, aura-cli 14); clippy --workspace --all-targets -D warnings exit 0; cargo doc
--workspace --no-deps clean.
refs #69
Architect drift review over b920362..HEAD (the 0043 monte-carlo cycle). Cycle is
sound: C12 axis 4 faithfully realized (monte_carlo varies the seed, the 0042
seeded source genuinely perturbs each run, so distinct_seeds_produce_distinct_draws
is a real property); C1 honoured (one run_indexed core, output sorted by job index
= input order independent of completion, 1-vs-8-thread determinism pinned,
parallelism across draws never within); the run_indexed extraction is a genuine
behaviour-preserving refactor (C11/C23 — the three pre-existing sweep tests stay
green, and both family executors are thin adapters over the single core; the
points.len() -> n.max(1) clamp is equivalent since GridSpace rejects empty axes);
the #71 eager-agnostic firewall is respected (API takes seeds + a per-draw
closure, no materialized stream Vec crosses the boundary).
One medium doc-lag found and fixed here: aura-engine/src/lib.rs module header
listed Monte-Carlo (and, already stale, the grid sweep) under "Still to come";
both orchestration axes ship now, so the header records the grid axis (sweep /
SweepFamily) and the seed axis (monte_carlo / McFamily) as delivered, with
walk-forward, random param-sweep, registry lineage, and the C10 position-event /
broker layer remaining future.
Regression gate (test suite — no dedicated regression scripts in project facts):
cargo test --workspace green (incl. 7 new mc::tests + the sweep guard tests);
clippy --workspace --all-targets -D warnings clean; cargo doc -p aura-engine
clean (the new intra-doc links resolve).
closes#68
`monte_carlo(base_point, seeds, run_one) -> McFamily`, the Monte-Carlo
orchestration family: a fixed base point run over a seed set, each seed a
disjoint C1 realization. C12 axis 4 — Monte-Carlo IS a sweep over seeds, so it
reuses the disjoint-parallel executor rather than forking a run loop.
The reuse is realized by extracting the index-parallel core out of
`sweep_with_threads` into a `pub(crate) run_indexed<T>` (generic over the
per-job result, over a 0..n job range); `sweep_with_threads` becomes a thin
adapter over it. Behaviour-preserving (C1/C11/C23): the three pre-existing sweep
tests stay green — they ARE the proof the refactor preserved behaviour. A small
incidental win: the adapter consumes `points` via zip, dropping the old
per-point clone.
mc.rs adds McDraw{seed,report} / McFamily{draws,aggregate} (draws in seed-INPUT
order, independent of thread completion) / McAggregate (mean + p5/p25/p50/p75/p95
of all three run metrics — the V1 decision: covers every metric, not a "chosen"
one) / MetricStats, a private type-7 linear-interpolation `quantile`, and
`McAggregate::from_draws` (a pure post-run reduction recomputable from the
retained raw draws). Eager-agnostic firewall (#71): the API takes seeds + a
per-draw closure `Fn(u64,&[Scalar])->RunReport`, never a materialized stream Vec;
seed->Source construction stays a closure-body concern. An empty seed set is a
debug-asserted precondition, preserving the no-Result `-> McFamily` signature.
7 RED-first mc::tests pin: one draw per seed in input order; determinism across
1 vs 8 threads (C1); draw == an independent seeded run; distinct seeds perturb
the family; aggregate == from_draws(draws); type-7 quantile/mean math on a known
fixture; quantile endpoints + singleton. Workspace green, clippy -D warnings
clean, cargo doc clean.
refs #68
Architect drift review over 682e459..HEAD (the seed-as-input cycle plus the
prior unaudited `aura run --real` work). Cycle is sound: Fork B respected (seed
forks at the data-generation edge into the source and the manifest; the engine
loop sees only a Source — C12/C1/C3 intact), the contract names Source never Vec
(precondition for #68/#52 delivered as specced), and the four seed-free callers
thread 0 in lockstep.
One medium doc-lag item found and fixed here: aura-engine/src/lib.rs module-doc
listed the seed axis under "still to come"; seed-as-input shipped this cycle and
RunManifest.seed is now live, so the headline doc now records SyntheticSpec as a
delivered seeded Source producer (params + orchestration families remain future).
One low carry-on note (not drift): run_sample_seeded exists only under
#[cfg(test)] — intended this cycle (no --seed CLI flag per spec non-goals); the
only non-test seed value remains 0. A scaling marker for #68, not debt.
Regression gate (test suite — no dedicated regression scripts in project facts):
cargo test --workspace green; clippy --workspace --all-targets -D warnings clean;
cargo doc -p aura-engine clean (the new [`SyntheticSpec`] intra-doc link resolves).
A seeded source whose stream is fully determined by a u64 seed, making
RunManifest.seed a live captured input instead of the dead 0 it was. This is
C12's reconciliation of stochastic runs with C1: a run is bit-identical for a
fixed seed (the seed is a captured input, not hidden nondeterminism), and a
different seed gives a different-but-reproducible run. Precondition for the
Monte-Carlo family (#68) and random param-sweep (#52).
Engine (aura-engine):
- SplitMix64: a ~10-line private, dependency-free PRNG. Chosen over a rand-family
crate for guaranteed bit-stability across toolchains/crate versions — the whole
value of seed-as-input is reproducibility, and a crate whose algorithm is not
version-stable would silently break a recorded seed on a dep bump. No new direct
dep.
- SyntheticSpec + source(seed) -> impl Source: the Fn(u64) -> impl Source contract
(Fork B — seed at the data-generation edge, outside the engine graph). Returns a
Source, never a Vec, so the MC family can re-seed N times without materializing
N streams. First cut collects to a VecSource internally; the signature does not
name Vec, so a lazy generator is a drop-in replacement. Re-exported from lib.rs
beside VecSource.
CLI (aura-cli):
- sim_optimal_manifest gains a seed: u64 param recorded into the manifest (the
single Fork B capture site). All four existing seed-free callers (run_sample,
run_sample_real, the sweep grid-report, run_macd) pass 0 unchanged — their
determinism tests stay green.
- A test-only run_sample_seeded + SeededTrace exercise a live seed and expose the
drained sink trace, so the three e2e tests assert bit-identity at the trace
level (strictly stronger than the folded 3-field metrics).
Tests (RED-first): 2 engine producer tests (same-seed identical / different-seed
differs) + 3 cli e2e tests (bit-identical trace, different-seed metrics differ,
seed recorded in manifest). Full workspace green; clippy -D warnings clean.
Errata vs plan 0042 (two compile-forced deviations, neither alters the spec
contract):
- source returns `impl Source + use<>`: under edition 2024 a bare `impl Source`
captures the &self lifetime, which blocks boxing as Box<dyn Source + 'static>
for run() (E0597). The source owns its data, so use<> (captures nothing) is
correct.
- SyntheticSpec is imported inside `#[cfg(test)] mod tests` in main.rs: it is
used only by the seeded test vehicle, so a top-level import is unused in the
non-test build under clippy -D warnings.
closes#66
The first real-data backtest from the CLI: `aura run --real <SYMBOL>
[--from <ms>] [--to <ms>]` runs the existing built-in sample signal-quality
harness over real M1 *close* bars instead of synthetic prices, printing the
same RunReport JSON as `aura run`. Bars are streamed lazily through
aura_ingest::M1FieldSource (a Box<dyn Source>) — dogfooding the #71 Source
seam, O(one chunk) resident, never eager. aura-ingest + data-server enter
aura-cli's deps (data-server git line mirrored from aura-ingest).
M1 only this cycle, deliberately. The TickSource is deferred: raw ticks are
the wrong input for the bar-semantic SMA sample (an SMA over ticks is
microstructure noise until a resampler node exists, C2), and tick's real
driver is the realistic broker's bid/ask execution (C10) — it should arrive
with that consumer, not speculatively here. Tick data is also local to only
EURUSD/XAUUSD/GER40, whereas M1 covers the test symbol AAPL.US.
Shape: run_sample_real builds sample_harness(), guards has_symbol /
M1FieldSource::open (unknown symbol or empty window -> 'aura: no local data'
+ exit 2, never a panic), and reuses sim_optimal_manifest + the run_sample
fold. parse_real_args is a pure, unit-tested grammar (mandatory symbol, then
--from/--to in any order; bad/missing/duplicate flag -> Err). Data path is
data_server::DEFAULT_DATA_PATH (no --data-path flag this cycle).
Known simplification: the manifest window is derived by draining a *separate*
single-pass probe source for first/last ts, so an unbounded window streams
the archive twice (once to bound, once to run). Acceptable and O(1)-resident
for now; a future Harness::run could surface the first/last cycle ts to drop
the probe pass.
Verified: cargo build/test/clippy --workspace all green; the gated
run_sample_real_streams_real_close_bars_deterministically RAN (not skipped)
over AAPL.US 2006-08; manual CLI smoke of the success + two error paths.
Architect drift review over 8b330e3..HEAD (the Source-seam cycle #71 plus the
interim #67 optimize work and refactors). No semantic or contract drift: the
seam faithfully preserves C3 (ms→epoch-ns at the one ingestion boundary), C4
(tie-by-source-index, byte-for-byte), and C7 (field-per-source SoA); the full
suite is byte-identical green and the gated streaming tests pass on real data.
Regression gate: `cargo test --workspace` green (the project declares no
dedicated regression script; the suite is the gate).
Tidied stale prose the cycle left behind (no behaviour change):
- docs/design/INDEX.md (C12): replaced the cycle-0011 "deliberate eager gap"
status note with a cycle-0041 realization note — the producer `Source` seam
+ streaming `M1FieldSource` now deliver per-source O(one-chunk) streaming;
precisely scopes what remains open (cross-*sim* Arc<[T]> window sharing,
still unbuilt until the orchestration families #66/#68/#69 consume it).
- aura-core/src/lib.rs: dropped the now-shipped "Source trait + data-server
ingestion" from the module doc's "still to come" list (aura-engine's twin
line was fixed in the cycle; aura-core's was missed).
- aura-ingest/src/lib.rs: module header now documents both coexisting paths
(eager load_m1_window/M1Columns vs lazy streaming M1FieldSource).
- aura-registry/src/lib.rs: rank_by/optimize doc comments linked to the
private `metric_cmp` (a public→private intra-doc-link warning from the #67
commit); demoted to plain code spans, so `cargo doc --workspace` is now
warning-clean.
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes#71
The engine's ingestion input was the only type encoding an eager-dataflow
assumption: `Harness::run(Vec<Vec<(Timestamp, Scalar)>>)` — one fully
materialized stream per source. At realistic research scale (20y, 3 tick
streams, ~7.9e9 ticks) that layout is ~190 GB resident, non-residable. Yet
the merge loop never needs the whole stream: it only ever peeks the head
timestamp of each live source and pops one record. So the eager Vec is
gratuitous.
This lands the first half of the Source seam (plan Tasks 1-2):
- A `Source` trait (`peek(&self) -> Option<Timestamp>`,
`next(&mut) -> Option<(Timestamp, Scalar)>`), object-safe so the merge
holds `Vec<Box<dyn Source>>`. The peek/next split is the producer contract
the k-way merge drives.
- `VecSource`, an adapter over the old `Vec<(Timestamp, Scalar)>` — the
cycle-0011 eager shape, named. It IS the old behaviour.
- `Harness::run` re-typed to `Vec<Box<dyn Source>>`; the merge loop's index
arithmetic becomes peek/next. C4 tie-by-source-index is preserved (the
strictly-`<` replace + source-order scan is byte-for-byte the old
semantics). The forward+eval body is unchanged.
- All 53 call sites threaded VecSource-wrapped in the same change, so the
workspace compiles atomically and run output is byte-identical (C1).
Behaviour-preserving is the load-bearing guarantee: the full suite is green
and unchanged (the two-run C1 determinism pairs — real_bars r1/r2, blueprint
sweep-equality, harness h/h2 — stay byte-identical); aura-engine unit tests
129 -> 132 (+3 VecSource unit tests). No residency change yet: VecSource holds
the same Vec as before. The lazy data-server-backed streaming source and its
end-to-end residency proof are the second half (plan Tasks 3-4).
refs #71
Ships C12 orchestration axis 2 (optimize): pick the single best point of
a sweep family by a named RunMetrics field. The GREEN half of the
RED-first iteration whose executable-spec landed in the previous commit.
optimize(&SweepFamily, metric) -> Result<SweepPoint, RegistryError>
returns the whole winning SweepPoint — its params coordinate AND its
RunReport — because the caller needs the params that won, not just the
metric. "Best" is fixed per metric (total_pips higher-is-better;
max_drawdown / exposure_sign_flips lower-is-better); ties resolve to the
earliest enumeration-order point (the family is in odometer order, and
only a strictly-better later point displaces the incumbent).
Single source of truth for "best", as #67 required ("reuse rank_by; do
not fork"): rank_by's per-metric direction is extracted into a private
`metric_cmp` helper that resolves the boundary metric *name* to a closed
`enum Metric` once and returns one comparator closure. Both rank_by
(sort by it) and optimize (argmax via `reduce`, strictly-better-displaces)
now call it, so the per-metric direction lives in exactly one match.
rank_by is behaviour-preserving — its existing tests pass unchanged.
The sweep executor is untouched.
No caller-supplied `direction`: one definition of best per metric,
consistent with the shipped rank_by invariant (a caller direction would
admit incoherent asks like the worst max_drawdown). An unknown metric is
UnknownMetric. A SweepFamily is non-empty by construction (EmptyAxis is
rejected upstream), so the argmax always yields a winner.
CLI surface: the "pick the best" view is already served by
`aura runs rank <metric>`, whose first line is the optimum over the
persisted registry; optimize() is the new *programmatic* axis-2 entry,
for World programs that hold a live SweepFamily and need the winning
params back in-process — which the registry/RunReport path cannot give.
No redundant CLI command added.
closes#67
The RED half of a RED-first iteration for the optimize/argmax
orchestration axis (C12 axis 2). Pins the headline behaviour before
the feature exists: optimize(&SweepFamily, metric) returns the single
winning SweepPoint — its params coordinate AND its RunReport — chosen
by the metric's fixed per-metric sense of best (reusing rank_by's
direction semantics), with ties resolving to the earliest
enumeration-order point.
No caller-supplied direction: that would contradict the shipped
rank_by invariant that "best" is fixed by each metric's meaning, and
would admit incoherent asks (e.g. the worst max_drawdown). One
definition of best per metric.
RED for the right reason — optimize is absent (E0425), the sole
error; cargo build --workspace stays green (test-only module). GREEN
lands in the following commit, which closes#67.
Three RunReport builders hand-rolled the same RunManifest with identical
commit/seed/broker fields (only params/window varied), drifting
independently. Extract sim_optimal_manifest(params, window) — one source for
the broker label, so #22's per-asset pip work has a single edit point.
kind_str and collect_distinct_composites no longer exist in aura-cli (it
consumes model_to_json directly), so the "mirrors aura-cli's ..." notes
pointed at nothing. Keep the substantive rationale (Debug is PascalCase).
resolve (single Scalar per slot) and resolve_axes (a Vec<Scalar> axis per
slot) were the same two-phase named-binding resolution against param_space(),
duplicated almost verbatim — the live instance of the "two raise-sites for
one invariant" hazard (cf. #45/#53). A change to binding precedence or error
ordering had to touch both in lockstep.
Collapse the shared skeleton into resolve_into<T>, parameterized by two
closures: claim_ok (phase-1 claim-time gate, axis-only EmptyAxis, before the
duplicate check) and kind_ok (phase-2 per-slot kind gate). The error total
order now lives in exactly one place. Behaviour-preserving: same error for
same input, all 231 tests green (the resolve/bind precedence tests pin it).
FieldSpec.name was &'static str, forcing derive_signature to Box::leak
every re-exported composite OutField name. Because compile_with_params
reruns derive_signature per sweep point, an N-point sweep over an
M-output-field composite leaked N×M boundary names unbounded — a scaling
wall the moment the World/sweep layer compiles families per point.
Pull FieldSpec.name to String, mirroring the already-owned PortSpec.name
and ParamSpec.name: a composite re-exports interior fields under runtime
boundary names, so ownership is the correct model, not a workaround.
FieldSpec loses Copy; the only fallout is one borrow at the render site.
leak_name/Box::leak deleted.
Behaviour-preserving: signature contents unchanged, all 231 tests green.
"Compilat" (German "Kompilat") was a coined noun for the product of the
bootstrap compilation — neither English nor a natural fit. The runtime
artifact already has a code identifier for exactly this thing: the
`FlatGraph` struct (harness.rs). Replace the coinage with that identifier:
- prose mentions -> "flat graph" (mirrors the type, reads plainly)
- definitional anchors -> `FlatGraph` (C11, C23, the running-graph line)
- `render_compilat` -> `render_flat_graph` (historical render symbol;
keeps the `render_blueprint` / `render_flat_graph`
source-vs-product pairing)
- `intra-compilat` -> `intra-graph`
- `from_compilat` (test) -> `from_flat`
"compilation" / "re-compilation" / "bootstrap-as-compilation" (the process,
ordinary English) are deliberately left untouched. Behaviour-preserving:
only comments, design ledger, specs/plans, one test-local variable and its
assert messages change. Full workspace test suite green; clippy clean.
Brummel