Phase 4 of the Stratification milestone. aura-std held four C28 ladder
layers in one roster; this cuts them into layer-aligned, aura-core-only
node crates so the import direction is enforced by the crate graph:
- aura-std — engine nodes only (arithmetic/logic/rolling + sinks)
- aura-market — session, resample
- aura-strategy — bias, stops, sizer, cost-model machinery
- aura-backtest — sim_broker, position_management
- aura-vocabulary — the relocated closed std_vocabulary roster
Node modules move verbatim (byte-identical renames); consumers are
rewired by import path only. A new structural test
(aura-vocabulary/tests/c28_layering.rs) asserts each node crate's
[dependencies] stay within its C28-permitted inner set, catching the
acyclic-but-outward violation the compiler misses.
Behaviour byte-identical: full workspace suite green (1448 tests), no
golden edited, clippy -D warnings clean. C28 Status block updated.
closes#288
probe_window drained a source's entire close column just to learn its
first and last bar timestamp; both live callers (open_real_source and
campaign_window_ms on the campaign trunk) now resolve through
aura-ingest's new archive_extent(): list the symbol's SYM_YYYY_MM.m1
files, load only the boundary months via the existing loader seam, and
walk across gap months forward/backward — O(2 file loads) typical
instead of O(archive). Empty-window refusal semantics are unchanged
(pinned). Two characterization tests captured the resolved windows
against the OLD implementation over the synthetic per-test archive and
stay green unchanged after the swap.
Measured: the probe itself drops 366ms -> 13ms (~28x) on the full
2014-2026 GER40 archive; end-to-end command wall-clock moves only
~2-3% because the sim loop dominates — the fix removes an
O(archive-size) term, not the dominant cost.
closes#252
Run the Phase-1 session-breakout node vocabulary over REAL GER40 M1 bars
from the data-server archive — the first real-data exercise of the shipped
strategy nodes (Resample/Delay/Gt/Session/EqConst/And/Latch + SimBroker).
It is a pure composition of shipped aura-std nodes (no project-specific
signal code), so it lives in the engine repo's examples/ carveout (C9), not
a separate consumer crate. The breakout FlatGraph is reused VERBATIM from
aura-engine/tests/ger40_breakout.rs; only the sources change — four real
M1FieldSource (open/high/low/close, in the fixed C4 merge order) replace the
synthetic VecSources, with pip_size = 1.0 (GER40 is an index).
- examples/ger40_breakout_real.rs — runnable demo over a Sept-2024 window:
prints the RunReport metrics plus a per-session entry/exit trace.
- examples/shared/breakout_real.rs — the single 13-node wiring (the proven
11 + two trace taps), shared by the example and the test via #[path] so it
is never duplicated.
- tests/ger40_breakout_real.rs — gated determinism test (mirrors
real_bars.rs): skips where the archive is absent; where present, asserts
finite metrics, binary Latch exposure (held in {0.0, 1.0}), and
bit-identical C1 reruns of both the report and the recorded series.
chrono / chrono-tz added as aura-ingest dev-dependencies (the Session
timezone and the UTC window bounds), test-only — never on the normal path.
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes#71
Iteration 1 of the run-registry cycle (#33): lay the dependency + serde
foundation the registry's typed read-path needs, and amend the contract that
forbade it.
Contract change (load-bearing — C18/C16). C16's blanket "zero-external-
dependency by commitment" + "aura-ingest is the sole external-dependency
firewall" framing is struck and replaced by a considered, per-case dependency
policy: dependencies are admitted by deliberate review (what a crate pulls in
vs. what it buys), with particular scrutiny for anything entering the frozen
deploy artifact (C13); standard vetted crates (serde, rayon, ...) pass that
review and are used wherever they do the job, including in the bot; hand-rolling
what a vetted crate does is the anti-pattern. C16's engine/project split +
three-tier node reuse are unchanged. The five source comments that asserted the
struck framing (aura-engine/aura-ingest Cargo.toml + aura-ingest/report.rs docs)
are rewritten to match; no live source or ledger text still asserts it.
Per-case justification for serde (the policy now requires one): it gives the
run-report types a typed (de)serialization path — exactly what ranking/compare
over stored runs needs, and what the writer-only hand-rolled JSON (C14) could
never provide; closes the typed-read-path gap (#17). Its closure is tiny,
ubiquitous, heavily audited, and its output deterministic (C1-safe). serde_json
is test-only this iteration (dev-dependency); no production serde_json yet.
Changes: [workspace.dependencies] serde (derive) + serde_json; serde derives on
Timestamp (aura-core) and RunMetrics/RunManifest/RunReport (aura-engine), with
serde_json round-trip tests (window renders as a [from,to] integer array via the
transparent Timestamp newtype). The hand-rolled to_json writers are untouched
(still drive stdout). No aura-registry crate, no SweepPoint change — those are
iterations 2 and 3.
Verified: cargo test --workspace green (incl. the two new round-trip tests);
clippy --workspace --all-targets -D warnings clean; no surviving assertion of
the struck zero-dep framing in live source or the ledger.
refs #33
aura's first real data source (closes#7, Walking-skeleton milestone). A new
aura-ingest workspace crate transposes data-server's AoS M1Parsed records into
SoA base columns (C7), normalizes Unix-ms to canonical epoch-ns at the one
ingestion boundary (C3), and feeds the existing k-way merge a real close-price
stream so a backtest runs over real bars, deterministically (C1).
Surface:
- unix_ms_to_epoch_ns(time_ms) -> Timestamp (= ms * 1_000_000), the single C3
unit normalization.
- M1Columns: the OHLCV bar as a bundle of base columns (open/high/low/close/
spread: f64, volume: i64, ts: epoch-ns) — SoA, C7.
- transpose_m1(&[M1Parsed]) -> M1Columns: pure AoS->SoA (C1).
- M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>: the price input the
SMA-cross sample strategy consumes.
- load_m1_window(server, symbol, from_ms, to_ms): drains data-server's
chronological chunks, transposes once at the boundary.
Design (per spec 0011):
- Eager materialization, not a lazy/shared Source abstraction: C12's cross-sim
Arc<[T]> sharing has no consumer until the multi-sim orchestration cycle;
deferred, the transpose logic carries over.
- aura-ingest is the external-dependency firewall. data-server transitively
pulls chrono + regex + zip (+ their trees) into Cargo.lock; isolating it in
this one crate keeps aura-core/std/engine zero-external-dependency and the
frozen deploy artifact (C6 replays recorded streams, never re-ingests) clean.
cargo test --workspace now needs a populated cargo cache (one Gitea fetch,
done).
- Scope: boundary + tests only, no aura run CLI arg surface (M1 first; tick and
the CLI wiring are follow-ups).
Tests: 6 hermetic unit tests on hand-built M1Parsed (normalization, field-wise
transpose, purity, close_stream order, empty edges) + one gated integration
test (tests/real_bars.rs) that runs the cycle-0007 sample harness over real
AAPL.US 2006-08 close bars and asserts finite metrics + two-run bit-identical
JSON (C1); it skips cleanly where /mnt/tickdata is absent. On this machine it
ran the real path. Workspace gates green: test (86), clippy -D warnings, doc
-D warnings.
Minor: transient unused-import warnings in the new lib.rs across the additive
build steps resolved once load_m1_window consumed the imports; final -D warnings
gate clean.
claude
Brummel