aura's first real data source (closes#7, Walking-skeleton milestone). A new
aura-ingest workspace crate transposes data-server's AoS M1Parsed records into
SoA base columns (C7), normalizes Unix-ms to canonical epoch-ns at the one
ingestion boundary (C3), and feeds the existing k-way merge a real close-price
stream so a backtest runs over real bars, deterministically (C1).
Surface:
- unix_ms_to_epoch_ns(time_ms) -> Timestamp (= ms * 1_000_000), the single C3
unit normalization.
- M1Columns: the OHLCV bar as a bundle of base columns (open/high/low/close/
spread: f64, volume: i64, ts: epoch-ns) — SoA, C7.
- transpose_m1(&[M1Parsed]) -> M1Columns: pure AoS->SoA (C1).
- M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>: the price input the
SMA-cross sample strategy consumes.
- load_m1_window(server, symbol, from_ms, to_ms): drains data-server's
chronological chunks, transposes once at the boundary.
Design (per spec 0011):
- Eager materialization, not a lazy/shared Source abstraction: C12's cross-sim
Arc<[T]> sharing has no consumer until the multi-sim orchestration cycle;
deferred, the transpose logic carries over.
- aura-ingest is the external-dependency firewall. data-server transitively
pulls chrono + regex + zip (+ their trees) into Cargo.lock; isolating it in
this one crate keeps aura-core/std/engine zero-external-dependency and the
frozen deploy artifact (C6 replays recorded streams, never re-ingests) clean.
cargo test --workspace now needs a populated cargo cache (one Gitea fetch,
done).
- Scope: boundary + tests only, no aura run CLI arg surface (M1 first; tick and
the CLI wiring are follow-ups).
Tests: 6 hermetic unit tests on hand-built M1Parsed (normalization, field-wise
transpose, purity, close_stream order, empty edges) + one gated integration
test (tests/real_bars.rs) that runs the cycle-0007 sample harness over real
AAPL.US 2006-08 close bars and asserts finite metrics + two-run bit-identical
JSON (C1); it skips cleanly where /mnt/tickdata is absent. On this machine it
ran the real path. Workspace gates green: test (86), clippy -D warnings, doc
-D warnings.
Minor: transient unused-import warnings in the new lib.rs across the additive
build steps resolved once load_m1_window consumed the imports; final -D warnings
gate clean.
The Walking-skeleton milestone's closing seam: a real `aura run` binary that
bootstraps a built-in sample harness, runs it deterministically, and prints the
cycle-0009 metrics+manifest report as canonical JSON to stdout — the headline
C14 "run a sim, emit structured metrics" move, end-to-end, from a binary for the
first time.
Two deliverables:
- aura-std::Recorder — a reusable recording sink node (the glossary *sink* role,
C8/C22): a pure consumer (output: vec![]) over kinds.len() input columns,
holding an mpsc::Sender<(Timestamp, Vec<Scalar>)> as its out-of-graph
destination. mpsc keeps the engine's purity invariant (C7): no Rc/RefCell.
Supports all four base scalar kinds; returns None until every column is warm.
Promotes the shape the #[cfg(test)] fixtures already proved into a shipped,
reusable block (the fixtures stay as historical snapshots; de-dup is a later
tidy).
- aura-cli `run` subcommand — synthetic_prices / sample_harness / run_sample /
main. The sample harness (synthetic source → SMA(2)/SMA(4) → Sub → Exposure →
SimBroker → two Recorder sinks) is authored in plain Rust over the raw
Harness::bootstrap(nodes, sources, edges) API (C17/C20) — the open
experiment-builder DSL thread is deliberately not committed this cycle. main
hand-parses one subcommand: `run` prints run_sample().to_json() (exit 0),
anything else prints a one-line usage to stderr (exit 2). aura-cli gains
aura-std + aura-core path deps; the workspace stays zero-(external-)dependency.
The built-in synthetic stream (7 ticks, rises then reverses) yields a non-trivial
demo trace: equity [0,0,0,0,-0.08,-0.17,-0.13] → total_pips -0.13, max_drawdown
0.17, exposure_sign_flips 1. The run_sample unit test pins the integer flip count
exactly and the two f64 metrics within 1e-9 (the real run's float dust is ~7e-15,
confirming the tolerance); determinism is pinned exactly (two runs → identical
JSON). tests/cli_run.rs drives the built binary for the observable exit/stdout
contract.
manifest.commit is filled from option_env!("AURA_COMMIT") (defaults to
"unknown"); real HEAD capture via build.rs is deferred. Non-goals untouched:
experiment-builder DSL, aura new, Aura.toml schema, the data-server source (#7).
One deviation from the plan's verbatim code: a scoped
#[allow(clippy::type_complexity)] on sample_harness (its (Harness, Receiver,
Receiver) return tuple trips the lint under -D warnings) — consistent with the
identical allow on the build_two_sink_harness fixture in aura-engine. Verified
myself: cargo test --workspace (61 green, 0 red), clippy --all-targets
-D warnings (clean), cargo doc -D warnings (clean), and both smoke runs.
closes#8
Cycle 0003: aura-engine's first real content — a Sim that runs a wired DAG of
nodes deterministically, cycle by cycle. First point in the project where
authored nodes execute against data, not just under a hand-fed Ctx.
- `Sim` (aura-engine) — a bootstrapped, frozen root graph: a flat Vec<NodeBox>
(each node owns its input columns, the cycle-0002 shape) + an index edge
table, topologically ordered (Kahn). `bootstrap` sizes every input column from
its node's schema, kind-checks each edge and source target, and rejects
directed cycles — C7's "type check paid once at wiring" generalized to the
whole topology (`BootstrapError::{KindMismatch, BadIndex, Cycle}`).
- `run` — the deterministic loop: per record, forward the source value into its
target slots, evaluate nodes in topo order, capture the observed node's output
at eval time, and forward each `Some` output into its consumers' input columns
(a `None` forwards nothing — the structural seed of sample-and-hold). The loop
destructures `&mut self` into disjoint field borrows and allocates nothing on
the per-cycle path. This is the flat, monomorphized sharpening of RustAst's
reference-counted, interior-mutable observer push graph (C1/C7).
- `Edge` / `Target` (aura-engine) — producer->consumer and source->consumer wiring.
- `Sub` (aura-std) — a 2-input f64-difference node, so the loop is proven on a
real fan-out + join DAG (source -> {SMA(2), SMA(4)} -> Sub), with a determinism
assertion (C1: a second identical run is bit-identical).
Engine library depends only on aura-core; a test-only dev-dependency on aura-std
lets the integration test wire real Sma/Sub nodes. Sim carries a hand-written,
node-opaque `Debug` impl (Box<dyn Node> is not Debug; the impl prints
nodes.len() + the index/topology fields, needed by the bootstrap-rejection tests'
`unwrap_err`).
Deliberately deferred (recorded as decisions): freshness-gated recompute /
sample-and-hold (C5 -> cycle 0004, with the second source that makes it testable);
the explicit monotonic cycle_id counter (its first reader is freshness, 0004);
the Source trait + data-server ingestion + k-way merge (C3/C11); the builder API
(C19); the real sink + run registry (C18/C22).
Gates green: cargo build/test (26: 18 aura-core + 3 aura-std + 5 aura-engine)/
clippy -D warnings clean; surface-purity grep (no dyn-Any / Rc / RefCell) clean.
refs walking-skeleton
Refine the structure per interview: aura is the reusable engine, research projects are separate external repos depending on it (C16). Add aura-std (universal standard-node tier) to the workspace; remove nodes/ from the engine (project concept). Record authoring-surface = Claude Code + skills pipeline, no embedded coding-LLM (C17), and project management = one-repo-one-project + Aura-native run registry (C18). Add CLAUDE.md invariants 9-10, code_roots -> [crates], and docs/project-layout.md documenting the day-in-the-life and project repo layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cargo workspace aura-core -> aura-engine -> aura-cli (bin `aura`) plus nodes/ for hot-reloadable cdylib node crates. Crate bodies are intentionally API-free; types arrive from the first spec. Adds the skills profile (.claude/dev-cycle-profile.yml), the project CLAUDE.md with the eight domain invariants, and the design-ledger skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brummel