bcd1072ca8515cb22d2a2a05ade71f120fff7608
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Brummel
|
1d9555c468 |
feat(aura-engine,aura-registry): producer-supplied window + registry lineage (0045 iter 1)
Iteration 1 of spec 0045 — the engine + registry core for storing orchestration families as linked records. aura-engine: add Source::bounds() — the inclusive (from, to) data extent a producer will stream, known without materialization (#71 firewall) — and a free window_of() that folds the union extent across a run's sources. This is the producer-supplied replacement for the prices.first()/.last() Vec scans at the CLI call sites (those migrate in iteration 2); a lazy producer now reports its window so stored lineage stays byte-identical whether a source is eager or streamed. aura-ingest: M1FieldSource stores its requested [from_ms, to_ms] window at open and reports it via bounds() (normalized to epoch-ns; None when open-ended — the archive-extent query is a deferred non-goal). aura-registry: new lineage module — FamilyKind / FamilyRunRecord (a RunReport stamped with family_id + kind + ordinal) / Family, plus Registry::append_family (assigns family_id = "{name}-{counter}" via a per-name generate-and-check counter, writes members to a sibling families.jsonl) and load_family_members; group_families re-derives the families (the round-trip), and the three *_member_reports extractors pull the per-kind member reports. The flat runs.jsonl store and its append/load/rank_by/optimize API are byte-for-byte unchanged (a test pins the two stores' disjointness). C9 preserved: aura-engine gains no registry dependency. New dep: serde derive on aura-registry (per-case policy, same basis as serde_json already is). Verification: cargo build --workspace; cargo test --workspace (228 green, incl. 3 new engine bounds/window_of, 1 data-gated ingest bounds run against real archive data, 5 new registry lineage round-trip/counter/disjointness tests); cargo clippy --workspace --all-targets -D warnings clean. The CLI surface (aura mc, family-aware aura runs, the window_of migration) is iteration 2. refs #70 |
||
|
Brummel
|
8158e97dc6 |
feat(aura-ingest): streaming M1FieldSource — lazy data-server window through the seam
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes #71
|
||
|
Brummel
|
eec129ee51 |
feat(aura-engine): Source ingestion seam — run() takes Vec<Box<dyn Source>>
The engine's ingestion input was the only type encoding an eager-dataflow
assumption: `Harness::run(Vec<Vec<(Timestamp, Scalar)>>)` — one fully
materialized stream per source. At realistic research scale (20y, 3 tick
streams, ~7.9e9 ticks) that layout is ~190 GB resident, non-residable. Yet
the merge loop never needs the whole stream: it only ever peeks the head
timestamp of each live source and pops one record. So the eager Vec is
gratuitous.
This lands the first half of the Source seam (plan Tasks 1-2):
- A `Source` trait (`peek(&self) -> Option<Timestamp>`,
`next(&mut) -> Option<(Timestamp, Scalar)>`), object-safe so the merge
holds `Vec<Box<dyn Source>>`. The peek/next split is the producer contract
the k-way merge drives.
- `VecSource`, an adapter over the old `Vec<(Timestamp, Scalar)>` — the
cycle-0011 eager shape, named. It IS the old behaviour.
- `Harness::run` re-typed to `Vec<Box<dyn Source>>`; the merge loop's index
arithmetic becomes peek/next. C4 tie-by-source-index is preserved (the
strictly-`<` replace + source-order scan is byte-for-byte the old
semantics). The forward+eval body is unchanged.
- All 53 call sites threaded VecSource-wrapped in the same change, so the
workspace compiles atomically and run output is byte-identical (C1).
Behaviour-preserving is the load-bearing guarantee: the full suite is green
and unchanged (the two-run C1 determinism pairs — real_bars r1/r2, blueprint
sweep-equality, harness h/h2 — stay byte-identical); aura-engine unit tests
129 -> 132 (+3 VecSource unit tests). No residency change yet: VecSource holds
the same Vec as before. The lazy data-server-backed streaming source and its
end-to-end residency proof are the second half (plan Tasks 3-4).
refs #71
|
||
|
Brummel
|
e304dbaae1 |
feat(aura-core): name input ports
PortSpec gains a non-load-bearing `name: String`, so an input port is named just as FieldSpec.name (output) and ParamSpec.name (param) already are — input ports were the lone unnamed member of the node signature. Identity stays positional by slot (C23); the name is render/debug only, never read by bootstrap or the run loop. PortSpec drops Copy (String is not Copy), exactly as ParamSpec already does. - Every aura-std node names its input slots: SMA/EMA "series", Sub/Add "lhs"/"rhs", Exposure "signal", SimBroker "exposure"/"price" (the slots become self-documenting), LinComb "term[i]" and Recorder "col[i]" generated in their build loops (mirroring LinComb's existing weights[i] param loop). - derive_signature carries a composite's Role.name into the derived input port (it was dropped before — the output side already carried FieldSpec.name), so the graph model is homogeneously named at both levels. - model_to_json (port_json + the composite-header inputs in scope_json) emits the name as a third tuple element: ["f64","any","exposure"]. The byte golden was re-captured (machine bytes) and its substring twins updated; the model is now fully named across inputs/outputs/params. - All 16 PortSpec construction sites threaded in one compile-gate change; test fixtures carry fixture names. C8 realization note added to the design ledger. Why name-only, no validation: the name is a pure debug symbol. Wire-by-name was rejected (it would be a C23 contract change). Bootstrap slot-wiring validation (which would close #21's same-kind swap footgun) is deferred to its own cycle — a name alone does not catch the swap; it makes the slots self-documenting and gives a future validation something to check against. Verified: cargo test --workspace 168 green; clippy --all-targets -D warnings clean; cargo build --workspace clean. Read-only render path (C9), no serde (C14), scalar kinds unchanged (C4). closes #50 refs #21 refs #51 |
||
|
Brummel
|
1b3909316e |
feat(aura): node signature lives in the blueprint; collapse Blueprint into Composite
Consolidate the node data structure so every node's signature (NodeSchema:
inputs/output/params) is declared once and exists in the blueprint pre-build,
and dissolve the special "root graph" type. Behaviour-preserving (C1).
Signature vs sizing
- NodeSchema is now the static signature only: InputSpec -> PortSpec{kind,firing},
with lookback removed. The signature is fully static per blueprint (input
kinds/firing, output fields, params); LinComb's variable arity is a builder arg,
not an injected param.
- The one param-dependent quantity, an input's buffer lookback (e.g. Sma's window =
its injected length), moves out of the signature to Node::lookbacks() -> Vec<usize>,
read only by bootstrap for sizing. Node::schema() is removed.
- LeafFactory -> PrimitiveBuilder, which carries the full NodeSchema. The built node
no longer re-declares it: closes the params-declared-twice drift (#36, the 8
per-node factory_params_match_built_node_schema lockstep tests are deleted — their
subject is now structurally impossible) and a value-empty recipe exposes its full
I/O interface pre-build (#43).
Root is just a bound composite
- struct Blueprint is deleted; its compile/bootstrap/param_space methods move onto
Composite. Role gains source: Option<ScalarKind> (None = open interior port,
Some = bound ingestion feed). A composite is runnable iff every root role is bound;
the "main graph" is no longer a category, only the fully-source-bound composite.
New error CompileError::UnboundRootRole for an open root role.
- BlueprintNode::signature() answers uniformly for both arms: Primitive returns the
builder's declared schema, Composite derives it from the interior (role kinds in,
OutField kinds out, aggregated params), pre-build, no build.
compile -> FlatGraph -> bootstrap
- compile validates structure pre-build via signature() (validate_wiring: range +
kind, returning the same variants as before, so an edge kind fault is now caught
before any build closure fires) and emits FlatGraph{nodes,signatures,sources,edges}.
- bootstrap consumes the FlatGraph: kinds/firing/output from the carried signatures,
buffer depth from node.lookbacks(). SourceSpec survives as the flat descriptor.
Renames: BlueprintNode::Leaf -> Primitive, LeafFactory -> PrimitiveBuilder.
Render (aura-cli/src/graph.rs) is migrated compile-only: it takes &Composite, maps
bound roles to the same source-entry shape, so both render goldens reproduce
byte-identical output (no re-capture needed). Render-fidelity tuning is the next cycle.
Verification (orchestrator-run, not agent-reported): cargo build --workspace green;
cargo test --workspace 150 passed / 0 failed; cargo clippy --workspace --all-targets
-D warnings clean. All pinned determinism/run-output tests pass with values unchanged;
no behavioural assertion was altered to go green. 5 new tests assert the signature is
pre-build and uniform, that compile rejects a kind mismatch without building (via a
panicking builder), UnboundRootRole, and lookbacks()/signature arity agreement.
Deferred to cycle-close audit (per plan): docs/design/INDEX.md and some aura-std
module docs still name the old Node::schema()/LeafFactory/BlueprintNode::Leaf/
Blueprint::param_space contracts; prose reconciliation is the architect's at audit.
closes #43 #36
|
||
|
Brummel
|
8bd5d0a3f9 |
fix(aura-ingest): express unbounded M1 window bounds via Option<i64>
load_m1_window took (from_ms: i64, to_ms: i64) and forwarded
Some(from_ms)/Some(to_ms) into stream_m1_windowed, throwing away the
underlying API's per-bound Option<i64> (None = unbounded). There was thus
no way to express an open upper bound, and the natural i64::MAX sentinel
overflowed chrono's from_timestamp_millis inside data-server's coarse
unix_ms_to_year_month file filter, panicking the whole ingest.
Thread Option<i64> through both bounds unchanged, so None = unbounded
matches the underlying contract and 'to the end of history' needs no
sentinel. Rejected the alternative of clamping a too-large bound to a
ceiling constant: a ceiling is not actually unbounded (it silently drops
any post-ceiling bar) and reintroduces a magic literal; restoring the
Option makes the open bound structurally expressible instead.
The committed RED test (
|
||
|
Brummel
|
2e381060fa |
test(aura-ingest): RED for unbounded M1 upper bound
load_m1_window cannot express an open upper bound: it hardcodes Some(from_ms)/Some(to_ms) into stream_m1_windowed, so callers reach for the i64::MAX sentinel, which panics upstream in chrono's from_timestamp_millis (data-server records.rs:28). This compile-RED pins the desired Option<i64> contract (None = unbounded) and asserts an unbounded read equals the finite sane bound bar-for-bar. refs #23 |
||
|
Brummel
|
a24729e97f |
feat(aura-ingest): data-server M1 ingestion at the one merge boundary
aura's first real data source (closes #7, Walking-skeleton milestone). A new aura-ingest workspace crate transposes data-server's AoS M1Parsed records into SoA base columns (C7), normalizes Unix-ms to canonical epoch-ns at the one ingestion boundary (C3), and feeds the existing k-way merge a real close-price stream so a backtest runs over real bars, deterministically (C1). Surface: - unix_ms_to_epoch_ns(time_ms) -> Timestamp (= ms * 1_000_000), the single C3 unit normalization. - M1Columns: the OHLCV bar as a bundle of base columns (open/high/low/close/ spread: f64, volume: i64, ts: epoch-ns) — SoA, C7. - transpose_m1(&[M1Parsed]) -> M1Columns: pure AoS->SoA (C1). - M1Columns::close_stream() -> Vec<(Timestamp, Scalar)>: the price input the SMA-cross sample strategy consumes. - load_m1_window(server, symbol, from_ms, to_ms): drains data-server's chronological chunks, transposes once at the boundary. Design (per spec 0011): - Eager materialization, not a lazy/shared Source abstraction: C12's cross-sim Arc<[T]> sharing has no consumer until the multi-sim orchestration cycle; deferred, the transpose logic carries over. - aura-ingest is the external-dependency firewall. data-server transitively pulls chrono + regex + zip (+ their trees) into Cargo.lock; isolating it in this one crate keeps aura-core/std/engine zero-external-dependency and the frozen deploy artifact (C6 replays recorded streams, never re-ingests) clean. cargo test --workspace now needs a populated cargo cache (one Gitea fetch, done). - Scope: boundary + tests only, no aura run CLI arg surface (M1 first; tick and the CLI wiring are follow-ups). Tests: 6 hermetic unit tests on hand-built M1Parsed (normalization, field-wise transpose, purity, close_stream order, empty edges) + one gated integration test (tests/real_bars.rs) that runs the cycle-0007 sample harness over real AAPL.US 2006-08 close bars and asserts finite metrics + two-run bit-identical JSON (C1); it skips cleanly where /mnt/tickdata is absent. On this machine it ran the real path. Workspace gates green: test (86), clippy -D warnings, doc -D warnings. Minor: transient unused-import warnings in the new lib.rs across the additive build steps resolved once load_m1_window consumed the imports; final -D warnings gate clean. |