Brummel
8158e97dc6
Completes the Source ingestion seam (plan Tasks 3-4): a real, lazily-streamed
data-server source proving the producer seam against the actual data source,
and closing the cycle-0011 deliberate eager-materialization gap.
- M1Field + a free decode(field, bar) — a pure projection of one M1 bar into
(epoch-ns ts, Scalar), reusing unix_ms_to_epoch_ns so ms->ns stays the one
C3 normalization point. Pure ⇒ hermetically unit-tested without an iterator.
- M1FieldSource: a streaming Source over a data-server M1 window. Holds one
Arc<[M1Parsed]> chunk (a zero-copy clone of the cache's chunk) + a cursor +
a pre-decoded head (so peek(&self) is cheap over a &mut next_chunk refill);
constructs each Scalar per-pull, refills when the chunk drains. Resident
footprint is O(one chunk), independent of window length.
- aura-engine moved [dev-dependencies] -> [dependencies] (M1FieldSource is a
library item implementing aura_engine::Source, not a test-only type).
- A gated streaming integration test (skips where local data is absent):
* end-to-end backtest through the seam to a RunReport, C1-deterministic
(two runs bit-identical);
* residency predicate — driving the source the way run() does (peek/next)
over the *unbounded* archive (~1.77M bars locally), peak resident_records
stays <= one data-server chunk (CHUNK_SIZE) while total >> CHUNK_SIZE:
O(one chunk), NOT O(window). A O(window) source would peak at `total`;
* two-field sharing (close + volume over one window, same ts axis).
The eager load_m1_window / close_stream path is kept (still valid for bounded
loads); the gap is closed by a streaming path now existing, not by deleting the
eager one. The residency test drives the full archive rather than the plan's
narrow 2006-08 window, which holds only ~21 bars locally — too few for the
> CHUNK_SIZE non-vacuity precondition; the unbounded archive is the stronger
"independent of window length" proof.
Verified: cargo build/test/clippy --workspace clean; the 3 gated tests ran
against real local data and passed; aura-ingest/aura-engine docs clean (two
pre-existing aura-registry doc-link warnings are from an earlier commit, not
this cycle).
closes #71
21 lines
788 B
TOML
21 lines
788 B
TOML
[package]
|
|
name = "aura-ingest"
|
|
edition.workspace = true
|
|
version.workspace = true
|
|
license.workspace = true
|
|
publish.workspace = true
|
|
|
|
[dependencies]
|
|
aura-core = { path = "../aura-core" }
|
|
# data-server: aura's first real data source. It enters the workspace at the
|
|
# ingestion edge (transitively chrono + regex + zip). Dependencies are admitted
|
|
# by the amended C16 per-case policy (INDEX.md), not a blanket zero-dep wall.
|
|
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }
|
|
# M1FieldSource impls aura_engine::Source in the lib, so the engine is a regular
|
|
# (non-dev) dependency.
|
|
aura-engine = { path = "../aura-engine" }
|
|
|
|
[dev-dependencies]
|
|
# the integration tests bootstrap a sample harness + fold a RunReport
|
|
aura-std = { path = "../aura-std" }
|