Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

engine+ingest: realize the Source trait as the run() ingestion seam, co-designed with a real DataServer-backed source (Ledger 0004/0011) #71

New Issue
Closed
opened 2026-06-15 09:26:04 +02:00 by Brummel · 0 comments
Brummel commented 2026-06-15 09:26:04 +02:00
Owner
Copy Link
Copy Source

Foundational cycle for World-II. Unifies the deferred Source trait (ledger 0004) with the DataServer ingestion edge (cycle 0011) in ONE integration, so the producer seam is designed against the real data source from the first line — surfacing boundary friction before the orchestration families (#66 / #68 / #69) build on it.

Why now

Harness::run(streams: Vec<Vec<(Timestamp, Scalar)>>) is the only engine type that encodes a materialized-Vec (eager) assumption. At realistic scale (20y, 3 streams, ~7.9e9 ticks) the eager layout is ~190 GB resident (24 B/tick) — non-residable. INDEX.md (cycle-0011 status) already declares today's eager materialization a "deliberate gap" whose Arc<[T]> zero-copy cross-sim sharing is "the target for the orchestration cycle that introduces [the axes]" — i.e. this milestone. The DataServer is purpose-built for it: lock-free Arc<[T]> 1024-chunks via SymbolChunkIter::next_chunk, windowed stream_*_windowed(from_ms, to_ms).

Deliverable

  1. trait Source { fn peek(&self) -> Option<Timestamp>; fn next(&mut self) -> Option<(Timestamp, Scalar)>; } — the ingestion-boundary producer the k-way merge drives (peek-head to pick the min across live sources, next to pop). Object-safe; SourceSpec stays the wiring metadata (kind + targets).
  2. Re-type Harness::run(&mut self, sources: Vec<Box<dyn Source>>). A VecSource wrapping today's Vec<(Timestamp, Scalar)> makes every existing test byte-identical — behaviour-preserving, no residency change.
  3. A DataServer-backed Source (in aura-ingest): windowed via stream_*_windowed, per-chunk AoS->SoA transpose, BORROWS the Arc<[M1Parsed]> chunk (no owned-Vec materialization), constructs each Scalar per-pull. One base-column Source per consumed field (a composite window -> N sources sharing the chunk's ts axis). next() -> Option (errors surface at source construction, matching stream_*_windowed's Option).
  4. An end-to-end real-window backtest: bootstrap a harness, feed it DataServer-backed Sources over a real [from, to] window, run to a RunReport.

Acceptance (friction-first)

  • Behaviour-preserving: the full suite is byte-identical with VecSource.
  • A real short-window DataServer backtest runs end-to-end through the seam to a RunReport.
  • Resident memory across the run is O(sum of node lookbacks), demonstrated on chunked real data — NOT O(window length) (the bounded-window proof on real chunks).
  • Window-bounds currency at the seam is Timestamp (epoch-ns), normalized to the DataServer's Unix-ms at the ingest edge (C3).

Surfaces these boundary frictions deliberately

AoS->SoA transpose granularity (per-chunk vs per-window); Arc<[T]> borrow lifetime across the merge and across disjoint sims (C1/C12); composite -> N base-column sources sharing one chunk; window-bounds type + ms<->epoch-ns normalization; error channel (Option at source open vs iteration); the lazy refill boundary (next_chunk when a chunk drains).

Contracts

C2 (bounded read-only windows), C3 (one merge at ingestion), C4 (SoA), C12 (Arc<[T]> cross-sim sharing). Closes the cycle-0011 deliberate-eager gap. Prerequisite for #66 / #68 / #69 (all inherit the Source shape).

Foundational cycle for World-II. Unifies the deferred `Source` trait (ledger 0004) with the DataServer ingestion edge (cycle 0011) in ONE integration, so the producer seam is designed against the real data source from the first line — surfacing boundary friction before the orchestration families (#66 / #68 / #69) build on it. ## Why now `Harness::run(streams: Vec<Vec<(Timestamp, Scalar)>>)` is the only engine type that encodes a materialized-Vec (eager) assumption. At realistic scale (20y, 3 streams, ~7.9e9 ticks) the eager layout is ~190 GB resident (24 B/tick) — non-residable. INDEX.md (cycle-0011 status) already declares today's eager materialization a "deliberate gap" whose `Arc<[T]>` zero-copy cross-sim sharing is "the target for the orchestration cycle that introduces [the axes]" — i.e. this milestone. The DataServer is purpose-built for it: lock-free `Arc<[T]>` 1024-chunks via `SymbolChunkIter::next_chunk`, windowed `stream_*_windowed(from_ms, to_ms)`. ## Deliverable 1. `trait Source { fn peek(&self) -> Option<Timestamp>; fn next(&mut self) -> Option<(Timestamp, Scalar)>; }` — the ingestion-boundary producer the k-way merge drives (peek-head to pick the min across live sources, next to pop). Object-safe; `SourceSpec` stays the wiring metadata (kind + targets). 2. Re-type `Harness::run(&mut self, sources: Vec<Box<dyn Source>>)`. A `VecSource` wrapping today's `Vec<(Timestamp, Scalar)>` makes every existing test byte-identical — behaviour-preserving, no residency change. 3. A DataServer-backed `Source` (in aura-ingest): windowed via `stream_*_windowed`, per-chunk AoS->SoA transpose, BORROWS the `Arc<[M1Parsed]>` chunk (no owned-Vec materialization), constructs each `Scalar` per-pull. One base-column `Source` per consumed field (a composite window -> N sources sharing the chunk's ts axis). `next() -> Option` (errors surface at source construction, matching `stream_*_windowed`'s `Option`). 4. An end-to-end real-window backtest: bootstrap a harness, feed it DataServer-backed `Source`s over a real `[from, to]` window, run to a `RunReport`. ## Acceptance (friction-first) - Behaviour-preserving: the full suite is byte-identical with `VecSource`. - A real short-window DataServer backtest runs end-to-end through the seam to a `RunReport`. - Resident memory across the run is O(sum of node lookbacks), demonstrated on chunked real data — NOT O(window length) (the bounded-window proof on real chunks). - Window-bounds currency at the seam is `Timestamp` (epoch-ns), normalized to the DataServer's Unix-ms at the ingest edge (C3). ## Surfaces these boundary frictions deliberately AoS->SoA transpose granularity (per-chunk vs per-window); `Arc<[T]>` borrow lifetime across the merge and across disjoint sims (C1/C12); composite -> N base-column sources sharing one chunk; window-bounds type + ms<->epoch-ns normalization; error channel (Option at source open vs iteration); the lazy refill boundary (`next_chunk` when a chunk drains). ## Contracts C2 (bounded read-only windows), C3 (one merge at ingestion), C4 (SoA), C12 (`Arc<[T]>` cross-sim sharing). Closes the cycle-0011 deliberate-eager gap. Prerequisite for #66 / #68 / #69 (all inherit the `Source` shape).
Brummel added this to the The World, part II — orchestration families milestone 2026-06-15 09:26:04 +02:00
Brummel added the feature label 2026-06-15 09:26:04 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone The World, part II — orchestration families
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#71
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?