docs: narrow streaming residency claim from whole-process RSS to the source ring
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes #95
This commit is contained in:
@@ -181,7 +181,10 @@ fn decode(field: M1Field, bar: &M1Parsed) -> (Timestamp, Scalar) {
|
||||
/// A streaming [`Source`](aura_engine::Source) over a data-server M1 window. Holds
|
||||
/// at most one `Arc` chunk (a zero-copy clone of the cache's chunk) and a cursor;
|
||||
/// constructs each `Scalar` per-pull; refills via `next_chunk()` when the chunk
|
||||
/// drains. Resident footprint is O(one chunk), independent of window length.
|
||||
/// drains. The **source**'s resident footprint is O(one chunk), independent of
|
||||
/// window length (see [`resident_records`](Self::resident_records)) — a per-source
|
||||
/// bound, not whole-process RSS: the data-server cache below retains the loaded
|
||||
/// window's parsed chunks for the pass (the replay-many sharing model, #95).
|
||||
pub struct M1FieldSource {
|
||||
iter: SymbolChunkIter<M1Parsed>,
|
||||
chunk: Option<Arc<[M1Parsed]>>,
|
||||
@@ -237,7 +240,9 @@ impl M1FieldSource {
|
||||
/// Records resident in this source right now: the current chunk's length (or
|
||||
/// 0 at exhaustion). The residency probe — bounded by one chunk length by
|
||||
/// construction (the source holds at most one `Arc` chunk and no accumulating
|
||||
/// field), so it can never grow with window length.
|
||||
/// field), so it can never grow with window length. This probes the **source
|
||||
/// ring** only; the data-server `FileCache` below retains the loaded window's
|
||||
/// parsed chunks for the pass and is not counted here (#95).
|
||||
pub fn resident_records(&self) -> usize {
|
||||
self.chunk.as_ref().map_or(0, |c| c.len())
|
||||
}
|
||||
|
||||
@@ -130,6 +130,12 @@ fn streaming_close_source_backtests_end_to_end_deterministically() {
|
||||
assert_eq!(r1.to_json(), r2.to_json());
|
||||
}
|
||||
|
||||
/// Measures the **aura source ring** residency — `M1FieldSource::resident_records()`,
|
||||
/// the single chunk the source holds — NOT whole-process RSS. The ring is bounded by
|
||||
/// one chunk at every window length (the assert below). The data-server `FileCache`
|
||||
/// beneath retains the loaded window's parsed chunks for the pass (the replay-many
|
||||
/// sharing model), so process RSS grows O(records-touched); that gap is by design,
|
||||
/// not a leak, and is tracked as #95.
|
||||
#[test]
|
||||
fn residency_is_bounded_by_one_chunk_independent_of_window_length() {
|
||||
let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
|
||||
@@ -157,8 +163,9 @@ fn residency_is_bounded_by_one_chunk_independent_of_window_length() {
|
||||
// multi-chunk window: more records than a single chunk holds, so the bound
|
||||
// below is non-vacuous (a O(window) source would have peaked at `total`).
|
||||
assert!(total > CHUNK_SIZE, "window must span multiple chunks (got {total} records)");
|
||||
// O(one chunk), NOT O(window): the per-pull ceiling never exceeds one chunk,
|
||||
// regardless of how many chunks the window spans.
|
||||
// O(one chunk) source ring, NOT O(window): the per-pull ceiling never exceeds
|
||||
// one chunk, regardless of how many chunks the window spans (whole-process RSS
|
||||
// is a separate quantity — see this test's doc comment and #95).
|
||||
assert!(peak <= CHUNK_SIZE, "resident records {peak} exceeded one chunk ({CHUNK_SIZE})");
|
||||
assert!(peak > 0, "a non-empty window resided at least one record");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user