docs: narrow streaming residency claim from whole-process RSS to the source ring
The C12 realization and the 0041 spec claimed streaming residency is O(one chunk) at the process level ("a 20-year window streams in the same memory as a one-day one"). That holds only for the aura source ring (M1FieldSource::resident_records(), bounded by one chunk and correctly tested); whole-process RSS grows O(records-touched) because data-server's FileCache retains each window's parsed chunks (~56 B/record) read-only for the pass. Reproduced: 6 MiB @ 1mo -> 173 MiB @ ~10y, linear.
That retention is the replay-many optimization (load-once; close+volume share one parse via the field-agnostic FileKey), not a leak, and no always-on eviction can bound a single forward pass without regressing it. This narrows the docs/comments to the true per-source-ring property and names the FileCache per-window retention as the real, replay-amortized process cost. The streaming_seam assert is unchanged (it was always the ring bound; only its labelling confused ring vs process). The deferred opt-in non-retaining streaming read path is parked as Brummel/data-server#2.
closes #95
This commit is contained in:
@@ -102,9 +102,10 @@ let mut h = Harness::bootstrap(sma_cross_signal_quality_graph())?;
|
||||
let close = M1FieldSource::open(&server, "AAPL.US", Some(from_ms), Some(to_ms), M1Field::Close)
|
||||
.expect("window overlaps real data");
|
||||
|
||||
// Drive the seam. The source streams Arc<[M1Parsed]> chunks lazily: resident
|
||||
// footprint is one chunk + the engine's per-node lookback windows, NOT the
|
||||
// whole window. A 20-year window streams in the same memory as a one-day one.
|
||||
// Drive the seam. The source streams Arc<[M1Parsed]> chunks lazily: the source's
|
||||
// resident footprint is one chunk + the engine's per-node lookback windows, NOT
|
||||
// the whole window — a per-source bound (the data-server cache below still retains
|
||||
// the loaded window for the pass; see "Data-server cache file retention" + #95).
|
||||
h.run(vec![Box::new(close)]);
|
||||
|
||||
let report = fold_recorded_into_report(/* sinks */);
|
||||
@@ -433,8 +434,10 @@ buffers (already O(Σ lookbacks), unchanged). Nothing scales with window length.
|
||||
O(window length), measured on real multi-chunk data (Testing strategy §4).
|
||||
The struct carries no accumulating field, so this bound holds by
|
||||
construction (§5). The engine's per-node lookback ring buffers are
|
||||
unchanged — already O(Σ lookbacks), pinned by the existing suite — so total
|
||||
resident memory is flat across the horizon.
|
||||
unchanged — already O(Σ lookbacks), pinned by the existing suite. (This is the
|
||||
*source-ring* footprint; whole-process RSS is **not** flat across the horizon —
|
||||
the data-server cache retains the loaded window for the pass, see "Data-server
|
||||
cache file retention" below + #95.)
|
||||
- [ ] `cargo clippy --workspace --all-targets -- -D warnings` clean;
|
||||
`cargo doc --workspace --no-deps` clean.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user