Opt-in non-retaining streaming read path (single-pass residency, no FileCache retention) #2

New Issue

2026-06-17T22:03:31+02:00

Brummel commented

2026-06-17 22:03:31 +02:00

Surfaced by aura issue Brummel/Aura#95 (process-residency deep-dive).

Finding

FileCache (src/cache.rs) retains every touched file's parsed chunks
(HashMap<FileKey, Arc<Vec<Arc<[M1Parsed]>>>>, keyed symbol/year/month/format)
for the whole streaming pass — the only eviction is release_symbol when a
symbol's refcount hits 0 (run end). A SymbolChunkIter holds a SymbolGuard
for its lifetime, so a single long forward pass keeps refcount ≥ 1 throughout and
nothing is evicted behind the cursor.

Consequence: a single sequential pass over a multi-year window resides
O(records-touched) ≈ records × sizeof(M1Parsed)=56 B. Measured (GER40 M1, one
symbol): VmHWM 6 MiB (1 month) → 173 MiB (~10 years), linear. This is correct
and desirable for the replay-many access pattern (many sims sharing one fixed
window: load-each-file-once, close+volume sharing one parse via the
field-agnostic FileKey). It is only wasteful for a lone forward pass that never
revisits a file.

Why no always-on fix

The retention is the replay-many optimization. Any unconditional evictor that
bounds a single pass to O(one file) must drop files while a reference is still
live — exactly what cross-sim sharing depends on. Cursor-aware per-file eviction
and a bounded LRU both re-load a file a lagging sim still needs, breaking the
load-once (loads) invariant. So single-pass and replay-many residency policies
must be distinguished by intent, never unified under one always-on policy.

Proposed (only if a genuine single-pass-over-decades consumer appears)

An opt-in non-retaining streaming read path: a second iterator/entrypoint
that does NOT insert into the shared FileCache — load file N, hand out its
chunks, drop them on advance to N+1 (the consumer's held chunk Arcs bound
residency to one file). The existing caching stream_m1_windowed path
(load-once, cross-field FileKey sharing, SymbolGuard refcount eviction) stays
bit-for-bit unchanged; the single-pass caller (aura's M1FieldSource) opts in,
replay-many keeps the retaining path.

Parked as idea (no commitment): aura's World is replay-many, which wants the
retention; the leak only bites the currently-unused single-pass pattern. Promote
to feature if a single-pass-over-decades consumer lands on the roadmap.

On the aura side, #95 is resolved by narrowing the docs from a (false)
whole-process O(one-chunk) claim to the true per-source-ring property.

Surfaced by aura issue Brummel/Aura#95 (process-residency deep-dive). ## Finding `FileCache` (`src/cache.rs`) retains every touched file's **parsed** chunks (`HashMap<FileKey, Arc<Vec<Arc<[M1Parsed]>>>>`, keyed `symbol/year/month/format`) for the whole streaming pass — the only eviction is `release_symbol` when a symbol's refcount hits 0 (run end). A `SymbolChunkIter` holds a `SymbolGuard` for its lifetime, so a single long forward pass keeps refcount ≥ 1 throughout and nothing is evicted behind the cursor. Consequence: a single sequential pass over a multi-year window resides O(records-touched) ≈ `records × sizeof(M1Parsed)=56 B`. Measured (GER40 M1, one symbol): VmHWM 6 MiB (1 month) → 173 MiB (~10 years), linear. This is **correct and desirable** for the replay-many access pattern (many sims sharing one fixed window: load-each-file-once, `close`+`volume` sharing one parse via the field-agnostic `FileKey`). It is only wasteful for a lone forward pass that never revisits a file. ## Why no always-on fix The retention **is** the replay-many optimization. Any unconditional evictor that bounds a single pass to O(one file) must drop files while a reference is still live — exactly what cross-sim sharing depends on. Cursor-aware per-file eviction and a bounded LRU both re-load a file a lagging sim still needs, breaking the load-once (`loads`) invariant. So single-pass and replay-many residency policies must be distinguished **by intent**, never unified under one always-on policy. ## Proposed (only if a genuine single-pass-over-decades consumer appears) An **opt-in non-retaining streaming read path**: a second iterator/entrypoint that does NOT insert into the shared `FileCache` — load file N, hand out its chunks, drop them on advance to N+1 (the consumer's held chunk `Arc`s bound residency to one file). The existing caching `stream_m1_windowed` path (load-once, cross-field `FileKey` sharing, `SymbolGuard` refcount eviction) stays bit-for-bit unchanged; the single-pass caller (aura's `M1FieldSource`) opts in, replay-many keeps the retaining path. Parked as `idea` (no commitment): aura's World is replay-many, which **wants** the retention; the leak only bites the currently-unused single-pass pattern. Promote to `feature` if a single-pass-over-decades consumer lands on the roadmap. On the aura side, #95 is resolved by narrowing the docs from a (false) whole-process O(one-chunk) claim to the true per-source-ring property.

Brummel added the idea label 2026-06-17 22:03:31 +02:00

Brummel referenced this issue from Brummel/Aura

2026-06-17 22:04:00 +02:00

Streaming residency is O(window) at process level — contradicts the documented O(one-chunk) claim #95

Brummel referenced this issue from a commit

2026-06-17 23:33:28 +02:00

docs: narrow streaming residency claim from whole-process RSS to the source ring

Sign in to join this conversation.