Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Streaming residency is O(window) at process level — contradicts the documented O(one-chunk) claim #95

New Issue
Closed
opened 2026-06-17 18:53:03 +02:00 by Brummel · 1 comment
Brummel commented 2026-06-17 18:53:03 +02:00
Owner
Copy Link
Copy Source

Surfaced by the GER40 research deep-dive fieldtest (docs/specs/fieldtest-research-breakout-deepdive.md).

The docs (C12 cycle-0041 realization; the test
aura-ingest/tests/streaming_seam.rs :: residency_is_bounded_by_one_chunk_independent_of_window_length) make a
load-bearing claim: streaming residency is O(one chunk), so "a 20-year window
streams in the same memory as a one-day one".

Measured, against real GER40 M1 (verified by re-reading the transcripts):
the aura-level predicate resident_records <= 1024 DOES hold at every window
length — but process peak RSS (VmHWM) grows ~linearly with the window:

1 month    records=24466     aura-resident<=1024? true   VmHWM= 6 MiB
6 months   records=159156    aura-resident<=1024? true   VmHWM=15 MiB
1 year     records=305878    aura-resident<=1024? true   VmHWM=24 MiB
4 years    records=1295897   aura-resident<=1024? true   VmHWM=78 MiB
~10 years  records=3064025   aura-resident<=1024? true   VmHWM=173 MiB

Isolation run — 4 raw M1FieldSources, NO harness, NO sinks — streaming
8.6M records over 7 years still peaks at 125 MiB (sinks add the rest: the
full breakout + drain_trace reaches 210 MiB, but that part is expected trace
accumulation). So the growth is source-/data-server-side, not the recorders.

The streaming_seam test only measures the aura M1FieldSource ring, not the
underlying data-server chunk/decode retention, so it passes green while the
process-level promise is false. A researcher who budgets RAM on the doc's
promise (one chunk ≈ tens of KiB) is off by 100–1000×; a real multi-symbol
sweep over decades would OOM where the docs say it streams flat.

Repro: cargo run --release --manifest-path fieldtests/research-breakout-deepdive/Cargo.toml --bin t1c_residency_scaling
→ the VmHWM column grows monotonically with the record count while
aura-resident<=1024? stays true.

Recommend (debug): either the data-server decoded chunks/buffers are
retained across pulls and must be released (root cause likely in
Brummel/data-server), OR the residency claim must be narrowed in the
docs/test to "aura-level ring residency" (all the predicate actually proves) and
the true per-window process cost documented. Minor adjacent friction:
resident_records() lives on the concrete M1FieldSource, not the public
Source trait, so a Box<dyn Source> consumer cannot probe residency without
downcasting.

**Surfaced by the GER40 research deep-dive fieldtest** (`docs/specs/fieldtest-research-breakout-deepdive.md`). The docs (C12 cycle-0041 realization; the test `aura-ingest/tests/streaming_seam.rs :: residency_is_bounded_by_one_chunk_independent_of_window_length`) make a load-bearing claim: streaming residency is O(one chunk), so "a 20-year window streams in the same memory as a one-day one". **Measured, against real GER40 M1 (verified by re-reading the transcripts):** the aura-level predicate `resident_records <= 1024` DOES hold at every window length — but process peak RSS (`VmHWM`) grows ~linearly with the window: ``` 1 month records=24466 aura-resident<=1024? true VmHWM= 6 MiB 6 months records=159156 aura-resident<=1024? true VmHWM=15 MiB 1 year records=305878 aura-resident<=1024? true VmHWM=24 MiB 4 years records=1295897 aura-resident<=1024? true VmHWM=78 MiB ~10 years records=3064025 aura-resident<=1024? true VmHWM=173 MiB ``` Isolation run — **4 raw `M1FieldSource`s, NO harness, NO sinks** — streaming 8.6M records over 7 years still peaks at **125 MiB** (sinks add the rest: the full breakout + `drain_trace` reaches 210 MiB, but that part is expected trace accumulation). So the growth is **source-/data-server-side**, not the recorders. The `streaming_seam` test only measures the aura `M1FieldSource` ring, not the underlying `data-server` chunk/decode retention, so it passes green while the process-level promise is false. A researcher who budgets RAM on the doc's promise (one chunk ≈ tens of KiB) is off by 100–1000×; a real multi-symbol sweep over decades would OOM where the docs say it streams flat. **Repro:** `cargo run --release --manifest-path fieldtests/research-breakout-deepdive/Cargo.toml --bin t1c_residency_scaling` → the `VmHWM` column grows monotonically with the record count while `aura-resident<=1024?` stays `true`. **Recommend** (debug): either the `data-server` decoded chunks/buffers are retained across pulls and must be released (root cause likely in `Brummel/data-server`), OR the residency claim must be narrowed in the docs/test to "aura-level ring residency" (all the predicate actually proves) and the true per-window process cost documented. Minor adjacent friction: `resident_records()` lives on the concrete `M1FieldSource`, not the public `Source` trait, so a `Box<dyn Source>` consumer cannot probe residency without downcasting.
Brummel added the bug label 2026-06-17 18:53:03 +02:00
Brummel commented 2026-06-17 22:04:00 +02:00
Author
Owner
Copy Link
Copy Source

Diagnosed + resolved (doc-truth correction)

Root cause confirmed (adversarially verified, high confidence; reproduced fresh:
6 MiB @ 1mo → 173 MiB @ ~10y, linear):

  • data-server's FileCache (src/cache.rs) retains every touched file's
    parsed chunks (HashMap<FileKey, Arc<Vec<Arc<[M1Parsed]>>>>, keyed
    symbol/year/month/format). The only eviction is release_symbol at symbol
    refcount→0 (run end); M1FieldSource holds a SymbolGuard for the whole pass,
    so nothing is evicted behind the cursor → live retention is O(records-touched).
  • sizeof(M1Parsed) = 56 B (7×8); VmHWM−baseline converges on records × 56 B
    (3.06M × 56 = 171 MiB ≈ measured 173). No mmap, no raw-buffer retention, not an
    allocator artifact (loader drops the raw Vec<u8> after parse).
  • The aura resident_records() predicate is honestly O(one chunk) — it measures
    the source ring, which is structurally blind to the FileCache beneath. So
    the predicate holding and VmHWM growing are not contradictory; they measure
    different quantities.

This is not a broken design promise. The true property (per-source ring ≤ one
chunk) is real and tested. The over-claim was the process-level gloss layered on
top of it — INDEX.md ("a 20-year window streams in the same memory as a one-day
one") and 0041 ("resident memory is flat across the horizon"). The 0041 spec
already scoped it correctly at its "Data-server cache file retention" friction note
("owned by that lib, not this cycle"); the summary prose simply widened past that
scope.

No clean always-on code fix exists — structurally, not for effort. The
whole-symbol retention is the replay-many optimization (load-once, close+volume
sharing one parse via the field-agnostic FileKey). Any unconditional evictor that
bounds a single forward pass must drop files a lagging sim still needs, breaking
load-once. Single-pass vs replay-many residency policies must be split by intent.

Resolution (working tree, to land with this issue's close)

Narrowed the docs/comments from the false whole-process claim to the true
per-source-ring property, naming the FileCache per-window retention as the real
(desirable, replay-amortized) process cost:

  • docs/design/INDEX.md — C12 realization gloss
  • docs/specs/0041-source-ingestion-seam.md — example comment + residency criterion
  • crates/aura-ingest/src/lib.rs — M1FieldSource + resident_records doc-comments
  • crates/aura-ingest/tests/streaming_seam.rs — residency-test comments (the assert
    is correct and unchanged; only its labelling was process-vs-ring confused)

The deferred capability (an opt-in non-retaining streaming read path, only needed if
a genuine single-pass-over-decades consumer appears — the World is replay-many and
wants the retention) is parked as Brummel/data-server#2 (idea).

Adjacent friction noted in the OP (resident_records() on concrete M1FieldSource,
not the Source trait) left as a separate small follow-up.

## Diagnosed + resolved (doc-truth correction) Root cause confirmed (adversarially verified, high confidence; reproduced fresh: 6 MiB @ 1mo → 173 MiB @ ~10y, linear): - data-server's `FileCache` (`src/cache.rs`) retains every touched file's **parsed** chunks (`HashMap<FileKey, Arc<Vec<Arc<[M1Parsed]>>>>`, keyed `symbol/year/month/format`). The only eviction is `release_symbol` at symbol refcount→0 (run end); `M1FieldSource` holds a `SymbolGuard` for the whole pass, so nothing is evicted behind the cursor → live retention is O(records-touched). - `sizeof(M1Parsed) = 56 B` (7×8); VmHWM−baseline converges on `records × 56 B` (3.06M × 56 = 171 MiB ≈ measured 173). No mmap, no raw-buffer retention, not an allocator artifact (loader drops the raw `Vec<u8>` after parse). - The aura `resident_records()` predicate is honestly O(one chunk) — it measures the **source ring**, which is structurally blind to the `FileCache` beneath. So the predicate holding *and* VmHWM growing are not contradictory; they measure different quantities. **This is not a broken design promise.** The true property (per-source ring ≤ one chunk) is real and tested. The over-claim was the *process-level* gloss layered on top of it — `INDEX.md` ("a 20-year window streams in the same memory as a one-day one") and `0041` ("resident memory is flat across the horizon"). The 0041 spec already scoped it correctly at its "Data-server cache file retention" friction note ("owned by that lib, not this cycle"); the summary prose simply widened past that scope. **No clean always-on code fix exists — structurally, not for effort.** The whole-symbol retention *is* the replay-many optimization (load-once, `close`+`volume` sharing one parse via the field-agnostic `FileKey`). Any unconditional evictor that bounds a single forward pass must drop files a lagging sim still needs, breaking load-once. Single-pass vs replay-many residency policies must be split **by intent**. ### Resolution (working tree, to land with this issue's close) Narrowed the docs/comments from the false whole-process claim to the true per-source-ring property, naming the `FileCache` per-window retention as the real (desirable, replay-amortized) process cost: - `docs/design/INDEX.md` — C12 realization gloss - `docs/specs/0041-source-ingestion-seam.md` — example comment + residency criterion - `crates/aura-ingest/src/lib.rs` — `M1FieldSource` + `resident_records` doc-comments - `crates/aura-ingest/tests/streaming_seam.rs` — residency-test comments (the assert is correct and unchanged; only its labelling was process-vs-ring confused) The deferred capability (an opt-in non-retaining streaming read path, only needed if a genuine single-pass-over-decades consumer appears — the World is replay-many and *wants* the retention) is parked as **Brummel/data-server#2** (`idea`). Adjacent friction noted in the OP (`resident_records()` on concrete `M1FieldSource`, not the `Source` trait) left as a separate small follow-up.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#95
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?