BLOCKER: stage1-r sweep uses ~2 GiB/member — full-history sweeps exhaust RAM+swap #138

New Issue

2026-06-25T09:15:12+02:00

Brummel commented

2026-06-25 09:15:12 +02:00

Summary

A single multi-member stage1-r sweep over full M1 history consumes
~2 GiB per swept member, so a full-history sweep nearly exhausts RAM +
swap and gets SIGKILL-137'd. This blocks the edge-research milestone: the
walk-forward OOS phase runs a sweep per window across many windows, which
multiplies this footprint.

Evidence (measured, reproduced)

The original cross-symbol screen died with exit 137 mid-run. Reproduced the
exact failing command (EURUSD, 9-member grid, default rayon = 24 threads,
no throttle) under per-second memory sampling:

RESULT EXIT=0 WALL=107s PEAK_RSS_MB=18974 MIN_AVAIL_MB=2717 MAX_SWAP_MB=30012

Peak RSS ≈ 18.5 GiB for one aura process.
Free RAM dropped from ~21 GiB to 2.7 GiB; swap climbed to 30 / 31 GiB.
It only completed this time because it ran in the background (no foreground
timeout) and had marginally more headroom; the original died at the edge.
Throttling RAYON_NUM_THREADS=2 (fewer concurrent members) survives — which
localises the cost to per-member, materialised concurrently.

18.5 GiB / 9 members ≈ 2 GiB/member over ~5.5 M one-minute cycles
(EURUSD, ~15 y).

Likely cause (to confirm RED-first when tackled)

Hypothesis: the per-cycle data is retained in full rather than folded
incrementally — summarize_r consumes the whole dense PositionManagement
record (14 columns × ~5.5 M cycles), and/or the recorders hold the full
equity / exposure / r_equity series per member, all resident at once across
the concurrently-running sweep members. ~2 GiB/member is far more than the
16 bytes/cycle that equity+exposure alone would need, so something is holding
the dense record (or all taps) for every cycle. This violates the C7 /
streaming-source spirit (the M1 source is lazy; the reduction is not).

Direction (not a prescription)

Make the reductions online/streaming: fold summarize_r and the metric
recorders incrementally so per-cycle rows are not retained
(O(1)/O(trades) memory, not O(cycles)).
Failing that, cap concurrency or window the data — but the streaming fold is
the real fix and keeps full-history multi-member sweeps (and walk-forward)
viable.

Process

RED-first (debug): pin the footprint with a test that fails on the current
retain-everything path (e.g. a bounded-memory assertion or a record-length
invariant), before the GREEN fix. Behaviour-preserving for the metric values
(goldens unchanged).

refs #137 (the research run this blocks)

## Summary A single multi-member stage1-r sweep over full M1 history consumes **~2 GiB per swept member**, so a full-history sweep nearly exhausts RAM + swap and gets SIGKILL-137'd. This blocks the edge-research milestone: the walk-forward OOS phase runs a sweep *per window* across many windows, which multiplies this footprint. ## Evidence (measured, reproduced) The original cross-symbol screen died with exit 137 mid-run. Reproduced the exact failing command (EURUSD, 9-member grid, default rayon = 24 threads, no throttle) under per-second memory sampling: ``` RESULT EXIT=0 WALL=107s PEAK_RSS_MB=18974 MIN_AVAIL_MB=2717 MAX_SWAP_MB=30012 ``` - **Peak RSS ≈ 18.5 GiB** for one `aura` process. - Free RAM dropped from ~21 GiB to **2.7 GiB**; swap climbed to **30 / 31 GiB**. - It only completed this time because it ran in the background (no foreground timeout) and had marginally more headroom; the original died at the edge. - Throttling `RAYON_NUM_THREADS=2` (fewer concurrent members) survives — which localises the cost to **per-member, materialised concurrently**. 18.5 GiB / 9 members ≈ **2 GiB/member** over ~5.5 M one-minute cycles (EURUSD, ~15 y). ## Likely cause (to confirm RED-first when tackled) Hypothesis: the per-cycle data is **retained in full** rather than folded incrementally — `summarize_r` consumes the whole dense PositionManagement record (14 columns × ~5.5 M cycles), and/or the recorders hold the full equity / exposure / r_equity series per member, all resident at once across the concurrently-running sweep members. ~2 GiB/member is far more than the 16 bytes/cycle that equity+exposure alone would need, so something is holding the dense record (or all taps) for every cycle. This violates the C7 / streaming-source spirit (the M1 source is lazy; the *reduction* is not). ## Direction (not a prescription) - Make the reductions **online/streaming**: fold `summarize_r` and the metric recorders incrementally so per-cycle rows are not retained (O(1)/O(trades) memory, not O(cycles)). - Failing that, cap concurrency or window the data — but the streaming fold is the real fix and keeps full-history multi-member sweeps (and walk-forward) viable. ## Process RED-first (`debug`): pin the footprint with a test that fails on the current retain-everything path (e.g. a bounded-memory assertion or a record-length invariant), before the GREEN fix. Behaviour-preserving for the metric values (goldens unchanged). refs #137 (the research run this blocks)

Brummel added this to the Edge research: prove a real tradeable edge milestone 2026-06-25 09:15:12 +02:00

Brummel added the BLOCKER bug labels 2026-06-25 09:15:12 +02:00

Brummel referenced this issue

2026-06-25 09:15:52 +02:00

Strategy research: find a Stage-1 positive-E[R] edge (honest OOS) #137

Brummel commented

2026-06-25 09:33:57 +02:00

Diagnosis update (RED-first) — the obvious fix is insufficient; the real fix is an engine-contract change

The RED-first diagnosis confirmed the root cause but also surfaced that the
dominant O(cycles) retainer is the unbounded mpsc channel buffering during
the run, not (only) summarize_r's slice API.

Confirmed

run_one (aura-cli/src/main.rs ~1207-1240) wires each recorder to an
unbounded mpsc::Sender; Harness::run (aura-engine/src/harness.rs:371)
is one synchronous call with no end-of-stream / finalize hook, so during the
run the channels accumulate all ~5.5M rows (dominant: the 14-column R-record
≈ 1.4 GB/member) before anything drains them. The post-run
rx.try_iter().collect() is a second O(cycles) copy, but the channel-full peak
at end-of-run is the real ceiling.

Why "just stream the consumer" does NOT fix it

Folding rx.try_iter() lazily instead of .collect() removes the second copy,
but the channel already holds every row when h.run() returns — so the peak
(channel-full ≈ 1.4 GB/member) is essentially unchanged. A streaming
summarize_r alone makes the attached RED test pass while leaving the BLOCKER
unfixed (green-but-wrong).

The fork: every O(1) fix touches the engine contract or invariant C1

Rows must be consumed during production; production is inside the synchronous
h.run(), so:

(M2) bounded channel + concurrent consumer thread — folds during the run,
but adds within-sim concurrency, in tension with invariant C1
("parallelism across sims, never within one"). Whether a sink-draining consumer
counts as within-sim parallelism is an invariant-interpretation call
(ledger-level), not a silent refactor.
(M3, recommended) end-of-stream finalize lifecycle on sinks + online-folding
reduction sink nodes — the reducer folds per eval into owned
O(trades)+O(1) state and emits a single summary row on finalize;
Harness::run calls finalize after the source loop; run_one then drains one
summary, not 5.5M raw rows. Idiomatic streaming-dataflow flush, reuses the
existing channel, aligns with C8/C22, no within-sim threads. Touches the Node
contract (additive finalize, default no-op) + Harness::run.
(M5) folding reduction node read back post-run via downcast — needs
Node: Any + a harness node accessor; no threads; a less idiomatic back-channel.
Filter-recorder (trade-events only) — insufficient alone: the open-at-end
window-end force-close (a golden) needs the final row, which the recorder cannot
identify without a finalize hook.

Decision needed (engine direction)

This is no longer a localized bug-fix — it is a small engine execution-model
change (streaming sink reductions). Recommend M3 as a bounded design cycle
(specify → plan → implement, all metric goldens byte-identical). It also unblocks
#139 (walk-forward multiplies the per-member footprint). Alternatives: accept M2
and amend C1's interpretation in the ledger, or ship the RAYON_NUM_THREADS cap
as a documented interim and defer the proper fix.

The RED test from this diagnosis
(crates/aura-engine/tests/r_reduction_streaming.rs, unstaged) pins the
summarize_r slice-API sub-condition but not the channel-buffer site; the
real RED must exercise the recorder → channel → reduction path end-to-end.

## Diagnosis update (RED-first) — the obvious fix is insufficient; the real fix is an engine-contract change The RED-first diagnosis confirmed the root cause but also surfaced that the **dominant O(cycles) retainer is the unbounded `mpsc` channel buffering during the run**, not (only) `summarize_r`'s slice API. ### Confirmed `run_one` (`aura-cli/src/main.rs` ~1207-1240) wires each recorder to an **unbounded `mpsc::Sender`**; `Harness::run` (`aura-engine/src/harness.rs:371`) is one synchronous call with **no end-of-stream / finalize hook**, so during the run the channels accumulate **all ~5.5M rows** (dominant: the 14-column R-record ≈ 1.4 GB/member) before anything drains them. The post-run `rx.try_iter().collect()` is a *second* O(cycles) copy, but the channel-full peak at end-of-run is the real ceiling. ### Why "just stream the consumer" does NOT fix it Folding `rx.try_iter()` lazily instead of `.collect()` removes the second copy, but the channel **already holds every row** when `h.run()` returns — so the peak (channel-full ≈ 1.4 GB/member) is essentially unchanged. A streaming `summarize_r` alone makes the attached RED test pass while leaving the BLOCKER unfixed (green-but-wrong). ### The fork: every O(1) fix touches the engine contract or invariant C1 Rows must be consumed *during* production; production is inside the synchronous `h.run()`, so: - **(M2) bounded channel + concurrent consumer thread** — folds during the run, but adds **within-sim concurrency**, in tension with invariant **C1** ("parallelism across sims, never within one"). Whether a sink-draining consumer counts as within-sim parallelism is an invariant-interpretation call (ledger-level), not a silent refactor. - **(M3, recommended) end-of-stream `finalize` lifecycle on sinks + online-folding reduction sink nodes** — the reducer folds per `eval` into owned O(trades)+O(1) state and emits a single summary row on `finalize`; `Harness::run` calls `finalize` after the source loop; `run_one` then drains one summary, not 5.5M raw rows. Idiomatic streaming-dataflow flush, reuses the existing channel, aligns with C8/C22, no within-sim threads. Touches the `Node` contract (additive `finalize`, default no-op) + `Harness::run`. - **(M5) folding reduction node read back post-run via downcast** — needs `Node: Any` + a harness node accessor; no threads; a less idiomatic back-channel. - **Filter-recorder (trade-events only)** — insufficient alone: the open-at-end window-end force-close (a golden) needs the final row, which the recorder cannot identify without a finalize hook. ### Decision needed (engine direction) This is no longer a localized bug-fix — it is a small **engine execution-model change** (streaming sink reductions). Recommend **M3** as a bounded design cycle (specify → plan → implement, all metric goldens byte-identical). It also unblocks #139 (walk-forward multiplies the per-member footprint). Alternatives: accept M2 and amend C1's interpretation in the ledger, or ship the `RAYON_NUM_THREADS` cap as a documented interim and defer the proper fix. The RED test from this diagnosis (`crates/aura-engine/tests/r_reduction_streaming.rs`, unstaged) pins the `summarize_r` slice-API sub-condition but **not** the channel-buffer site; the real RED must exercise the recorder → channel → reduction path end-to-end.

Brummel commented

2026-06-25 09:50:00 +02:00

Direction decided (user): M3

Minuted from chat — the user reviewed the fork and chose M3: an end-of-stream
finalize lifecycle on sinks plus online-folding reduction sink nodes, to be
implemented autonomously. Proceeding spec → plan → implement, all metric goldens
byte-identical.

Mechanism (settled):

Node::finalize(&mut self) {} — additive default no-op on the trait
(aura-core/src/node.rs); only a folding sink overrides it. Existing nodes
compile untouched.
Harness::run (aura-engine/src/harness.rs) gains a post-loop epilogue: after
the sources drain, call finalize() on every node once, in topo order.
New folding reduction sink node(s) (sibling of aura-std/src/recorder.rs):
fold per eval into O(trades)+O(1) owned state (no Rc/RefCell — C7
preserved, like today's Recorder), emit a compacted summary on finalize.
The R path filters to CLOSED rows + the genuine final row, so summarize_r
stays unchanged and its goldens byte-identical; the pip equity/exposure path
is a true online fold (running peak/shortfall, Welford) that must replicate
summarize's arithmetic exactly.
run_one (sweep) and run_stage1_r (single) drain O(trades)/one summary
instead of ~5.5M raw rows.

Determinism: finalize runs once after the deterministic source loop; no
within-sim concurrency (C1 intact). This also unblocks #139 (walk-forward
multiplies the per-member footprint).

The debug-run RED test (r_reduction_streaming.rs) under-pinned the bug
(summarize_r slice API only, not the channel buffer); it has been lifted out
of the tree. The spec defines the stronger end-to-end RED — the
recorder → channel → reduction peak is O(trades), not O(cycles) — plus the
golden-preservation tests.

## Direction decided (user): M3 Minuted from chat — the user reviewed the fork and chose **M3**: an end-of-stream `finalize` lifecycle on sinks plus online-folding reduction sink nodes, to be implemented autonomously. Proceeding spec → plan → implement, all metric goldens byte-identical. Mechanism (settled): - `Node::finalize(&mut self) {}` — additive default no-op on the trait (`aura-core/src/node.rs`); only a folding sink overrides it. Existing nodes compile untouched. - `Harness::run` (`aura-engine/src/harness.rs`) gains a post-loop epilogue: after the sources drain, call `finalize()` on every node once, in topo order. - New folding reduction sink node(s) (sibling of `aura-std/src/recorder.rs`): fold per `eval` into O(trades)+O(1) owned state (no `Rc`/`RefCell` — C7 preserved, like today's `Recorder`), emit a compacted summary on `finalize`. The R path filters to CLOSED rows + the genuine final row, so `summarize_r` stays unchanged and its goldens byte-identical; the pip equity/exposure path is a true online fold (running peak/shortfall, Welford) that must replicate `summarize`'s arithmetic exactly. - `run_one` (sweep) and `run_stage1_r` (single) drain O(trades)/one summary instead of ~5.5M raw rows. Determinism: `finalize` runs once after the deterministic source loop; no within-sim concurrency (C1 intact). This also unblocks #139 (walk-forward multiplies the per-member footprint). The debug-run RED test (`r_reduction_streaming.rs`) under-pinned the bug (`summarize_r` slice API only, not the channel buffer); it has been lifted out of the tree. The spec defines the stronger end-to-end RED — the recorder → channel → reduction peak is O(trades), not O(cycles) — plus the golden-preservation tests.

Brummel commented

2026-06-25 11:30:38 +02:00

Fixed — M3 streaming sink reductions landed (cycle 0070)

The no-trace stage1-r sweep now folds its reductions online via a new end-of-stream
finalize lifecycle: SeriesReducer folds the equity/exposure series to one
summary row each, GatedRecorder retains only the gated (closed) trade rows + the
genuine final row. Per-member memory drops from O(cycles) (~2 GiB over ~5.5M
one-minute bars) to O(trades). The --trace path and the single run keep the raw
recorders as the byte-for-byte reference.

Verified: cargo test --workspace green (42 suites); cargo clippy --workspace --all-targets -- -D warnings clean; stage1-r goldens byte-identical (a new
CLI-seam test pins folded-no-trace == raw---trace metrics); a counting-allocator
capstone holds peak flat across a 770x cycle-count growth.

Landed in cfe7bad (machinery: finalize hook + SeriesFold + folding sinks +
integration tests) and eb86534 (CLI wiring). Cycle closed in ffd18b9 — the
finalize lifecycle is recorded in ledger C8. This unblocks #139 (walk-forward,
which multiplies the per-member footprint). Related: #77 (cross-referenced there).

## Fixed — M3 streaming sink reductions landed (cycle 0070) The no-trace stage1-r sweep now folds its reductions online via a new end-of-stream `finalize` lifecycle: `SeriesReducer` folds the equity/exposure series to one summary row each, `GatedRecorder` retains only the gated (closed) trade rows + the genuine final row. Per-member memory drops from O(cycles) (~2 GiB over ~5.5M one-minute bars) to O(trades). The `--trace` path and the single run keep the raw recorders as the byte-for-byte reference. Verified: `cargo test --workspace` green (42 suites); `cargo clippy --workspace --all-targets -- -D warnings` clean; stage1-r goldens byte-identical (a new CLI-seam test pins folded-no-trace == raw-`--trace` metrics); a counting-allocator capstone holds peak flat across a 770x cycle-count growth. Landed in `cfe7bad` (machinery: finalize hook + SeriesFold + folding sinks + integration tests) and `eb86534` (CLI wiring). Cycle closed in `ffd18b9` — the `finalize` lifecycle is recorded in ledger C8. This unblocks #139 (walk-forward, which multiplies the per-member footprint). Related: #77 (cross-referenced there).

Brummel referenced this issue

2026-06-25 11:30:38 +02:00

Rename Recorder -> Probe (the edge tap); decide whether a Probe must heap-allocate a record per cycle #77

Brummel referenced this issue from a commit

2026-06-25 11:49:59 +02:00

spec: 0070 streaming sink reductions (boss-signed)

Brummel referenced this issue from a commit

2026-06-25 11:50:00 +02:00

plan: 0070 streaming sink reductions

Brummel referenced this issue from a commit

2026-06-25 11:50:00 +02:00

feat(0070): streaming sink reductions — finalize hook + folding sinks (#138)