Bound the campaign executor fan-out to a worker pool #268
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
During a 42-cell campaign run (21 instruments x 2 regimes, 9-member axes, 90/30 walk-forward + 1000-resample bootstrap, engine
84e1075release) on a 24-core box, the executor process averages ~1410% CPU (~14 cores, ~58% of the box) while its thread count oscillates hard. Twenty 0.5-s samples ofnlwp:Three phases alternate: bursts of 150-240 threads (10x oversubscription of the cores), multi-second plateaus of exactly 10-11 threads (suspiciously close to 9 sweep members + main), and 1-thread valleys where 23 cores idle. RSS stays flat at ~2.9 GB.
Claim (unverified): the bursts are unbounded thread-per-sim fan-out (e.g. all walk-forward windows x members spawned at once) and the valleys are serial seams (ingest/merge, bootstrap, record writing). A bounded worker pool sized to the cores, plus a look at what the serial seams spend their time on, is worth roughly 40% of campaign wall time on this box if the valleys can overlap with sim execution. Per-cell wall time is otherwise healthy (~11-13 s per cell over 8.5 y of m1 with costs; the Arc-1-era baseline in the external research project recorded 9.5 s/cell without cost nodes).
Ruled-out alternative: the 1-thread valleys are not network/data-paging stalls. The archive is an NFS mount (nfs4, rsize 1M), but the m1 working set is 1.4 GB against ~12 GB of page cache. Measured over 10 s mid-run: the process reads ~15 MB/s via syscalls (
rchar+152 MB) while the NIC receives ~73 KB/s — i.e. reads are served from the local page cache, nothing is re-paged over the wire, and local block reads are zero. The valleys are therefore CPU-bound single-threaded work; the ingest parse/transpose of the next instrument (AoS -> SoA before sim fan-out) is the prime suspect. Claim (unverified): a profile of the valley phases would confirm the seam.Root cause (verified from the code, not the profile)
The thread oscillation traces to nested
run_indexed, the shareddisjoint-parallel core behind both
sweepandwalk_forward. Each entryindependently sizes its pool to
available_parallelism()(sweep.rs:381,mc.rs:127,walkforward.rs:189), and the campaign nests two of them:run_walk_forward_stagecallswalk_forward(exec.rs:988), which fans thewindows across
available_parallelism()workers (walkforward.rs:210);sweepfor the in-sample optimize (exec.rs:994),fanning the survivor points across
available_parallelism()again(
sweep.rs:394).The two pools multiply: on a 24-core box, ~24 windows x ~9 members ~= 216
concurrent threads — matching the observed 150-240 bursts. The 10-11 plateaus
are the flat sweep stage with no walk-forward above it (
exec.rs:653, members +main). The 1-thread valleys are genuinely serial seams — the cell loop runs the
42 cells sequentially (
exec.rs:132, C1) andr_bootstrapis single-threaded(
mc.rs:193) — not the fan-out. The process averaged ~1410% CPU (~14 of 24cores): ~42% of the box was idle, and that idle is the valleys, not the bursts.
Direction
Set by the maintainer, 2026-07-15: maximize throughput regardless of
dependency footprint — rayon accepted, the std-only-kernel constraint lifted
for this path. That moves the objective from "bound the fan-out" to "saturate
the cores", which makes the idle valleys — not the oversubscription bursts — the
primary target, since the idle is the larger loss.
Fix: one shared work-stealing pool over all disjoint work
std::thread::scopefan-out inrun_indexedwith asingle process-wide rayon pool. rayon's pool is persistent and global:
nested
par_iterenqueues into the same N workers instead of spawning freshOS threads, so the multiplicative oversubscription disappears and work-stealing
keeps every worker busy while any work remains. (This is the deadlock-free
nesting the earlier "shared pool risks deadlock" note misjudged — rayon steals
inner work rather than blocking on a join.)
exec.rs:132), which is C1-disjoint. This isthe main lever: with 42 cells' work in one pool, the per-cell serial seams
(the next instrument's ingest transpose, the single-threaded bootstrap) fill
with parallel work from other cells, so the cores stay saturated across the
seams that currently idle 23 of them.
Rationale: a single shared pool over the coarsest-to-finest disjoint work
(cells x members x windows x seeds) is the structure that saturates the cores
regardless of nesting shape — work-stealing balances flat across levels and
across the serial seams, which a per-level bounded pool cannot.
Constraints held: C1 determinism is non-negotiable — jobs stay pure and
disjoint, results are collected in input order, so runs remain bit-identical
across worker counts (the existing
_with_threads(1) == _with_threads(8)testsstay green, extended to the parallel cell loop). Parallel cell execution
multiplies the resident working set, so the count of concurrently-active cells
is bounded to a RAM budget — swap thrashing would be slower, i.e. the opposite
of the goal (this box: ~1.4 GB working set per instrument, 32 GB total).
Alternative held back
run_indexedlevel (std-only, no dependency, minimal). It removes the burstsbut leaves the valleys idle and under-utilizes when the outer level has few
jobs — insufficient for the saturate-the-cores objective. It was the right fix
for the narrower "bound the fan-out" framing, superseded by the maintainer's
maximize-throughput direction.
Scope
The shared pool, the parallel cell loop, and a reproducible local benchmark that
measures core utilization / wall time before and after (the real 42-cell
workload's data is external and not reproducible here). RAM-aware bounding of
concurrently-active cells is part of the pool work, not a separate track.
Recon: the parallel cell loop is a correctness rework — scope split
A working-set / thread-safety recon of the executor surfaced two facts that
reshape the scope of the maximize-throughput direction.
1. The FileCache resident set scales with concurrently-active distinct
symbols, not pool workers. The external
data-serverFileCache (pinned694f96f) has no capacity/LRU eviction — a symbol's parsed month-files stayresident until its refcount hits 0 (
cache.rsrelease_symbol/evict_symbol).Today the cell loop is sequential, so ~1 instrument's files are resident at a
time (each drops before the next opens). Parallel cells over distinct
instruments would hold K instruments' parsed files at once (~1.4 GB each here),
so a parallel cell loop needs a RAM budget bounding concurrently-active
instruments, not just a worker cap. The per-member SoA buffer itself is local
to one
run_memberand drops on return (bounded by worker count) — it is theFileCache retention, not the ring buffer, that drives RAM.
2. The Registry write path is not concurrency-safe.
Registry::append_familyis called once per cell per stage from inside the hot loop (
exec.rs:310-312)and does an unsynchronized
OpenOptions::append().open()+writeln!to ashared
families.jsonl(lineage.rs:263-288); its own doc states "single-processCLI invocations do not race". Parallel cells would interleave concurrent appends
to the same file — a correctness bug, not a perf issue. A registry-write fault
also escapes #272's per-cell fault isolation (it aborts the whole run via the
top-level
?,exec.rs:148-156).Confirmed harmless:
SilencedPanic(exec.rs:594-627) is alreadymulti-thread-correct (refcounted hook swap, no recursive lock, no deadlock) — a
parallel cell loop adds contention on its one global mutex but no correctness
hazard. No
thread_local!/RefCellanywhere in the hot path. The cell-loopaccumulators (
cells_out/cells_rec/nominees,exec.rs:118-123) aresequential-loop artifacts a parallel loop must rebuild with the established
Mutex<Vec>+ index-resort pattern (run_membersalready uses it for faults).Revised sequencing (derived)
The oversubscription fix and the valley fix separate cleanly, and the recon shows
the valley fix carries its own correctness work — so the direction lands as a
short sequence rather than one par_iter:
std::thread::scopefan-out inrun_indexedwith one process-wide rayon pool;nested
par_itershares the same N workers, so the multiplicative nesting(walk_forward windows x inner-sweep members => ~216 threads) collapses to
<= available_parallelism(). Self-contained in aura-engine, C1-preserving,
RAM-neutral (one cell = one symbol; the shared pool only reduces concurrent
threads vs today). This is also the prerequisite for any cell-loop parallelism —
without a shared pool, parallel cells would spawn worker-count x cell-count
threads.
by a failing test first (concurrent appends corrupt
families.jsonl). Theprerequisite for a parallel cell loop.
rebuild and a RAM budget bounding concurrently-active instruments. The valley
lever (the ~42% idle).
Basis: derived. The maximize-throughput direction (maintainer, 2026-07-15) is
unchanged; the recon just shows it is a sequence whose later steps carry
correctness work, and that the shared pool both stands alone (it fixes the
nesting oversubscription regardless) and is the prerequisite for the rest. The
benchmark accompanies each step. Follow-up issues will be filed as this pool
work lands.
Spec auto-signed (grounding-check PASS)
The design spec for the shared worker pool — replacing the per-call
std::thread::scopefan-out in the engine's disjoint-parallel core with oneprocess-wide rayon pool — was auto-signed under
/bosson agrounding-checkPASS, with no human in the loop. The PASS ratified every load-bearing assumption
about current behaviour against named, currently-green tests: the
run_indexedenumeration-order contract, the
_with_threads(1) == _with_threads(8) == publicdeterminism tests across sweep / monte_carlo / walk_forward, the
n == 0empty-vec contract, and the campaign
catch_unwindwrapping that keeps a memberpanic out of the pool. The rayon-library properties the change relies on
(order-preserving indexed collect, deadlock-free nested work-stealing, 1-worker
inline progress) are the new dependency's, out of scope for ratification — but
the retained
_with_threads(1)test goes red on a 1-worker nested-join deadlock,so even that risk sits under an existing green test kept green.
Design settled — ready for planning.
Delivered: the shared pool (commit b19eba6, branch worktree-issue-268-executor-pool)
run_indexednow drives one process-wide rayon pool instead of a per-callstd::thread::scopebatch. The publicsweep/monte_carlo/walk_forwardcallit on the ambient pool through a shared
assemble_*helper; the_with_threadsvariants are now thin, test-only (
#[cfg(test)]) wrappers running the sameassembly inside a local pool. The nested
walk_forward->sweepno longermultiplies OS-thread batches — nested
par_iterwork-steals within the pool, solive workers stay <=
available_parallelism().C1 held bit-for-bit: the
_with_threads(1) == _with_threads(8) == publicdeterminism tests stay green (now driving local rayon pools of 1 and N workers).
Added a nested-oversubscription regression test (peak concurrency <= pool size,
where the pre-rayon shape ran POOL^2) and an ignored wall-time benchmark.
Verified: aura-engine 276 passed, clippy -D warnings clean, workspace tests no
failures.
The two follow-ups the recon surfaced are filed:
unsynchronized
append_familyraces on a shared file). RED-first.concurrently-active instruments (the valley lever, the ~42% idle). Needs a
spec for the budget mechanic.
The pool is the first of three steps; the valleys — the main throughput win —
land with #276 then #277, both building on this branch once merged. Awaiting
review + merge.
Rebased onto the new main after #275 landed (main advanced
ad4249f→88667a6, the FF #275 merge). The branch was based on ad4249f; it now sits linearly on88667a6as commitb048923(wasb19eba6).The integration is clean: source files are disjoint from #275 (#268 touches only
aura-engine/src/{sweep,mc,walkforward}.rs+Cargo.toml; #275 touchedharness.rs/lib.rs/blueprint.rs/report.rs+ the CLI), the only shared file is the auto-regeneratedCargo.lock, andgit merge-treereported no conflicts.Re-verified green on the combined tree:
cargo build --workspaceclean,aura-engine283 passed / 0 failed, the 4 C1deterministic_across_thread_countstests green (the executor's_with_threads(1)==_with_threads(8)==publicinvariant holds over #275's reworked harness), the nested-oversubscription regression test green, full workspace suite no failures,clippy --workspace --all-targets -D warningsclean. The rayonF: Syncbound also type-proves that #275'sbind_sources/SourceSpec.rolepath introduced nothing non-Syncinto whatrun_onecloses over.The branch stays FF-mergeable onto current main. Still awaiting review + merge; #276 → #277 build on it once merged.