Parallelize the campaign cell loop for full core saturation #277
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Motivation
The #268 shared pool fixes the thread oversubscription (the 150-240-thread
bursts), but the larger loss is the idle valleys: a 42-cell campaign averaged
~1410% CPU of a possible 2400% — ~42% of the box idle. The idle is the
sequential cell loop (
crates/aura-campaign/src/exec.rs:132, cells C1-disjointbut run one at a time) and the single-threaded bootstrap. Parallelizing the
C1-disjoint cell loop over the #268 shared pool is the lever that fills the
valleys and saturates the cores — the maximize-throughput direction's main win.
Problem
The cell loop is a sequential 4-nested
forover strategy x instrument x windowx regime (
exec.rs:132-166). C1 makes the cells disjoint and concurrentlyexecutable. Parallelizing it on the shared rayon pool requires three pieces the
recon surfaced (all recorded on #268):
separately) — concurrent
append_familycurrently corrupts the store.data-serverFileCache retains a symbol's parsed month-files while anyreference is live (refcount eviction, no capacity cap;
cache.rsrelease_symbol/evict_symbol). Sequential today holds ~1 instrument resident;
parallel cells over distinct instruments would hold K at once (~1.4 GB each,
32 GB box). The budget must bound concurrently-active instruments, not just
pool workers — the per-member SoA buffers are pool-worker-bounded and drop on
return; it is the FileCache retention that scales with active instruments.
Swap thrashing is slower than sequential, so the bound is load-bearing.
cells_out/cells_rec/nominees(
exec.rs:118-123) are sequential-loop artifacts; a parallel loop needs theestablished
Mutex<Vec>+ index-resort pattern (run_membersalready uses itfor faults) to preserve doc-order and deterministic fault attribution (C1).
Needs a spec — the RAM-budget mechanic (how concurrently-active instruments are
bounded, the default, whether it is configurable) is the load-bearing design
choice.
Context: #268 (the shared pool — the prerequisite and the diagnosis); the recon
findings (FileCache retention, Registry no-lock, accumulator artifacts) are
recorded on #268.
Design reconciliation — the spec's load-bearing forks, resolved (2026-07-16)
Spec: parallel-cell-loop. The issue body lists the RAM-budget mechanic as the undesigned piece ("Needs a spec"); this records the resolution of that fork and its siblings before the spec is written. The run's branch is
worktree-issue-277-parallel-cell-loop.Fork: RAM-bound mechanic → instrument-major chunking. Options considered: (a) structural chunking — group the flattened cell list by instrument, walk chunks of K distinct instruments sequentially, run everything inside a chunk in parallel on the shared pool (cells × members × windows nest freely); (b) a counting semaphore gating cell starts per distinct instrument; (c) a capacity/LRU cap inside the external data-server FileCache. Derived: (a). Blocking inside rayon tasks (b) wastes workers while they wait and risks pool starvation; (c) is the wrong layer — live symbol guards make eviction-under-reference semantically impossible, and the FileCache is a separate pinned library. Chunking makes the bound a structural property of what can be in flight at all — the same make-it-structurally-impossible pattern the engine's causality contract uses — and costs only a chunk-boundary barrier that the surviving cells' own inner fan-out (members × windows) keeps filled. A sliding-window scheduler (start instrument N+1 the moment one finishes) was held back as unjustified complexity for the first landing; the chunk barrier is cheap for the same reason just given.
Fork: bound unit, surface, default → K bounds distinct concurrently-active instruments (the FileCache retention scales with those, not with pool workers); exposed as a CLI flag on
campaign runwith default 4; K=1 reproduces today's single-instrument residency while still parallelizing within the instrument. Not a campaign-document field: the campaign doc is content-addressed experiment identity, and a box-resource knob inside it would change the campaign id without changing the experiment. Default 4 is derived, not measured: ~1.4 GB per resident instrument on the reference dataset keeps 4 well inside a 16 GB box beside the ~3 GB process footprint, while already carrying 4 instruments' cells × members × windows of parallel work.Fork: fault routing for non-containable faults in a parallel loop → they stay run-fatal; the loop collects per-cell results, and after the join the fault with the lowest document-order cell index propagates (deterministic fault attribution — the same sort-by-index attribution the member-level fault capture already uses). An atomic abort flag stops later cells from starting once a fatal fault exists, saving wall clock. A registry-write fault stays in this class rather than being contained as a cell fault: it is an infrastructure failure, not a property of the cell, and recording it as a cell pathology would be false provenance — the per-cell containment introduced for member/window faults (a failed cell is recorded, never a global abort) continues to cover exactly the cell-owned pathologies. This resolves the fault-containment gap deferred from the registry-write-safety prerequisite (issues/276#issuecomment-3694): the gap was the mid-loop
?abort, which a parallel loop removes structurally; the class of the fault stays run-level.Fork: where the parallel primitive comes from → the campaign crate takes a direct rayon dependency and drives its own indexed fan-out. rayon's global pool is process-wide by construction, so campaign-level and engine-level fan-out share the one pool (the shared-pool structure the executor rework established) without the engine exporting a generic parallelism primitive — its public API stays domain verbs (sweep / monte_carlo / walk_forward). rayon is already a vetted workspace dependency.
Fork: accumulator rebuild → the four nested loops flatten into a pre-enumerated cell list carrying each cell's document-order index; results land in index-addressed slots; the ordered outputs (cell outcomes, realizations, per-(strategy, window, regime) nominee groups) are rebuilt sequentially from the slots after the join. Byte-identical outputs to the sequential loop across worker counts, per the determinism contract (C1).
Assumption to verify during implementation: per-cell family names in the registry store are unique per (cell, stage), so concurrent appends cannot interleave run-index assignment within one family name; the store's line order is not part of any read contract.
Status: design settled — ready for spec production.
Decision-log addendum — one more fork, surfaced by the spec's grounding review (2026-07-16).
Fork: family-name collision under duplicate campaign instruments → refuse duplicates at the campaign-validate seam. The independent grounding review of the spec draft found that
validate_campaigndoes not reject a campaign document listing the same instrument twice, while the per-cell registry family name embeds the raw instrument string — so two duplicate-instrument cells would share one family name and race its run-index assignment under a parallel cell loop, breaking byte-identical outputs (C1). Options considered: (a) refuse duplicate instruments at validation — a duplicate is a document error in its own right (it enumerates two identical cells: the same work twice, and an ambiguous per-instrument nominee group for the campaign-scope generalization); (b) embed a positional ordinal into the family name to make collisions structurally impossible — rejected because the family-name scheme is persisted store vocabulary with downstream readers, and reshaping it for an input that is never legitimate is the wrong trade. Derived: (a), with its own validation test. This makes the spec's family-name-uniqueness premise structural (all four cell axes plus the stage ordinal appear in the name, and instruments are then distinct by construction) instead of an unratified assumption.The grounding review also noted the FileCache eviction premise (per-reference retention in the external pinned data-server) has no in-workspace ratifying test; accepted as an externality — the residency-bound test the spec plans pins the structural instrument-count bound this repo owns, and the eviction behaviour is the external library's documented contract.
Spec auto-signed (2026-07-16). The spec for the parallel campaign cell loop (instrument-residency-bounded, deterministic outputs, run-fatal fault propagation, duplicate-instrument refusal,
--parallel-instrumentsflag) was signed autonomously: the signature is an independent fresh-context grounding review returning PASS — every load-bearing assumption about current codebase behaviour ratified against currently-green tests — not the producing side's confidence. A first grounding review returned BLOCK (the duplicate-instrument family-name race, logged in the decision-log addendum); the spec was repaired to specify the refusal and re-reviewed from scratch before signing. No human signed; the user was notified with a standing veto.Delivered: the campaign cell loop runs in parallel, RAM-bounded (branch
worktree-issue-277-parallel-cell-loop, five commits)Cells run concurrently on the process-global work-stealing pool the engine's member/window fan-out already shares, so cells × members × windows feed one scheduler and the per-cell serial seams (ingest transpose, single-threaded bootstrap) overlap with other cells' work — the fix for the ~42%-idle valleys measured on the 42-cell campaign. The residency bound is structural, per the decision log (issues/277#issuecomment-3703): the flattened, document-order-indexed cell list is grouped by instrument ordinal and walked in sequential chunks of K instrument groups, so no cell of an instrument outside the current chunk can start at all. K is
--parallel-instrumentsonaura campaign run(default 4; NonZeroUsize, 0 refused at parse time in domain prose; K=1 reproduces the sequential residency footprint; oversizing beyond the instrument count is safe). Results land in index-addressed slots and the ordered outputs are rebuilt in document order: outcomes are byte-identical across worker counts and bounds (C1), pinned by a determinism test (1-thread/K=1 vs 8-thread/K=2).Fault routing, restructured for a loop that cannot mid-iterate abort: member/window faults stay contained per cell (the established containment contract); non-containable faults — including registry writes, which are infrastructure, not cell pathology — latch an atomic abort flag and propagate as the lowest document-order fault after the join, with nothing partial persisted at run level. Duplicate campaign instruments are refused at both the validate tier and the executor's preflight (they would collide on registry family names and race the run-index assignment — the gap the spec's grounding review surfaced).
Cycle-close audit: substantively sound, four low/medium drift items all resolved in-cycle — the executor-side duplicate refusal (defense in depth, RED-first), a C1 realization note in the design ledger recording the parallel cell loop and its two scheduling-dependent fatal-path carve-outs, and a test pinning the zero-bound usage error.
Field test at the complex's close (the shared-pool → registry-safety → parallel-loop sequence, issues/268 / issues/276 / this issue): a downstream-consumer run over the public surface only — a 4-instrument campaign authored, validated, run and read back; determinism probed across
--parallel-instruments1/2/4/10 plus a same-K repeat (byte-identical modulo the run-counter suffix); duplicate refusal and faulted-cell isolation (exit 3, healthy cells persist) confirmed under the parallel loop. Five findings working, zero bugs; the two remaining findings (the zero-bound diagnostic leaked Rust type wording; the exit semantics of a gate-emptied cell were undocumented) are both resolved on the branch. Fixtures live in the tracked fieldtest corpus.Gates green throughout: workspace suite, clippy -D warnings, doc build. The branch awaits review + merge;
closes #277fires with it.Measured: the parallel cell loop is ~1.4× faster on a real workload, 95% box utilization
Benchmark at the user's request, after delivery (2026-07-16). Workload: the research project's real 4-instrument campaign (2 strategies × 4 instruments × 1 window × 1 risk = 8 cells; 9-point sweep + rolling walk-forward + 1000-resample bootstrap + generalize per cell), real archive data, run in a scratch copy of the project. Release builds, serial measurements on a quiet 24-core box, 1 discarded warmup + 2 measured runs per config (run-to-run spread <2%),
/usr/bin/time -v.ad4249f(sequential cells, oversubscribed fan-out)98ddcd3(shared pool, sequential cells)--parallel-instruments 1--parallel-instruments 2--parallel-instruments 4(default)Findings beyond the headline:
Limitations, honestly: 8 heterogeneous cells (the slowest cell gates the last chunk at K=4), cache-warm compute-bound runs (a cold NFS run would be I/O-bound), N=2 per config. The 1.42× is a lower bound for this workload shape, not a universal constant.
Bench artifacts (scratch project, per-run time/out files, driver script) are retained outside the repo for reproduction.