feat(campaign): parallel cell loop on the shared rayon pool

The flip: within each K-instrument chunk (the structural residency bound
from the previous commit), all cells run concurrently on rayon's process-
global pool — the same pool the engine's member/window fan-out already
shares, so cells x members x windows feed one work-stealing scheduler and
the per-cell serial seams (ingest transpose, single-threaded bootstrap)
overlap with other cells' work. Results land in index-addressed slots and
the ordered outputs are rebuilt in document order: byte-identical outcomes
across worker counts and bounds (C1).

Fault routing, restructured for a loop that cannot mid-iterate abort:
member/window faults stay #272-contained inside run_cell; non-containable
faults (incl. registry writes — infrastructure, not cell pathology) latch
an atomic abort flag so unstarted cells never run, and after the join the
lowest document-order fault among completed cells propagates as the run's
error. Nothing partial is persisted at run level on the fatal path.

Property pins: determinism across thread counts and bounds (1-thread/K=1
== 8-thread/K=2), distinct live instruments never exceed K (refcounting
stub runner, 8-thread pool), a dead family store is run-fatal with no
partial campaign-run record, member faults stay contained under the
parallel loop. E2E fixtures (previous commit) pin the CLI prose and
document-order persistence across K.

Verified: workspace suite green, clippy -D warnings clean. rayon enters
aura-campaign under the amended C16 per-case policy (comment in
Cargo.toml, mirroring aura-engine).

closes #277
This commit is contained in:
2026-07-16 15:10:31 +02:00
parent 7fa3ef4e26
commit cf94377f30
5 changed files with 347 additions and 20 deletions
+54
View File
@@ -1727,6 +1727,60 @@ fn campaign_run_synthetic_e2e_cell_order_is_document_order_under_parallel_instru
}
}
/// #277 CLI-level containment under REAL cross-cell parallelism: with
/// `--parallel-instruments 2` both instruments land in the same chunk and run
/// genuinely concurrently on the ambient rayon pool (not the crate-level
/// `execute()` tests' `FakeRunner`, which never touches the real engine or
/// the CLI's exit-code/record-presentation layer). The fixture picks a window
/// that lies inside SYMA's synthetic span (2024-01..08) but strictly before
/// SYMB's (2024-03..06 — SYMB has no data in January), so SYMA's cell
/// succeeds while SYMB's cell hits a real per-cell data fault. A regression
/// that let the parallel chunk walk leak a fault out of `contain` (aborting
/// the whole run) or drop/misplace a completed cell would show up here as the
/// wrong exit code, a missing cell, or a fault on the wrong instrument.
/// Expected: exit 3 ("completed with failed cells"), both cells present in
/// document order, SYMA's carries no fault, SYMB's does.
#[test]
fn campaign_run_synthetic_e2e_parallel_instruments_contains_a_real_per_cell_fault() {
let (dir, _fixture) = fresh_project_with_data();
let runs_dir = dir.join("runs");
std::fs::remove_dir_all(&runs_dir).ok();
let _cleanup = ScratchGuard(vec![
ScratchPath::Dir(runs_dir.clone()),
ScratchPath::File(dir.join("parfault.process.json")),
ScratchPath::File(dir.join("parfault.campaign.json")),
]);
let bp_id = seed_blueprint(&dir, "campaign-run-parfault-seed");
let proc_id = register_process_doc(&dir, "parfault.process.json", SWEEP_ONLY_PROCESS_DOC);
// 2024-01-10 .. 2024-01-20: inside SYMA's Jan-Aug span, strictly before
// SYMB's Mar-Jun span.
let doc = campaign_doc_json_two(
("SYMA", "SYMB"),
"",
&bp_id,
&proc_id,
(1704844800000, 1705708800000),
);
write_doc(&dir, "parfault.campaign.json", &doc);
let (out, code) = run_code_in(
&dir,
&["campaign", "run", "parfault.campaign.json", "--parallel-instruments", "2"],
);
assert_eq!(code, Some(3), "one contained fault, one success: {out}");
let line = out
.lines()
.find(|l| l.starts_with("{\"campaign_run\":"))
.expect("the always-on final campaign_run line");
let v: serde_json::Value = serde_json::from_str(line).expect("record parses");
let cells = v["campaign_run"]["cells"].as_array().expect("cells array");
assert_eq!(cells.len(), 2, "one cell per instrument: {line}");
assert_eq!(cells[0]["instrument"].as_str(), Some("SYMA"), "doc order preserved: {line}");
assert_eq!(cells[1]["instrument"].as_str(), Some("SYMB"), "doc order preserved: {line}");
assert!(cells[0]["fault"].is_null(), "SYMA has data in January, its cell succeeds: {line}");
assert!(!cells[1]["fault"].is_null(), "SYMB has no January data, its cell is faulted: {line}");
}
/// The v2 boundary, campaign side: a generalize-bearing process is an
/// executable shape, but `std::generalize` needs >= 2 instruments in the
/// campaign — a STATIC preflight fact (the referential gate has run, no data