feat(campaign): parallel cell loop on the shared rayon pool
The flip: within each K-instrument chunk (the structural residency bound from the previous commit), all cells run concurrently on rayon's process- global pool — the same pool the engine's member/window fan-out already shares, so cells x members x windows feed one work-stealing scheduler and the per-cell serial seams (ingest transpose, single-threaded bootstrap) overlap with other cells' work. Results land in index-addressed slots and the ordered outputs are rebuilt in document order: byte-identical outcomes across worker counts and bounds (C1). Fault routing, restructured for a loop that cannot mid-iterate abort: member/window faults stay #272-contained inside run_cell; non-containable faults (incl. registry writes — infrastructure, not cell pathology) latch an atomic abort flag so unstarted cells never run, and after the join the lowest document-order fault among completed cells propagates as the run's error. Nothing partial is persisted at run level on the fatal path. Property pins: determinism across thread counts and bounds (1-thread/K=1 == 8-thread/K=2), distinct live instruments never exceed K (refcounting stub runner, 8-thread pool), a dead family store is run-fatal with no partial campaign-run record, member faults stay contained under the parallel loop. E2E fixtures (previous commit) pin the CLI prose and document-order persistence across K. Verified: workspace suite green, clippy -D warnings clean. rayon enters aura-campaign under the amended C16 per-case policy (comment in Cargo.toml, mirroring aura-engine). closes #277
This commit is contained in:
@@ -1727,6 +1727,60 @@ fn campaign_run_synthetic_e2e_cell_order_is_document_order_under_parallel_instru
|
||||
}
|
||||
}
|
||||
|
||||
/// #277 CLI-level containment under REAL cross-cell parallelism: with
|
||||
/// `--parallel-instruments 2` both instruments land in the same chunk and run
|
||||
/// genuinely concurrently on the ambient rayon pool (not the crate-level
|
||||
/// `execute()` tests' `FakeRunner`, which never touches the real engine or
|
||||
/// the CLI's exit-code/record-presentation layer). The fixture picks a window
|
||||
/// that lies inside SYMA's synthetic span (2024-01..08) but strictly before
|
||||
/// SYMB's (2024-03..06 — SYMB has no data in January), so SYMA's cell
|
||||
/// succeeds while SYMB's cell hits a real per-cell data fault. A regression
|
||||
/// that let the parallel chunk walk leak a fault out of `contain` (aborting
|
||||
/// the whole run) or drop/misplace a completed cell would show up here as the
|
||||
/// wrong exit code, a missing cell, or a fault on the wrong instrument.
|
||||
/// Expected: exit 3 ("completed with failed cells"), both cells present in
|
||||
/// document order, SYMA's carries no fault, SYMB's does.
|
||||
#[test]
|
||||
fn campaign_run_synthetic_e2e_parallel_instruments_contains_a_real_per_cell_fault() {
|
||||
let (dir, _fixture) = fresh_project_with_data();
|
||||
let runs_dir = dir.join("runs");
|
||||
std::fs::remove_dir_all(&runs_dir).ok();
|
||||
let _cleanup = ScratchGuard(vec![
|
||||
ScratchPath::Dir(runs_dir.clone()),
|
||||
ScratchPath::File(dir.join("parfault.process.json")),
|
||||
ScratchPath::File(dir.join("parfault.campaign.json")),
|
||||
]);
|
||||
let bp_id = seed_blueprint(&dir, "campaign-run-parfault-seed");
|
||||
let proc_id = register_process_doc(&dir, "parfault.process.json", SWEEP_ONLY_PROCESS_DOC);
|
||||
// 2024-01-10 .. 2024-01-20: inside SYMA's Jan-Aug span, strictly before
|
||||
// SYMB's Mar-Jun span.
|
||||
let doc = campaign_doc_json_two(
|
||||
("SYMA", "SYMB"),
|
||||
"",
|
||||
&bp_id,
|
||||
&proc_id,
|
||||
(1704844800000, 1705708800000),
|
||||
);
|
||||
write_doc(&dir, "parfault.campaign.json", &doc);
|
||||
|
||||
let (out, code) = run_code_in(
|
||||
&dir,
|
||||
&["campaign", "run", "parfault.campaign.json", "--parallel-instruments", "2"],
|
||||
);
|
||||
assert_eq!(code, Some(3), "one contained fault, one success: {out}");
|
||||
let line = out
|
||||
.lines()
|
||||
.find(|l| l.starts_with("{\"campaign_run\":"))
|
||||
.expect("the always-on final campaign_run line");
|
||||
let v: serde_json::Value = serde_json::from_str(line).expect("record parses");
|
||||
let cells = v["campaign_run"]["cells"].as_array().expect("cells array");
|
||||
assert_eq!(cells.len(), 2, "one cell per instrument: {line}");
|
||||
assert_eq!(cells[0]["instrument"].as_str(), Some("SYMA"), "doc order preserved: {line}");
|
||||
assert_eq!(cells[1]["instrument"].as_str(), Some("SYMB"), "doc order preserved: {line}");
|
||||
assert!(cells[0]["fault"].is_null(), "SYMA has data in January, its cell succeeds: {line}");
|
||||
assert!(!cells[1]["fault"].is_null(), "SYMB has no January data, its cell is faulted: {line}");
|
||||
}
|
||||
|
||||
/// The v2 boundary, campaign side: a generalize-bearing process is an
|
||||
/// executable shape, but `std::generalize` needs >= 2 instruments in the
|
||||
/// campaign — a STATIC preflight fact (the referential gate has run, no data
|
||||
|
||||
Reference in New Issue
Block a user