Make the Registry write path concurrency-safe for a parallel cell loop #276
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Motivation
Parallelizing the campaign cell loop (the #268 follow-up, for core saturation)
requires the Registry write path to tolerate concurrent writers. It does not
today, so this is a prerequisite for that work.
Problem
Registry::append_familyis called once per cell per stage from inside theexecutor's hot loop (
crates/aura-campaign/src/exec.rs:310) and does anunsynchronized
OpenOptions::new().append().open()+writeln!to a sharedfamilies.jsonl(crates/aura-registry/src/lineage.rs:263-288). Its own docstates "single-process CLI invocations do not race" — concurrent writers were
never designed for. A parallel cell loop would interleave concurrent appends to
the same file, corrupting the store.
append_campaign_run(
lineage.rs:322-340) shares the identical no-lock pattern.A registry-write fault also escapes #272's per-cell fault isolation: it
propagates as
ExecFault::Registryvia the top-level?inexecute's loop(
exec.rs:148-156), aborting the whole run rather than being contained to thecell — a second gap a parallel loop makes reachable.
Pin it RED-first (a test driving concurrent
append_familyand asserting thestore stays well-formed / every line is intact), then add write synchronization
(a mutex, an advisory file lock, or a per-cell buffer merged serially at the
loop boundary).
Context: #268 (the shared worker pool — the first step of the maximize-throughput
direction); the working-set / thread-safety recon that surfaced this is recorded
on #268.
Decision log — work started (2026-07-16; the run's branch is
worktree-issue-276-registry-write-safety).Entry shape. The behaviour is pinned RED-first as one executable spec, as the issue prescribes:
concurrent_appends_keep_the_family_store_well_formed(crates/aura-registry/src/lib.rs) drives 8 threads × 50 appends × 4 members through one sharedRegistryhandle and asserts the store stays well-formed — parseable AND complete (successful appends × members lines). Verified failing deterministically (3/3 runs, a "trailing characters" merged-line parse error — two records on one physical line, exactly the corruption the issue predicts).Fork decided: test sharing topology. Options: one shared handle across threads, or one handle per thread. Chosen: one shared handle — it models the parallel cell loop's actual caller shape (the executor hands each cell the same registry reference) and is the least constraining on the fix, admitting all three candidate mechanisms the issue names (internal mutex, advisory file lock, serial per-cell merge), where a handle-per-thread topology would silently rule out an in-process mutex.
Fork decided: synchronization scope stays in-process. The write path is made safe for concurrent writers within one process (the type carries its own synchronization); cross-process concurrency stays out of contract, as the module doc already states — the threatened caller is the in-process parallel cell loop, and widening the contract to multi-process coordination is a separate feature, not this prerequisite.
Scope decided: both append functions, no fault-routing change. The deliverable covers
append_familyandappend_campaign_run(the issue's Problem section names both as the identical pattern). The fault-containment gap also noted there (a registry-write fault escaping per-cell isolation) is deliberately NOT part of this issue's delivery: containment is a property of the executor loop's fault routing, which the parallel-loop work restructures anyway, while write synchronization is a property of the registry type itself.Status: headline behaviour pinned as a failing executable spec; the green side is next.
Delivered: the Registry write path is concurrency-safe (branch
worktree-issue-276-registry-write-safety, three commits)All three JSONL append paths —
append_family(families.jsonl),append_campaign_run(campaign_runs.jsonl), and the flatappend(runs.jsonl) — are serialized through one internalappend_write_lock: Mutex<()>onRegistry, held across each append's full read-the-run-counter-then-write critical section. That fixes both failure shapes concurrent writers could produce: torn/merged lines (two records interleaved onto one physical line) and double-assigned run indices from the read-before-write counter derivation. A poisoned lock is entered anyway (PoisonError::into_inner): store lines are independent, so one panicked writer must not wedge later appends. In-process only, per the decision log (issues/276#issuecomment-3694); the doc comments now state the actual guarantee instead of the old "single-process CLI invocations do not race" disclaimer.RED-first throughout: three property tests (one per store) each drive 8 threads × 50 appends through one shared
&Registryand assert the store parses with every successful append's record intact — each was verified failing against the unguarded path (deterministic "trailing characters" merged-line corruption) before its fix landed.Cycle-close audit (architect review): C1 holds — the lock sits at the results-persistence edge, outside the lock-free sim event loop; the mechanism is correct (no re-entrancy via the load paths). The one drift item it surfaced — the flat
appendstanding unguarded beside its two fixed siblings, while being public API with external consumers — was resolved within the cycle on the fix path.put_blueprint/put_docandTraceStore::writeare race-tolerant by shape and correctly outside the append scope. Gates green: workspace suite, clippy -D warnings, doc build.The fault-containment gap noted in the issue body (a registry-write fault escaping per-cell isolation) stays deliberately out of this delivery, per the decision log — it belongs to the parallel-loop fault-routing restructure (#277).
The branch awaits review + merge; #277 (the parallel cell loop) builds on it once merged.