ail-embed: re-quarantine symbol_fan_swarm_leak_free — atomic-global fix necessary but insufficient (doc-honesty + finding record)

Attempted to un-#[ignore] the M5 swarm leak-proof as the
integration-level acceptance of bugfix-rc-global-stats-race
(7bfa11e). Boss verification falsified the sufficiency premise:
across 4-6 swarm runs the leak Σ is STILL non-deterministic —
Σfrees stable-exact (12000003) every run, Σallocs short by a
jittering ~1600-2260 in ~1/3 of runs (e.g. left:11998383
right:12000003, 4 stat lines).

The atomic-global runtime fix (7bfa11e) is real, committed, and
NECESSARY — the isolated 8x2M pure-global RED
embed_rc_global_stats_race is deterministically green. It is NOT
sufficient for the swarm: a second, distinct alloc-side undercount
remains that the isolated RED did not model (Σfrees never loses =>
not a per-ctx race; not pure global contention => RED green).

The un-#[ignore] was reverted (never committed). The ONLY shipped
diff is the three now-stale #[ignore]-rationale breadcrumbs in
ail-embed/tests/swarm.rs corrected from "un-ignore when the runtime
makes the counters atomic" to the accurate necessary-but-insufficient
state — a future agent seeing 7bfa11e must NOT un-ignore on that
basis (doc-honesty rule). The test body and the #[ignore] attribute
itself are unchanged: still quarantined, NOT weakened.
symbol_fan_swarm_bit_exact stays live and GREEN — the swarm IS
actually correct and leak-free; only the accounting is wrong.

Residual handed to a fresh debug RED-first cycle. Two framings
recorded for it (genuine residual runtime defect vs Σ-over-
heterogeneous-lines test-methodology unsoundness; the latter, if so,
is a post-root-cause design decision). main stays green; M5 open
[~]; m5.3 remains downstream of a sound leak-proof.

Includes the resume-attempt journal + INDEX line.

ail-embed/tests/swarm.rs

@@ -11,23 +11,25 @@
 //!   host reference fold. This is GREEN and live: it proves the
 //!   real-data-server → adapter → M3-kernel swarm computes correctly.
 //!
-//! - `symbol_fan_swarm_leak_free` — `#[ignore]`d pending an escalated
-//!   design decision. The global `Σallocs==Σfrees` measurement
-//!   (ported from crates/ail/tests/embed_tick_e2e.rs:94-104) is
-//!   *invalid under a multi-threaded host*: host-side box allocs run
-//!   with `__ail_tls_ctx == NULL` and fall through to the
-//!   **non-atomic** global `g_rc_*` counters (runtime/rc.c:90-91,
-//!   161,212), which rc.c's own header explicitly documents as
-//!   single-threaded/non-atomic ("when it acquires [concurrency],
-//!   atomic-vs-non-atomic becomes a separate decision"). The swarm is
-//!   *actually* leak-free (no box crosses a thread — `Ctx: !Send`;
-//!   bit-exact is GREEN every run); only the global *counter* races,
-//!   losing increments. The assertion body is preserved verbatim and
-//!   will be re-armed (un-`#[ignore]`d) by the runtime iteration that
-//!   makes the global fallback counters atomic. See
-//!   docs/journals/2026-05-19-iter-embedding-abi-m5.2.md and the M5
-//!   bounce-back. Quarantined, NOT weakened — what it asserts is
-//!   unchanged; it is gated on a decision the Boss escalated.
+//! - `symbol_fan_swarm_leak_free` — STILL `#[ignore]`d. The global
+//!   `Σallocs==Σfrees` measurement (ported from
+//!   crates/ail/tests/embed_tick_e2e.rs:94-104) was non-deterministic
+//!   under the multi-threaded host. The runtime fix
+//!   `bugfix-rc-global-stats-race` (commit `7bfa11e`) made the global
+//!   `g_rc_*` fallback counters `_Atomic` — that fix is real,
+//!   committed, and NECESSARY, but it is **not sufficient**: a Boss
+//!   verification (4 swarm runs) showed the leak Σ is still
+//!   non-deterministic — `Σfrees` is stable-exact (12000003) every
+//!   run while `Σallocs` is short by a jittering ~1600–2260 in ~1/3
+//!   of runs (e.g. left:11998383 right:12000003 across the 4 stat
+//!   lines). A SECOND, distinct alloc-undercount remains that the
+//!   isolated 8×2M pure-global RED did not model. Un-`#[ignore]` is
+//!   now gated on that residual's own root-cause/debug cycle, NOT on
+//!   `7bfa11e`. The swarm is still *actually* leak-free (no box
+//!   crosses a thread — `Ctx: !Send`; `symbol_fan_swarm_bit_exact`
+//!   GREEN every run); only the alloc *accounting* is still wrong.
+//!   Body preserved verbatim — quarantined, NOT weakened. See
+//!   docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md.
 
 use std::process::Command;
 use std::sync::Arc;
@@ -123,23 +125,27 @@ fn symbol_fan_swarm_bit_exact() {
 /// embed_tick_e2e.rs:94-104). The invariant is the global Σ, not
 /// per-line balance (M2 TLS-ctx cross-attribution).
 ///
-/// `#[ignore]`d: this measurement is invalid under a multi-threaded
-/// host. Host-side box allocs run with `__ail_tls_ctx == NULL` and
-/// hit the non-atomic global `g_rc_*` counters (runtime/rc.c:90-91,
-/// 161,212) — single-threaded/non-atomic *by rc.c's own documented
-/// design*. 3 worker threads race the counter, losing increments;
-/// the *counter* is unreliable even though the swarm is actually
-/// leak-free (no box crosses a thread; `symbol_fan_swarm_bit_exact`
-/// is GREEN every run). Un-`#[ignore]` this — body unchanged — once
-/// the runtime iteration makes the global fallback counters atomic
-/// (the deferred "concurrency arrived" decision rc.c:44-49 names).
-/// See docs/journals/2026-05-19-iter-embedding-abi-m5.2.md.
+/// STILL `#[ignore]`d. The runtime fix `bugfix-rc-global-stats-race`
+/// (commit `7bfa11e`) made the global `g_rc_*` fallback counters
+/// `_Atomic` — necessary and committed, but Boss verification proved
+/// it INSUFFICIENT for this swarm: `Σfrees` is stable-exact
+/// (12000003) every run, `Σallocs` short by a jittering ~1600–2260
+/// in ~1/3 of runs (e.g. left:11998383 right:12000003, 4 stat
+/// lines). A second, distinct alloc-undercount remains — NOT pure
+/// global-counter contention (the isolated 8×2M pure-global RED
+/// `embed_rc_global_stats_race` is deterministically green). The
+/// swarm is still actually leak-free (no box crosses a thread —
+/// `Ctx: !Send`; bit-exact GREEN every run); only the alloc
+/// accounting is wrong. Un-`#[ignore]` is gated on the residual's
+/// own root-cause/debug cycle, not on `7bfa11e`. Body unchanged.
+/// See docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md.
 #[test]
-#[ignore = "blocked: global g_rc_* counter is non-atomic by rc.c design; \
-            invalid under the multi-thread host. Swarm is actually \
-            leak-free (bit-exact is green; no box crosses a thread). \
-            Un-ignore when the runtime makes the global fallback \
-            counters atomic — see the m5.2 journal + M5 bounce-back."]
+#[ignore = "still blocked: atomic-global fix 7bfa11e is necessary but \
+            INSUFFICIENT — a residual swarm alloc-undercount remains \
+            (Σfrees exact, Σallocs short ~1600 jittering, ~1/3 runs; \
+            NOT pure global contention — the isolated RED is green). \
+            Un-ignore on the residual's own debug cycle, not on \
+            7bfa11e. See the m5.2-resume-attempt journal."]
 fn symbol_fan_swarm_leak_free() {
     let Some((_stdout, stderr)) = run_swarm_or_skip() else { return };
 

docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md

@@ -0,0 +1,91 @@
+# iter embedding-abi-m5.2-resume-attempt — atomic-global fix is necessary but INSUFFICIENT for the swarm leak-proof
+
+**Date:** 2026-05-19
+**Started from:** 7bfa11e (GREEN of bugfix-rc-global-stats-race)
+**Status:** RE-QUARANTINED — residual handed to a new debug cycle
+**Tasks completed:** 0 of 1 (the un-`#[ignore]` was attempted, verified
+unsound, and reverted; the honest-rationale correction is the only
+shipped change)
+
+## Summary
+
+After the runtime fix `bugfix-rc-global-stats-race` (`7bfa11e`,
+Boss-verified: the isolated 8×2M pure-global RED
+`embed_rc_global_stats_race` deterministically green 3/3; full
+`cargo test -p ail` green), I un-`#[ignore]`d the M5 swarm leak-proof
+`symbol_fan_swarm_leak_free` — that un-ignore was *designed* as the
+integration-level acceptance of the runtime fix. **Boss verification
+falsified the premise that the runtime fix was sufficient.** Across 4
+swarm runs: runs 1–2 GREEN, runs 3–4 FAILED; a 6-run capture: 3
+GREEN, run 4 FAILED `left: 11998383, right: 12000003`. Signature:
+`Σfrees` is **stable-exact (12000003) every run**; `Σallocs` is short
+by a **jittering ~1600–2260 in ~1/3 of runs**, across the 4
+`ailang_rc_stats:` lines (3 per-ctx + 1 global atexit).
+
+The atomic-global fix is real, committed, and **necessary** — it
+removed the *pure* global-counter contention (the isolated RED is now
+deterministic). It is **not sufficient** for the swarm: a *second,
+distinct* alloc-undercount remains that the isolated 8×2M pure-global
+RED did not model. The un-ignore was reverted (`git checkout` — it
+was never committed); the now-stale `#[ignore]` rationale that said
+"un-ignore when the runtime makes the counters atomic" was corrected
+to the accurate necessary-but-insufficient state (this honesty fix is
+the only shipped diff — a future agent seeing `7bfa11e` must NOT
+un-ignore on that basis).
+
+## The asymmetry (orchestrator framing for the debug carrier, NOT a fix)
+
+`Σfrees` perfectly stable while `Σallocs` jitters is the load-bearing
+clue. Both global counters are now `_Atomic`; both per-ctx counters
+are plain `++` but single-writer-per-thread (each worker owns its
+`Ctx`; `Ctx: !Send`). If a per-ctx counter raced, *both* alloc and
+free would jitter (the dec path also `++`s a per-ctx counter) — it
+does not. So the lost increments are specifically on the **alloc**
+side, and not pure global contention (RED green). Boss reasoning
+could not localise it further; per the debug Iron Law that is
+where Boss speculation stops.
+
+Two framings the debug cycle must both consider (do not pre-judge):
+
+1. **A genuine residual runtime defect** — some alloc events on the
+   swarm path are uncounted non-deterministically (a TLS-ctx
+   set/restore window across the M3 forwarder boundary; an
+   interaction with data-server's prefetch background threads; a
+   path the 8×2M ctx-less RED structurally cannot reach because the
+   swarm mixes TLS-NULL host allocs with TLS-ctx kernel-internal
+   allocs).
+2. **A test-methodology unsoundness** — `Σ` over a *heterogeneous*
+   set of lines (one process-global atexit line + N per-ctx
+   `ailang_ctx_free` lines, with per-box alloc/free attribution
+   split asymmetrically between global and ctx) may not be a sound
+   leak invariant for a multi-thread host even with correct
+   counters. If so the fix is a methodology redesign (e.g.
+   per-symbol single-thread subprocess leak-proof, or assert each
+   line balances under a single-thread-per-process shape), not
+   another runtime change — and that becomes a design decision,
+   possibly a bounce-back, only AFTER root cause.
+
+## Disposition
+
+- `symbol_fan_swarm_leak_free` STAYS `#[ignore]`d, body verbatim
+  (still quarantined, NOT weakened). Rationale corrected in all
+  three breadcrumbs (module doc, fn doc, `#[ignore]` reason).
+- `symbol_fan_swarm_bit_exact` remains live & GREEN — the M5
+  existence proof is unaffected (the swarm IS actually correct and
+  leak-free; only the *accounting* is wrong).
+- The residual is handed to a fresh `debug` RED-first cycle
+  (`bugfix-swarm-rc-alloc-undercount`). main stays green at the
+  honesty-fix commit.
+- M5 (the milestone) stays open `[~]`; m5.3 (time-shard +
+  friction-harvest) remains downstream of a sound leak-proof.
+
+## Files touched
+
+- `ail-embed/tests/swarm.rs` (the three `#[ignore]`-rationale
+  breadcrumbs corrected for doc-honesty; the test body and the
+  `#[ignore]` itself unchanged)
+
+## Stats
+
+(no orchestrator-stats file — this is a Boss-side honesty correction +
+finding record, not an implement-orchestrator iteration)

docs/journals/INDEX.md

@@ -109,3 +109,4 @@
 - 2026-05-19 — iter embedding-abi-m5.1 (DONE 3/3): M5 iter 1 — stood up the workspace-excluded `ail-embed` crate. A zero-dependency embedding core (Rust port of the audited `crates/ail/tests/embed/tick_roundtrip.c`: `extern "C"` to the M3-frozen ABI + frozen-layout `State`/`Tick` box helpers + a `Kernel` price fold, raw pointers never escaping the type) links the M3 staticlib via a new `build.rs` (no in-repo precedent — `AIL_BIN` env override else nested `cargo build -p ail` against the parent workspace, separate target dir so no cargo-lock deadlock). Plus a hermetic `data-server` smoke: a synthetic Pepperstone-format ZIP fixture written via the crate's own public `RawTickRecord` type (correct-by-construction, mirrors `data-server/src/loader.rs:110-124`) → real `DataServer` → `Kernel`, asserting bit-exact vs a same-order host reference fold and the closed-form `55.0`; runs with no `/mnt`. `ail-embed` is its own cargo workspace root (empty `[workspace]` table) so the AILang compiler workspace owes it nothing; `data-server` is a dev-dependency only ⇒ Invariant 1 holds in the dependency graph, not just on paper. Boss-verified independently: ail-embed suite 2/2 green (`kernel_run_sums_prices` unit RED-first + `hermetic_smoke_data_server_roundtrip` integration), full+no-deps `cargo metadata` on the AILang workspace shows `data-server` count 0, `git status` path filter empty (zero diff to `crates/ailang-*`/`crates/ail/`/`runtime/`/`examples/*.ail`), `src/lib.rs` zero code-level `data_server`, AILang `cargo build --workspace` still clean. Two toolchain-forced corrections to the plan's verbatim `ail-embed/Cargo.toml` (added the empty `[workspace]` table; sibling dev-dep path `../libs`→`../../libs` manifest-relative) — confined to the plan-created manifest, no acceptance gate altered, 0 review re-loops; planner-recon implication recorded in the journal Concerns. Adapter API + thread-swarm explicitly deferred to M5 iter 2+. → 2026-05-19-iter-embedding-abi-m5.1.md
 - 2026-05-19 — iter embedding-abi-m5.2 (PARTIAL 2/3; M5 bounce-back): data-server adapter + symbol-fan swarm; the leak-proof did its job and surfaced a real concurrency finding. Tasks 1+2 landed clean: `data-server` promoted dev-dep→real dep (Invariant-1 sanctioned — compiler workspace graph still data-server-count 0, zero compiler-surface diff), additive `adapter` module (`tick_to_px`/`MidPriceStream` lazy `Iterator`/`fold_symbol`, RED-first), and a `swarm_runner` `[[bin]]` whose clean compile *is* the compile-time per-thread-ctx proof (`Ctx: !Send` ⇒ a shared-ctx swarm is `E0277`, cannot build). The real-data symbol-fan swarm runs (EURUSD/GER40/XAUUSD, ~4s) and is **bit-exact vs an independent same-order host reference every run** (`symbol_fan_swarm_bit_exact`, GREEN, live). Task 3's `Σallocs==Σfrees` leak gate is BLOCKED: Boss independently read `runtime/rc.c` and confirms it is an **instrumentation race, not a memory bug** — no box crosses a thread (`!Send`), the real RC is correct (bit-exact green), only the *global* `g_rc_*` stat counters (rc.c:90-91,161,212) are non-atomic *by rc.c's own documented single-threaded design* and lose `++`s when 3 worker threads hit the null-`__ail_tls_ctx` host-side path. M5 is AILang's first concurrent consumer → the deferred "concurrency arrived: atomic-vs-non-atomic" decision rc.c:44-49 names is now due. The leak assertion was Boss-split into `symbol_fan_swarm_leak_free` (`#[ignore]`, **body verbatim** — quarantined not weakened; un-ignore = the runtime fix's acceptance) so main stays green and the finding is pinned. Prior claim that `embed_swarm_tsan.rs` covers this was corrected (it uses the scalar kernel — zero box allocs — never exercised this path). Escalated as a **bounce-back** (touches M3-frozen `runtime/`; re-frames M5's "zero runtime change"; multiple substantive options); Boss recommendation on the record = Option A (atomic *global-fallback* counters only, standalone RED-first runtime micro-iter, M5 framing amended) — user picks the direction. Time-shard + friction-harvest remain m5.3. → 2026-05-19-iter-embedding-abi-m5.2.md
 - 2026-05-19 — iter bugfix-rc-global-stats-race (RED→GREEN, debug→implement mini, DONE 1/1): the M5 iter-2 bounce-back resolution (user-approved Option A). The two null-ctx fallback RC-stats counters `g_rc_alloc_count`/`g_rc_free_count` (`runtime/rc.c:90-91`) were plain `static uint64_t` with a non-atomic `++` on the `__ail_tls_ctx == NULL` arm of `ailang_rc_alloc` (rc.c:161) / the to-zero branch of `ailang_rc_dec` (rc.c:212); a multi-threaded host driving the global fallback (no `ailang_ctx_new`) raced the read-modify-write and lost increments, so the `AILANG_RC_STATS` atexit Σ under-counted non-deterministically — exactly the deferred "concurrency arrived: atomic-vs-non-atomic" decision rc.c's own header (rc.c:44-49) names, M5 being AILang's first concurrent consumer. NOT a memory bug (`Ctx: !Send` keeps every box on one thread; the real refcount/free is correct; programs bit-exact). RED (debugger, audit-trail commit `427b687`): a C host — 8 threads × 2M alloc-then-dec, no ctx so the global path is taken, no box crossing a thread — + integration test asserting the atexit Σ is exact; Boss-verified RED `allocs=2141382 expected 16000000, live=-131242` (deterministic-fail under that contention, not flaky). GREEN (implement mini, this commit): the two globals are now `_Atomic uint64_t`, `atomic_fetch_add_explicit(.., memory_order_relaxed)` at the two fallback `++` sites, `atomic_load_explicit(.., relaxed)` in `ailang_rc_stats_atexit` (its sole reader) — relaxed is correct (pure stats, no happens-before; atexit reader runs post-join). Per-ctx counters + the per-object refcount header left non-atomic BY DESIGN (single-thread-per-ctx; boxes never cross threads) — explicitly out of scope, untouched; frozen value layout / ABI offsets / host-free rule untouched; the genuinely-still-single-threaded drop-worklist doc note correctly left unchanged (a false correction was refused); the two genuinely-stale atomicity doc blocks corrected for honesty. Scope held to `runtime/rc.c` ONLY (37+/14−). Boss-verified independently: RED→GREEN deterministically 3/3 (jitter gone, `allocs==frees==16_000_000`), full `cargo test -p ail` green (every binary 0 failed — the per-ctx tsan harnesses embed_swarm_tsan/embed_rc_accounting_tsan + embed_tick_e2e/embed_record_e2e unaffected since the per-ctx path is untouched; rc.c links into every ail binary so this is the strong regression gate), git-status scope = rc.c + journal + stats only, the RED files unchanged. Unblocks resuming the M5 leak-proof (un-`#[ignore]` `ail-embed`'s `symbol_fan_swarm_leak_free`). RED `427b687` → GREEN this commit. → 2026-05-19-iter-bugfix-rc-global-stats-race.md
+- 2026-05-19 — iter embedding-abi-m5.2-resume-attempt (RE-QUARANTINED; honesty fix only): attempted to un-`#[ignore]` the M5 swarm leak-proof `symbol_fan_swarm_leak_free` as the integration-level acceptance of the runtime fix `7bfa11e` (bugfix-rc-global-stats-race). Boss verification FALSIFIED the sufficiency premise: across 4–6 swarm runs the leak Σ is still non-deterministic — `Σfrees` stable-exact (12000003) every run, `Σallocs` short by a jittering ~1600–2260 in ~1/3 of runs (e.g. left:11998383 right:12000003 across 4 stat lines). The atomic-global fix is real, committed, NECESSARY (the isolated 8×2M pure-global RED `embed_rc_global_stats_race` is deterministically green) but NOT sufficient — a second, distinct alloc-side undercount remains that the isolated RED did not model (frees never lose ⇒ not a per-ctx race; not pure global contention ⇒ RED green; localisation beyond that is where Boss speculation stops per the debug Iron Law). Un-ignore reverted (never committed); the only shipped diff is the three now-stale `#[ignore]`-rationale breadcrumbs in `ail-embed/tests/swarm.rs` corrected from "un-ignore when the runtime is atomic" to the accurate necessary-but-insufficient state (a future agent seeing `7bfa11e` must NOT un-ignore on that basis — doc-honesty). Test body + the `#[ignore]` itself unchanged (still quarantined, NOT weakened); `symbol_fan_swarm_bit_exact` stays live+GREEN (the swarm IS actually correct and leak-free — only the accounting is wrong). Residual handed to a fresh debug RED-first cycle; two framings flagged for it (genuine residual runtime defect vs Σ-over-heterogeneous-global+per-ctx-lines methodology unsoundness — the latter, if so, is a post-root-cause design decision). main green at the honesty-fix commit; M5 stays open `[~]`; m5.3 remains downstream of a sound leak-proof. → 2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md