diff --git a/ail-embed/tests/swarm.rs b/ail-embed/tests/swarm.rs index 4cdf41e..db21a15 100644 --- a/ail-embed/tests/swarm.rs +++ b/ail-embed/tests/swarm.rs @@ -11,23 +11,25 @@ //! host reference fold. This is GREEN and live: it proves the //! real-data-server → adapter → M3-kernel swarm computes correctly. //! -//! - `symbol_fan_swarm_leak_free` — `#[ignore]`d pending an escalated -//! design decision. The global `Σallocs==Σfrees` measurement -//! (ported from crates/ail/tests/embed_tick_e2e.rs:94-104) is -//! *invalid under a multi-threaded host*: host-side box allocs run -//! with `__ail_tls_ctx == NULL` and fall through to the -//! **non-atomic** global `g_rc_*` counters (runtime/rc.c:90-91, -//! 161,212), which rc.c's own header explicitly documents as -//! single-threaded/non-atomic ("when it acquires [concurrency], -//! atomic-vs-non-atomic becomes a separate decision"). The swarm is -//! *actually* leak-free (no box crosses a thread — `Ctx: !Send`; -//! bit-exact is GREEN every run); only the global *counter* races, -//! losing increments. The assertion body is preserved verbatim and -//! will be re-armed (un-`#[ignore]`d) by the runtime iteration that -//! makes the global fallback counters atomic. See -//! docs/journals/2026-05-19-iter-embedding-abi-m5.2.md and the M5 -//! bounce-back. Quarantined, NOT weakened — what it asserts is -//! unchanged; it is gated on a decision the Boss escalated. +//! - `symbol_fan_swarm_leak_free` — STILL `#[ignore]`d. The global +//! `Σallocs==Σfrees` measurement (ported from +//! crates/ail/tests/embed_tick_e2e.rs:94-104) was non-deterministic +//! under the multi-threaded host. The runtime fix +//! `bugfix-rc-global-stats-race` (commit `7bfa11e`) made the global +//! `g_rc_*` fallback counters `_Atomic` — that fix is real, +//! committed, and NECESSARY, but it is **not sufficient**: a Boss +//! verification (4 swarm runs) showed the leak Σ is still +//! non-deterministic — `Σfrees` is stable-exact (12000003) every +//! run while `Σallocs` is short by a jittering ~1600–2260 in ~1/3 +//! of runs (e.g. left:11998383 right:12000003 across the 4 stat +//! lines). A SECOND, distinct alloc-undercount remains that the +//! isolated 8×2M pure-global RED did not model. Un-`#[ignore]` is +//! now gated on that residual's own root-cause/debug cycle, NOT on +//! `7bfa11e`. The swarm is still *actually* leak-free (no box +//! crosses a thread — `Ctx: !Send`; `symbol_fan_swarm_bit_exact` +//! GREEN every run); only the alloc *accounting* is still wrong. +//! Body preserved verbatim — quarantined, NOT weakened. See +//! docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md. use std::process::Command; use std::sync::Arc; @@ -123,23 +125,27 @@ fn symbol_fan_swarm_bit_exact() { /// embed_tick_e2e.rs:94-104). The invariant is the global Σ, not /// per-line balance (M2 TLS-ctx cross-attribution). /// -/// `#[ignore]`d: this measurement is invalid under a multi-threaded -/// host. Host-side box allocs run with `__ail_tls_ctx == NULL` and -/// hit the non-atomic global `g_rc_*` counters (runtime/rc.c:90-91, -/// 161,212) — single-threaded/non-atomic *by rc.c's own documented -/// design*. 3 worker threads race the counter, losing increments; -/// the *counter* is unreliable even though the swarm is actually -/// leak-free (no box crosses a thread; `symbol_fan_swarm_bit_exact` -/// is GREEN every run). Un-`#[ignore]` this — body unchanged — once -/// the runtime iteration makes the global fallback counters atomic -/// (the deferred "concurrency arrived" decision rc.c:44-49 names). -/// See docs/journals/2026-05-19-iter-embedding-abi-m5.2.md. +/// STILL `#[ignore]`d. The runtime fix `bugfix-rc-global-stats-race` +/// (commit `7bfa11e`) made the global `g_rc_*` fallback counters +/// `_Atomic` — necessary and committed, but Boss verification proved +/// it INSUFFICIENT for this swarm: `Σfrees` is stable-exact +/// (12000003) every run, `Σallocs` short by a jittering ~1600–2260 +/// in ~1/3 of runs (e.g. left:11998383 right:12000003, 4 stat +/// lines). A second, distinct alloc-undercount remains — NOT pure +/// global-counter contention (the isolated 8×2M pure-global RED +/// `embed_rc_global_stats_race` is deterministically green). The +/// swarm is still actually leak-free (no box crosses a thread — +/// `Ctx: !Send`; bit-exact GREEN every run); only the alloc +/// accounting is wrong. Un-`#[ignore]` is gated on the residual's +/// own root-cause/debug cycle, not on `7bfa11e`. Body unchanged. +/// See docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md. #[test] -#[ignore = "blocked: global g_rc_* counter is non-atomic by rc.c design; \ - invalid under the multi-thread host. Swarm is actually \ - leak-free (bit-exact is green; no box crosses a thread). \ - Un-ignore when the runtime makes the global fallback \ - counters atomic — see the m5.2 journal + M5 bounce-back."] +#[ignore = "still blocked: atomic-global fix 7bfa11e is necessary but \ + INSUFFICIENT — a residual swarm alloc-undercount remains \ + (Σfrees exact, Σallocs short ~1600 jittering, ~1/3 runs; \ + NOT pure global contention — the isolated RED is green). \ + Un-ignore on the residual's own debug cycle, not on \ + 7bfa11e. See the m5.2-resume-attempt journal."] fn symbol_fan_swarm_leak_free() { let Some((_stdout, stderr)) = run_swarm_or_skip() else { return }; diff --git a/docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md b/docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md new file mode 100644 index 0000000..a2ff0ab --- /dev/null +++ b/docs/journals/2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md @@ -0,0 +1,91 @@ +# iter embedding-abi-m5.2-resume-attempt — atomic-global fix is necessary but INSUFFICIENT for the swarm leak-proof + +**Date:** 2026-05-19 +**Started from:** 7bfa11e (GREEN of bugfix-rc-global-stats-race) +**Status:** RE-QUARANTINED — residual handed to a new debug cycle +**Tasks completed:** 0 of 1 (the un-`#[ignore]` was attempted, verified +unsound, and reverted; the honest-rationale correction is the only +shipped change) + +## Summary + +After the runtime fix `bugfix-rc-global-stats-race` (`7bfa11e`, +Boss-verified: the isolated 8×2M pure-global RED +`embed_rc_global_stats_race` deterministically green 3/3; full +`cargo test -p ail` green), I un-`#[ignore]`d the M5 swarm leak-proof +`symbol_fan_swarm_leak_free` — that un-ignore was *designed* as the +integration-level acceptance of the runtime fix. **Boss verification +falsified the premise that the runtime fix was sufficient.** Across 4 +swarm runs: runs 1–2 GREEN, runs 3–4 FAILED; a 6-run capture: 3 +GREEN, run 4 FAILED `left: 11998383, right: 12000003`. Signature: +`Σfrees` is **stable-exact (12000003) every run**; `Σallocs` is short +by a **jittering ~1600–2260 in ~1/3 of runs**, across the 4 +`ailang_rc_stats:` lines (3 per-ctx + 1 global atexit). + +The atomic-global fix is real, committed, and **necessary** — it +removed the *pure* global-counter contention (the isolated RED is now +deterministic). It is **not sufficient** for the swarm: a *second, +distinct* alloc-undercount remains that the isolated 8×2M pure-global +RED did not model. The un-ignore was reverted (`git checkout` — it +was never committed); the now-stale `#[ignore]` rationale that said +"un-ignore when the runtime makes the counters atomic" was corrected +to the accurate necessary-but-insufficient state (this honesty fix is +the only shipped diff — a future agent seeing `7bfa11e` must NOT +un-ignore on that basis). + +## The asymmetry (orchestrator framing for the debug carrier, NOT a fix) + +`Σfrees` perfectly stable while `Σallocs` jitters is the load-bearing +clue. Both global counters are now `_Atomic`; both per-ctx counters +are plain `++` but single-writer-per-thread (each worker owns its +`Ctx`; `Ctx: !Send`). If a per-ctx counter raced, *both* alloc and +free would jitter (the dec path also `++`s a per-ctx counter) — it +does not. So the lost increments are specifically on the **alloc** +side, and not pure global contention (RED green). Boss reasoning +could not localise it further; per the debug Iron Law that is +where Boss speculation stops. + +Two framings the debug cycle must both consider (do not pre-judge): + +1. **A genuine residual runtime defect** — some alloc events on the + swarm path are uncounted non-deterministically (a TLS-ctx + set/restore window across the M3 forwarder boundary; an + interaction with data-server's prefetch background threads; a + path the 8×2M ctx-less RED structurally cannot reach because the + swarm mixes TLS-NULL host allocs with TLS-ctx kernel-internal + allocs). +2. **A test-methodology unsoundness** — `Σ` over a *heterogeneous* + set of lines (one process-global atexit line + N per-ctx + `ailang_ctx_free` lines, with per-box alloc/free attribution + split asymmetrically between global and ctx) may not be a sound + leak invariant for a multi-thread host even with correct + counters. If so the fix is a methodology redesign (e.g. + per-symbol single-thread subprocess leak-proof, or assert each + line balances under a single-thread-per-process shape), not + another runtime change — and that becomes a design decision, + possibly a bounce-back, only AFTER root cause. + +## Disposition + +- `symbol_fan_swarm_leak_free` STAYS `#[ignore]`d, body verbatim + (still quarantined, NOT weakened). Rationale corrected in all + three breadcrumbs (module doc, fn doc, `#[ignore]` reason). +- `symbol_fan_swarm_bit_exact` remains live & GREEN — the M5 + existence proof is unaffected (the swarm IS actually correct and + leak-free; only the *accounting* is wrong). +- The residual is handed to a fresh `debug` RED-first cycle + (`bugfix-swarm-rc-alloc-undercount`). main stays green at the + honesty-fix commit. +- M5 (the milestone) stays open `[~]`; m5.3 (time-shard + + friction-harvest) remains downstream of a sound leak-proof. + +## Files touched + +- `ail-embed/tests/swarm.rs` (the three `#[ignore]`-rationale + breadcrumbs corrected for doc-honesty; the test body and the + `#[ignore]` itself unchanged) + +## Stats + +(no orchestrator-stats file — this is a Boss-side honesty correction + +finding record, not an implement-orchestrator iteration) diff --git a/docs/journals/INDEX.md b/docs/journals/INDEX.md index 23cd9d9..751903c 100644 --- a/docs/journals/INDEX.md +++ b/docs/journals/INDEX.md @@ -109,3 +109,4 @@ - 2026-05-19 — iter embedding-abi-m5.1 (DONE 3/3): M5 iter 1 — stood up the workspace-excluded `ail-embed` crate. A zero-dependency embedding core (Rust port of the audited `crates/ail/tests/embed/tick_roundtrip.c`: `extern "C"` to the M3-frozen ABI + frozen-layout `State`/`Tick` box helpers + a `Kernel` price fold, raw pointers never escaping the type) links the M3 staticlib via a new `build.rs` (no in-repo precedent — `AIL_BIN` env override else nested `cargo build -p ail` against the parent workspace, separate target dir so no cargo-lock deadlock). Plus a hermetic `data-server` smoke: a synthetic Pepperstone-format ZIP fixture written via the crate's own public `RawTickRecord` type (correct-by-construction, mirrors `data-server/src/loader.rs:110-124`) → real `DataServer` → `Kernel`, asserting bit-exact vs a same-order host reference fold and the closed-form `55.0`; runs with no `/mnt`. `ail-embed` is its own cargo workspace root (empty `[workspace]` table) so the AILang compiler workspace owes it nothing; `data-server` is a dev-dependency only ⇒ Invariant 1 holds in the dependency graph, not just on paper. Boss-verified independently: ail-embed suite 2/2 green (`kernel_run_sums_prices` unit RED-first + `hermetic_smoke_data_server_roundtrip` integration), full+no-deps `cargo metadata` on the AILang workspace shows `data-server` count 0, `git status` path filter empty (zero diff to `crates/ailang-*`/`crates/ail/`/`runtime/`/`examples/*.ail`), `src/lib.rs` zero code-level `data_server`, AILang `cargo build --workspace` still clean. Two toolchain-forced corrections to the plan's verbatim `ail-embed/Cargo.toml` (added the empty `[workspace]` table; sibling dev-dep path `../libs`→`../../libs` manifest-relative) — confined to the plan-created manifest, no acceptance gate altered, 0 review re-loops; planner-recon implication recorded in the journal Concerns. Adapter API + thread-swarm explicitly deferred to M5 iter 2+. → 2026-05-19-iter-embedding-abi-m5.1.md - 2026-05-19 — iter embedding-abi-m5.2 (PARTIAL 2/3; M5 bounce-back): data-server adapter + symbol-fan swarm; the leak-proof did its job and surfaced a real concurrency finding. Tasks 1+2 landed clean: `data-server` promoted dev-dep→real dep (Invariant-1 sanctioned — compiler workspace graph still data-server-count 0, zero compiler-surface diff), additive `adapter` module (`tick_to_px`/`MidPriceStream` lazy `Iterator`/`fold_symbol`, RED-first), and a `swarm_runner` `[[bin]]` whose clean compile *is* the compile-time per-thread-ctx proof (`Ctx: !Send` ⇒ a shared-ctx swarm is `E0277`, cannot build). The real-data symbol-fan swarm runs (EURUSD/GER40/XAUUSD, ~4s) and is **bit-exact vs an independent same-order host reference every run** (`symbol_fan_swarm_bit_exact`, GREEN, live). Task 3's `Σallocs==Σfrees` leak gate is BLOCKED: Boss independently read `runtime/rc.c` and confirms it is an **instrumentation race, not a memory bug** — no box crosses a thread (`!Send`), the real RC is correct (bit-exact green), only the *global* `g_rc_*` stat counters (rc.c:90-91,161,212) are non-atomic *by rc.c's own documented single-threaded design* and lose `++`s when 3 worker threads hit the null-`__ail_tls_ctx` host-side path. M5 is AILang's first concurrent consumer → the deferred "concurrency arrived: atomic-vs-non-atomic" decision rc.c:44-49 names is now due. The leak assertion was Boss-split into `symbol_fan_swarm_leak_free` (`#[ignore]`, **body verbatim** — quarantined not weakened; un-ignore = the runtime fix's acceptance) so main stays green and the finding is pinned. Prior claim that `embed_swarm_tsan.rs` covers this was corrected (it uses the scalar kernel — zero box allocs — never exercised this path). Escalated as a **bounce-back** (touches M3-frozen `runtime/`; re-frames M5's "zero runtime change"; multiple substantive options); Boss recommendation on the record = Option A (atomic *global-fallback* counters only, standalone RED-first runtime micro-iter, M5 framing amended) — user picks the direction. Time-shard + friction-harvest remain m5.3. → 2026-05-19-iter-embedding-abi-m5.2.md - 2026-05-19 — iter bugfix-rc-global-stats-race (RED→GREEN, debug→implement mini, DONE 1/1): the M5 iter-2 bounce-back resolution (user-approved Option A). The two null-ctx fallback RC-stats counters `g_rc_alloc_count`/`g_rc_free_count` (`runtime/rc.c:90-91`) were plain `static uint64_t` with a non-atomic `++` on the `__ail_tls_ctx == NULL` arm of `ailang_rc_alloc` (rc.c:161) / the to-zero branch of `ailang_rc_dec` (rc.c:212); a multi-threaded host driving the global fallback (no `ailang_ctx_new`) raced the read-modify-write and lost increments, so the `AILANG_RC_STATS` atexit Σ under-counted non-deterministically — exactly the deferred "concurrency arrived: atomic-vs-non-atomic" decision rc.c's own header (rc.c:44-49) names, M5 being AILang's first concurrent consumer. NOT a memory bug (`Ctx: !Send` keeps every box on one thread; the real refcount/free is correct; programs bit-exact). RED (debugger, audit-trail commit `427b687`): a C host — 8 threads × 2M alloc-then-dec, no ctx so the global path is taken, no box crossing a thread — + integration test asserting the atexit Σ is exact; Boss-verified RED `allocs=2141382 expected 16000000, live=-131242` (deterministic-fail under that contention, not flaky). GREEN (implement mini, this commit): the two globals are now `_Atomic uint64_t`, `atomic_fetch_add_explicit(.., memory_order_relaxed)` at the two fallback `++` sites, `atomic_load_explicit(.., relaxed)` in `ailang_rc_stats_atexit` (its sole reader) — relaxed is correct (pure stats, no happens-before; atexit reader runs post-join). Per-ctx counters + the per-object refcount header left non-atomic BY DESIGN (single-thread-per-ctx; boxes never cross threads) — explicitly out of scope, untouched; frozen value layout / ABI offsets / host-free rule untouched; the genuinely-still-single-threaded drop-worklist doc note correctly left unchanged (a false correction was refused); the two genuinely-stale atomicity doc blocks corrected for honesty. Scope held to `runtime/rc.c` ONLY (37+/14−). Boss-verified independently: RED→GREEN deterministically 3/3 (jitter gone, `allocs==frees==16_000_000`), full `cargo test -p ail` green (every binary 0 failed — the per-ctx tsan harnesses embed_swarm_tsan/embed_rc_accounting_tsan + embed_tick_e2e/embed_record_e2e unaffected since the per-ctx path is untouched; rc.c links into every ail binary so this is the strong regression gate), git-status scope = rc.c + journal + stats only, the RED files unchanged. Unblocks resuming the M5 leak-proof (un-`#[ignore]` `ail-embed`'s `symbol_fan_swarm_leak_free`). RED `427b687` → GREEN this commit. → 2026-05-19-iter-bugfix-rc-global-stats-race.md +- 2026-05-19 — iter embedding-abi-m5.2-resume-attempt (RE-QUARANTINED; honesty fix only): attempted to un-`#[ignore]` the M5 swarm leak-proof `symbol_fan_swarm_leak_free` as the integration-level acceptance of the runtime fix `7bfa11e` (bugfix-rc-global-stats-race). Boss verification FALSIFIED the sufficiency premise: across 4–6 swarm runs the leak Σ is still non-deterministic — `Σfrees` stable-exact (12000003) every run, `Σallocs` short by a jittering ~1600–2260 in ~1/3 of runs (e.g. left:11998383 right:12000003 across 4 stat lines). The atomic-global fix is real, committed, NECESSARY (the isolated 8×2M pure-global RED `embed_rc_global_stats_race` is deterministically green) but NOT sufficient — a second, distinct alloc-side undercount remains that the isolated RED did not model (frees never lose ⇒ not a per-ctx race; not pure global contention ⇒ RED green; localisation beyond that is where Boss speculation stops per the debug Iron Law). Un-ignore reverted (never committed); the only shipped diff is the three now-stale `#[ignore]`-rationale breadcrumbs in `ail-embed/tests/swarm.rs` corrected from "un-ignore when the runtime is atomic" to the accurate necessary-but-insufficient state (a future agent seeing `7bfa11e` must NOT un-ignore on that basis — doc-honesty). Test body + the `#[ignore]` itself unchanged (still quarantined, NOT weakened); `symbol_fan_swarm_bit_exact` stays live+GREEN (the swarm IS actually correct and leak-free — only the accounting is wrong). Residual handed to a fresh debug RED-first cycle; two framings flagged for it (genuine residual runtime defect vs Σ-over-heterogeneous-global+per-ctx-lines methodology unsoundness — the latter, if so, is a post-root-cause design decision). main green at the honesty-fix commit; M5 stays open `[~]`; m5.3 remains downstream of a sound leak-proof. → 2026-05-19-iter-embedding-abi-m5.2-resume-attempt.md