# iter embedding-abi-m5.2 — data-server adapter + symbol-fan swarm; leak-free gate surfaced a runtime race

**Date:** 2026-05-19
**Started from:** 9cc9d9c5170eb8d52152dcef071726c7a6ffd05d
**Status:** PARTIAL
**Tasks completed:** 2 of 3

## Summary

Tasks 1 and 2 landed clean and verbatim-to-plan: `data-server` was
promoted dev-dep → real dep, the additive `adapter` module
(`tick_to_px` / `MidPriceStream` / `fold_symbol`) was added with a
genuine compile-error RED → GREEN cycle, and the `swarm_runner` bin
compiled — that compile *is* the compile-time per-thread-ctx proof
(`Ctx: !Send`; a shared-ctx design is `error[E0277]` and could not
have built). Invariant 1 holds independently: the AILang workspace
graph has zero `data-server` (Step 4 = `0`) and there is zero
compiler-surface diff (Step 5 path filter empty — only `ail-embed/**`
touched, not even the root `Cargo.toml`).

Task 3 is BLOCKED. The plan's verbatim leak-free assertion
(`Σallocs == Σfrees` across all `ailang_rc_stats:` lines) is correct
and is doing its job: it surfaced a genuine concurrency defect. The
swarm E2E passed once in isolation by luck (the racing counter
happened to land on the balanced value) and then FAILED in the
full-suite run and on 4 of 5 direct re-runs. The host-side box
allocations in `ail-embed/src/lib.rs` (`make_state` / `make_tick` /
the final `ailang_rc_dec`) run on the Rust worker thread where
`__ail_tls_ctx == NULL` — the M3 forwarder only sets the TLS ctx for
the synchronous duration of the kernel call, not for the host's own
allocs. With `__ail_tls_ctx` null those allocs fall through to the
**non-atomic global statics** `g_rc_alloc_count` / `g_rc_free_count`
(`runtime/rc.c:161,212`), incremented concurrently by 3 worker
threads with no synchronisation. The atexit `g_rc` line's `allocs`
is consequently non-deterministic across runs
(5,993,321 … 5,999,318 over 5 runs; should be a stable 6,000,000)
while `frees` stays at a stable 3. The bit-exactness half of the
test is sound and passed every run; only the global leak-accounting
is racy.

This is exactly the class of finding the symbol-fan swarm milestone
exists to expose — and it is out of m5.2 scope to fix (see Blocked
detail). The Boss decides: extend the spec / m5.3 to cover the
host-side-alloc TLS-ctx routing (or atomic global counters), or
re-scope the leak-free acceptance.

## Per-task notes

- iter embedding-abi-m5.2.1: promote `data-server` dev-dep → real
  dep + additive `adapter` module. RED (`E0432 unresolved import
  super::tick_to_px`) → impl `tick_to_px`/`MidPriceStream`/
  `fold_symbol` → GREEN (`tick_to_px_is_mid`, +
  `kernel_run_sums_prices` unregressed under `--lib`). Cargo.toml +
  `lib.rs:7` verbatim-to-plan. Stale m5.1 "ZERO dependencies"
  manifest comment corrected for present-tense honesty (quality
  phase).
- iter embedding-abi-m5.2.2: `src/bin/swarm_runner.rs` written
  verbatim; `cargo build --bin swarm_runner` → exit 0. The clean
  compile IS the compile-time per-thread-ctx proof (`Ctx: !Send`;
  no `E0277`; per-thread-owned-Kernel structure intact, not
  collapsed to a shared ctx).
- iter embedding-abi-m5.2.3: `tests/swarm.rs` written verbatim. The
  E2E genuinely executed (NOT the skip path — 3 tick symbols
  present: EURUSD, GER40, XAUUSD; ~4 s, 3×2M FFI folds). Per-symbol
  bit-exactness vs independent host reference: sound, passed every
  run. Global `Σallocs==Σfrees` leak-free assertion: BLOCKED —
  correctly detects a non-atomic global-counter race (see Summary +
  Blocked detail). Not re-loopable (the plan's verbatim code is
  exactly as specified; no re-attempt makes a racy C counter
  deterministic) and not fixable in-scope (Step 5 mandates zero
  `runtime/` diff; `runtime/` is M3-frozen).

## Concerns

- Plan Task 1 Step 6's literal command
  (`cargo test --manifest-path ail-embed/Cargo.toml`, unfiltered)
  is unsatisfiable in Task 1's isolation: the `[[bin]] swarm_runner`
  declaration added by Step 2's verbatim Cargo.toml replacement
  references `src/bin/swarm_runner.rs`, which Task 2 Step 1 creates.
  The plan Self-review point 7 ("No later task is required to make
  an earlier compile gate pass") is incorrect for this command.
  Task 1's no-core-regression intent was fully proven via `--lib`
  (`kernel_run_sums_prices` + `tick_to_px_is_mid` GREEN); the
  smoke + full-suite gate is covered by Task 3 Step 3 once the bin
  exists. The verbatim Cargo.toml was NOT split to dodge this
  (splitting it would be the spec-compliance violation; the
  `[[bin]]` + dep-promotion is one cohesive manifest change). No
  code defect — purely a plan command-ordering defect.
- Quality phase corrected the pre-existing m5.1 Cargo.toml comment
  ("the library itself has ZERO dependencies … data-server is
  dev-only") which became false the moment `data-server` moved into
  `[dependencies]`. The plan's verbatim replacement boundary started
  *at* `[dependencies]` and did not script the preceding comment's
  removal; leaving a now-false manifest comment violates the
  DESIGN.md honesty rule (comments mirror current state). The
  replacement comment states the present-tense m5.2 framing,
  consistent with the plan's own new block + Boss decision.

## Known debt

- The host-side embedding allocation path (`ail-embed/src/lib.rs`
  `make_state`/`make_tick`/final `ailang_rc_dec`) executes with
  `__ail_tls_ctx == NULL` (the M3 `@<sym>` forwarder sets the TLS
  ctx only for the synchronous kernel-call duration). Under the
  multi-thread swarm these allocs/frees land on the non-atomic
  global `g_rc_alloc_count`/`g_rc_free_count` fallback
  (`runtime/rc.c:161,212`), which races. Not touched here:
  `runtime/` is M3-frozen and Step 5 mandates zero `runtime/` diff;
  the host-side-routing alternative is a design change beyond
  "additive adapter + swarm bin". Reserved for Boss/m5.3 design.

## Blocked detail

- **Task:** 3 (Step 2 / Step 3 — the leak-free `Σallocs==Σfrees`
  gate).
- **Reason:** `worker-blocked` — the plan's own verbatim acceptance
  gate correctly detects a runtime concurrency defect whose fix is
  out of m5.2 scope and would violate this iter's Invariant-1
  Step-5 zero-`runtime/`-diff constraint.
- **Worker says (verbatim):** Task 3's `tests/swarm.rs` is verbatim
  to plan and genuinely executes (bit-exact half sound, GREEN every
  run). The `Σallocs==Σfrees` half is non-deterministic: full-suite
  run FAILED `left: 11997743, right: 12000003`; 5 direct child runs
  of the atexit `g_rc` line gave `allocs=` 5993699/5996029/5993321/
  5999318/5996019 with `frees=3` stable. Root cause: host-side
  `ailang_rc_alloc`/`ailang_rc_dec` from `lib.rs` run with
  `__ail_tls_ctx==NULL`, falling through to the non-atomic global
  `g_rc_*` counters (`rc.c:161,212`) raced by 3 worker threads. Not
  re-loopable (verbatim plan code; a racy C counter is not made
  deterministic by re-attempting the same code) and not
  in-scope-fixable (zero `runtime/` diff mandated; `runtime/` is
  M3-frozen).
- **Suggested next step:** Boss decision — either (a) extend the
  spec/m5.3 to route host-side `ail-embed` allocs through a
  per-thread `ailang_ctx_t` so the swarm's leak accounting is
  per-thread-attributed (no global fallback), or make
  `g_rc_alloc_count`/`g_rc_free_count` atomic in a *separate*
  runtime iteration outside m5.2's zero-`runtime/`-diff constraint;
  or (b) re-scope the m5.2 leak-free acceptance. Tasks 1 + 2 are
  clean and committable as a PARTIAL; the architect should see the
  TSan-discharge reasoning below at milestone close.

## Boss-decision rationale carried forward (mirrored from the plan, for the architect at milestone close)

- **TSan discharge (compile-time + cited green precedent, no new
  harness).** The spec's "sanitiser-clean (no data race across
  per-thread ctxs)" acceptance is discharged by (1) `Ctx`/`Kernel`
  being `!Send` (raw `*mut c_void`) — the swarm is *structurally
  forced* one-ctx-per-thread; a shared-ctx design is `error[E0277]`
  and cannot compile, so Task 2 Step 2's clean build is a
  compile-time race-freedom proof for the *per-ctx kernel calls*;
  (2) the C-runtime race-freedom of that exact per-thread-ctx
  pattern is ratified green by `crates/ail/tests/embed_swarm_tsan.rs`
  (C host, 8 pthreads, own ctx each, `clang -fsanitize=thread`,
  exit-clean), cited not re-run. NOTE FOR THE ARCHITECT: this
  discharge covers the *per-thread-ctx kernel path*. The Task 3
  finding is a *different* surface — the host-side `ail-embed`
  allocs that bypass the TLS ctx and hit the non-atomic global
  `g_rc_*` fallback. The TSan precedent does not cover that path
  (the C host in `embed_swarm_tsan.rs` does not exercise a Rust
  host allocating boxes with a null TLS ctx). The compile-time
  `!Send` proof and the cited green TSan remain valid for what they
  claim; they simply do not reach the global-fallback race m5.2
  newly exposed.
- **`data-server` dev-dep → real dep is sanctioned.** Invariant 1
  governs the *compiler* crates (`ailang-*`, `crates/ail`,
  `runtime/`), which gain nothing — verified clean this iter
  (Step 4 = `0`, Step 5 empty). `ail-embed` is explicitly the
  *sole* data-server↔AILang meeting point.
- **Deferred to m5.3 (not this plan):** time-shard
  boundary-invisibility, friction-harvest timing, milestone
  close-out. Not implemented here (scope discipline held — no
  time-shard, no perf timing, no close-out crept in).

## Files touched

- `ail-embed/Cargo.toml` (modified — dep promotion + `[[bin]]` +
  stale-comment honesty fix)
- `ail-embed/src/lib.rs` (modified — `pub mod adapter;`, additive)
- `ail-embed/src/adapter.rs` (new — the sole data-server-knowing layer)
- `ail-embed/src/bin/swarm_runner.rs` (new — symbol-fan swarm bin)
- `ail-embed/tests/swarm.rs` (new — symbol-fan leak-free + bit-exact
  E2E; bit-exact half sound, leak-free half surfaces the race)

## Stats

bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.2.json

## Boss disposition (2026-05-19)

Boss independently read `runtime/rc.c` and confirms the diagnosis is
**precise and the finding is an instrumentation-measurement race, NOT
a memory bug**: the per-object refcount header ops (`*hdr +=/-= 1`,
rc.c:180/208) are non-atomic but **no box crosses a thread** in the
swarm (`Ctx: !Send` structurally forces alloc/use/free within one
worker), so there is no real refcount race, no double-free, no leak —
the bit-exact half is GREEN every run, confirming the memory
management is actually correct. The *only* raced object is the global
`g_rc_*` statistics counter (rc.c:90-91,161,212), **non-atomic by
rc.c's own explicit design** (header comment rc.c:44-49,85: "AILang
has no concurrency primitives yet; when it acquires them,
atomic-vs-non-atomic becomes a separate decision"). M5 is AILang's
first concurrent consumer — that deferred decision is now due.

Committed as a PARTIAL (Tasks 1+2 + the GREEN existence proof) so the
sound work + the finding land on the record, main stays green:
`tests/swarm.rs` was Boss-split into `symbol_fan_swarm_bit_exact`
(live, GREEN — the real-data existence proof) and
`symbol_fan_swarm_leak_free` (`#[ignore]`, **body preserved
verbatim** — quarantined, not weakened; un-`#[ignore]` is the
acceptance criterion for the runtime fix). Earlier plan/journal claim
that `embed_swarm_tsan.rs` covers this path was wrong and is
corrected here: that test uses the *scalar* kernel (zero box allocs)
so it never exercised the host-side global-counter path; the
compile-time `!Send` proof + the cited green TSan remain valid only
for the per-thread-ctx *kernel* path, which they always claimed.

Escalated to the user as an M5 **bounce-back** (touches M3-frozen
`runtime/`; re-frames M5's "zero runtime change" commitment;
multiple substantive options — not unilaterally Boss's to pick in
frozen-runtime territory). Boss recommendation on the record:
**Option A** — make *only the global fallback* `g_rc_*` counters
atomic (leave the per-thread ctx path plain — no contention there),
as its own RED-first runtime micro-iteration (`debug` skill), outside
M5's "zero runtime change" framing; then resume the M5 leak-proof on
the now-valid measurement (un-`#[ignore]` `symbol_fan_swarm_leak_free`
as that iter's GREEN). Rationale: it discharges exactly the deferred
decision rc.c's own comment names, at the defect's actual location,
without touching the frozen value layout / ABI offsets / host-free
rule (the literal content of the M3 freeze); Option B (host-side
TLS-ctx routing) adds a new runtime API surface; Option C (per-symbol
single-thread subprocess leak-proof) is sound but weaker and leaves
the runtime's documented single-thread-stats footgun for every future
concurrent consumer. User picks the direction.
