AILang/docs/journals/2026-05-19-iter-embedding-abi-m5.2.md

iter embedding-abi-m5.2 — data-server adapter + symbol-fan swarm; leak-free gate surfaced a runtime race

Date: 2026-05-19 Started from: 9cc9d9c517 Status: PARTIAL Tasks completed: 2 of 3

Summary

Tasks 1 and 2 landed clean and verbatim-to-plan: data-server was promoted dev-dep → real dep, the additive adapter module (tick_to_px / MidPriceStream / fold_symbol) was added with a genuine compile-error RED → GREEN cycle, and the swarm_runner bin compiled — that compile is the compile-time per-thread-ctx proof (Ctx: !Send; a shared-ctx design is error[E0277] and could not have built). Invariant 1 holds independently: the AILang workspace graph has zero data-server (Step 4 = 0) and there is zero compiler-surface diff (Step 5 path filter empty — only ail-embed/** touched, not even the root Cargo.toml).

Task 3 is BLOCKED. The plan's verbatim leak-free assertion (Σallocs == Σfrees across all ailang_rc_stats: lines) is correct and is doing its job: it surfaced a genuine concurrency defect. The swarm E2E passed once in isolation by luck (the racing counter happened to land on the balanced value) and then FAILED in the full-suite run and on 4 of 5 direct re-runs. The host-side box allocations in ail-embed/src/lib.rs (make_state / make_tick / the final ailang_rc_dec) run on the Rust worker thread where __ail_tls_ctx == NULL — the M3 forwarder only sets the TLS ctx for the synchronous duration of the kernel call, not for the host's own allocs. With __ail_tls_ctx null those allocs fall through to the non-atomic global statics g_rc_alloc_count / g_rc_free_count (runtime/rc.c:161,212), incremented concurrently by 3 worker threads with no synchronisation. The atexit g_rc line's allocs is consequently non-deterministic across runs (5,993,321 … 5,999,318 over 5 runs; should be a stable 6,000,000) while frees stays at a stable 3. The bit-exactness half of the test is sound and passed every run; only the global leak-accounting is racy.

This is exactly the class of finding the symbol-fan swarm milestone exists to expose — and it is out of m5.2 scope to fix (see Blocked detail). The Boss decides: extend the spec / m5.3 to cover the host-side-alloc TLS-ctx routing (or atomic global counters), or re-scope the leak-free acceptance.

Per-task notes

• iter embedding-abi-m5.2.1: promote data-server dev-dep → real dep + additive adapter module. RED (E0432 unresolved import super::tick_to_px) → impl tick_to_px/MidPriceStream/ fold_symbol → GREEN (tick_to_px_is_mid, + kernel_run_sums_prices unregressed under --lib). Cargo.toml + lib.rs:7 verbatim-to-plan. Stale m5.1 "ZERO dependencies" manifest comment corrected for present-tense honesty (quality phase).
• iter embedding-abi-m5.2.2: src/bin/swarm_runner.rs written verbatim; cargo build --bin swarm_runner → exit 0. The clean compile IS the compile-time per-thread-ctx proof (Ctx: !Send; no E0277; per-thread-owned-Kernel structure intact, not collapsed to a shared ctx).
• iter embedding-abi-m5.2.3: tests/swarm.rs written verbatim. The E2E genuinely executed (NOT the skip path — 3 tick symbols present: EURUSD, GER40, XAUUSD; ~4 s, 3×2M FFI folds). Per-symbol bit-exactness vs independent host reference: sound, passed every run. Global Σallocs==Σfrees leak-free assertion: BLOCKED — correctly detects a non-atomic global-counter race (see Summary + Blocked detail). Not re-loopable (the plan's verbatim code is exactly as specified; no re-attempt makes a racy C counter deterministic) and not fixable in-scope (Step 5 mandates zero runtime/ diff; runtime/ is M3-frozen).

Concerns

• Plan Task 1 Step 6's literal command (cargo test --manifest-path ail-embed/Cargo.toml, unfiltered) is unsatisfiable in Task 1's isolation: the [[bin]] swarm_runner declaration added by Step 2's verbatim Cargo.toml replacement references src/bin/swarm_runner.rs, which Task 2 Step 1 creates. The plan Self-review point 7 ("No later task is required to make an earlier compile gate pass") is incorrect for this command. Task 1's no-core-regression intent was fully proven via --lib (kernel_run_sums_prices + tick_to_px_is_mid GREEN); the smoke + full-suite gate is covered by Task 3 Step 3 once the bin exists. The verbatim Cargo.toml was NOT split to dodge this (splitting it would be the spec-compliance violation; the [[bin]] + dep-promotion is one cohesive manifest change). No code defect — purely a plan command-ordering defect.
• Quality phase corrected the pre-existing m5.1 Cargo.toml comment ("the library itself has ZERO dependencies … data-server is dev-only") which became false the moment data-server moved into [dependencies]. The plan's verbatim replacement boundary started at [dependencies] and did not script the preceding comment's removal; leaving a now-false manifest comment violates the DESIGN.md honesty rule (comments mirror current state). The replacement comment states the present-tense m5.2 framing, consistent with the plan's own new block + Boss decision.

Known debt

• The host-side embedding allocation path (ail-embed/src/lib.rs make_state/make_tick/final ailang_rc_dec) executes with __ail_tls_ctx == NULL (the M3 @<sym> forwarder sets the TLS ctx only for the synchronous kernel-call duration). Under the multi-thread swarm these allocs/frees land on the non-atomic global g_rc_alloc_count/g_rc_free_count fallback (runtime/rc.c:161,212), which races. Not touched here: runtime/ is M3-frozen and Step 5 mandates zero runtime/ diff; the host-side-routing alternative is a design change beyond "additive adapter + swarm bin". Reserved for Boss/m5.3 design.

Blocked detail

• Task: 3 (Step 2 / Step 3 — the leak-free Σallocs==Σfrees gate).
• Reason: worker-blocked — the plan's own verbatim acceptance gate correctly detects a runtime concurrency defect whose fix is out of m5.2 scope and would violate this iter's Invariant-1 Step-5 zero-runtime/-diff constraint.
• Worker says (verbatim): Task 3's tests/swarm.rs is verbatim to plan and genuinely executes (bit-exact half sound, GREEN every run). The Σallocs==Σfrees half is non-deterministic: full-suite run FAILED left: 11997743, right: 12000003; 5 direct child runs of the atexit g_rc line gave allocs= 5993699/5996029/5993321/ 5999318/5996019 with frees=3 stable. Root cause: host-side ailang_rc_alloc/ailang_rc_dec from lib.rs run with __ail_tls_ctx==NULL, falling through to the non-atomic global g_rc_* counters (rc.c:161,212) raced by 3 worker threads. Not re-loopable (verbatim plan code; a racy C counter is not made deterministic by re-attempting the same code) and not in-scope-fixable (zero runtime/ diff mandated; runtime/ is M3-frozen).
• Suggested next step: Boss decision — either (a) extend the spec/m5.3 to route host-side ail-embed allocs through a per-thread ailang_ctx_t so the swarm's leak accounting is per-thread-attributed (no global fallback), or make g_rc_alloc_count/g_rc_free_count atomic in a separate runtime iteration outside m5.2's zero-runtime/-diff constraint; or (b) re-scope the m5.2 leak-free acceptance. Tasks 1 + 2 are clean and committable as a PARTIAL; the architect should see the TSan-discharge reasoning below at milestone close.

Boss-decision rationale carried forward (mirrored from the plan, for the architect at milestone close)

• TSan discharge (compile-time + cited green precedent, no new harness). The spec's "sanitiser-clean (no data race across per-thread ctxs)" acceptance is discharged by (1) Ctx/Kernel being !Send (raw *mut c_void) — the swarm is structurally forced one-ctx-per-thread; a shared-ctx design is error[E0277] and cannot compile, so Task 2 Step 2's clean build is a compile-time race-freedom proof for the per-ctx kernel calls; (2) the C-runtime race-freedom of that exact per-thread-ctx pattern is ratified green by crates/ail/tests/embed_swarm_tsan.rs (C host, 8 pthreads, own ctx each, clang -fsanitize=thread, exit-clean), cited not re-run. NOTE FOR THE ARCHITECT: this discharge covers the per-thread-ctx kernel path. The Task 3 finding is a different surface — the host-side ail-embed allocs that bypass the TLS ctx and hit the non-atomic global g_rc_* fallback. The TSan precedent does not cover that path (the C host in embed_swarm_tsan.rs does not exercise a Rust host allocating boxes with a null TLS ctx). The compile-time !Send proof and the cited green TSan remain valid for what they claim; they simply do not reach the global-fallback race m5.2 newly exposed.
• data-server dev-dep → real dep is sanctioned. Invariant 1 governs the compiler crates (ailang-*, crates/ail, runtime/), which gain nothing — verified clean this iter (Step 4 = 0, Step 5 empty). ail-embed is explicitly the sole data-server↔AILang meeting point.
• Deferred to m5.3 (not this plan): time-shard boundary-invisibility, friction-harvest timing, milestone close-out. Not implemented here (scope discipline held — no time-shard, no perf timing, no close-out crept in).

Files touched

• ail-embed/Cargo.toml (modified — dep promotion + [[bin]] + stale-comment honesty fix)
• ail-embed/src/lib.rs (modified — pub mod adapter;, additive)
• ail-embed/src/adapter.rs (new — the sole data-server-knowing layer)
• ail-embed/src/bin/swarm_runner.rs (new — symbol-fan swarm bin)
• ail-embed/tests/swarm.rs (new — symbol-fan leak-free + bit-exact E2E; bit-exact half sound, leak-free half surfaces the race)

Stats

bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.2.json

Boss disposition (2026-05-19)

Boss independently read runtime/rc.c and confirms the diagnosis is precise and the finding is an instrumentation-measurement race, NOT a memory bug: the per-object refcount header ops (*hdr +=/-= 1, rc.c:180/208) are non-atomic but no box crosses a thread in the swarm (Ctx: !Send structurally forces alloc/use/free within one worker), so there is no real refcount race, no double-free, no leak — the bit-exact half is GREEN every run, confirming the memory management is actually correct. The only raced object is the global g_rc_* statistics counter (rc.c:90-91,161,212), non-atomic by rc.c's own explicit design (header comment rc.c:44-49,85: "AILang has no concurrency primitives yet; when it acquires them, atomic-vs-non-atomic becomes a separate decision"). M5 is AILang's first concurrent consumer — that deferred decision is now due.

Committed as a PARTIAL (Tasks 1+2 + the GREEN existence proof) so the sound work + the finding land on the record, main stays green: tests/swarm.rs was Boss-split into symbol_fan_swarm_bit_exact (live, GREEN — the real-data existence proof) and symbol_fan_swarm_leak_free (#[ignore], body preserved verbatim — quarantined, not weakened; un-#[ignore] is the acceptance criterion for the runtime fix). Earlier plan/journal claim that embed_swarm_tsan.rs covers this path was wrong and is corrected here: that test uses the scalar kernel (zero box allocs) so it never exercised the host-side global-counter path; the compile-time !Send proof + the cited green TSan remain valid only for the per-thread-ctx kernel path, which they always claimed.

Escalated to the user as an M5 bounce-back (touches M3-frozen runtime/; re-frames M5's "zero runtime change" commitment; multiple substantive options — not unilaterally Boss's to pick in frozen-runtime territory). Boss recommendation on the record: Option A — make only the global fallback g_rc_* counters atomic (leave the per-thread ctx path plain — no contention there), as its own RED-first runtime micro-iteration (debug skill), outside M5's "zero runtime change" framing; then resume the M5 leak-proof on the now-valid measurement (un-#[ignore] symbol_fan_swarm_leak_free as that iter's GREEN). Rationale: it discharges exactly the deferred decision rc.c's own comment names, at the defect's actual location, without touching the frozen value layout / ABI offsets / host-free rule (the literal content of the M3 freeze); Option B (host-side TLS-ctx routing) adds a new runtime API surface; Option C (per-symbol single-thread subprocess leak-proof) is sound but weaker and leaves the runtime's documented single-thread-stats footgun for every future concurrent consumer. User picks the direction.