AILang/docs/journals/2026-05-18-iter-embedding-abi-m2.1.md

iter embedding-abi-m2.1 — per-thread runtime context + concurrency safety

Date: 2026-05-18 Started from: c9a84b33b3 (Tasks 1–4 committed) Original start (first dispatch): b3388c8350 Status: DONE Tasks completed: 9 of 9 (Tasks 1–4 in the first dispatch, committed at c9a84b3; Tasks 5–9 in this re-dispatch)

Summary

The per-thread embedding-ctx ABI is complete. Tasks 1–4 (first dispatch, committed at c9a84b3): the FIXED-FIRST alloc-guard baseline pin, the de-globalised ailang_ctx_t runtime (per-thread TLS slot + conditional accounting, g_rc_*+atexit retained verbatim as the null-ctx fallback), the codegen forwarder gaining a leading ptr %ctx + @__ail_tls_ctx save/store/restore around the byte-unchanged internal call, and the CLI RC-only guard. Tasks 5–9 (this re-dispatch): the per-thread-ctx scalar-swarm capability demo (swarm.c, no SHARED_CTX, GREEN), the de-globalisation negative-control standing test in the rc-accounting harness (rc_accounting_shared_ctx_is_flagged_by_tsan — shared-ctx genuinely raced under tsan, teeth proven), the M1 host.c migration to the ctx ABI (s == 25 holds through the ctx-threaded forwarder), the DESIGN.md current-state update (provisional-until-M3 narrowed to value/record layout; Decision-10 per-thread-ctx note; pinned "Export parameters are written bare…" sentence + (con Int) snippet preserved verbatim), and the full regression + bench gate (workspace 0 failed; bench-trio causally exonerated). The milestone's concurrency-safety invariant is sanitiser-verified.

Boss spec-defect repair (2026-05-18)

The first dispatch (Tasks 1–4 clean) correctly BLOCKED on Task 5, surfacing a genuine spec defect: the original Task 5 paired a -DSHARED_CTX negative control onto swarm.c. That is structurally impossible — examples/embed_backtest_step.ail is a non-allocating scalar kernel (state += sample*sample); it never calls ailang_rc_alloc/ailang_rc_dec, never writes a ctx field, and @__ail_tls_ctx is __thread (per-thread, never shared even when the ailang_ctx_t* value is passed to a shared ctx). A shared-ctx scalar swarm therefore has no shared-memory write to race on; tsan correctly stays clean. Asserting a structurally-unreachable race would have been a vacuous (and dishonest) test. The Boss adjudicated: the de-globalisation negative-control teeth belong to the item-1 rc-accounting harness (rc_accounting_tsan.c -DSHARED_CTX, where a shared ctx genuinely races on ctx->alloc_count), NOT swarm.c. The spec was amended in lockstep and the plan's Task 5 restructured: swarm.c is the per-thread-ctx capability demo ONLY (one green positive test); the negative control became rc_accounting_shared_ctx_is_flagged_by_tsan added to the committed embed_rc_accounting_tsan.rs. This re-dispatch implemented the corrected Task 5 — no swarm.c negative control reintroduced. The plan's Task 3 expected-string transcription error (recorded below) was likewise Boss-corrected (@ail_embed_backtest_step_step, the module-qualified convention-correct symbol).

Per-task notes

• iter embedding-abi-m2.1.1 (first dispatch, committed): FIXED-FIRST alloc-guard pin. New crates/ail/tests/embed_staticlib_alloc_guard.rs (verbatim from plan); RED captured (build currently succeeds), the deliverable. RED→GREEN across the iter (Task 4 closes it).
• iter embedding-abi-m2.1.2 (first dispatch, committed): runtime/rc.c gains ailang_ctx_t {alloc_count, free_count} + ailang_ctx_new/ ailang_ctx_free + __thread __ail_tls_ctx; the two increment sites become if (_ctx) _ctx->… else g_rc_…. g_rc_* statics + atexit + constructor retained verbatim. New rc_accounting_tsan.c + embed_rc_accounting_tsan.rs: RED (link error) → GREEN (per-ctx, 8×200000, tsan-clean).
• iter embedding-abi-m2.1.3 (first dispatch, committed): codegen Target::StaticLib arm emits @__ail_tls_ctx = external thread_local global ptr once; forwarder signature gains leading ptr %ctx; body does load-save / store-ctx / unchanged call @ail_<mname>_<fn>(args) / store-restore. _adapter/_clos
  • internal arg vector byte-untouched. Doc-comment narrowed to value/record-layout provisionality. embed_e2e left RED by design (Task 6 migrates it).
• iter embedding-abi-m2.1.4 (first dispatch, committed): crates/ail/src/main.rs build_staticlib rejects --alloc != Rc after the has_export check, before lowering. Task-1 pin RED→GREEN; embed_staticlib_cli default path unaffected.
• iter embedding-abi-m2.1.5 (this re-dispatch): per-thread-ctx capability demo. New crates/ail/tests/embed/swarm.c (no SHARED_CTX) + crates/ail/tests/embed_swarm_tsan.rs; scalar_swarm_per_ctx_is_tsan_clean_and_matches_oracle GREEN (8 threads × 200k, tsan-clean, every result == serial oracle). De-globalisation negative control rc_accounting_shared_ctx_is_flagged_by_tsan appended to the committed embed_rc_accounting_tsan.rs (reuses its ws_root()/ cc() helpers); GREEN — shared-ctx build genuinely raced under tsan (teeth proven, not vacuous). No swarm.c negative control reintroduced (the Boss spec-defect repair).
• iter embedding-abi-m2.1.6 (this re-dispatch): crates/ail/tests/ embed/host.c replaced with the ctx-ABI version (opaque ailang_ctx_t, ailang_ctx_new/_free lifecycle, ctx-prefixed backtest_step). Confirmed expected RED post-Task-3 first (s != 25 from the garbage-ctx arg shift). embed_e2e.rs intentionally NOT modified: the M1 harness already links libailang_rt.a by absolute path (.arg(outdir.join( "libailang_rt.a"))), so the ctx symbols resolve without an -lailang_rt add — the task block's own "no change is needed beyond Step 2" conditional branch. GREEN, s == 25 holds.
• iter embedding-abi-m2.1.7 (this re-dispatch): docs/DESIGN.md — provisional-until-M3 sentence (:2282) replaced with the M2 ctx prose (mandatory leading ailang_ctx_t*, per-thread, RC-only swarm artefact, value/record layout still provisional); Decision-10 atomicity bullet (:1528) gains the per-thread-ctx soundness note. Pinned "Export parameters are written bare…" sentence + canonical (con Int) snippet preserved verbatim (the edit ends before them; no soft-wrap split risk). docs_honesty_pin GREEN unmodified (5 passed), design_schema_drift GREEN (8 passed) — edits outside the ## Data model scan window.
• iter embedding-abi-m2.1.8 (this re-dispatch): regression gate, no code. print_no_leak_pin (1) + e2e -- rc_stats (4) GREEN — null-ctx fallback prints unchanged. embed_export_hash_stable (1, fn_without_export_hash_is_unchanged) + embed_export_gate (6, all pass) GREEN — no schema field, export gate unchanged. Full cargo test --workspace: every suite ok, 0 failing suites.
• iter embedding-abi-m2.1.9 (this re-dispatch): bench gate, no code. compile_check.py exit=0 (24/24 stable); cross_lang.py exit=0 (25/25 stable); check.py exit=1 on exactly 4 tracked-P2 noise metrics (bench_list_sum.bump_s +16.53%, latency.explicit_at_rc.max_us +40.41%, latency.implicit_at_rc.p99_9_us +31.66%, latency.implicit_at_rc.max_us +92.35%). Causally exonerated by the source-level invariant (see Concerns); no baseline ratified.

Concerns

• iter embedding-abi-m2.1.3 (first dispatch — carried forward): PLAN TRANSCRIPTION ERROR. Task 3 Step 1's plan code block asserted ir.contains("call i64 @ail_backtest_step(i64 %a0, i64 %a1)"). The internal symbol is @ail_embed_backtest_step_step (module-qualified @ail_<mname>_<fn>), which the plan's own Step-4 parenthetical and Boss constraint 3 require to stay byte-unchanged. Written instead with the convention-correct symbol (Boss-corrected in the re-dispatched plan). The plan file should remain corrected.
• iter embedding-abi-m2.1.6: embed_e2e.rs is in the task's Files: block but was intentionally left unchanged — the M1 harness already links libailang_rt.a by absolute path, so the ctx symbols resolve without an -lailang_rt add. This is the task block's own sanctioned "no change is needed beyond Step 2" conditional branch, not a missed requirement (observation, not correctness).
• iter embedding-abi-m2.1.7: the section header is still ## Embedding ABI (M1) while the body now describes the M2 ctx ABI as current state — a header/content honesty mismatch. NOT fixed: renaming the header is outside the task block's named edit regions (:2282–2285, :1528–1529) and could perturb a doc-honesty pin tracking the (M1) token; the controller curates scope, not the implementer. Routed here for Boss adjudication — a candidate follow-up (header rename + fieldtest/pin audit) but explicitly NOT in this iter's plan scope.
• iter embedding-abi-m2.1.8: the task block expected embed_export_gate to be "8 passed"; the suite at HEAD c9a84b3 has exactly 6 tests, all passing, 0 filtered. The literal "8" is a plan-transcription artefact (pre-existing — M2 added/removed no test in this suite). The protected invariant (export gate unchanged by M2) holds unambiguously; the implementer correctly did NOT pad the suite to 8 (that would fabricate coverage). Plan number vs reality, not a regression.
• iter embedding-abi-m2.1.9: check.py exit=1 on 4 metrics, all in the tracked-P2 noise families (*.bump_s, latency.*.max_us, latency.*.p99_9_us). CAUSALLY EXONERATED by a source-level invariant stronger than the plan's binary cmp: this re-dispatch's entire working-tree footprint is test fixtures + test drivers + docs/DESIGN.md — ZERO files under crates/*/src/ or runtime/. The ail codegen binary and the linked rc.c/str.c runtime are byte-identical to HEAD c9a84b3 by construction, so the firing metrics cannot be caused by this iter. The plan's git stash/ git stash pop recipe was deliberately NOT run: on a re-dispatch it would discard the iter's uncommitted Tasks 5–8 work and violate the working-tree-as-handoff contract. No baseline ratified (consistent with roadmap-P2 noise policy + the 2026-05-14 audit-pd "baseline pristine, Nth consecutive observation" envelope).

Known debt

• ## Embedding ABI (M1) section header in DESIGN.md still says "(M1)" though its body is now M2 current-state. Out of this iter's plan scope (Task 7 named only :2282–2285 and :1528–1529). Candidate follow-up: header rename with a doc-honesty-pin audit for any (M1) token dependency. Not touched to keep the diff scope-compliant and avoid perturbing a pin.

Blocked detail

None. All 9 tasks completed (Tasks 1–4 committed at c9a84b3 in the first dispatch; Tasks 5–9 in this re-dispatch, all reaching approved quality with zero review re-loops). The first dispatch's Task-5 BLOCK was a genuine spec defect, Boss-adjudicated and repaired before this re-dispatch (see "Boss spec-defect repair").

Files touched (this re-dispatch — Tasks 5–9 working tree)

Modified:

• crates/ail/tests/embed/host.c (Task 6: ctx-ABI migration)
• crates/ail/tests/embed_rc_accounting_tsan.rs (Task 5: + rc_accounting_shared_ctx_is_flagged_by_tsan negative control)
• docs/DESIGN.md (Task 7: M2 current-state + Decision-10 note)

Created:

• crates/ail/tests/embed/swarm.c (Task 5: per-thread-ctx capability demo, no SHARED_CTX)
• crates/ail/tests/embed_swarm_tsan.rs (Task 5: one green positive test)

(Tasks 1–4's files — embed_staticlib_alloc_guard.rs, rc.c, codegen lib.rs, embed_staticlib_lowering.rs, main.rs, rc_accounting_tsan.c, the Task-2 embed_rc_accounting_tsan.rs base — are already committed at c9a84b3.)

Stats

bench/orchestrator-stats/2026-05-18-iter-embedding-abi-m2.1.json