From b3388c83506a314fbd87f3354f80ae8cc212f501 Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 18 May 2026 16:58:43 +0200 Subject: [PATCH] =?UTF-8?q?plan:=20embedding-abi-m2.1=20=E2=80=94=20per-th?= =?UTF-8?q?read=20runtime=20context=20+=20concurrency=20safety=20(9=20task?= =?UTF-8?q?s,=20RED-first,=20task=201=20fixed-first=20per=20spec=20sequenc?= =?UTF-8?q?ing)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/plans/embedding-abi-m2.1.md | 809 +++++++++++++++++++++++++++++++ 1 file changed, 809 insertions(+) create mode 100644 docs/plans/embedding-abi-m2.1.md diff --git a/docs/plans/embedding-abi-m2.1.md b/docs/plans/embedding-abi-m2.1.md new file mode 100644 index 0000000..c293591 --- /dev/null +++ b/docs/plans/embedding-abi-m2.1.md @@ -0,0 +1,809 @@ +# Embedding ABI — M2.1: per-thread runtime context + concurrency safety — Implementation Plan + +> **Parent spec:** `docs/specs/2026-05-18-embedding-abi-m2.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement` +> to run this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Make the compiled scalar kernel callable concurrently from a host thread swarm without an RC-runtime data race, by threading a mandatory `ailang_ctx_t*` first parameter through the C `@` forwarder (TLS transport, internal convention byte-untouched) and de-globalising the RC accounting onto the ctx for the swarm path while retaining `g_rc_*`+atexit as the null-ctx executable fallback. + +**Architecture:** Three code layers change — `runtime/rc.c` (new ctx struct + lifecycle + `__thread` slot + conditional accounting split), `crates/ailang-codegen` `Target::StaticLib` forwarder only (leading `ptr %ctx`, TLS save/store/restore around the unchanged internal call), `crates/ail` `build_staticlib` (reject `--alloc != rc`). Plus DESIGN.md current-state update. No schema field, no `ailang-check` change, no Form-A change; the internal `@ail__` convention and the `_adapter`/`_clos` pair stay byte-identical. + +**Tech Stack:** C11 (`runtime/rc.c`, `__thread`, pthread test harnesses, `-fsanitize=thread`), Rust (`ailang-codegen` IR emission, `ailang` CLI, integration tests), LLVM IR text, `clang`/`ar`. + +--- + +## Files this plan creates or modifies + +- Modify: `runtime/rc.c:86–103,113–127,161–177` — `ailang_ctx_t` struct + `ailang_ctx_new`/`ailang_ctx_free` + `__thread __ail_tls_ctx`; conditional `if (ctx) ctx->… else g_rc_…` accounting split; statics+atexit+constructor retained as null-ctx fallback. +- Modify: `crates/ailang-codegen/src/lib.rs:600–638` — `Target::StaticLib` forwarder: leading `ptr %ctx`, one `@__ail_tls_ctx = external thread_local global ptr` decl, TLS save/store/restore around the unchanged `call @ail__`. +- Modify: `crates/ailang-codegen/src/lib.rs:165–176` — `Target::StaticLib` doc comment: narrow "provisional until M3" to the value/record layout. +- Modify: `crates/ail/src/main.rs:2485–2493` — `build_staticlib`: reject `alloc != AllocStrategy::Rc` with the RC-only diagnostic before lowering. +- Modify: `docs/DESIGN.md:2263–2307` — §"Embedding ABI (M1)" updated in place to post-M2 current state (ctx mandatory; per-thread RC-only swarm; accounting per ctx; "provisional until M3" narrowed to value/record layout). The "Export parameters are written **bare**…" sentence + the canonical `(con Int)` snippet preserved verbatim. +- Modify: `docs/DESIGN.md:1528–1529` — Decision 10 atomicity clause: one-line per-thread-ctx note. +- Modify: `crates/ailang-codegen/tests/embed_staticlib_lowering.rs:20–33` — `staticlib_emits_forwarder_no_main` extended for the ctx forwarder shape. +- Modify: `crates/ail/tests/embed/host.c` — migrated to the ctx ABI (M1 value oracle, single-threaded). +- Modify: `crates/ail/tests/embed_e2e.rs:17–46` — `c_host_calls_exported_scalar_kernel` migrated to drive the ctx-ABI `host.c`. +- Create: `crates/ail/tests/embed/rc_accounting_tsan.c` — direct `ailang_rc_alloc`/`ailang_rc_dec`+`ailang_ctx_*` N-thread harness (links `libailang_rt.a` only; the de-globalisation teeth — item 1). +- Create: `crates/ail/tests/embed/swarm.c` — N-pthread scalar-kernel swarm; per-thread-ctx mode + `-DSHARED_CTX` negative control (items 2, 3). +- Create: `crates/ail/tests/embed_swarm_tsan.rs` — driver: builds the staticlib + the two new C harnesses under `-fsanitize=thread`, asserts the matrix. +- Create: `crates/ail/tests/embed_staticlib_alloc_guard.rs` — RED-first pin: `--emit=staticlib --alloc=gc|bump` rejected (Task 1). +- Test (regression, unmodified): `crates/ail/tests/print_no_leak_pin.rs`, `crates/ail/tests/e2e.rs` (leak-stat helper), `crates/ailang-core/tests/embed_export_hash_stable.rs`, `crates/ailang-core/tests/design_schema_drift.rs`, `crates/ailang-check/tests/embed_export_gate.rs`, `crates/ailang-core/tests/docs_honesty_pin.rs`. + +--- + +## Task 1: Baseline pins (FIXED FIRST — spec §Iteration sequencing) + +Ships before any runtime/codegen/CLI change. (a) names the existing executable-path readback tests as the standing null-ctx regression guard; (b) RED-first pins the `--alloc`-guard final behaviour (RED now ⇒ proves current success). + +**Files:** +- Create: `crates/ail/tests/embed_staticlib_alloc_guard.rs` + +- [ ] **Step 1: Confirm the executable-path readback baseline is green (0a guard, no new test)** + +Run: `cargo test -p ail --test print_no_leak_pin -- alloc_rc_print_int_does_not_leak_show_result_str` +Expected: PASS (`test result: ok. 1 passed`) + +Run: `cargo test -p ail --test e2e -- alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells` +Expected: PASS (`test result: ok. 1 passed`) + +These two named tests are the null-ctx-fallback regression guard. The plan's runtime task (Task 2) must keep them green unmodified; Task 8 re-asserts. + +- [ ] **Step 2: Write the RED-first alloc-guard pin** + +Create `crates/ail/tests/embed_staticlib_alloc_guard.rs`: + +```rust +//! M2: `ail build --emit=staticlib` is RC-only. `--alloc=gc`/`--alloc=bump` +//! must fail the build (the shared Boehm collector / bench stub are not +//! swarm-safe). RED until the Task-4 CLI guard lands: today the alloc +//! strategy is passed straight through and the build SUCCEEDS. +use std::process::Command; + +fn ail_bin() -> &'static str { env!("CARGO_BIN_EXE_ail") } + +fn build_staticlib_with_alloc(alloc: &str, outdir: &str) -> std::process::Output { + Command::new(ail_bin()) + .args([ + "build", "examples/embed_backtest_step.ail", + "--emit=staticlib", &format!("--alloc={alloc}"), "-o", outdir, + ]) + .current_dir(env!("CARGO_MANIFEST_DIR").to_string() + "/../..") + .output() + .expect("spawn ail build") +} + +#[test] +fn staticlib_gc_is_rejected() { + let out = build_staticlib_with_alloc("gc", "/tmp/ail_m2_guard_gc"); + assert!( + !out.status.success(), + "expected `--emit=staticlib --alloc=gc` to FAIL the build; exit was success" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("staticlib (swarm) artefact is RC-only"), + "expected the RC-only diagnostic; stderr was:\n{stderr}" + ); +} + +#[test] +fn staticlib_bump_is_rejected() { + let out = build_staticlib_with_alloc("bump", "/tmp/ail_m2_guard_bump"); + assert!( + !out.status.success(), + "expected `--emit=staticlib --alloc=bump` to FAIL the build; exit was success" + ); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + stderr.contains("staticlib (swarm) artefact is RC-only"), + "expected the RC-only diagnostic; stderr was:\n{stderr}" + ); +} +``` + +- [ ] **Step 3: Run the pin to verify it is RED (proves current success — spec item 0b)** + +Run: `cargo test -p ail --test embed_staticlib_alloc_guard` +Expected: FAIL — both tests fail at the first assertion with +`expected ... to FAIL the build; exit was success` (today `build_staticlib` passes the strategy through and the build succeeds, producing the two archives). This RED state IS the "currently succeeds" baseline capture. + +--- + +## Task 2: Runtime — `ailang_ctx_t` + lifecycle + `__ail_tls_ctx` + conditional accounting (de-globalisation teeth) + +`runtime/rc.c` is C — no cargo compile gate. Its regression coverage is the direct N-thread rc-accounting tsan harness (spec item 1), which this task ships RED-first alongside the change. Harness links **only** `libailang_rt.a` (never the kernel) — keeps the de-globalisation proof distinct from the capability demo. + +**Files:** +- Create: `crates/ail/tests/embed/rc_accounting_tsan.c` +- Modify: `runtime/rc.c:86–103,113–127,161–177` +- (driver test fn added to the new `crates/ail/tests/embed_swarm_tsan.rs` in Task 5; this task adds only the rc-accounting C file + its standalone build/run inside a `#[test]` here) +- Create: `crates/ail/tests/embed_rc_accounting_tsan.rs` + +- [ ] **Step 1: Write the direct rc-accounting tsan C harness (RED — symbols absent today)** + +Create `crates/ail/tests/embed/rc_accounting_tsan.c`: + +```c +/* M2 item 1: the de-globalisation teeth. Exercises ailang_rc_alloc / + * ailang_rc_dec DIRECTLY (not through any kernel) from N threads, each + * owning its own ailang_ctx_t. Per-ctx counters must be individually + * correct and -fsanitize=thread-clean. Build -DSHARED_CTX for the + * global-hammering negative control (must make tsan report a race). */ +#include +#include +#include +#include +#include + +typedef struct ailang_ctx ailang_ctx_t; +extern ailang_ctx_t *ailang_ctx_new(void); +extern void ailang_ctx_free(ailang_ctx_t *); +extern void *ailang_rc_alloc(size_t); +extern void ailang_rc_dec(void *); +/* set the per-call ctx the runtime accounts into (defined in rc.c) */ +extern __thread ailang_ctx_t *__ail_tls_ctx; + +#define NTHREADS 8 +#define NCYCLES 200000 + +#ifdef SHARED_CTX +static ailang_ctx_t *g_shared; /* negative control: one ctx, all threads */ +#endif + +static void *worker(void *_a) { + (void)_a; +#ifdef SHARED_CTX + ailang_ctx_t *ctx = g_shared; +#else + ailang_ctx_t *ctx = ailang_ctx_new(); +#endif + __ail_tls_ctx = ctx; + for (int i = 0; i < NCYCLES; i++) { + void *p = ailang_rc_alloc(16); + ailang_rc_dec(p); /* refcount 1 -> 0, frees, free_count++ */ + } +#ifndef SHARED_CTX + /* per-ctx balance must hold exactly */ + /* (read via the same struct layout the runtime defines) */ + ailang_ctx_free(ctx); +#endif + return NULL; +} + +int main(void) { +#ifdef SHARED_CTX + g_shared = ailang_ctx_new(); +#endif + pthread_t t[NTHREADS]; + for (int i = 0; i < NTHREADS; i++) pthread_create(&t[i], NULL, worker, NULL); + for (int i = 0; i < NTHREADS; i++) pthread_join(t[i], NULL); +#ifdef SHARED_CTX + ailang_ctx_free(g_shared); +#endif + printf("rc_accounting_tsan: ok\n"); + return 0; +} +``` + +- [ ] **Step 2: Write the driver and run it to verify RED** + +Create `crates/ail/tests/embed_rc_accounting_tsan.rs`: + +```rust +//! M2 item 1: direct rc-accounting de-globalisation proof. Compiles +//! runtime/rc.c + runtime/str.c into libailang_rt.a, links the +//! rc_accounting_tsan.c harness against it under -fsanitize=thread. +//! Per-thread-ctx build: tsan-clean + exit 0. RED until rc.c gains +//! ailang_ctx_* + __ail_tls_ctx (link error: undefined symbols). +use std::process::Command; +use std::path::PathBuf; + +fn ws_root() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..").canonicalize().unwrap() +} + +fn cc(args: &[&str], ctx: &str) { + let st = Command::new("clang").args(args).status().expect("spawn clang"); + assert!(st.success(), "clang failed: {ctx}"); +} + +#[test] +fn rc_accounting_per_ctx_is_tsan_clean() { + let root = ws_root(); + let tmp = std::env::temp_dir().join("ail_m2_rcacct"); + std::fs::create_dir_all(&tmp).unwrap(); + let rc_o = tmp.join("rc.o"); + let str_o = tmp.join("str.o"); + let rt_a = tmp.join("libailang_rt.a"); + cc(&["-O1", "-g", "-fsanitize=thread", "-c", "-o", rc_o.to_str().unwrap(), + root.join("runtime/rc.c").to_str().unwrap()], "rc.c"); + cc(&["-O1", "-g", "-fsanitize=thread", "-c", "-o", str_o.to_str().unwrap(), + root.join("runtime/str.c").to_str().unwrap()], "str.c"); + let st = Command::new("ar").args(["rcs", rt_a.to_str().unwrap(), + rc_o.to_str().unwrap(), str_o.to_str().unwrap()]).status().unwrap(); + assert!(st.success(), "ar failed"); + let bin = tmp.join("rcacct"); + cc(&["-O1", "-g", "-fsanitize=thread", "-pthread", + "-o", bin.to_str().unwrap(), + root.join("crates/ail/tests/embed/rc_accounting_tsan.c").to_str().unwrap(), + rt_a.to_str().unwrap()], "link rc_accounting_tsan"); + let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1") + .output().expect("run harness"); + assert!(out.status.success(), + "per-ctx rc-accounting harness must exit 0 tsan-clean; stderr:\n{}", + String::from_utf8_lossy(&out.stderr)); +} +``` + +Run: `cargo test -p ail --test embed_rc_accounting_tsan` +Expected: FAIL — `clang failed: link rc_accounting_tsan` (undefined +references to `ailang_ctx_new`, `ailang_ctx_free`, `__ail_tls_ctx` — +absent from today's `rc.c`). + +- [ ] **Step 3: Implement the rc.c ctx struct + lifecycle + TLS slot** + +In `runtime/rc.c`, immediately after the `g_rc_free_count` static (currently `rc.c:87`), insert: + +```c +/* M2: per-thread embedding context. A scalar kernel allocates nothing, + * so this carries only the de-globalised RC accounting. Set by the + * generated C `@` forwarder into __ail_tls_ctx for the synchronous + * duration of one call (never held across a suspension point). The + * g_rc_* statics + atexit below are RETAINED as the null-ctx + * (single-threaded executable) fallback. */ +typedef struct ailang_ctx { + uint64_t alloc_count; + uint64_t free_count; +} ailang_ctx_t; + +__thread ailang_ctx_t *__ail_tls_ctx = NULL; + +ailang_ctx_t *ailang_ctx_new(void) { + return (ailang_ctx_t *)calloc(1, sizeof(ailang_ctx_t)); +} + +void ailang_ctx_free(ailang_ctx_t *ctx) { + if (ctx != NULL) { + const char *flag = getenv("AILANG_RC_STATS"); + if (flag != NULL && flag[0] != '\0') { + fprintf(stderr, + "ailang_rc_stats: allocs=%llu frees=%llu live=%lld\n", + (unsigned long long)ctx->alloc_count, + (unsigned long long)ctx->free_count, + (long long)(ctx->alloc_count - ctx->free_count)); + } + } + free(ctx); +} +``` + +- [ ] **Step 4: Split the two increment sites (swarm → ctx, else → global)** + +In `ailang_rc_alloc`, replace the line `g_rc_alloc_count++;` (currently `rc.c:125`) with: + +```c + ailang_ctx_t *_ctx = __ail_tls_ctx; + if (_ctx != NULL) _ctx->alloc_count++; else g_rc_alloc_count++; +``` + +In `ailang_rc_dec`, inside the `if (*hdr == 0)` block, replace the line `g_rc_free_count++;` (currently `rc.c:175`) with: + +```c + ailang_ctx_t *_ctx = __ail_tls_ctx; + if (_ctx != NULL) _ctx->free_count++; else g_rc_free_count++; +``` + +Leave `g_rc_alloc_count`/`g_rc_free_count` (`:86–87`), `ailang_rc_stats_atexit` (`:89–95`), and `ailang_rc_stats_install` (`:97–103`) **verbatim** — they are now the null-ctx fallback only. + +- [ ] **Step 5: Run the item-1 harness to verify GREEN** + +Run: `cargo test -p ail --test embed_rc_accounting_tsan` +Expected: PASS (`test result: ok. 1 passed`) — links cleanly, +per-ctx counters correct, zero tsan reports. + +--- + +## Task 3: Codegen — `Target::StaticLib` forwarder gains `ptr %ctx` + TLS save/store/restore + +`lower_workspace_staticlib*` Rust signatures are unchanged (recon-confirmed) — no E0061 caller cascade; the only Rust gate is the codegen crate's own build + the extended IR-shape test. `embed_e2e.rs` is expected RED between this task and Task 6 (M1 `host.c` calls the now-`ptr`-prefixed symbol); its migration is Task 6 — do NOT add `embed_e2e` to this task's GREEN gate. + +**Files:** +- Modify: `crates/ailang-codegen/src/lib.rs:600–638` and `:165–176` +- Modify: `crates/ailang-codegen/tests/embed_staticlib_lowering.rs:20–33` + +- [ ] **Step 1: Extend the RED IR-shape pin** + +In `crates/ailang-codegen/tests/embed_staticlib_lowering.rs`, in `staticlib_emits_forwarder_no_main`, after the existing assertions on the forwarder, add (keep all existing assertions, including the no-`@main` one and the `_adapter`/`_clos` ones, unchanged): + +```rust + // M2: forwarder takes a leading ptr %ctx and routes it through TLS + // around the UNCHANGED internal call. + assert!(ir.contains("@__ail_tls_ctx = external thread_local global ptr"), + "staticlib module must declare the external thread-local ctx slot;\nIR:\n{ir}"); + assert!(ir.contains("define i64 @backtest_step(ptr %ctx, i64 %a0, i64 %a1)"), + "forwarder must take a leading ptr %ctx;\nIR:\n{ir}"); + assert!(ir.contains("store ptr %ctx, ptr @__ail_tls_ctx"), + "forwarder must publish ctx into the TLS slot;\nIR:\n{ir}"); + assert!(ir.contains("call i64 @ail_backtest_step(i64 %a0, i64 %a1)"), + "internal call must stay byte-unchanged (no ctx arg);\nIR:\n{ir}"); +``` + +- [ ] **Step 2: Run to verify RED** + +Run: `cargo test -p ailang-codegen --test embed_staticlib_lowering -- staticlib_emits_forwarder_no_main` +Expected: FAIL — first new assertion fails: `staticlib module must declare the external thread-local ctx slot` (today's forwarder is the no-ctx M1 shape). + +- [ ] **Step 3: Emit the external thread-local decl once in the StaticLib arm** + +In `crates/ailang-codegen/src/lib.rs`, in the `Target::StaticLib => {` arm (currently opens at `:600`), immediately after the arm opens and before the `for (mname, m) in &ws.modules {` loop, insert: + +```rust + // M2: the per-thread embedding ctx slot. Defined by + // runtime/rc.c (__thread ailang_ctx_t *__ail_tls_ctx); + // the forwarder publishes its explicit ctx param here for + // the synchronous duration of the internal call. + out.push_str("\n@__ail_tls_ctx = external thread_local global ptr\n"); +``` + +- [ ] **Step 4: Thread `ptr %ctx` through the forwarder body** + +In the same arm, replace the forwarder `format!` block (currently `lib.rs:626–633`, the `out.push_str(&format!("\ndefine {cret} @{sym}(...` … `ret {cret} %r\n}}\n", …))`) with: + +```rust + let mut sig_params = vec!["ptr %ctx".to_string()]; + sig_params.extend( + param_tys.iter().enumerate() + .map(|(i, t)| format!("{} %a{i}", llvm_scalar(t)))); + out.push_str(&format!( + "\ndefine {cret} @{sym}({}) {{\n \ + %saved = load ptr, ptr @__ail_tls_ctx\n \ + store ptr %ctx, ptr @__ail_tls_ctx\n \ + %r = call {cret} @ail_{mname}_{}({})\n \ + store ptr %saved, ptr @__ail_tls_ctx\n \ + ret {cret} %r\n}}\n", + sig_params.join(", "), + f.name, + args.join(", "), + )); +``` + +(`args` — the internal-call argument vector built at `lib.rs:623–625` — is left exactly as is: the internal `@ail__` call takes the scalars only, byte-unchanged. `_adapter`/`_clos` emission is untouched.) + +- [ ] **Step 5: Narrow the provisional doc on the `Target` enum** + +In `crates/ailang-codegen/src/lib.rs:165–176`, in the `Target::StaticLib` doc comment, replace the phrase that calls the signature "provisional until M3" with wording that scopes provisionality to the value/record layout. Exact replacement — change the sentence reading (around `:173`) to: + +```rust + /// emit one external C entrypoint per `(export …)` fn forwarding + /// to the internal `@ail__`. M2: the entrypoint takes + /// a mandatory leading `ptr %ctx` (per-thread embedding context) + /// published via `@__ail_tls_ctx` around the unchanged internal + /// call. The ctx-threaded C signature is the M2 shape; only the + /// value/record layout remains provisional until M3. +``` + +(Match and replace only the existing `StaticLib` doc lines; do not alter the `Executable` variant or its doc.) + +- [ ] **Step 6: Run to verify GREEN (codegen IR-shape only)** + +Run: `cargo test -p ailang-codegen --test embed_staticlib_lowering` +Expected: PASS (`test result: ok. N passed`) — both +`staticlib_emits_forwarder_no_main` (extended) and +`executable_target_still_emits_main` (unmodified) pass. + +Run: `cargo build -p ailang-codegen` +Expected: `Finished` (0 errors). + +--- + +## Task 4: CLI — `build_staticlib` `--alloc != rc` reject guard + +RED test already shipped in Task 1 (`embed_staticlib_alloc_guard`). This task is the GREEN side: the guard. + +**Files:** +- Modify: `crates/ail/src/main.rs:2485–2493` + +- [ ] **Step 1: Insert the reject guard before lowering** + +In `crates/ail/src/main.rs`, in `fn build_staticlib`, immediately before the `let ir = ailang_codegen::lower_workspace_staticlib_with_alloc(&ws, alloc)?;` line (currently `:2493`), insert: + +```rust + if !matches!(alloc, ailang_codegen::AllocStrategy::Rc) { + anyhow::bail!( + "staticlib (swarm) artefact is RC-only — `--alloc=gc` links the \ + shared Boehm collector, which is not swarm-safe; use `--alloc=rc`" + ); + } +``` + +(Placed after the `has_export` check and before lowering, so a non-RC staticlib request fails before any `.ll`/archive work. The `--alloc` clap default is already `rc`, so the default path is unaffected; this rejects only an *explicit* `--alloc=gc`/`--alloc=bump`.) + +- [ ] **Step 2: Run the Task-1 pin to verify RED→GREEN** + +Run: `cargo test -p ail --test embed_staticlib_alloc_guard` +Expected: PASS (`test result: ok. 2 passed`) — both +`staticlib_gc_is_rejected` and `staticlib_bump_is_rejected` now see a +non-zero exit with the `staticlib (swarm) artefact is RC-only` string. + +- [ ] **Step 3: Confirm the default (rc) staticlib build still works** + +Run: `cargo test -p ail --test embed_staticlib_cli` +Expected: PASS (`test result: ok. N passed`) — default-alloc +two-archive build path unaffected by the guard. + +--- + +## Task 5: Scalar-swarm capability harness + negative control (items 2, 3) + +Depends on Tasks 2+3 GREEN (ctx ABI present). Links **both** archives (kernel + runtime). Separate file from Task 2's harness — different link surface, distinct spec role (capability demo vs de-globalisation proof). + +**Files:** +- Create: `crates/ail/tests/embed/swarm.c` +- Create: `crates/ail/tests/embed_swarm_tsan.rs` + +- [ ] **Step 1: Write the swarm harness C file** + +Create `crates/ail/tests/embed/swarm.c`: + +```c +/* M2 items 2+3: scalar-kernel swarm. Per-thread-ctx (default) must be + * tsan-clean and every thread's result must match the serial oracle. + * Built -DSHARED_CTX = negative control: all threads share one ctx; + * tsan MUST report a race (proves the clean result is not vacuous). */ +#include +#include +#include +#include + +typedef struct ailang_ctx ailang_ctx_t; +extern ailang_ctx_t *ailang_ctx_new(void); +extern void ailang_ctx_free(ailang_ctx_t *); +extern int64_t backtest_step(ailang_ctx_t *ctx, int64_t state, int64_t sample); + +#define NTHREADS 8 +#define NITERS 200000 + +static int64_t serial_oracle(void) { + /* mirrors examples/embed_backtest_step.ail: state += sample*sample, + * sample = (i & 7) over NITERS iterations. */ + int64_t s = 0; + for (int i = 0; i < NITERS; i++) { int64_t x = i & 7; s = s + x * x; } + return s; +} + +#ifdef SHARED_CTX +static ailang_ctx_t *g_shared; +#endif + +static void *worker(void *out) { +#ifdef SHARED_CTX + ailang_ctx_t *ctx = g_shared; +#else + ailang_ctx_t *ctx = ailang_ctx_new(); +#endif + int64_t s = 0; + for (int i = 0; i < NITERS; i++) s = backtest_step(ctx, s, (i & 7)); +#ifndef SHARED_CTX + ailang_ctx_free(ctx); +#endif + *(int64_t *)out = s; + return NULL; +} + +int main(void) { +#ifdef SHARED_CTX + g_shared = ailang_ctx_new(); +#endif + pthread_t t[NTHREADS]; + int64_t res[NTHREADS]; + for (int i = 0; i < NTHREADS; i++) pthread_create(&t[i], NULL, worker, &res[i]); + for (int i = 0; i < NTHREADS; i++) pthread_join(t[i], NULL); +#ifdef SHARED_CTX + ailang_ctx_free(g_shared); +#endif + int64_t oracle = serial_oracle(); + for (int i = 0; i < NTHREADS; i++) { + if (res[i] != oracle) { fprintf(stderr, "thread %d: %lld != %lld\n", + i, (long long)res[i], (long long)oracle); return 1; } + } + printf("swarm: ok (%lld)\n", (long long)oracle); + return 0; +} +``` + +- [ ] **Step 2: Write the driver (per-thread-ctx GREEN; shared-ctx flagged)** + +Create `crates/ail/tests/embed_swarm_tsan.rs`: + +```rust +//! M2 items 2+3. Builds the staticlib (rc) via the `ail` CLI, then +//! links swarm.c against lib.a + libailang_rt.a under +//! -fsanitize=thread. Per-thread-ctx: tsan-clean, exit 0. Negative +//! control (-DSHARED_CTX): tsan MUST report (non-zero / report text). +use std::process::Command; +use std::path::PathBuf; + +fn ws_root() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..").canonicalize().unwrap() +} +fn ail_bin() -> &'static str { env!("CARGO_BIN_EXE_ail") } + +fn build_staticlib(outdir: &PathBuf) { + std::fs::create_dir_all(outdir).unwrap(); + let st = Command::new(ail_bin()) + .args(["build", "examples/embed_backtest_step.ail", + "--emit=staticlib", "--alloc=rc", "-o", outdir.to_str().unwrap()]) + .current_dir(ws_root()).status().expect("ail build"); + assert!(st.success(), "ail build --emit=staticlib failed"); +} + +fn link_swarm(outdir: &PathBuf, bin: &PathBuf, shared: bool) { + let root = ws_root(); + let mut args: Vec = vec![ + "-O1".into(), "-g".into(), "-fsanitize=thread".into(), "-pthread".into(), + "-o".into(), bin.to_string_lossy().into(), + root.join("crates/ail/tests/embed/swarm.c").to_string_lossy().into(), + ]; + if shared { args.push("-DSHARED_CTX".into()); } + args.push(format!("-L{}", outdir.to_string_lossy())); + args.push("-lembed_backtest_step".into()); + args.push("-lailang_rt".into()); + let st = Command::new("clang").args(&args).status().expect("clang"); + assert!(st.success(), "linking swarm.c failed (shared={shared})"); +} + +#[test] +fn scalar_swarm_per_ctx_is_tsan_clean_and_matches_oracle() { + let outdir = std::env::temp_dir().join("ail_m2_swarm_ok"); + build_staticlib(&outdir); + let bin = outdir.join("swarm_ok"); + link_swarm(&outdir, &bin, false); + let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1") + .output().expect("run swarm"); + assert!(out.status.success(), + "per-thread-ctx swarm must exit 0 tsan-clean; stderr:\n{}", + String::from_utf8_lossy(&out.stderr)); +} + +#[test] +fn shared_ctx_negative_control_is_flagged_by_tsan() { + let outdir = std::env::temp_dir().join("ail_m2_swarm_neg"); + build_staticlib(&outdir); + let bin = outdir.join("swarm_neg"); + link_swarm(&outdir, &bin, true); + let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1") + .output().expect("run swarm neg"); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!( + !out.status.success() || stderr.contains("WARNING: ThreadSanitizer"), + "shared-ctx negative control MUST be flagged by tsan (the harness \ + has teeth); exit={:?} stderr:\n{stderr}", out.status.code() + ); +} +``` + +- [ ] **Step 3: Run the swarm matrix** + +Run: `cargo test -p ail --test embed_swarm_tsan` +Expected: PASS (`test result: ok. 2 passed`) — +`scalar_swarm_per_ctx_is_tsan_clean_and_matches_oracle` exits 0 +tsan-clean with every thread == oracle; +`shared_ctx_negative_control_is_flagged_by_tsan` observes the tsan +race report (teeth proven). + +--- + +## Task 6: Migrate M1 `host.c` + `embed_e2e.rs` to the ctx ABI (item 5) + +After Task 3 the M1 `host.c` (no-ctx `backtest_step`) is RED (wrong call ABI). Migrate it; keep the M1 value assertion (`s == 25`). + +**Files:** +- Modify: `crates/ail/tests/embed/host.c` +- Modify: `crates/ail/tests/embed_e2e.rs:17–46` + +- [ ] **Step 1: Confirm `embed_e2e` is RED post-Task-3 (expected)** + +Run: `cargo test -p ail --test embed_e2e -- c_host_calls_exported_scalar_kernel` +Expected: FAIL — the M1 `host.c` calls `backtest_step(s, 3)` against +the now-`(ptr,i64,i64)` symbol (garbage ctx; `s != 25` or crash). This +is the expected RED that Task 3 deliberately left for this task. + +- [ ] **Step 2: Migrate `host.c` to the ctx ABI** + +Replace the entire contents of `crates/ail/tests/embed/host.c` with: + +```c +#include +#include +typedef struct ailang_ctx ailang_ctx_t; +extern ailang_ctx_t *ailang_ctx_new(void); +extern void ailang_ctx_free(ailang_ctx_t *); +extern int64_t backtest_step(ailang_ctx_t *ctx, int64_t state, int64_t sample); +int main(void) { + ailang_ctx_t *ctx = ailang_ctx_new(); + int64_t s = 0; + s = backtest_step(ctx, s, 3); /* 0 + 9 */ + s = backtest_step(ctx, s, 4); /* 9 + 16 */ + ailang_ctx_free(ctx); + assert(s == 25); + return 0; +} +``` + +- [ ] **Step 3: Update `embed_e2e.rs` link line for the runtime archive** + +In `crates/ail/tests/embed_e2e.rs`, in `c_host_calls_exported_scalar_kernel`, ensure the harness links `libailang_rt.a` alongside `lib.a` (M1 already produces both archives; the host now references `ailang_ctx_new`/`_free` which live in `libailang_rt.a`). If the existing link command already passes `-lailang_rt` (M1 two-archive harness), no change is needed beyond Step 2; otherwise add `-lailang_rt` to the `clang` link args next to the existing `-lembed_backtest_step` (the exact existing arg vector is in this fn — append `-lailang_rt` if absent). Do not change the asserted value (`s == 25`) or exit-0 expectation. + +- [ ] **Step 4: Run to verify GREEN** + +Run: `cargo test -p ail --test embed_e2e -- c_host_calls_exported_scalar_kernel` +Expected: PASS (`test result: ok. 1 passed`) — ctx-ABI host links, +runs, `s == 25` holds. + +--- + +## Task 7: DESIGN.md current-state update (preserve `docs_honesty_pin.rs:135`) + +`docs_honesty_pin.rs:135` asserts (via the `norm()` whitespace-collapse helper) that DESIGN.md contains `Export parameters are written **bare**: a scalar type carries no `own`/`borrow` mode`. M2's §"Embedding ABI (M1)" rewrite MUST keep that sentence and the canonical `(con Int)` snippet verbatim; edit only the provisional-until-M3 sentence and add the ctx/swarm prose. The pin then stays green unmodified. + +**Files:** +- Modify: `docs/DESIGN.md:2282–2285` +- Modify: `docs/DESIGN.md:1528–1529` + +- [ ] **Step 1: Replace only the provisional-until-M3 sentence** + +In `docs/DESIGN.md`, in §"Embedding ABI (M1)", replace the sentence currently at `:2282–2285` reading: + +``` +M1 ships no runtime lifecycle API: the per-thread +context parameter that M2 threads through every exported call will +change this C signature; do not treat the M1 signature as frozen. +The value/record layout freeze is M3. +``` + +with: + +``` +Every exported entrypoint takes a mandatory leading `ailang_ctx_t*` +(M2): a per-thread embedding context created by `ailang_ctx_new()` +and released by `ailang_ctx_free()`, owned by the calling thread for +its lifetime. The host links one `ailang_ctx_t` per OS worker thread +and the runtime accounts RC alloc/free into it (no shared mutable +runtime state — the swarm artefact is data-race-free, sanitiser- +verified). The staticlib swarm artefact is **RC-only**: +`ail build --emit=staticlib` rejects `--alloc=gc`/`--alloc=bump` +(the shared Boehm collector is not swarm-safe). Only the value/record +layout remains provisional until M3; the ctx-threaded C signature is +the M2 shape. +``` + +Do NOT touch the paragraph at `:2287–2289` ("Export parameters are written **bare**: a scalar type carries no `own`/`borrow` mode …") or the canonical `(con Int)` snippet at `:2292–2302` — they are pinned and orthogonal to M2. + +- [ ] **Step 2: Add the Decision-10 per-thread-ctx note** + +In `docs/DESIGN.md`, the Decision 10 atomicity bullet currently at `:1528–1529`: + +``` +- **Does not commit to atomic refcounts.** AILang is + single-threaded; refcounts are non-atomic. +``` + +Replace with: + +``` +- **Does not commit to atomic refcounts.** AILang is + single-threaded; refcounts are non-atomic. The embedding ABI's + per-thread `ailang_ctx_t` keeps this correct under a host swarm: + each thread's allocations are private to its ctx, no RC cell + crosses a thread, so non-atomic refcounts stay sound. +``` + +- [ ] **Step 3: Verify the honesty pin + schema-drift stay green unmodified** + +Run: `cargo test -p ailang-core --test docs_honesty_pin` +Expected: PASS (`test result: ok. N passed`) — the preserved +"Export parameters are written **bare**…" sentence keeps the +`:135` assertion green; no pin edit. + +Run: `cargo test -p ailang-core --test design_schema_drift` +Expected: PASS — M2 edits `DESIGN.md` only outside the `## Data +model` scan window (2263–2307 and 1528–1529); anchors unaffected. + +--- + +## Task 8: Executable-path + schema/hash invariance regression gate (items 6, 7) + +Pure verification — no code. Asserts the null-ctx fallback and the no-schema-change promises hold. + +**Files:** (none modified — regression gate) + +- [ ] **Step 1: Executable-path AILANG_RC_STATS readback unchanged** + +Run: `cargo test -p ail --test print_no_leak_pin` +Expected: PASS (`test result: ok. N passed`) — unmodified; +the `g_rc_*`+atexit null-ctx fallback prints exactly as before. + +Run: `cargo test -p ail --test e2e -- rc_stats` +Expected: PASS — the leak-stat helper consumers +(`alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells` and the +sibling `*rc_stats*` tests) green; the `ailang_rc_stats:` atexit line +shape preserved. + +- [ ] **Step 2: No schema field — hash + drift invariant** + +Run: `cargo test -p ailang-core --test embed_export_hash_stable` +Expected: PASS (`fn_without_export_hash_is_unchanged` green) — M2 +adds no schema field. + +Run: `cargo test -p ailang-check --test embed_export_gate` +Expected: PASS (`test result: ok. 8 passed`) — the M1 scalar/effect +export gate is unchanged by M2. + +- [ ] **Step 3: Full workspace gate** + +Run: `cargo test --workspace 2>&1 | tail -5` +Expected: every suite `test result: ok`; 0 failed. (Diff against the +pre-M2 baseline: the only test files added are +`embed_staticlib_alloc_guard`, `embed_rc_accounting_tsan`, +`embed_swarm_tsan`; the only behaviour-migrated test is +`embed_e2e::c_host_calls_exported_scalar_kernel`.) + +--- + +## Task 9: Bench-trio carry-on + null-ctx-fallback bench-neutrality (item 9) + +The null-ctx fallback adds `load __thread ptr; brif null` ahead of the executable path's `g_rc_*++`. Demonstrate bench-neutral, not hand-waved. + +**Files:** (none modified — regression gate) + +- [ ] **Step 1: Run the regression trio** + +Run: `python3 bench/check.py; echo "exit=$?"` +Expected: `exit=0`. If exit=1 on a `*.bump_s` or +`latency.*.max_us` metric, that is a tracked-P2 noise family +(roadmap P2) — proceed to Step 2 for causal exoneration; do NOT +ratify a baseline. + +Run: `python3 bench/compile_check.py; echo "exit=$?"` +Expected: `exit=0`. + +Run: `python3 bench/cross_lang.py; echo "exit=$?"` +Expected: `exit=0`. + +- [ ] **Step 2: Causal exoneration if a tracked-noise metric fires** + +If and only if `check.py` exited 1 on a tracked-noise metric, build +the benched binary at the pre-M2 commit and at HEAD and byte-compare: + +Run: +``` +git stash; git rev-parse HEAD > /tmp/m2_head +cargo build --release -p ail 2>/dev/null +./target/release/ail build examples/list_sum.ail --alloc=rc -o /tmp/bench_head +git stash pop +cargo build --release -p ail 2>/dev/null +./target/release/ail build examples/list_sum.ail --alloc=rc -o /tmp/bench_m2 +cmp /tmp/bench_head/list_sum /tmp/bench_m2/list_sum && echo "BYTE-IDENTICAL: causally exonerated (no ratify)" +``` +Expected: for the *executable* path, `list_sum` does not allocate +across a ctx (null-ctx fallback), and the rc.c change adds only the +`load/brif null` ahead of `g_rc_*++`; if the metric is a tracked +noise family the binaries differ only by the fallback branch — record +the exoneration in the iter journal, do NOT update any baseline. +(If the binaries are NOT byte-identical AND a non-noise metric +regressed, STOP and bounce to the Boss: that is a real codegen +regression, not carry-on.) + +--- + +## Self-review (Step 5 — completed inline) + +1. **Spec coverage:** §Architecture → Tasks 2–4,7; §Concrete code shapes (impl) → Tasks 2,3,4; §Components table → every row mapped (rc.c T2; forwarder T3; CLI T4; DESIGN.md T7; embed_e2e T6; tests T1,T2,T5,T8); §Testing items 0→T1, 1→T2, 2/3→T5, 4→T3, 5→T6, 6→T8, 7→T8, 8→T8/architect-at-audit, 9→T9; §Acceptance/§Iteration sequencing → T1 fixed-first honoured. ✓ +2. **Placeholder scan:** no "TBD/TODO/implement later/similar to/add appropriate" — grep-clean. ✓ +3. **Type/name consistency:** `ailang_ctx_t`, `ailang_ctx_new`, `ailang_ctx_free`, `__ail_tls_ctx`, `@__ail_tls_ctx`, `backtest_step`, `staticlib_gc_is_rejected`, `embed_swarm_tsan` used identically across T1–T9 and the file-map. ✓ +4. **Step granularity:** every step is one 2–5 min action (one file edit / one cargo run). ✓ +5. **No commit steps:** none present (Boss commits at iter end). ✓ +6. **Pin/replacement contiguity:** the only presence-pin paired with a verbatim doc edit is `docs_honesty_pin.rs:135` ↔ DESIGN.md §"Embedding ABI (M1)". Task 7 Step 1 edits ONLY `:2282–2285` and explicitly forbids touching `:2287–2289` (the pinned sentence) — the pinned substring is never inside the replacement body, so no soft-wrap split is possible; `norm()` additionally collapses whitespace. ✓ +7. **Compile-gate vs deferred-caller:** no task changes a Rust fn signature (`lower_workspace_staticlib*` unchanged; `build_staticlib` adds an internal guard, signature unchanged). The one cross-task RED interval (Task 3 leaves `embed_e2e` RED until Task 6) is a *link/runtime* gate, explicitly named in both Task 3's preamble and Task 6 Step 1, and Task 3's GREEN gate scopes to `embed_staticlib_lowering` only — no silent resequencing. ✓ +8. **Verification filter strings resolve:** existing-test filters use recon-verified names (`alloc_rc_print_int_does_not_leak_show_result_str`, `alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells`, `staticlib_emits_forwarder_no_main`, `executable_target_still_emits_main`, `c_host_calls_exported_scalar_kernel`, `fn_without_export_hash_is_unchanged`); new-test filters resolve by construction (defined in the same task); the `-- rc_stats` filter (T8 S1) is backed by the unfiltered `print_no_leak_pin` run in the same step and `e2e.rs:2178` helper consumers are `*rc_stats*`-named (recon-confirmed) — and T8 S3 runs the unfiltered `cargo test --workspace` so "nothing ran" cannot masquerade as green. ✓