plan: embedding-abi-m2.1 — per-thread runtime context + concurrency safety (9 tasks, RED-first, task 1 fixed-first per spec sequencing)

This commit is contained in:
2026-05-18 16:58:43 +02:00
parent 1c58055170
commit b3388c8350
+809
View File
@@ -0,0 +1,809 @@
# Embedding ABI — M2.1: per-thread runtime context + concurrency safety — Implementation Plan
> **Parent spec:** `docs/specs/2026-05-18-embedding-abi-m2.md`
>
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
> to run this plan. Steps use `- [ ]` checkboxes for tracking.
**Goal:** Make the compiled scalar kernel callable concurrently from a host thread swarm without an RC-runtime data race, by threading a mandatory `ailang_ctx_t*` first parameter through the C `@<sym>` forwarder (TLS transport, internal convention byte-untouched) and de-globalising the RC accounting onto the ctx for the swarm path while retaining `g_rc_*`+atexit as the null-ctx executable fallback.
**Architecture:** Three code layers change — `runtime/rc.c` (new ctx struct + lifecycle + `__thread` slot + conditional accounting split), `crates/ailang-codegen` `Target::StaticLib` forwarder only (leading `ptr %ctx`, TLS save/store/restore around the unchanged internal call), `crates/ail` `build_staticlib` (reject `--alloc != rc`). Plus DESIGN.md current-state update. No schema field, no `ailang-check` change, no Form-A change; the internal `@ail_<mod>_<fn>` convention and the `_adapter`/`_clos` pair stay byte-identical.
**Tech Stack:** C11 (`runtime/rc.c`, `__thread`, pthread test harnesses, `-fsanitize=thread`), Rust (`ailang-codegen` IR emission, `ailang` CLI, integration tests), LLVM IR text, `clang`/`ar`.
---
## Files this plan creates or modifies
- Modify: `runtime/rc.c:86103,113127,161177``ailang_ctx_t` struct + `ailang_ctx_new`/`ailang_ctx_free` + `__thread __ail_tls_ctx`; conditional `if (ctx) ctx->… else g_rc_…` accounting split; statics+atexit+constructor retained as null-ctx fallback.
- Modify: `crates/ailang-codegen/src/lib.rs:600638``Target::StaticLib` forwarder: leading `ptr %ctx`, one `@__ail_tls_ctx = external thread_local global ptr` decl, TLS save/store/restore around the unchanged `call @ail_<mname>_<fn>`.
- Modify: `crates/ailang-codegen/src/lib.rs:165176``Target::StaticLib` doc comment: narrow "provisional until M3" to the value/record layout.
- Modify: `crates/ail/src/main.rs:24852493``build_staticlib`: reject `alloc != AllocStrategy::Rc` with the RC-only diagnostic before lowering.
- Modify: `docs/DESIGN.md:22632307` — §"Embedding ABI (M1)" updated in place to post-M2 current state (ctx mandatory; per-thread RC-only swarm; accounting per ctx; "provisional until M3" narrowed to value/record layout). The "Export parameters are written **bare**…" sentence + the canonical `(con Int)` snippet preserved verbatim.
- Modify: `docs/DESIGN.md:15281529` — Decision 10 atomicity clause: one-line per-thread-ctx note.
- Modify: `crates/ailang-codegen/tests/embed_staticlib_lowering.rs:2033``staticlib_emits_forwarder_no_main` extended for the ctx forwarder shape.
- Modify: `crates/ail/tests/embed/host.c` — migrated to the ctx ABI (M1 value oracle, single-threaded).
- Modify: `crates/ail/tests/embed_e2e.rs:1746``c_host_calls_exported_scalar_kernel` migrated to drive the ctx-ABI `host.c`.
- Create: `crates/ail/tests/embed/rc_accounting_tsan.c` — direct `ailang_rc_alloc`/`ailang_rc_dec`+`ailang_ctx_*` N-thread harness (links `libailang_rt.a` only; the de-globalisation teeth — item 1).
- Create: `crates/ail/tests/embed/swarm.c` — N-pthread scalar-kernel swarm; per-thread-ctx mode + `-DSHARED_CTX` negative control (items 2, 3).
- Create: `crates/ail/tests/embed_swarm_tsan.rs` — driver: builds the staticlib + the two new C harnesses under `-fsanitize=thread`, asserts the matrix.
- Create: `crates/ail/tests/embed_staticlib_alloc_guard.rs` — RED-first pin: `--emit=staticlib --alloc=gc|bump` rejected (Task 1).
- Test (regression, unmodified): `crates/ail/tests/print_no_leak_pin.rs`, `crates/ail/tests/e2e.rs` (leak-stat helper), `crates/ailang-core/tests/embed_export_hash_stable.rs`, `crates/ailang-core/tests/design_schema_drift.rs`, `crates/ailang-check/tests/embed_export_gate.rs`, `crates/ailang-core/tests/docs_honesty_pin.rs`.
---
## Task 1: Baseline pins (FIXED FIRST — spec §Iteration sequencing)
Ships before any runtime/codegen/CLI change. (a) names the existing executable-path readback tests as the standing null-ctx regression guard; (b) RED-first pins the `--alloc`-guard final behaviour (RED now ⇒ proves current success).
**Files:**
- Create: `crates/ail/tests/embed_staticlib_alloc_guard.rs`
- [ ] **Step 1: Confirm the executable-path readback baseline is green (0a guard, no new test)**
Run: `cargo test -p ail --test print_no_leak_pin -- alloc_rc_print_int_does_not_leak_show_result_str`
Expected: PASS (`test result: ok. 1 passed`)
Run: `cargo test -p ail --test e2e -- alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells`
Expected: PASS (`test result: ok. 1 passed`)
These two named tests are the null-ctx-fallback regression guard. The plan's runtime task (Task 2) must keep them green unmodified; Task 8 re-asserts.
- [ ] **Step 2: Write the RED-first alloc-guard pin**
Create `crates/ail/tests/embed_staticlib_alloc_guard.rs`:
```rust
//! M2: `ail build --emit=staticlib` is RC-only. `--alloc=gc`/`--alloc=bump`
//! must fail the build (the shared Boehm collector / bench stub are not
//! swarm-safe). RED until the Task-4 CLI guard lands: today the alloc
//! strategy is passed straight through and the build SUCCEEDS.
use std::process::Command;
fn ail_bin() -> &'static str { env!("CARGO_BIN_EXE_ail") }
fn build_staticlib_with_alloc(alloc: &str, outdir: &str) -> std::process::Output {
Command::new(ail_bin())
.args([
"build", "examples/embed_backtest_step.ail",
"--emit=staticlib", &format!("--alloc={alloc}"), "-o", outdir,
])
.current_dir(env!("CARGO_MANIFEST_DIR").to_string() + "/../..")
.output()
.expect("spawn ail build")
}
#[test]
fn staticlib_gc_is_rejected() {
let out = build_staticlib_with_alloc("gc", "/tmp/ail_m2_guard_gc");
assert!(
!out.status.success(),
"expected `--emit=staticlib --alloc=gc` to FAIL the build; exit was success"
);
let stderr = String::from_utf8_lossy(&out.stderr);
assert!(
stderr.contains("staticlib (swarm) artefact is RC-only"),
"expected the RC-only diagnostic; stderr was:\n{stderr}"
);
}
#[test]
fn staticlib_bump_is_rejected() {
let out = build_staticlib_with_alloc("bump", "/tmp/ail_m2_guard_bump");
assert!(
!out.status.success(),
"expected `--emit=staticlib --alloc=bump` to FAIL the build; exit was success"
);
let stderr = String::from_utf8_lossy(&out.stderr);
assert!(
stderr.contains("staticlib (swarm) artefact is RC-only"),
"expected the RC-only diagnostic; stderr was:\n{stderr}"
);
}
```
- [ ] **Step 3: Run the pin to verify it is RED (proves current success — spec item 0b)**
Run: `cargo test -p ail --test embed_staticlib_alloc_guard`
Expected: FAIL — both tests fail at the first assertion with
`expected ... to FAIL the build; exit was success` (today `build_staticlib` passes the strategy through and the build succeeds, producing the two archives). This RED state IS the "currently succeeds" baseline capture.
---
## Task 2: Runtime — `ailang_ctx_t` + lifecycle + `__ail_tls_ctx` + conditional accounting (de-globalisation teeth)
`runtime/rc.c` is C — no cargo compile gate. Its regression coverage is the direct N-thread rc-accounting tsan harness (spec item 1), which this task ships RED-first alongside the change. Harness links **only** `libailang_rt.a` (never the kernel) — keeps the de-globalisation proof distinct from the capability demo.
**Files:**
- Create: `crates/ail/tests/embed/rc_accounting_tsan.c`
- Modify: `runtime/rc.c:86103,113127,161177`
- (driver test fn added to the new `crates/ail/tests/embed_swarm_tsan.rs` in Task 5; this task adds only the rc-accounting C file + its standalone build/run inside a `#[test]` here)
- Create: `crates/ail/tests/embed_rc_accounting_tsan.rs`
- [ ] **Step 1: Write the direct rc-accounting tsan C harness (RED — symbols absent today)**
Create `crates/ail/tests/embed/rc_accounting_tsan.c`:
```c
/* M2 item 1: the de-globalisation teeth. Exercises ailang_rc_alloc /
* ailang_rc_dec DIRECTLY (not through any kernel) from N threads, each
* owning its own ailang_ctx_t. Per-ctx counters must be individually
* correct and -fsanitize=thread-clean. Build -DSHARED_CTX for the
* global-hammering negative control (must make tsan report a race). */
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <assert.h>
typedef struct ailang_ctx ailang_ctx_t;
extern ailang_ctx_t *ailang_ctx_new(void);
extern void ailang_ctx_free(ailang_ctx_t *);
extern void *ailang_rc_alloc(size_t);
extern void ailang_rc_dec(void *);
/* set the per-call ctx the runtime accounts into (defined in rc.c) */
extern __thread ailang_ctx_t *__ail_tls_ctx;
#define NTHREADS 8
#define NCYCLES 200000
#ifdef SHARED_CTX
static ailang_ctx_t *g_shared; /* negative control: one ctx, all threads */
#endif
static void *worker(void *_a) {
(void)_a;
#ifdef SHARED_CTX
ailang_ctx_t *ctx = g_shared;
#else
ailang_ctx_t *ctx = ailang_ctx_new();
#endif
__ail_tls_ctx = ctx;
for (int i = 0; i < NCYCLES; i++) {
void *p = ailang_rc_alloc(16);
ailang_rc_dec(p); /* refcount 1 -> 0, frees, free_count++ */
}
#ifndef SHARED_CTX
/* per-ctx balance must hold exactly */
/* (read via the same struct layout the runtime defines) */
ailang_ctx_free(ctx);
#endif
return NULL;
}
int main(void) {
#ifdef SHARED_CTX
g_shared = ailang_ctx_new();
#endif
pthread_t t[NTHREADS];
for (int i = 0; i < NTHREADS; i++) pthread_create(&t[i], NULL, worker, NULL);
for (int i = 0; i < NTHREADS; i++) pthread_join(t[i], NULL);
#ifdef SHARED_CTX
ailang_ctx_free(g_shared);
#endif
printf("rc_accounting_tsan: ok\n");
return 0;
}
```
- [ ] **Step 2: Write the driver and run it to verify RED**
Create `crates/ail/tests/embed_rc_accounting_tsan.rs`:
```rust
//! M2 item 1: direct rc-accounting de-globalisation proof. Compiles
//! runtime/rc.c + runtime/str.c into libailang_rt.a, links the
//! rc_accounting_tsan.c harness against it under -fsanitize=thread.
//! Per-thread-ctx build: tsan-clean + exit 0. RED until rc.c gains
//! ailang_ctx_* + __ail_tls_ctx (link error: undefined symbols).
use std::process::Command;
use std::path::PathBuf;
fn ws_root() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..").canonicalize().unwrap()
}
fn cc(args: &[&str], ctx: &str) {
let st = Command::new("clang").args(args).status().expect("spawn clang");
assert!(st.success(), "clang failed: {ctx}");
}
#[test]
fn rc_accounting_per_ctx_is_tsan_clean() {
let root = ws_root();
let tmp = std::env::temp_dir().join("ail_m2_rcacct");
std::fs::create_dir_all(&tmp).unwrap();
let rc_o = tmp.join("rc.o");
let str_o = tmp.join("str.o");
let rt_a = tmp.join("libailang_rt.a");
cc(&["-O1", "-g", "-fsanitize=thread", "-c", "-o", rc_o.to_str().unwrap(),
root.join("runtime/rc.c").to_str().unwrap()], "rc.c");
cc(&["-O1", "-g", "-fsanitize=thread", "-c", "-o", str_o.to_str().unwrap(),
root.join("runtime/str.c").to_str().unwrap()], "str.c");
let st = Command::new("ar").args(["rcs", rt_a.to_str().unwrap(),
rc_o.to_str().unwrap(), str_o.to_str().unwrap()]).status().unwrap();
assert!(st.success(), "ar failed");
let bin = tmp.join("rcacct");
cc(&["-O1", "-g", "-fsanitize=thread", "-pthread",
"-o", bin.to_str().unwrap(),
root.join("crates/ail/tests/embed/rc_accounting_tsan.c").to_str().unwrap(),
rt_a.to_str().unwrap()], "link rc_accounting_tsan");
let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1")
.output().expect("run harness");
assert!(out.status.success(),
"per-ctx rc-accounting harness must exit 0 tsan-clean; stderr:\n{}",
String::from_utf8_lossy(&out.stderr));
}
```
Run: `cargo test -p ail --test embed_rc_accounting_tsan`
Expected: FAIL — `clang failed: link rc_accounting_tsan` (undefined
references to `ailang_ctx_new`, `ailang_ctx_free`, `__ail_tls_ctx`
absent from today's `rc.c`).
- [ ] **Step 3: Implement the rc.c ctx struct + lifecycle + TLS slot**
In `runtime/rc.c`, immediately after the `g_rc_free_count` static (currently `rc.c:87`), insert:
```c
/* M2: per-thread embedding context. A scalar kernel allocates nothing,
* so this carries only the de-globalised RC accounting. Set by the
* generated C `@<sym>` forwarder into __ail_tls_ctx for the synchronous
* duration of one call (never held across a suspension point). The
* g_rc_* statics + atexit below are RETAINED as the null-ctx
* (single-threaded executable) fallback. */
typedef struct ailang_ctx {
uint64_t alloc_count;
uint64_t free_count;
} ailang_ctx_t;
__thread ailang_ctx_t *__ail_tls_ctx = NULL;
ailang_ctx_t *ailang_ctx_new(void) {
return (ailang_ctx_t *)calloc(1, sizeof(ailang_ctx_t));
}
void ailang_ctx_free(ailang_ctx_t *ctx) {
if (ctx != NULL) {
const char *flag = getenv("AILANG_RC_STATS");
if (flag != NULL && flag[0] != '\0') {
fprintf(stderr,
"ailang_rc_stats: allocs=%llu frees=%llu live=%lld\n",
(unsigned long long)ctx->alloc_count,
(unsigned long long)ctx->free_count,
(long long)(ctx->alloc_count - ctx->free_count));
}
}
free(ctx);
}
```
- [ ] **Step 4: Split the two increment sites (swarm → ctx, else → global)**
In `ailang_rc_alloc`, replace the line `g_rc_alloc_count++;` (currently `rc.c:125`) with:
```c
ailang_ctx_t *_ctx = __ail_tls_ctx;
if (_ctx != NULL) _ctx->alloc_count++; else g_rc_alloc_count++;
```
In `ailang_rc_dec`, inside the `if (*hdr == 0)` block, replace the line `g_rc_free_count++;` (currently `rc.c:175`) with:
```c
ailang_ctx_t *_ctx = __ail_tls_ctx;
if (_ctx != NULL) _ctx->free_count++; else g_rc_free_count++;
```
Leave `g_rc_alloc_count`/`g_rc_free_count` (`:8687`), `ailang_rc_stats_atexit` (`:8995`), and `ailang_rc_stats_install` (`:97103`) **verbatim** — they are now the null-ctx fallback only.
- [ ] **Step 5: Run the item-1 harness to verify GREEN**
Run: `cargo test -p ail --test embed_rc_accounting_tsan`
Expected: PASS (`test result: ok. 1 passed`) — links cleanly,
per-ctx counters correct, zero tsan reports.
---
## Task 3: Codegen — `Target::StaticLib` forwarder gains `ptr %ctx` + TLS save/store/restore
`lower_workspace_staticlib*` Rust signatures are unchanged (recon-confirmed) — no E0061 caller cascade; the only Rust gate is the codegen crate's own build + the extended IR-shape test. `embed_e2e.rs` is expected RED between this task and Task 6 (M1 `host.c` calls the now-`ptr`-prefixed symbol); its migration is Task 6 — do NOT add `embed_e2e` to this task's GREEN gate.
**Files:**
- Modify: `crates/ailang-codegen/src/lib.rs:600638` and `:165176`
- Modify: `crates/ailang-codegen/tests/embed_staticlib_lowering.rs:2033`
- [ ] **Step 1: Extend the RED IR-shape pin**
In `crates/ailang-codegen/tests/embed_staticlib_lowering.rs`, in `staticlib_emits_forwarder_no_main`, after the existing assertions on the forwarder, add (keep all existing assertions, including the no-`@main` one and the `_adapter`/`_clos` ones, unchanged):
```rust
// M2: forwarder takes a leading ptr %ctx and routes it through TLS
// around the UNCHANGED internal call.
assert!(ir.contains("@__ail_tls_ctx = external thread_local global ptr"),
"staticlib module must declare the external thread-local ctx slot;\nIR:\n{ir}");
assert!(ir.contains("define i64 @backtest_step(ptr %ctx, i64 %a0, i64 %a1)"),
"forwarder must take a leading ptr %ctx;\nIR:\n{ir}");
assert!(ir.contains("store ptr %ctx, ptr @__ail_tls_ctx"),
"forwarder must publish ctx into the TLS slot;\nIR:\n{ir}");
assert!(ir.contains("call i64 @ail_backtest_step(i64 %a0, i64 %a1)"),
"internal call must stay byte-unchanged (no ctx arg);\nIR:\n{ir}");
```
- [ ] **Step 2: Run to verify RED**
Run: `cargo test -p ailang-codegen --test embed_staticlib_lowering -- staticlib_emits_forwarder_no_main`
Expected: FAIL — first new assertion fails: `staticlib module must declare the external thread-local ctx slot` (today's forwarder is the no-ctx M1 shape).
- [ ] **Step 3: Emit the external thread-local decl once in the StaticLib arm**
In `crates/ailang-codegen/src/lib.rs`, in the `Target::StaticLib => {` arm (currently opens at `:600`), immediately after the arm opens and before the `for (mname, m) in &ws.modules {` loop, insert:
```rust
// M2: the per-thread embedding ctx slot. Defined by
// runtime/rc.c (__thread ailang_ctx_t *__ail_tls_ctx);
// the forwarder publishes its explicit ctx param here for
// the synchronous duration of the internal call.
out.push_str("\n@__ail_tls_ctx = external thread_local global ptr\n");
```
- [ ] **Step 4: Thread `ptr %ctx` through the forwarder body**
In the same arm, replace the forwarder `format!` block (currently `lib.rs:626633`, the `out.push_str(&format!("\ndefine {cret} @{sym}(...``ret {cret} %r\n}}\n", …))`) with:
```rust
let mut sig_params = vec!["ptr %ctx".to_string()];
sig_params.extend(
param_tys.iter().enumerate()
.map(|(i, t)| format!("{} %a{i}", llvm_scalar(t))));
out.push_str(&format!(
"\ndefine {cret} @{sym}({}) {{\n \
%saved = load ptr, ptr @__ail_tls_ctx\n \
store ptr %ctx, ptr @__ail_tls_ctx\n \
%r = call {cret} @ail_{mname}_{}({})\n \
store ptr %saved, ptr @__ail_tls_ctx\n \
ret {cret} %r\n}}\n",
sig_params.join(", "),
f.name,
args.join(", "),
));
```
(`args` — the internal-call argument vector built at `lib.rs:623625` — is left exactly as is: the internal `@ail_<mname>_<fn>` call takes the scalars only, byte-unchanged. `_adapter`/`_clos` emission is untouched.)
- [ ] **Step 5: Narrow the provisional doc on the `Target` enum**
In `crates/ailang-codegen/src/lib.rs:165176`, in the `Target::StaticLib` doc comment, replace the phrase that calls the signature "provisional until M3" with wording that scopes provisionality to the value/record layout. Exact replacement — change the sentence reading (around `:173`) to:
```rust
/// emit one external C entrypoint per `(export …)` fn forwarding
/// to the internal `@ail_<module>_<def>`. M2: the entrypoint takes
/// a mandatory leading `ptr %ctx` (per-thread embedding context)
/// published via `@__ail_tls_ctx` around the unchanged internal
/// call. The ctx-threaded C signature is the M2 shape; only the
/// value/record layout remains provisional until M3.
```
(Match and replace only the existing `StaticLib` doc lines; do not alter the `Executable` variant or its doc.)
- [ ] **Step 6: Run to verify GREEN (codegen IR-shape only)**
Run: `cargo test -p ailang-codegen --test embed_staticlib_lowering`
Expected: PASS (`test result: ok. N passed`) — both
`staticlib_emits_forwarder_no_main` (extended) and
`executable_target_still_emits_main` (unmodified) pass.
Run: `cargo build -p ailang-codegen`
Expected: `Finished` (0 errors).
---
## Task 4: CLI — `build_staticlib` `--alloc != rc` reject guard
RED test already shipped in Task 1 (`embed_staticlib_alloc_guard`). This task is the GREEN side: the guard.
**Files:**
- Modify: `crates/ail/src/main.rs:24852493`
- [ ] **Step 1: Insert the reject guard before lowering**
In `crates/ail/src/main.rs`, in `fn build_staticlib`, immediately before the `let ir = ailang_codegen::lower_workspace_staticlib_with_alloc(&ws, alloc)?;` line (currently `:2493`), insert:
```rust
if !matches!(alloc, ailang_codegen::AllocStrategy::Rc) {
anyhow::bail!(
"staticlib (swarm) artefact is RC-only — `--alloc=gc` links the \
shared Boehm collector, which is not swarm-safe; use `--alloc=rc`"
);
}
```
(Placed after the `has_export` check and before lowering, so a non-RC staticlib request fails before any `.ll`/archive work. The `--alloc` clap default is already `rc`, so the default path is unaffected; this rejects only an *explicit* `--alloc=gc`/`--alloc=bump`.)
- [ ] **Step 2: Run the Task-1 pin to verify RED→GREEN**
Run: `cargo test -p ail --test embed_staticlib_alloc_guard`
Expected: PASS (`test result: ok. 2 passed`) — both
`staticlib_gc_is_rejected` and `staticlib_bump_is_rejected` now see a
non-zero exit with the `staticlib (swarm) artefact is RC-only` string.
- [ ] **Step 3: Confirm the default (rc) staticlib build still works**
Run: `cargo test -p ail --test embed_staticlib_cli`
Expected: PASS (`test result: ok. N passed`) — default-alloc
two-archive build path unaffected by the guard.
---
## Task 5: Scalar-swarm capability harness + negative control (items 2, 3)
Depends on Tasks 2+3 GREEN (ctx ABI present). Links **both** archives (kernel + runtime). Separate file from Task 2's harness — different link surface, distinct spec role (capability demo vs de-globalisation proof).
**Files:**
- Create: `crates/ail/tests/embed/swarm.c`
- Create: `crates/ail/tests/embed_swarm_tsan.rs`
- [ ] **Step 1: Write the swarm harness C file**
Create `crates/ail/tests/embed/swarm.c`:
```c
/* M2 items 2+3: scalar-kernel swarm. Per-thread-ctx (default) must be
* tsan-clean and every thread's result must match the serial oracle.
* Built -DSHARED_CTX = negative control: all threads share one ctx;
* tsan MUST report a race (proves the clean result is not vacuous). */
#include <stdint.h>
#include <stdio.h>
#include <pthread.h>
#include <assert.h>
typedef struct ailang_ctx ailang_ctx_t;
extern ailang_ctx_t *ailang_ctx_new(void);
extern void ailang_ctx_free(ailang_ctx_t *);
extern int64_t backtest_step(ailang_ctx_t *ctx, int64_t state, int64_t sample);
#define NTHREADS 8
#define NITERS 200000
static int64_t serial_oracle(void) {
/* mirrors examples/embed_backtest_step.ail: state += sample*sample,
* sample = (i & 7) over NITERS iterations. */
int64_t s = 0;
for (int i = 0; i < NITERS; i++) { int64_t x = i & 7; s = s + x * x; }
return s;
}
#ifdef SHARED_CTX
static ailang_ctx_t *g_shared;
#endif
static void *worker(void *out) {
#ifdef SHARED_CTX
ailang_ctx_t *ctx = g_shared;
#else
ailang_ctx_t *ctx = ailang_ctx_new();
#endif
int64_t s = 0;
for (int i = 0; i < NITERS; i++) s = backtest_step(ctx, s, (i & 7));
#ifndef SHARED_CTX
ailang_ctx_free(ctx);
#endif
*(int64_t *)out = s;
return NULL;
}
int main(void) {
#ifdef SHARED_CTX
g_shared = ailang_ctx_new();
#endif
pthread_t t[NTHREADS];
int64_t res[NTHREADS];
for (int i = 0; i < NTHREADS; i++) pthread_create(&t[i], NULL, worker, &res[i]);
for (int i = 0; i < NTHREADS; i++) pthread_join(t[i], NULL);
#ifdef SHARED_CTX
ailang_ctx_free(g_shared);
#endif
int64_t oracle = serial_oracle();
for (int i = 0; i < NTHREADS; i++) {
if (res[i] != oracle) { fprintf(stderr, "thread %d: %lld != %lld\n",
i, (long long)res[i], (long long)oracle); return 1; }
}
printf("swarm: ok (%lld)\n", (long long)oracle);
return 0;
}
```
- [ ] **Step 2: Write the driver (per-thread-ctx GREEN; shared-ctx flagged)**
Create `crates/ail/tests/embed_swarm_tsan.rs`:
```rust
//! M2 items 2+3. Builds the staticlib (rc) via the `ail` CLI, then
//! links swarm.c against lib<entry>.a + libailang_rt.a under
//! -fsanitize=thread. Per-thread-ctx: tsan-clean, exit 0. Negative
//! control (-DSHARED_CTX): tsan MUST report (non-zero / report text).
use std::process::Command;
use std::path::PathBuf;
fn ws_root() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..").canonicalize().unwrap()
}
fn ail_bin() -> &'static str { env!("CARGO_BIN_EXE_ail") }
fn build_staticlib(outdir: &PathBuf) {
std::fs::create_dir_all(outdir).unwrap();
let st = Command::new(ail_bin())
.args(["build", "examples/embed_backtest_step.ail",
"--emit=staticlib", "--alloc=rc", "-o", outdir.to_str().unwrap()])
.current_dir(ws_root()).status().expect("ail build");
assert!(st.success(), "ail build --emit=staticlib failed");
}
fn link_swarm(outdir: &PathBuf, bin: &PathBuf, shared: bool) {
let root = ws_root();
let mut args: Vec<String> = vec![
"-O1".into(), "-g".into(), "-fsanitize=thread".into(), "-pthread".into(),
"-o".into(), bin.to_string_lossy().into(),
root.join("crates/ail/tests/embed/swarm.c").to_string_lossy().into(),
];
if shared { args.push("-DSHARED_CTX".into()); }
args.push(format!("-L{}", outdir.to_string_lossy()));
args.push("-lembed_backtest_step".into());
args.push("-lailang_rt".into());
let st = Command::new("clang").args(&args).status().expect("clang");
assert!(st.success(), "linking swarm.c failed (shared={shared})");
}
#[test]
fn scalar_swarm_per_ctx_is_tsan_clean_and_matches_oracle() {
let outdir = std::env::temp_dir().join("ail_m2_swarm_ok");
build_staticlib(&outdir);
let bin = outdir.join("swarm_ok");
link_swarm(&outdir, &bin, false);
let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1")
.output().expect("run swarm");
assert!(out.status.success(),
"per-thread-ctx swarm must exit 0 tsan-clean; stderr:\n{}",
String::from_utf8_lossy(&out.stderr));
}
#[test]
fn shared_ctx_negative_control_is_flagged_by_tsan() {
let outdir = std::env::temp_dir().join("ail_m2_swarm_neg");
build_staticlib(&outdir);
let bin = outdir.join("swarm_neg");
link_swarm(&outdir, &bin, true);
let out = Command::new(&bin).env("TSAN_OPTIONS", "halt_on_error=1")
.output().expect("run swarm neg");
let stderr = String::from_utf8_lossy(&out.stderr);
assert!(
!out.status.success() || stderr.contains("WARNING: ThreadSanitizer"),
"shared-ctx negative control MUST be flagged by tsan (the harness \
has teeth); exit={:?} stderr:\n{stderr}", out.status.code()
);
}
```
- [ ] **Step 3: Run the swarm matrix**
Run: `cargo test -p ail --test embed_swarm_tsan`
Expected: PASS (`test result: ok. 2 passed`) —
`scalar_swarm_per_ctx_is_tsan_clean_and_matches_oracle` exits 0
tsan-clean with every thread == oracle;
`shared_ctx_negative_control_is_flagged_by_tsan` observes the tsan
race report (teeth proven).
---
## Task 6: Migrate M1 `host.c` + `embed_e2e.rs` to the ctx ABI (item 5)
After Task 3 the M1 `host.c` (no-ctx `backtest_step`) is RED (wrong call ABI). Migrate it; keep the M1 value assertion (`s == 25`).
**Files:**
- Modify: `crates/ail/tests/embed/host.c`
- Modify: `crates/ail/tests/embed_e2e.rs:1746`
- [ ] **Step 1: Confirm `embed_e2e` is RED post-Task-3 (expected)**
Run: `cargo test -p ail --test embed_e2e -- c_host_calls_exported_scalar_kernel`
Expected: FAIL — the M1 `host.c` calls `backtest_step(s, 3)` against
the now-`(ptr,i64,i64)` symbol (garbage ctx; `s != 25` or crash). This
is the expected RED that Task 3 deliberately left for this task.
- [ ] **Step 2: Migrate `host.c` to the ctx ABI**
Replace the entire contents of `crates/ail/tests/embed/host.c` with:
```c
#include <stdint.h>
#include <assert.h>
typedef struct ailang_ctx ailang_ctx_t;
extern ailang_ctx_t *ailang_ctx_new(void);
extern void ailang_ctx_free(ailang_ctx_t *);
extern int64_t backtest_step(ailang_ctx_t *ctx, int64_t state, int64_t sample);
int main(void) {
ailang_ctx_t *ctx = ailang_ctx_new();
int64_t s = 0;
s = backtest_step(ctx, s, 3); /* 0 + 9 */
s = backtest_step(ctx, s, 4); /* 9 + 16 */
ailang_ctx_free(ctx);
assert(s == 25);
return 0;
}
```
- [ ] **Step 3: Update `embed_e2e.rs` link line for the runtime archive**
In `crates/ail/tests/embed_e2e.rs`, in `c_host_calls_exported_scalar_kernel`, ensure the harness links `libailang_rt.a` alongside `lib<entry>.a` (M1 already produces both archives; the host now references `ailang_ctx_new`/`_free` which live in `libailang_rt.a`). If the existing link command already passes `-lailang_rt` (M1 two-archive harness), no change is needed beyond Step 2; otherwise add `-lailang_rt` to the `clang` link args next to the existing `-lembed_backtest_step` (the exact existing arg vector is in this fn — append `-lailang_rt` if absent). Do not change the asserted value (`s == 25`) or exit-0 expectation.
- [ ] **Step 4: Run to verify GREEN**
Run: `cargo test -p ail --test embed_e2e -- c_host_calls_exported_scalar_kernel`
Expected: PASS (`test result: ok. 1 passed`) — ctx-ABI host links,
runs, `s == 25` holds.
---
## Task 7: DESIGN.md current-state update (preserve `docs_honesty_pin.rs:135`)
`docs_honesty_pin.rs:135` asserts (via the `norm()` whitespace-collapse helper) that DESIGN.md contains `Export parameters are written **bare**: a scalar type carries no `own`/`borrow` mode`. M2's §"Embedding ABI (M1)" rewrite MUST keep that sentence and the canonical `(con Int)` snippet verbatim; edit only the provisional-until-M3 sentence and add the ctx/swarm prose. The pin then stays green unmodified.
**Files:**
- Modify: `docs/DESIGN.md:22822285`
- Modify: `docs/DESIGN.md:15281529`
- [ ] **Step 1: Replace only the provisional-until-M3 sentence**
In `docs/DESIGN.md`, in §"Embedding ABI (M1)", replace the sentence currently at `:22822285` reading:
```
M1 ships no runtime lifecycle API: the per-thread
context parameter that M2 threads through every exported call will
change this C signature; do not treat the M1 signature as frozen.
The value/record layout freeze is M3.
```
with:
```
Every exported entrypoint takes a mandatory leading `ailang_ctx_t*`
(M2): a per-thread embedding context created by `ailang_ctx_new()`
and released by `ailang_ctx_free()`, owned by the calling thread for
its lifetime. The host links one `ailang_ctx_t` per OS worker thread
and the runtime accounts RC alloc/free into it (no shared mutable
runtime state — the swarm artefact is data-race-free, sanitiser-
verified). The staticlib swarm artefact is **RC-only**:
`ail build --emit=staticlib` rejects `--alloc=gc`/`--alloc=bump`
(the shared Boehm collector is not swarm-safe). Only the value/record
layout remains provisional until M3; the ctx-threaded C signature is
the M2 shape.
```
Do NOT touch the paragraph at `:22872289` ("Export parameters are written **bare**: a scalar type carries no `own`/`borrow` mode …") or the canonical `(con Int)` snippet at `:22922302` — they are pinned and orthogonal to M2.
- [ ] **Step 2: Add the Decision-10 per-thread-ctx note**
In `docs/DESIGN.md`, the Decision 10 atomicity bullet currently at `:15281529`:
```
- **Does not commit to atomic refcounts.** AILang is
single-threaded; refcounts are non-atomic.
```
Replace with:
```
- **Does not commit to atomic refcounts.** AILang is
single-threaded; refcounts are non-atomic. The embedding ABI's
per-thread `ailang_ctx_t` keeps this correct under a host swarm:
each thread's allocations are private to its ctx, no RC cell
crosses a thread, so non-atomic refcounts stay sound.
```
- [ ] **Step 3: Verify the honesty pin + schema-drift stay green unmodified**
Run: `cargo test -p ailang-core --test docs_honesty_pin`
Expected: PASS (`test result: ok. N passed`) — the preserved
"Export parameters are written **bare**…" sentence keeps the
`:135` assertion green; no pin edit.
Run: `cargo test -p ailang-core --test design_schema_drift`
Expected: PASS — M2 edits `DESIGN.md` only outside the `## Data
model` scan window (22632307 and 15281529); anchors unaffected.
---
## Task 8: Executable-path + schema/hash invariance regression gate (items 6, 7)
Pure verification — no code. Asserts the null-ctx fallback and the no-schema-change promises hold.
**Files:** (none modified — regression gate)
- [ ] **Step 1: Executable-path AILANG_RC_STATS readback unchanged**
Run: `cargo test -p ail --test print_no_leak_pin`
Expected: PASS (`test result: ok. N passed`) — unmodified;
the `g_rc_*`+atexit null-ctx fallback prints exactly as before.
Run: `cargo test -p ail --test e2e -- rc_stats`
Expected: PASS — the leak-stat helper consumers
(`alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells` and the
sibling `*rc_stats*` tests) green; the `ailang_rc_stats:` atexit line
shape preserved.
- [ ] **Step 2: No schema field — hash + drift invariant**
Run: `cargo test -p ailang-core --test embed_export_hash_stable`
Expected: PASS (`fn_without_export_hash_is_unchanged` green) — M2
adds no schema field.
Run: `cargo test -p ailang-check --test embed_export_gate`
Expected: PASS (`test result: ok. 8 passed`) — the M1 scalar/effect
export gate is unchanged by M2.
- [ ] **Step 3: Full workspace gate**
Run: `cargo test --workspace 2>&1 | tail -5`
Expected: every suite `test result: ok`; 0 failed. (Diff against the
pre-M2 baseline: the only test files added are
`embed_staticlib_alloc_guard`, `embed_rc_accounting_tsan`,
`embed_swarm_tsan`; the only behaviour-migrated test is
`embed_e2e::c_host_calls_exported_scalar_kernel`.)
---
## Task 9: Bench-trio carry-on + null-ctx-fallback bench-neutrality (item 9)
The null-ctx fallback adds `load __thread ptr; brif null` ahead of the executable path's `g_rc_*++`. Demonstrate bench-neutral, not hand-waved.
**Files:** (none modified — regression gate)
- [ ] **Step 1: Run the regression trio**
Run: `python3 bench/check.py; echo "exit=$?"`
Expected: `exit=0`. If exit=1 on a `*.bump_s` or
`latency.*.max_us` metric, that is a tracked-P2 noise family
(roadmap P2) — proceed to Step 2 for causal exoneration; do NOT
ratify a baseline.
Run: `python3 bench/compile_check.py; echo "exit=$?"`
Expected: `exit=0`.
Run: `python3 bench/cross_lang.py; echo "exit=$?"`
Expected: `exit=0`.
- [ ] **Step 2: Causal exoneration if a tracked-noise metric fires**
If and only if `check.py` exited 1 on a tracked-noise metric, build
the benched binary at the pre-M2 commit and at HEAD and byte-compare:
Run:
```
git stash; git rev-parse HEAD > /tmp/m2_head
cargo build --release -p ail 2>/dev/null
./target/release/ail build examples/list_sum.ail --alloc=rc -o /tmp/bench_head
git stash pop
cargo build --release -p ail 2>/dev/null
./target/release/ail build examples/list_sum.ail --alloc=rc -o /tmp/bench_m2
cmp /tmp/bench_head/list_sum /tmp/bench_m2/list_sum && echo "BYTE-IDENTICAL: causally exonerated (no ratify)"
```
Expected: for the *executable* path, `list_sum` does not allocate
across a ctx (null-ctx fallback), and the rc.c change adds only the
`load/brif null` ahead of `g_rc_*++`; if the metric is a tracked
noise family the binaries differ only by the fallback branch — record
the exoneration in the iter journal, do NOT update any baseline.
(If the binaries are NOT byte-identical AND a non-noise metric
regressed, STOP and bounce to the Boss: that is a real codegen
regression, not carry-on.)
---
## Self-review (Step 5 — completed inline)
1. **Spec coverage:** §Architecture → Tasks 24,7; §Concrete code shapes (impl) → Tasks 2,3,4; §Components table → every row mapped (rc.c T2; forwarder T3; CLI T4; DESIGN.md T7; embed_e2e T6; tests T1,T2,T5,T8); §Testing items 0→T1, 1→T2, 2/3→T5, 4→T3, 5→T6, 6→T8, 7→T8, 8→T8/architect-at-audit, 9→T9; §Acceptance/§Iteration sequencing → T1 fixed-first honoured. ✓
2. **Placeholder scan:** no "TBD/TODO/implement later/similar to/add appropriate" — grep-clean. ✓
3. **Type/name consistency:** `ailang_ctx_t`, `ailang_ctx_new`, `ailang_ctx_free`, `__ail_tls_ctx`, `@__ail_tls_ctx`, `backtest_step`, `staticlib_gc_is_rejected`, `embed_swarm_tsan` used identically across T1T9 and the file-map. ✓
4. **Step granularity:** every step is one 25 min action (one file edit / one cargo run). ✓
5. **No commit steps:** none present (Boss commits at iter end). ✓
6. **Pin/replacement contiguity:** the only presence-pin paired with a verbatim doc edit is `docs_honesty_pin.rs:135` ↔ DESIGN.md §"Embedding ABI (M1)". Task 7 Step 1 edits ONLY `:22822285` and explicitly forbids touching `:22872289` (the pinned sentence) — the pinned substring is never inside the replacement body, so no soft-wrap split is possible; `norm()` additionally collapses whitespace. ✓
7. **Compile-gate vs deferred-caller:** no task changes a Rust fn signature (`lower_workspace_staticlib*` unchanged; `build_staticlib` adds an internal guard, signature unchanged). The one cross-task RED interval (Task 3 leaves `embed_e2e` RED until Task 6) is a *link/runtime* gate, explicitly named in both Task 3's preamble and Task 6 Step 1, and Task 3's GREEN gate scopes to `embed_staticlib_lowering` only — no silent resequencing. ✓
8. **Verification filter strings resolve:** existing-test filters use recon-verified names (`alloc_rc_print_int_does_not_leak_show_result_str`, `alloc_rc_explicit_mode_tail_sum_does_not_leak_outer_cells`, `staticlib_emits_forwarder_no_main`, `executable_target_still_emits_main`, `c_host_calls_exported_scalar_kernel`, `fn_without_export_hash_is_unchanged`); new-test filters resolve by construction (defined in the same task); the `-- rc_stats` filter (T8 S1) is backed by the unfiltered `print_no_leak_pin` run in the same step and `e2e.rs:2178` helper consumers are `*rc_stats*`-named (recon-confirmed) — and T8 S3 runs the unfiltered `cargo test --workspace` so "nothing ran" cannot masquerade as green. ✓