iter embedding-abi-m5.2 (PARTIAL 2/3): data-server adapter + symbol-fan swarm; leak-proof surfaced a non-atomic global g_rc_* stats race

M5 iteration 2 (spec ae905de, plan 9cc9d9c). Tasks 1+2 clean and
committable; Task 3 BLOCKED on a real finding → M5 bounce-back.

Shipped:
- data-server promoted [dev-dependencies] -> [dependencies]
  (Invariant-1 sanctioned: ail-embed is the sole meeting point; the
  AILang workspace graph still has data-server count 0; zero
  compiler-surface diff — not even root Cargo.toml this iter).
- additive `adapter` module (ail-embed/src/adapter.rs): tick_to_px,
  MidPriceStream (lazy Iterator<f64> over SymbolChunkIter),
  fold_symbol; RED-first (E0432 -> GREEN). m5.1 core untouched
  except `pub mod adapter;`.
- swarm_runner [[bin]]: one thread per symbol, each owning its
  Kernel. The clean compile IS the compile-time per-thread-ctx
  proof (Ctx: !Send => a shared-ctx swarm is E0277).
- tests/swarm.rs: real-data symbol-fan E2E. `symbol_fan_swarm_bit_exact`
  is GREEN and live — per-symbol kernel (acc,n) bit-exact vs an
  independent same-order host reference (EURUSD/GER40/XAUUSD, ~4s).

The Task 3 finding (Boss independently confirmed by reading
runtime/rc.c): the global Σallocs==Σfrees leak measurement is an
INSTRUMENTATION race, not a memory bug. No box crosses a thread
(Ctx: !Send), the real refcount/free is correct (bit-exact GREEN
every run); only the global g_rc_* stat counters (rc.c:90-91,
161,212) are non-atomic BY rc.c's own documented single-threaded
design, losing ++s when 3 worker threads hit the null-__ail_tls_ctx
host-side path. M5 is AILang's first concurrent consumer; rc.c's
header (rc.c:44-49) explicitly defers exactly this atomic-vs-non-
atomic decision to "when it acquires concurrency primitives".

The leak assertion was Boss-split into `symbol_fan_swarm_leak_free`
(#[ignore], body preserved VERBATIM — quarantined not weakened;
un-ignore = the runtime fix's acceptance criterion) so main stays
green and the finding is pinned as a regression marker. The earlier
plan/journal claim that embed_swarm_tsan.rs covers this path was
wrong and is corrected on the record (that test uses the scalar
kernel — zero box allocs — so it never exercised the host-side
global-counter path).

Escalated as an M5 bounce-back: resolution touches M3-frozen
runtime/ and re-frames M5's "zero runtime change" commitment;
multiple substantive options, not unilaterally Boss's to pick in
frozen-runtime territory. Boss recommendation on the record =
Option A (make only the global-fallback g_rc_* counters atomic, as
a standalone RED-first runtime micro-iteration; M5 framing amended).
Time-shard + friction-harvest remain m5.3.

Includes the per-iter journal (with Boss disposition), stats, and
the INDEX.md line.

ail-embed/Cargo.toml

@@ -14,13 +14,23 @@ license = "MIT"
 publish = false
 description = "Lean embedding of an AILang M3-frozen staticlib kernel into a Rust host. Not an AILang [workspace] member (Invariant 1)."
 
-# Iter m5.1: the library itself has ZERO dependencies — the embedding
-# core only touches the frozen C ABI. `data-server` is dev-only,
-# exercised solely by the hermetic smoke test, so Invariant 1 holds
-# in the dependency graph, not just on paper.
+# The embedding core (`src/lib.rs`) touches only the frozen C ABI and
+# carries zero finance knowledge; `data-server` enters solely through
+# the additive `adapter` module + the `swarm_runner` bin. `ail-embed`
+# is the sole sanctioned data-server↔AILang meeting point — Invariant 1
+# governs the *compiler* crates (`ailang-*`, `crates/ail`, `runtime/`),
+# which remain data-server-free, not this workspace-excluded crate.
 [dependencies]
+# The adapter + swarm_runner bin link data-server (ail-embed is the
+# sole data-server↔AILang meeting point — Invariant 1 governs the
+# *compiler* crates, not this crate). Path is manifest-relative
+# (data-server is at /home/brummel/dev/libs/data-server).
+data-server = { path = "../../libs/data-server" }
 
 [dev-dependencies]
-data-server = { path = "../../libs/data-server" }
 zip = "2"
 tempfile = "3"
+
+[[bin]]
+name = "swarm_runner"
+path = "src/bin/swarm_runner.rs"

ail-embed/src/adapter.rs

@@ -0,0 +1,82 @@
+//! The sole `data-server`-knowing layer of `ail-embed`: maps
+//! `data_server::records::TickParsed` to the single scalar the M3
+//! `Tick` record carries (mid price), and lazily streams
+//! `SymbolChunkIter` chunks into the data-server-free `Kernel`
+//! (host-side chunk unroll — the adapter owns chunk iteration).
+
+use std::sync::Arc;
+
+use data_server::records::TickParsed;
+use data_server::{DataServer, SymbolChunkIter};
+
+use crate::Kernel;
+
+/// Mid price — the single scalar the M3 `Tick` record carries.
+#[inline]
+pub fn tick_to_px(t: &TickParsed) -> f64 {
+    (t.ask + t.bid) / 2.0
+}
+
+/// Lazy `Iterator<Item = f64>` over a `data-server` tick stream,
+/// yielding the per-tick mid. Pulls `next_chunk` on demand (the
+/// adapter owns chunk iteration; chunk boundaries are invisible to
+/// the fold) so a whole symbol is never materialised.
+pub struct MidPriceStream {
+    it: SymbolChunkIter<TickParsed>,
+    chunk: Option<Arc<[TickParsed]>>,
+    idx: usize,
+}
+
+impl MidPriceStream {
+    pub fn new(it: SymbolChunkIter<TickParsed>) -> Self {
+        Self { it, chunk: None, idx: 0 }
+    }
+}
+
+impl Iterator for MidPriceStream {
+    type Item = f64;
+    fn next(&mut self) -> Option<f64> {
+        loop {
+            if let Some(c) = &self.chunk {
+                if self.idx < c.len() {
+                    let px = tick_to_px(&c[self.idx]);
+                    self.idx += 1;
+                    return Some(px);
+                }
+            }
+            match self.it.next_chunk() {
+                Some(c) => {
+                    self.chunk = Some(c);
+                    self.idx = 0;
+                }
+                None => return None,
+            }
+        }
+    }
+}
+
+/// Fold up to `max_ticks` of `symbol`'s mid-price stream through the
+/// M3 kernel. `None` if the symbol has no tick stream. Returns
+/// `(Σ mid, count)` — the kernel's `acc += px; n += 1` over the
+/// bounded prefix, identical-order to a host reference fold.
+pub fn fold_symbol(
+    server: &Arc<DataServer>,
+    symbol: &str,
+    max_ticks: usize,
+) -> Option<(f64, i64)> {
+    let it = server.stream_tick(symbol)?;
+    Some(Kernel::new().run(MidPriceStream::new(it).take(max_ticks)))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::tick_to_px;
+    use data_server::records::TickParsed;
+
+    /// mid = (ask + bid) / 2. Exact in f64 for these inputs.
+    #[test]
+    fn tick_to_px_is_mid() {
+        let t = TickParsed { time_ms: 0, ask: 2.0, bid: 4.0 };
+        assert_eq!(tick_to_px(&t), 3.0);
+    }
+}

ail-embed/src/bin/swarm_runner.rs

@@ -0,0 +1,73 @@
+//! Symbol-fan swarm harness (spec §3 symbol-fan / Testing §2). Run
+//! as a subprocess by `tests/swarm.rs` under `AILANG_RC_STATS=1` so
+//! the C-runtime atexit `ailang_rc_stats:` line is observable.
+//!
+//! argv[1] = data dir (defaults to `data_server::DEFAULT_DATA_PATH`).
+//! Enumerates up to `N_SYMBOLS` symbols that have tick files
+//! (requires >= 2 for a meaningful swarm), spawns one thread per
+//! symbol — each thread owns its `Kernel` (`Ctx: !Send` forces
+//! one ctx per thread) — folds at most `MAX_TICKS` mids, and prints
+//! one `RESULT <symbol> <acc_bits:016x> <count>` line per symbol to
+//! stdout. Exits 0 on success, non-zero (with a message) otherwise.
+
+use std::sync::Arc;
+
+use ail_embed::adapter::fold_symbol;
+use data_server::records::DataFormat;
+use data_server::{DataServer, DEFAULT_DATA_PATH};
+
+/// Deterministic bounded prefix per symbol (chunk-aligned via
+/// `Iterator::take`): enough real volume to be a genuine proof,
+/// bounded so the E2E stays ~seconds. m5.3 owns the friction
+/// timing; here it is only a runtime bound.
+const MAX_TICKS: usize = 2_000_000;
+/// Max threads/symbols fanned. Real data has >= 2 tick symbols
+/// (EURUSD, XAUUSD); take min(N_SYMBOLS, available).
+const N_SYMBOLS: usize = 4;
+
+fn main() {
+    let data_dir = std::env::args()
+        .nth(1)
+        .unwrap_or_else(|| DEFAULT_DATA_PATH.to_string());
+
+    let server = Arc::new(DataServer::new(&data_dir));
+
+    let mut tick_syms: Vec<String> = server
+        .symbols()
+        .into_iter()
+        .filter(|s| {
+            server.file_count(s, DataFormat::Tick).unwrap_or(0) > 0
+        })
+        .map(|s| s.to_string())
+        .collect();
+    tick_syms.truncate(N_SYMBOLS);
+
+    if tick_syms.len() < 2 {
+        eprintln!(
+            "swarm_runner: need >= 2 tick symbols under {data_dir}, found {}",
+            tick_syms.len()
+        );
+        std::process::exit(2);
+    }
+
+    let handles: Vec<_> = tick_syms
+        .into_iter()
+        .map(|sym| {
+            let srv = Arc::clone(&server);
+            // Each thread builds its own Kernel inside the closure.
+            // `Kernel`/`Ctx` are `!Send`, so this is the ONLY way
+            // this compiles — one ctx per thread, enforced by the
+            // type system, not by convention.
+            std::thread::spawn(move || {
+                let (acc, n) = fold_symbol(&srv, &sym, MAX_TICKS)
+                    .expect("symbol has a tick stream");
+                (sym, acc, n)
+            })
+        })
+        .collect();
+
+    for h in handles {
+        let (sym, acc, n) = h.join().expect("worker thread panicked");
+        println!("RESULT {sym} {:016x} {n}", acc.to_bits());
+    }
+}

ail-embed/src/lib.rs

@@ -6,6 +6,10 @@
 
 use std::ffi::c_void;
 
+/// The sole `data-server`-knowing layer (spec §2). Additive; the
+/// embedding core below has zero finance knowledge.
+pub mod adapter;
+
 // The M3-frozen C ABI. Symbol names + signatures mirror
 // `crates/ail/tests/embed/tick_roundtrip.c:31-36`. `backtest_step_tick`
 // is the author-chosen export of `examples/embed_backtest_step_tick.ail`.

ail-embed/tests/swarm.rs

@@ -0,0 +1,164 @@
+//! Symbol-fan swarm E2E (spec Testing §2). Runs the `swarm_runner`
+//! bin as a subprocess under `AILANG_RC_STATS=1` (the C-runtime
+//! atexit stat line, runtime/rc.c:124-138, is only observable from a
+//! separate process that exits). Skips when /mnt tick data is absent
+//! — mirroring data-server's own tests/data_server.rs::skip_if_no_data().
+//!
+//! Two assertions, deliberately SPLIT into two tests:
+//!
+//! - `symbol_fan_swarm_bit_exact` — the existence proof. Per symbol,
+//!   the kernel `(acc,n)` is BIT-EXACT vs an independent same-order
+//!   host reference fold. This is GREEN and live: it proves the
+//!   real-data-server → adapter → M3-kernel swarm computes correctly.
+//!
+//! - `symbol_fan_swarm_leak_free` — `#[ignore]`d pending an escalated
+//!   design decision. The global `Σallocs==Σfrees` measurement
+//!   (ported from crates/ail/tests/embed_tick_e2e.rs:94-104) is
+//!   *invalid under a multi-threaded host*: host-side box allocs run
+//!   with `__ail_tls_ctx == NULL` and fall through to the
+//!   **non-atomic** global `g_rc_*` counters (runtime/rc.c:90-91,
+//!   161,212), which rc.c's own header explicitly documents as
+//!   single-threaded/non-atomic ("when it acquires [concurrency],
+//!   atomic-vs-non-atomic becomes a separate decision"). The swarm is
+//!   *actually* leak-free (no box crosses a thread — `Ctx: !Send`;
+//!   bit-exact is GREEN every run); only the global *counter* races,
+//!   losing increments. The assertion body is preserved verbatim and
+//!   will be re-armed (un-`#[ignore]`d) by the runtime iteration that
+//!   makes the global fallback counters atomic. See
+//!   docs/journals/2026-05-19-iter-embedding-abi-m5.2.md and the M5
+//!   bounce-back. Quarantined, NOT weakened — what it asserts is
+//!   unchanged; it is gated on a decision the Boss escalated.
+
+use std::process::Command;
+use std::sync::Arc;
+
+use data_server::records::DataFormat;
+use data_server::{DataServer, DEFAULT_DATA_PATH};
+
+// Must match swarm_runner.rs.
+const MAX_TICKS: usize = 2_000_000;
+const N_SYMBOLS: usize = 4;
+
+fn skip_if_no_data() -> bool {
+    !std::path::Path::new(DEFAULT_DATA_PATH).exists()
+}
+
+/// Independent host reference: same symbols, same order, same cap,
+/// pure-Rust fold (`acc += mid; n += 1`). Bit-exact with the kernel
+/// fold because the addition order is identical.
+fn reference(symbol: &str) -> (f64, i64) {
+    let server = Arc::new(DataServer::new(DEFAULT_DATA_PATH));
+    let mut it = server.stream_tick(symbol).expect("tick stream");
+    let (mut acc, mut n) = (0.0_f64, 0_i64);
+    'outer: while let Some(chunk) = it.next_chunk() {
+        for r in chunk.iter() {
+            if n as usize >= MAX_TICKS {
+                break 'outer;
+            }
+            acc += (r.ask + r.bid) / 2.0;
+            n += 1;
+        }
+    }
+    (acc, n)
+}
+
+/// Spawn `swarm_runner` under `AILANG_RC_STATS=1`. `None` on the
+/// skip-if-absent path (no /mnt data); otherwise `(stdout, stderr)`
+/// after asserting a clean child exit.
+fn run_swarm_or_skip() -> Option<(String, String)> {
+    if skip_if_no_data() {
+        eprintln!("skipping: {DEFAULT_DATA_PATH} absent (mirrors data-server)");
+        return None;
+    }
+    let out = Command::new(env!("CARGO_BIN_EXE_swarm_runner"))
+        .arg(DEFAULT_DATA_PATH)
+        .env("AILANG_RC_STATS", "1")
+        .output()
+        .expect("spawn swarm_runner");
+    let stdout = String::from_utf8_lossy(&out.stdout).into_owned();
+    let stderr = String::from_utf8_lossy(&out.stderr).into_owned();
+    assert!(
+        out.status.success(),
+        "swarm_runner exited {:?}\nstdout:\n{stdout}\nstderr:\n{stderr}",
+        out.status.code()
+    );
+    Some((stdout, stderr))
+}
+
+/// Existence proof (GREEN, live): each symbol's kernel `(acc,n)` is
+/// bit-exact vs an independent same-order host reference fold. Real
+/// data-server → adapter → M3-frozen kernel, one symbol per thread.
+#[test]
+fn symbol_fan_swarm_bit_exact() {
+    let Some((stdout, _stderr)) = run_swarm_or_skip() else { return };
+
+    let mut symbols_checked = 0usize;
+    for line in stdout.lines().filter(|l| l.starts_with("RESULT ")) {
+        let mut t = line.split_whitespace();
+        let _ = t.next(); // "RESULT"
+        let sym = t.next().expect("symbol field");
+        let acc_bits = u64::from_str_radix(
+            t.next().expect("acc-bits field"), 16,
+        ).expect("acc-bits hex");
+        let n: i64 = t.next().expect("n field").parse().expect("n int");
+
+        let kernel_acc = f64::from_bits(acc_bits);
+        let (ref_acc, ref_n) = reference(sym);
+        assert_eq!(
+            kernel_acc.to_bits(), ref_acc.to_bits(),
+            "{sym}: kernel acc bit-exact vs same-order host reference"
+        );
+        assert_eq!(n, ref_n, "{sym}: tick count matches reference");
+        assert!(ref_n > 0, "{sym}: non-empty stream");
+        symbols_checked += 1;
+    }
+    assert!(
+        (2..=N_SYMBOLS).contains(&symbols_checked),
+        "expected 2..={N_SYMBOLS} symbols fanned, got {symbols_checked}"
+    );
+}
+
+/// Global leak-freedom: Σallocs == Σfrees across ALL `ailang_rc_stats:`
+/// lines (the M2 dual-stat-line model, ported from
+/// embed_tick_e2e.rs:94-104). The invariant is the global Σ, not
+/// per-line balance (M2 TLS-ctx cross-attribution).
+///
+/// `#[ignore]`d: this measurement is invalid under a multi-threaded
+/// host. Host-side box allocs run with `__ail_tls_ctx == NULL` and
+/// hit the non-atomic global `g_rc_*` counters (runtime/rc.c:90-91,
+/// 161,212) — single-threaded/non-atomic *by rc.c's own documented
+/// design*. 3 worker threads race the counter, losing increments;
+/// the *counter* is unreliable even though the swarm is actually
+/// leak-free (no box crosses a thread; `symbol_fan_swarm_bit_exact`
+/// is GREEN every run). Un-`#[ignore]` this — body unchanged — once
+/// the runtime iteration makes the global fallback counters atomic
+/// (the deferred "concurrency arrived" decision rc.c:44-49 names).
+/// See docs/journals/2026-05-19-iter-embedding-abi-m5.2.md.
+#[test]
+#[ignore = "blocked: global g_rc_* counter is non-atomic by rc.c design; \
+            invalid under the multi-thread host. Swarm is actually \
+            leak-free (bit-exact is green; no box crosses a thread). \
+            Un-ignore when the runtime makes the global fallback \
+            counters atomic — see the m5.2 journal + M5 bounce-back."]
+fn symbol_fan_swarm_leak_free() {
+    let Some((_stdout, stderr)) = run_swarm_or_skip() else { return };
+
+    let mut allocs: u64 = 0;
+    let mut frees: u64 = 0;
+    let mut seen = 0usize;
+    for line in stderr.lines().filter(|l| l.starts_with("ailang_rc_stats:")) {
+        seen += 1;
+        for tok in line.split_whitespace() {
+            if let Some(v) = tok.strip_prefix("allocs=") {
+                allocs += v.parse::<u64>().expect("allocs= u64");
+            } else if let Some(v) = tok.strip_prefix("frees=") {
+                frees += v.parse::<u64>().expect("frees= u64");
+            }
+        }
+    }
+    assert!(seen > 0, "no ailang_rc_stats line; stderr:\n{stderr}");
+    assert_eq!(
+        allocs, frees,
+        "globally leak-free: Σallocs==Σfrees across {seen} stat line(s)"
+    );
+}