AILang/docs/journals/2026-05-19-audit-embedding-abi-m5.md

audit embedding-abi-m5 — milestone close

Date: 2026-05-19 Scope: c75fb80..HEAD (M3-close → m5.3 0900f3f), focus the M5 arc Outcome: DRIFT (one doc-honesty tidy) + RATIFY (gc_rss trio, evidenced) + causally-exonerated pre-existing P2 bench noise

Architect (drift review) — drift_found

Gates that PASS:

• Invariant 1 — PASS (hard M5 spec Acceptance gate, spec:328-330). cargo metadata --no-deps on the AILang root manifest = zero data-server; ail-embed/ is its own [workspace] root, not an AILang member. No compiler crate / runtime/ gains a data_server/finance symbol. The only "finance" hits in compiler crates are the M3 ABI export symbol name backtest_step_tick in tests/docs — ABI knowledge, not data-server knowledge (correct).
• M3 frozen-layout SSOT unmoved. The one runtime/rc.c change in range (7bfa11e) touches only the two GLOBAL g_rc_* statistics counters; box offsets, p-8 header, ctor tag, host-ailang_rc_dec rule byte-unchanged; DESIGN.md §"Frozen value layout" correctly not touched.
• rc.c header-comment honesty accurate. The bugfix-rc-global-stats-race GREEN rewrite (rc.c ~:42-58 / ~:85-101) matches the actual _Atomic decls + atomic_fetch_add_explicit/ atomic_load_explicit sites + plain per-ctx ++; the rc.c:~270 drop-worklist "single-threaded" note is correctly still true (per-call worklist).

Drift list (prioritised):

• [medium] docs/DESIGN.md:2358-2360 — "an additive M4 concern" for a recursive typed-free of boxed-field records is stale: M4 was RETIRED 2026-05-18 (never speced), will never ship. PRE-EXISTING (M4-retirement era), correctly scoped out by the M5 spec §"Out of scope" and the M4-retirement journal forward note; NOT an M5 regression. Direction: doc-honesty tidy (drop the M4 reference; the boxed-field record is simply export-gate-rejected).
• [medium] docs/DESIGN.md §"Embedding ABI" — DESIGN.md already asserts "no shared mutable runtime state … data-race-free, sanitiser-verified". After 7bfa11e the global RC-stats fallback is shared mutable runtime state under concurrency (atomic-relaxed), and M5 (the embedding swarm) is AILang's first concurrent runtime consumer. The doc asserts a concurrency property whose mechanism it no longer fully/accurately describes — real silence-drift the contract line reaches. Direction: additive present-tense note (the global RC-stats fallback is atomic-relaxed; per-ctx counters + the per-object refcount header stay non-atomic by design because a box never crosses a thread and each ctx is single-thread-per-ctx) — a current-state mirror, not a history retelling.
• [low] runtime/rc.c:88 — "two unconditional ++" is now imprecise (the global path is a relaxed atomic add, conditional on _ctx == NULL). Pre-existing, self-flagged known debt in 2026-05-19-iter-bugfix-rc-global-stats-race.md; a performance aside, not an atomicity claim. Fold into the same tidy if rc.c is reopened.

Architect recommendation (adopted): ONE doc-honesty tidy iteration reconciling the stale-M4 phrasing and the atomic-global-stats / first-concurrent-consumer note (carrying rc.c:88 if rc.c is reopened) — same surface, same honesty axis; do not split.

Bencher (bench-regression localisation) — exit 1, hypothesis-driven

bench/check.py exit 1: 6 regressed / 1 improved / 56 stable on the first audit run. Bencher built ail at HEAD (0900f3f) and pre-atomic (427b687) in an isolated worktree; ail binary sha256-identical; emitted IR for the gc_rss fixtures byte-identical both sides (rc.o links into bench binaries, not the compiler).

• 3× *.gc_rss_kb +32% (list_sum, hof_pipeline, list_sum_explicit) — GENUINE, DETERMINISTIC, M5-ATTRIBUTABLE. My pre-audit causal read (allocator-orthogonal → environmental) was refuted by evidence. Mechanism: 7bfa11e's atomic split a branch in ailang_rc_alloc (called by str.c int→str at print, reached by GC-mode programs too), spilling %r14/%rbx; a stale interior list pointer in the new spill footprint is conservatively retained by Boehm, pinning a ~32 MB dead list spine across a collection (Boehm internal: collection #12 freed PRE 32,000,032 B vs HEAD 32 B). Bisects exactly on the commit with IR byte-identical — causation established, not inferred. RC/bump RSS dead-stable.
• bench_list_sum.bump_s / bench_list_sum_explicit.bump_s ~+12% — pre-existing tracked-P2 stale-0.046/0.048s-anchor; bump path is g_rc_*-orthogonal; HEAD-vs-PRE delta ~2%. NOT M5.
• latency.{explicit,implicit}_at_rc.max_us — 1-sample worst-case tail; median/p99/p99.9 flat both sides; PRE frequently worse; tracked-P2 -n5 tail-jitter. Re-run minutes later swung +65→+104% / +33→+71% — dispositively noise, NOT M5.

Adjudication (orchestrator)

RATIFY — the gc_rss trio (7bfa11e, evidenced)

Item: throughput.bench_list_sum.gc_rss_kb, bench_hof_pipeline.gc_rss_kb, bench_list_sum_explicit.gc_rss_kb, each ~+32% (103980→137448, 103788→137768, 103772→137636).

Iter that moved the metric: 7bfa11e (fix(rc): GREEN — atomic global g_rc_* stats fallback counters), the user-approved Option-A swarm-safety fix in the bugfix-rc-global-stats-race chain — necessary and load-bearing for M5's deterministic leak-proof (reverting it reintroduces the non-deterministic swarm-leak race).

Language reason (not effort): AILang's committed memory model is RC + uniqueness inference; Boehm is explicitly transitional (Decision 9). The measurable cost of the correct, necessary atomic fix is exclusively a Boehm conservative-stack-scan false-retention artifact on the transitional GC allocator — the bencher is explicit it is "a Boehm conservatism artefact, not an RC-semantics cost"; RC and bump RSS are dead-stable; no memory is actually leaked by AILang (Boehm conservatively retains a dead stack slot). The mechanism is "codegen-layout-fragile — a future unrelated rc.c edit could equally flip it back." Spending an out-of-scope codegen change (clear the dead root before print) to placate the conservative scanner of the allocator the language is migrating away from is investing in the path being removed; the committed RC path — the one that matters — is unaffected. Therefore ratify the new deterministic steady-state rather than chase it.

Action taken: selective baseline re-anchor — bench/baseline.json lines 14/44/64 only (the 3 evidenced M5-caused gc_rss_kb), set to the bencher-confirmed stable HEAD values (137448 / 137768 / 137636); every other metric byte-pristine (a wholesale --update-baseline would have absorbed the unrelated P2 noise and broken the established-envelope discipline). Re-run confirms the 3 are now ok (±0.3% vs the new anchor, ≪ 5% tol).

NO-RATIFY / causally exonerated — the pre-existing P2 noise

bench_list_sum.bump_s, bench_list_sum_explicit.bump_s, latency.explicit_at_rc.max_us, latency.implicit_at_rc.max_us remain firing (check.py still exit 1 on these). Bencher-established NOT M5 (bump path RC-orthogonal; max_us 1-sample tail-jitter, PRE often worse, +65→+104% swing between runs minutes apart). Baseline left pristine for these — the established tracked-P2 noise envelope, identical disposition to the M2 / M3 / prelude-decouple milestone closes ("check.py exit 1 causally exonerated, NO ratify, baseline pristine"). The standing P2 items (the stale bump_s anchor; the -n5 max_us tail) are not M5's and are not bundled here.

DRIFT → tidy iteration

The architect's [medium]+[medium]+[low] doc-honesty drift is resolved via one tidy iteration embedding-abi-m5.tidy (planner → implement; doc-only — DESIGN.md stale-M4 phrasing + the atomic-global-stats / first-concurrent-consumer present-tense note reconciling the "no shared mutable runtime state" claim; rc.c:88 carried along). Same surface, same honesty axis, not split.

Friction-harvest (recorded — the P2 flat-array-decision input)

m5.3 measured host-per-tick-FFI ≈ 658 ms / 3 192 562 ticks ≈ ~206 ns/tick at real EURUSD volume (one symbol-window per thread, own ctx, the M3 ABI per tick). This is the observation the P2 "flat-array/batch-crossing" roadmap item is now informed by: per-tick FFI at ~200 ns is the cost a future batch crossing would amortise; whether that is worth the cons-list-free flat-array path is the P2 decision, now anchored on a real number rather than the stale cons-list framing the M4 retirement obsoleted.

Close posture

Invariant 1 PASS; M3 frozen layout unmoved; bench dispositioned (gc_rss ratified evidenced, P2 noise exonerated baseline-pristine). M5 closes after the doc-honesty tidy lands and is Boss-verified. The baseline edit + this journal are committed together (ratify discipline: updated baseline JSON + paired JOURNAL ratify entry).