roadmap: close Embedding ABI — M5 ([~]->[x]) — M1–M5 arc complete, audited+ratified+doc-honest; WhatsNew + P2 reframed

M5 done 2026-05-19: real backtest over real Pepperstone tick data
through the real external data-server into the shipped M3-frozen
kernel, across a per-thread-ctx thread swarm; symbol-fan (headline
existence proof) + time-shard (first actual proof of chunk-boundary
invisibility); globally leak-free, deterministic. Zero
language/compiler change; one runtime change (7bfa11e atomic
global RC-stats fallback — user-adjudicated bounce-back, RED-first,
+ a build-dep staleness root-caused & fixed). Audit: Invariant 1
PASS, M3 frozen-layout unmoved, gc_rss trio evidenced-RATIFY
(Boehm conservative-scan artefact, Decision-9 transitional), P2
noise causally exonerated baseline-pristine, doc-honesty tidy
clean. Friction-harvest: host-per-tick-FFI ~= ~206 ns/tick at real
EURUSD volume.

- roadmap M5 [~]->[x] with the done-state summary (stays one cycle
  for context, then prune).
- roadmap P2 "Flat array/slice primitive" reframed: the
  M4-retirement forward note's deferred reconciliation is now
  done — the obsolete "1024-cell cons-chain per chunk / M4 ships
  it" framing replaced with the real residual (host-per-tick-FFI
  vs. batch crossing) anchored on M5's measured ~206 ns/tick; the
  M5-dependency is now satisfied (the justification input exists).
- WhatsNew.md done-state entry (verbatim == the notify text):
  user-facing, no internals — the embedding arc is complete.

Done-state notification fired (user away per /boss). The next
roadmap item is a new milestone (the P2 flat-array decision or
another P0) — a deliberate user call, not autonomous.

docs/WhatsNew.md

@@ -224,3 +224,13 @@ This is verified, not asserted: the kernels run clean under a thread-race checke
 Nothing about how programs are written changed — the source you write is byte-for-byte the same as before; only the generated calling boundary and its runtime became concurrency-safe, which is exactly the point: thread-safety here is a property of the compiled artefact, not a burden on the author. The build was reviewed clean; the one flagged benchmark number is, once again, the same pre-existing machine-drift item already on the list, proven unrelated here.
 
 The next milestone lets structured records — not just single numbers — cross that boundary with a frozen, stable memory layout. That is a fresh piece of work and a natural point to decide how to pick it up.
+
+## 2026-05-19 — A compiled AILang kernel now runs a real backtest over a live market-data feed, across a thread swarm
+
+The embedding arc is complete. A compiled AILang function can now be driven by a real external market-data source — streaming real historical tick data — with one worker thread per data stream, all running the same compiled kernel at the same time. An actual backtest (folding a price series into a running result) runs end-to-end this way over real data.
+
+It is proven, not asserted: each worker's result is bit-for-bit identical to an independent reference computed the slow, simple way; and splitting one data stream into separate time windows handled by separate threads produces exactly the same per-window numbers as processing each window alone — so how the data happens to be chunked or sharded is invisible to the result. The whole run is memory-leak-free and deterministic across repeated runs.
+
+Driving the kernel from many threads at once surfaced one real concurrency gap in the runtime's internal bookkeeping — caught by the leak check, diagnosed to root cause, fixed, and re-verified — so the safety guarantee has teeth rather than being a claim on paper. Nothing about how programs are written, type-checked, or compiled changed; only host-side glue and one runtime-internal counter. The build was reviewed clean; the one flagged benchmark number was traced to its root cause and is an accepted, documented side effect of that necessary concurrency fix on the transitional memory path, with the primary path unaffected.
+
+This closes the multi-step arc that began with making a single numeric function callable from a host: it is now a real, concurrent, data-fed compute kernel. The next direction — whether to add a batch/array crossing to amortise the now-measured per-call cost — is a fresh decision, now driven by a real measurement rather than speculation.

docs/roadmap.md

@@ -304,29 +304,45 @@ work progresses.
     `docs/journals/2026-05-18-audit-embedding-abi-m3.md`;
     `docs/journals/2026-05-18-iter-embedding-abi-m3.tidy.md`.
 
-- [~] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter +
-  `data-server` wiring + thread-swarm backtest. **In-flight**: spec
-  written + user-approved + grounding-check PASS 2026-05-19. The
-  host-side adapter — a lean reusable embedding module (`extern "C"`
-  to the M3-frozen ABI + frozen-layout box helpers, zero finance
-  knowledge) **plus** the real E2E on top; in-repo but
-  `[workspace]`-excluded; sole `data-server`↔AILang meeting point
-  (Invariant 1) — wires the **real external** `data-server`'s
-  `while let next_chunk` loop to the shipped M3 kernel by
-  **unrolling each chunk host-side** into per-tick
-  `(State, Tick) -> State` calls (the adapter, not the kernel, owns
-  chunk iteration; M4's cons-list crossing was retired). **Zero
-  language/compiler/runtime change.** A real backtest runs over real
-  Pepperstone tick data on a thread swarm: symbol-fan (headline) +
-  time-shard (first actual proof of the chunk-boundary invisibility
-  the M4 retirement rests on). Terminal embedding milestone:
-  existence proof of the whole goal + friction harvest feeding the
-  host-per-tick-FFI vs. batch-crossing perf decision (the P2
-  flat-array item). Coherent stop: the goal itself.
+- [x] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter +
+  `data-server` wiring + thread-swarm backtest — **DONE 2026-05-19**,
+  audited + ratified + doc-honest. Terminal milestone of the M1–M5
+  Embedding ABI arc: a real backtest runs over **real Pepperstone
+  tick data** through the real external `data-server` crate into the
+  shipped M3-frozen AILang kernel, across a thread swarm of
+  independent per-thread embedding contexts. `ail-embed` is a lean
+  reusable embedding module (`extern "C"` to the M3-frozen ABI +
+  frozen-layout box helpers, zero finance knowledge) **plus** the
+  real E2E on top — in-repo but its own cargo workspace root
+  (`[workspace]`-excluded), the sole `data-server`↔AILang meeting
+  point (Invariant 1, architect-confirmed PASS at close). The
+  adapter (not the kernel) owns chunk iteration, unrolling each
+  chunk host-side into per-tick `(State, Tick) -> State` calls (M4's
+  cons-list crossing was retired as speculative infra). Symbol-fan
+  swarm = the headline existence proof (per-thread bit-exact vs an
+  independent host reference); time-shard swarm = the **first actual
+  proof** of the chunk/window-boundary invisibility the M4
+  retirement rests on (per-shard bit-exact). Globally leak-free,
+  deterministic. Zero language/compiler change; one runtime change
+  (`7bfa11e`: the global RC-stats fallback counters made
+  atomic-relaxed — the swarm exposed that AILang's first concurrent
+  consumer needed it; user-adjudicated bounce-back → RED-first fix,
+  then a build-dependency staleness root-caused + fixed). Audit:
+  Invariant 1 PASS, M3 frozen-layout SSOT unmoved, bench `gc_rss`
+  trio evidenced-RATIFY (a Boehm conservative-scan artefact of the
+  necessary atomic fix; Decision-9 Boehm-transitional), P2 noise
+  causally exonerated baseline-pristine; doc-honesty tidy clean.
+  **Friction-harvest deliverable: host-per-tick-FFI ≈ ~206 ns/tick
+  at real EURUSD volume** — the P2 flat-array decision input.
+  Stays one cycle for context, then prune.
   - depends on: Embedding ABI — M3 (shipped); Tick-coverage on M3
     (shipped `170464f`).
-  - context: `docs/specs/2026-05-19-embedding-abi-m5.md`;
-    M4 retirement
+  - context: `docs/specs/2026-05-19-embedding-abi-m5.md`; iter
+    journals `2026-05-19-iter-embedding-abi-m5.{1,2,3,tidy}.md` +
+    `…-m5.2-resume-attempt.md` + bugfix
+    `…-rc-global-stats-race.md` / `…-swarm-rc-alloc-undercount.md`;
+    audit `docs/journals/2026-05-19-audit-embedding-abi-m5.md`; M4
+    retirement
     `docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md`.
 
 - [x] **\[milestone\]** Heap-`Str` ABI — runtime infrastructure for
@@ -465,19 +481,27 @@ work progresses.
 - [ ] **\[milestone\]** Flat array/slice primitive — performance
   follow-up to the Embedding ABI arc, *not* a capability gap.
   AILang's only sequence is the recursive `List` ADT (one RC cell
-  per element); marshalling a 1024-record chunk across the
-  embedding boundary builds a 1024-cell cons-chain per chunk on a
-  multi-GB hot path (Embedding ABI M4 ships the capability this
-  way). A contiguous array/slice primitive would remove that, but
-  it is a *language* change (interacts with uniqueness inference,
-  RC, and the structurally-decreasing-recursion totality story)
-  and is explicitly out of the ABI arc's scope. Sequenced after
-  M5 so the decision is driven by M5's fieldtest measurements,
-  not speculation.
-  - depends on: Embedding ABI — M5 (for the measured
-    justification).
-  - context: 2026-05-18 chat (user deferred this explicitly to
-    keep the ABI arc clean).
+  per element). The ABI arc resolved chunk-crossing by having the
+  M5 adapter unroll each `data-server` chunk host-side into per-tick
+  `(State, Tick) -> State` calls (M4's cons-list crossing was
+  retired as speculative infra) — so the residual perf question is
+  no longer "1024-cell cons-chain per chunk" but **host-per-tick-FFI
+  vs. an eventual batch/flat-array crossing**. M5's friction-harvest
+  measured the per-tick FFI at **≈ ~206 ns/tick at real EURUSD
+  volume** (≈ 658 ms / 3.19 M ticks). Open decision: whether a
+  contiguous array/slice primitive (to amortise that per-tick cost
+  via a batch crossing) is worth a *language* change — it interacts
+  with uniqueness inference, RC, and the
+  structurally-decreasing-recursion totality story, and is out of
+  the (now-closed) ABI arc's scope. Now decidable on the measured
+  number, not speculation.
+  - depends on: Embedding ABI — M5 (shipped; measured ~206 ns/tick
+    — the justification input now exists).
+  - context: 2026-05-18 chat (user deferred this to keep the ABI
+    arc clean); M5 audit
+    `docs/journals/2026-05-19-audit-embedding-abi-m5.md` (the
+    friction-harvest number); M4 retirement
+    `docs/journals/2026-05-18-brainstorm-embedding-abi-m4-retired.md`.
 
 - [ ] **\[milestone\]** Iteration-totality story — structural +
   Int-bounded total recursion with *enforced* non-negativity.