roadmap: Embedding ABI arc — M1–M5 (P1) + deferred array primitive (P2)

The data-server-driven goal: a compiled AILang kernel callable
in-process from a concurrent Rust host so a backtest swarm runs
each backtest as an independent single-threaded computation.
Decomposed into five chronologically-bounded milestones (M1
linkable artifact+scalar entrypoint, M2 per-thread ctx+concurrency
safety, M3 frozen value layout+ADT crossing, M4 List-ADT sequence
crossing, M5 ail-embed adapter+swarm fieldtest), each its own full
brainstorm->plan->implement->audit cycle. Shared motivation + three
audited invariants (clean core / single-threaded per computation /
native-AOT-only) carried on the M1 entry. The flat array/slice
primitive is recorded P2 as a measured performance follow-up
gated on M5's fieldtest, not a capability gap — keeping the ABI
arc's scope clean. Decision made in the 2026-05-18 data-server
analysis chat; no spec yet (brainstorm produces M1's spec at start).

docs/roadmap.md

@@ -129,6 +129,119 @@ work progresses.
 
 ## P1 — Next
 
+- [ ] **\[milestone\]** Embedding ABI — M1: linkable artifact +
+  scalar C entrypoint + `main`-free lifecycle. First of a
+  five-milestone arc (M1–M5) that makes a compiled AILang kernel
+  callable in-process from a concurrent Rust host, so a swarm of
+  backtests over `data-server`-supplied market data runs each
+  backtest as an independent single-threaded AILang computation.
+
+  **Motivation (shared by M1–M5).** The target deployment of
+  AILang is financial-data analysis: a host (Rust, owning
+  `data-server`'s shared zero-copy cache) drives a swarm of
+  backtests; each backtest is the pure per-chunk fold AILang is
+  built for (`(State, Chunk) -> State`, the `sum_sq` loop/recur
+  shape). Today AILang is a whole-program compiler whose only
+  output channel is `io/print_str` — there is no way to invoke a
+  compiled AILang function from a host with structured arguments
+  and a typed return. That missing call surface, not the math, is
+  the entire blocker. This is a backend/packaging capability, not
+  a language feature: the language semantics do not change. Its
+  justification is strategic (this is where the language is meant
+  to run), not the standard authoring-utility gate — a deliberate,
+  user-made direction call (2026-05-18 chat).
+
+  **Invariants (audited at every close in this arc).**
+  (1) *Clean core* — `ailang-core` / `ailang-codegen` / the
+  runtime gain no `data-server`/finance knowledge and no
+  dependency on either; the only place AILang and `data-server`
+  meet is a separate host-side adapter crate (`ail-embed`, M5)
+  that depends on both while neither depends on it. The `audit`
+  architect review verifies this each close.
+  (2) *Single-threaded per computation* — all concurrency is
+  structural and owned by the host; no AILang value ever crosses
+  a thread boundary, so non-atomic RC and "no shared mutable
+  refs" stay intact.
+  (3) *Native-AOT only* — no interpreter/playground is introduced
+  (the "emit LLVM → clang -O2" performance pillar is load-bearing
+  for the swarm).
+
+  **M1 scope.** CLI emit-mode producing a relocatable archive
+  instead of linking an executable; codegen exports an
+  author-designated `fn` under a stable C symbol with a
+  C-compatible signature instead of wrapping `main` (today
+  hard-coded `define i32 @main() { call i8 @ail_<mod>_main() }`,
+  `crates/ailang-codegen/src/lib.rs:556`); a minimal explicit
+  lifecycle so the runtime is usable with no `main`-time setup.
+  Scalar signatures only (`(Int)->Int`, `(Float,Float)->Float`)
+  — no ADT crosses the boundary yet. Coherent stop: AILang
+  arithmetic callable from C/Rust, proven by a host harness that
+  links the `.a`, calls the symbol, asserts the return.
+  - context: 2026-05-18 chat (`data-server` analysis →
+    embedding-ABI decomposition). No spec yet; brainstorm
+    produces `docs/specs/<date>-embedding-abi-m1.md` at M1 start.
+
+- [ ] **\[milestone\]** Embedding ABI — M2: per-thread runtime
+  context + concurrency safety. `ailang_ctx_new()/_free` threaded
+  through every call; neutralise the two process-global hazards
+  the runtime read surfaced — the non-atomic
+  `g_rc_alloc_count` / `g_rc_free_count` counters + `atexit`
+  stats hook (`runtime/rc.c:86–98`), and the allocator backing
+  (toward pure per-thread RC, no shared Boehm collector; aligns
+  with the standing P2 "Boehm full retirement" todo). Still
+  scalar-only so any failure is attributable to concurrency, not
+  marshalling. IO-at-the-boundary (`io/print_str` via a
+  per-instance sink) explicitly deferred here: pure
+  `(State,Chunk)->State` kernels have no effects; revisit only
+  if a logging kernel appears — deliberately not a separate
+  roadmap entry, by feature-acceptance discipline. Coherent stop:
+  swarm-safe scalar kernels, tsan-clean under N threads.
+  - depends on: Embedding ABI — M1.
+  - context: 2026-05-18 chat; runtime hazard read
+    (`runtime/rc.c` global stats + `atexit`).
+
+- [ ] **\[milestone\]** Embedding ABI — M3: frozen value layout +
+  single ADT/record crossing + RC ownership contract. Host
+  constructs a single-ctor record (Int/Float fields), passes it,
+  the kernel returns a record the host reads and frees; the RC
+  ownership-transfer contract at the call boundary is specified
+  and the `rc_header`/ADT-box layout documented as a *frozen*
+  ABI. Sequenced after M2 — a one-way stability commitment, so
+  the frozen ABI must already include the ctx parameter shape
+  (substantive ordering reason, not effort). Covers the `State`
+  value crossing in/out. Coherent stop: a record in/out with
+  alloc==free across many calls.
+  - depends on: Embedding ABI — M2.
+  - context: 2026-05-18 chat. `runtime/rc.c`'s header already
+    anticipates this ("The ABI defined here is stable … unless
+    the layout itself shifts").
+
+- [ ] **\[milestone\]** Embedding ABI — M4: sequence crossing via
+  the existing `List` ADT. Host builds a `List Record` per chunk;
+  `(State, List Record) -> State` callable end-to-end. Pure
+  ABI/marshalling, no language change — applies M3's frozen
+  layout repeatedly. Makes the real fold runnable over real
+  chunks. The array-vs-cons-list performance question (1024 RC
+  cons-cells per chunk on the hot path) is deliberately *not* in
+  scope here — capability now, performance as a separate later
+  non-ABI milestone (P2 below), keeping the user's "focus on the
+  ABI, ignore the array gap" cut. Coherent stop:
+  `(State, List M1) -> State` runs.
+  - depends on: Embedding ABI — M3.
+  - context: 2026-05-18 chat.
+
+- [ ] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter +
+  `data-server` wiring + thread-swarm backtest (fieldtest
+  target). The host-side adapter crate (sole meeting point of
+  `data-server` and AILang, per Invariant 1) wires
+  `data-server`'s `while let next_chunk` loop to the AILang
+  kernel via the ABI; an actual small backtest runs over a
+  thread swarm. Natural `fieldtest` milestone: existence proof
+  of the whole goal + friction harvest feeding the
+  array-primitive decision. Coherent stop: the goal itself.
+  - depends on: Embedding ABI — M4.
+  - context: 2026-05-18 chat.
+
 - [x] **\[milestone\]** Heap-`Str` ABI — runtime infrastructure for
   malloc-backed, refcounted `Str` values alongside the existing
   static `@.str_*` globals. Today the `Str` path is static-only
@@ -165,6 +278,23 @@ work progresses.
 
 ## P2 — Medium-term
 
+- [ ] **\[milestone\]** Flat array/slice primitive — performance
+  follow-up to the Embedding ABI arc, *not* a capability gap.
+  AILang's only sequence is the recursive `List` ADT (one RC cell
+  per element); marshalling a 1024-record chunk across the
+  embedding boundary builds a 1024-cell cons-chain per chunk on a
+  multi-GB hot path (Embedding ABI M4 ships the capability this
+  way). A contiguous array/slice primitive would remove that, but
+  it is a *language* change (interacts with uniqueness inference,
+  RC, and the structurally-decreasing-recursion totality story)
+  and is explicitly out of the ABI arc's scope. Sequenced after
+  M5 so the decision is driven by M5's fieldtest measurements,
+  not speculation.
+  - depends on: Embedding ABI — M5 (for the measured
+    justification).
+  - context: 2026-05-18 chat (user deferred this explicitly to
+    keep the ABI arc clean).
+
 - [ ] **\[milestone\]** Iteration-totality story — structural +
   Int-bounded total recursion with *enforced* non-negativity.
   AILang's iteration story stays as-is (structural / tail recursion;