From 064599e091d07abff6e02ed80bb5e8a53f236448 Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 18 May 2026 12:51:51 +0200 Subject: [PATCH] =?UTF-8?q?roadmap:=20Embedding=20ABI=20arc=20=E2=80=94=20?= =?UTF-8?q?M1=E2=80=93M5=20(P1)=20+=20deferred=20array=20primitive=20(P2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The data-server-driven goal: a compiled AILang kernel callable in-process from a concurrent Rust host so a backtest swarm runs each backtest as an independent single-threaded computation. Decomposed into five chronologically-bounded milestones (M1 linkable artifact+scalar entrypoint, M2 per-thread ctx+concurrency safety, M3 frozen value layout+ADT crossing, M4 List-ADT sequence crossing, M5 ail-embed adapter+swarm fieldtest), each its own full brainstorm->plan->implement->audit cycle. Shared motivation + three audited invariants (clean core / single-threaded per computation / native-AOT-only) carried on the M1 entry. The flat array/slice primitive is recorded P2 as a measured performance follow-up gated on M5's fieldtest, not a capability gap — keeping the ABI arc's scope clean. Decision made in the 2026-05-18 data-server analysis chat; no spec yet (brainstorm produces M1's spec at start). --- docs/roadmap.md | 130 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index 619df1a..7aeb299 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -129,6 +129,119 @@ work progresses. ## P1 — Next +- [ ] **\[milestone\]** Embedding ABI — M1: linkable artifact + + scalar C entrypoint + `main`-free lifecycle. First of a + five-milestone arc (M1–M5) that makes a compiled AILang kernel + callable in-process from a concurrent Rust host, so a swarm of + backtests over `data-server`-supplied market data runs each + backtest as an independent single-threaded AILang computation. + + **Motivation (shared by M1–M5).** The target deployment of + AILang is financial-data analysis: a host (Rust, owning + `data-server`'s shared zero-copy cache) drives a swarm of + backtests; each backtest is the pure per-chunk fold AILang is + built for (`(State, Chunk) -> State`, the `sum_sq` loop/recur + shape). Today AILang is a whole-program compiler whose only + output channel is `io/print_str` — there is no way to invoke a + compiled AILang function from a host with structured arguments + and a typed return. That missing call surface, not the math, is + the entire blocker. This is a backend/packaging capability, not + a language feature: the language semantics do not change. Its + justification is strategic (this is where the language is meant + to run), not the standard authoring-utility gate — a deliberate, + user-made direction call (2026-05-18 chat). + + **Invariants (audited at every close in this arc).** + (1) *Clean core* — `ailang-core` / `ailang-codegen` / the + runtime gain no `data-server`/finance knowledge and no + dependency on either; the only place AILang and `data-server` + meet is a separate host-side adapter crate (`ail-embed`, M5) + that depends on both while neither depends on it. The `audit` + architect review verifies this each close. + (2) *Single-threaded per computation* — all concurrency is + structural and owned by the host; no AILang value ever crosses + a thread boundary, so non-atomic RC and "no shared mutable + refs" stay intact. + (3) *Native-AOT only* — no interpreter/playground is introduced + (the "emit LLVM → clang -O2" performance pillar is load-bearing + for the swarm). + + **M1 scope.** CLI emit-mode producing a relocatable archive + instead of linking an executable; codegen exports an + author-designated `fn` under a stable C symbol with a + C-compatible signature instead of wrapping `main` (today + hard-coded `define i32 @main() { call i8 @ail__main() }`, + `crates/ailang-codegen/src/lib.rs:556`); a minimal explicit + lifecycle so the runtime is usable with no `main`-time setup. + Scalar signatures only (`(Int)->Int`, `(Float,Float)->Float`) + — no ADT crosses the boundary yet. Coherent stop: AILang + arithmetic callable from C/Rust, proven by a host harness that + links the `.a`, calls the symbol, asserts the return. + - context: 2026-05-18 chat (`data-server` analysis → + embedding-ABI decomposition). No spec yet; brainstorm + produces `docs/specs/-embedding-abi-m1.md` at M1 start. + +- [ ] **\[milestone\]** Embedding ABI — M2: per-thread runtime + context + concurrency safety. `ailang_ctx_new()/_free` threaded + through every call; neutralise the two process-global hazards + the runtime read surfaced — the non-atomic + `g_rc_alloc_count` / `g_rc_free_count` counters + `atexit` + stats hook (`runtime/rc.c:86–98`), and the allocator backing + (toward pure per-thread RC, no shared Boehm collector; aligns + with the standing P2 "Boehm full retirement" todo). Still + scalar-only so any failure is attributable to concurrency, not + marshalling. IO-at-the-boundary (`io/print_str` via a + per-instance sink) explicitly deferred here: pure + `(State,Chunk)->State` kernels have no effects; revisit only + if a logging kernel appears — deliberately not a separate + roadmap entry, by feature-acceptance discipline. Coherent stop: + swarm-safe scalar kernels, tsan-clean under N threads. + - depends on: Embedding ABI — M1. + - context: 2026-05-18 chat; runtime hazard read + (`runtime/rc.c` global stats + `atexit`). + +- [ ] **\[milestone\]** Embedding ABI — M3: frozen value layout + + single ADT/record crossing + RC ownership contract. Host + constructs a single-ctor record (Int/Float fields), passes it, + the kernel returns a record the host reads and frees; the RC + ownership-transfer contract at the call boundary is specified + and the `rc_header`/ADT-box layout documented as a *frozen* + ABI. Sequenced after M2 — a one-way stability commitment, so + the frozen ABI must already include the ctx parameter shape + (substantive ordering reason, not effort). Covers the `State` + value crossing in/out. Coherent stop: a record in/out with + alloc==free across many calls. + - depends on: Embedding ABI — M2. + - context: 2026-05-18 chat. `runtime/rc.c`'s header already + anticipates this ("The ABI defined here is stable … unless + the layout itself shifts"). + +- [ ] **\[milestone\]** Embedding ABI — M4: sequence crossing via + the existing `List` ADT. Host builds a `List Record` per chunk; + `(State, List Record) -> State` callable end-to-end. Pure + ABI/marshalling, no language change — applies M3's frozen + layout repeatedly. Makes the real fold runnable over real + chunks. The array-vs-cons-list performance question (1024 RC + cons-cells per chunk on the hot path) is deliberately *not* in + scope here — capability now, performance as a separate later + non-ABI milestone (P2 below), keeping the user's "focus on the + ABI, ignore the array gap" cut. Coherent stop: + `(State, List M1) -> State` runs. + - depends on: Embedding ABI — M3. + - context: 2026-05-18 chat. + +- [ ] **\[milestone\]** Embedding ABI — M5: `ail-embed` adapter + + `data-server` wiring + thread-swarm backtest (fieldtest + target). The host-side adapter crate (sole meeting point of + `data-server` and AILang, per Invariant 1) wires + `data-server`'s `while let next_chunk` loop to the AILang + kernel via the ABI; an actual small backtest runs over a + thread swarm. Natural `fieldtest` milestone: existence proof + of the whole goal + friction harvest feeding the + array-primitive decision. Coherent stop: the goal itself. + - depends on: Embedding ABI — M4. + - context: 2026-05-18 chat. + - [x] **\[milestone\]** Heap-`Str` ABI — runtime infrastructure for malloc-backed, refcounted `Str` values alongside the existing static `@.str_*` globals. Today the `Str` path is static-only @@ -165,6 +278,23 @@ work progresses. ## P2 — Medium-term +- [ ] **\[milestone\]** Flat array/slice primitive — performance + follow-up to the Embedding ABI arc, *not* a capability gap. + AILang's only sequence is the recursive `List` ADT (one RC cell + per element); marshalling a 1024-record chunk across the + embedding boundary builds a 1024-cell cons-chain per chunk on a + multi-GB hot path (Embedding ABI M4 ships the capability this + way). A contiguous array/slice primitive would remove that, but + it is a *language* change (interacts with uniqueness inference, + RC, and the structurally-decreasing-recursion totality story) + and is explicitly out of the ABI arc's scope. Sequenced after + M5 so the decision is driven by M5's fieldtest measurements, + not speculation. + - depends on: Embedding ABI — M5 (for the measured + justification). + - context: 2026-05-18 chat (user deferred this explicitly to + keep the ABI arc clean). + - [ ] **\[milestone\]** Iteration-totality story — structural + Int-bounded total recursion with *enforced* non-negativity. AILang's iteration story stays as-is (structural / tail recursion;