# Embedding ABI — M5: `ail-embed` adapter + `data-server` wiring + thread-swarm backtest — Design Spec

**Date:** 2026-05-19
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude

## Goal

Prove the whole embedding arc end-to-end against the **real** target
host: stream real Pepperstone tick data through the real external
`data-server` crate into the **already-shipped, M3-frozen** AILang
kernel, across a thread swarm of independent per-thread embedding
contexts, and harvest the host-per-tick-FFI friction that feeds the
later P2 batch-crossing perf decision.

M5 is the **terminal** embedding milestone. M4 (cons-list crossing)
was retired; nothing follows M5 in this arc. Its value is exactly
two things, both stated in the roadmap:

1. **Existence proof of the whole goal** — a real backtest runs over
   a real data-server stream on a real thread swarm via the frozen
   M3 ABI, leak-free and sanitiser-clean.
2. **Friction harvest** — honest, at real data scale, feeding the
   host-per-tick-FFI-vs-batch-crossing perf decision (the P2
   flat-array roadmap item).

**M5 ships zero language/compiler/runtime change.** No schema,
checker, codegen, `runtime/`, or `examples/*.ail` edit. The M3 ABI is
frozen and already sufficient: the kernel `(State, Tick) -> State`
two-record-param per-tick crossing is gate-accepted, forwarder-
supported, and E2E-pinned both `own` and `borrow` (commit `170464f`).
The only AILang artefact M5 consumes is the already-shipped
`examples/embed_backtest_step_tick.ail`. M5 is host-side code +
wiring + the proof.

## Architecture

Three things, in dependency order. None of `ailang-core`,
`ailang-check`, `ailang-codegen`, `crates/ail`, or `runtime/` is
touched — **Invariant 1** (the compiler crates gain no
`data-server`/finance knowledge or dependency) is preserved not only
in letter but in the dependency graph, and is audited at close.

1. **`ail-embed` — a lean, reusable AILang-kernel embedding module.**
   `extern "C"` declarations against the M3-frozen ABI
   (`ailang_ctx_new`/`ailang_ctx_free`, `ailang_rc_alloc`,
   `ailang_rc_dec`, the author-exported `backtest_step_tick`), plus
   frozen-value-layout box read/write helpers and safe `State`/`Tick`
   wrappers. **Zero finance/`data-server` knowledge.** It is the Rust
   analogue of the already-audited C host
   `crates/ail/tests/embed/tick_roundtrip.c` — the same frozen-layout
   `make_state`/`make_tick`/read-back, the same `own`-mode contract —
   exposed as a small safe API instead of an inline test loop. It is
   **not** a published crate with a stability contract and implies
   **no** successor milestone; its correctness is proven by the (b)
   E2E below, not by an API-versioning promise.

2. **`data-server` wiring (the adapter layer of `ail-embed`).** The
   only place that knows *both* sides. It maps the rich
   `data_server::records::TickParsed { time_ms, ask, bid }` to the
   single scalar the M3 `Tick` record carries — `px = (ask + bid) /
   2.0`, the mid price — and drives the per-tick fold by *unrolling
   each `Arc<[TickParsed]>` chunk host-side* into per-tick
   `(State, Tick) -> State` calls. The adapter, not the kernel, owns
   chunk iteration (the M4-retirement decision: chunk boundaries are
   a host artefact, semantically invisible to the fold).

3. **The thread-swarm backtest (the E2E proof + friction harvest).**
   N worker threads, each with its own `ailang_ctx_t`, each folding
   the shipped kernel over a data-server stream, in the two shapes
   the user selected ("volle Möhre"):
   - **Symbol-fan (headline).** One distinct symbol per thread; each
     thread's `(acc, n)` checked against a single-thread host
     reference fold for that symbol. This is data-server's own
     stated concurrency design ("designed to back hundreds of
     single-threaded VMs in parallel" — one symbol per VM).
   - **Time-shard (second assertion).** One symbol, N threads each a
     disjoint month-window. Boundary-invisibility is proven **per
     shard, bit-exact**: each shard's `(acc, n)` equals a
     single-thread host fold of *that exact window* in stream order
     — the kernel's result for a window is independent of how
     `data-server` chunked it. The host then sums the partials; the
     *same* summation over the reference partials matches bit-exact
     by construction. The whole-window single-thread total is a
     *secondary* cross-check with an f64-reassociation tolerance
     (cross-shard re-association is expected host arithmetic, **not**
     a kernel property — asserting it bit-exact would be wrong).
     The per-shard bit-exact equality *is* the first actual proof of
     the chunk/window-boundary-invisibility claim the entire M4
     retirement rests on.

### Topology / Invariant 1 (orchestrator decision, not a user fork)

`ail-embed/` lives **in the AILang repo but excluded from the
`[workspace]`** — its own `Cargo.toml`, built and tested explicitly.
Substantive reason (not effort): the compiler workspace
(`cargo build/test --workspace`) then owes nothing to
`../libs/data-server` or `/mnt/tickdata` — Invariant 1 holds in the
dependency graph, not just on paper — while the *sole consumer of the
frozen ABI* stays co-versioned and co-audited with the ABI it binds,
which is precisely the protection the M3 freeze exists to provide.
(Rejected: workspace member — couples the compiler's own CI to an
external sibling dir + a 6 GB data mount; out-of-repo crate —
sacrifices the co-audit the freeze is for.)

`ail-embed` depends on `data-server` via a path dependency
(`../libs/data-server`); the README also documents a git source. The
path dep is the in-repo-dev choice and is the meeting point Invariant
1 names.

## Concrete code shapes

### The AILang program M5 delivers (headline)

M5 delivers **no new `.ail`**. The delivered AILang program is the
already-shipped, M3-frozen kernel it consumes —
`examples/embed_backtest_step_tick.ail`, verbatim:

```
(module embed_backtest_step_tick
  (data State (ctor State (con Float) (con Int)))
  (data Tick  (ctor Tick  (con Float)))
  (fn step
    (export "backtest_step_tick")
    (type (fn-type
            (params (own (con State)) (own (con Tick)))
            (ret (con State))))
    (params st tick)
    (body
      (match st
        (case (pat-ctor State acc n)
          (match tick
            (case (pat-ctor Tick px)
              (term-ctor State State (app + acc px) (app + n 1)))))))))
```

`acc += px; n += 1` over the per-tick mid price ⇒ the run's result
is `(Σ mid, tick-count)`, i.e. the mean mid price over the window —
a real, exactly-verifiable backtest aggregate.

**No new must-fail fixture.** The feature-acceptance clause-3
discriminator ("wrong code fails to typecheck") is already owned by
the M3 export gate (`export-non-scalar-signature`,
`export-has-effects`); M5 adds no checker/codegen change, so there is
no new wrong-code obligation. The three-clause feature-acceptance
criterion gates *language features*; M5 ships none, so it is N/A in
form. The discipline that *was* applied is the same M4-lesson
skepticism turned on the **deliverable shape**: "does a host author
need a named reusable embedding crate, or is that speculative infra?"
— resolved to a lean module whose correctness the real E2E proves,
not a speculative published API (see Decisions).

The frozen box layout the kernel crosses (DESIGN.md §"Frozen value
layout", M3 one-way commitment):

- `State`: payload `8 + 2*8` = 24 bytes — `tag:i64 @0`,
  `acc:f64 @8`, `n:i64 @16`.
- `Tick`: payload `8 + 1*8` = 16 bytes — `tag:i64 @0`, `px:f64 @8`.
- 8-byte rc header at `p-8` (set to 1 by `ailang_rc_alloc`,
  payload zeroed). `own` ⇒ kernel consumes both inputs; the return
  is host-owned and host-freed via `ailang_rc_dec`.

### Implementation shape (secondary — supporting, not the point)

The lean embedding module — Rust port of the audited
`tick_roundtrip.c` helpers, exposed as a safe API:

```rust
// ail-embed/src/lib.rs  (no finance knowledge)
#[repr(transparent)] pub struct Ctx(*mut c_void);   // !Send by default
unsafe extern "C" {
    fn ailang_ctx_new() -> *mut c_void;
    fn ailang_ctx_free(c: *mut c_void);
    fn ailang_rc_alloc(n: usize) -> *mut c_void;     // header=1, payload zeroed
    fn ailang_rc_dec(p: *mut c_void);
    fn backtest_step_tick(c: *mut c_void, st: *mut c_void,
                          tick: *mut c_void) -> *mut c_void;
}

impl Ctx {
    pub fn new() -> Self { Self(unsafe { ailang_ctx_new() }) }
    /// State box: tag@0=0, acc:f64@8, n:i64@16  (frozen M3 layout).
    fn make_state(acc: f64, n: i64) -> *mut c_void { /* ailang_rc_alloc(24) + writes */ }
    /// Tick box: tag@0=0, px:f64@8  (frozen M3 layout).
    fn make_tick(px: f64) -> *mut c_void { /* ailang_rc_alloc(16) + write */ }
    /// One fold step. `own`: kernel consumes st+tick; returns new owned State.
    pub fn step(&self, st: *mut c_void, px: f64) -> *mut c_void {
        unsafe { backtest_step_tick(self.0, st, Self::make_tick(px)) }
    }
}
impl Drop for Ctx { fn drop(&mut self) { unsafe { ailang_ctx_free(self.0) } } }
```

The adapter + per-thread fold (the only code that knows `data-server`):

```rust
// the worker body, one per thread, own ctx:
let ctx = Ctx::new();
let mut st = Ctx::make_state(0.0, 0);
let mut it = server.stream_tick_windowed(symbol, from_ms, to_ms).unwrap();
while let Some(chunk) = it.next_chunk() {            // data-server's loop
    for r in chunk.iter() {                          // host-side unroll
        let px = (r.ask + r.bid) / 2.0;              // adapter's job
        st = ctx.step(st, px);                       // M3 ABI, per tick
    }
}
let (acc, n) = read_state(st); ailang_rc_dec(st);    // host-owned return
```

`ail-embed/build.rs` invokes the `ail` CLI to emit the staticlib and
links it (mirrors `embed_tick_e2e.rs:61-78` verbatim, but from a
workspace-excluded crate): resolve the `ail` binary via an `AIL_BIN`
env override else `cargo build -p ail` against the parent AILang
workspace; run `ail build examples/embed_backtest_step_tick.ail
--emit=staticlib -o <OUT_DIR>`; emit `cargo:rustc-link-search` +
`-l static=embed_backtest_step_tick` + `-l static=ailang_rt`.

## Components

| Component | Role | Knows data-server? | Knows ABI? |
|---|---|---|---|
| `ail-embed/build.rs` | emit + link the M3 staticlib | no | symbol names only |
| `ail-embed` core (`Ctx`, box helpers) | frozen-layout FFI, safe wrappers | **no** | yes (the only crate that does) |
| `ail-embed` adapter (`tick→px`, fold driver) | `TickParsed`→`Tick`, chunk unroll | **yes (sole site)** | via core |
| swarm E2E (symbol-fan + time-shard + hermetic smoke) | the proof + friction harvest | yes | via core |
| `examples/embed_backtest_step_tick.ail` | the delivered kernel (unchanged) | no | it *is* the ABI source |

## Data flow

`DataServer::new(path)` → `Arc` → per worker thread:
`stream_tick_windowed(symbol, win)` → `SymbolChunkIter` →
`while let next_chunk()` yields `Arc<[TickParsed]>` →
host unroll: each `TickParsed` → `px=(ask+bid)/2` → `make_tick` box →
`backtest_step_tick(ctx, st, tick)` (kernel consumes `st`+`tick`,
returns new owned `State`) → thread `st` forward → end-of-stream:
read `(acc, n)`, `ailang_rc_dec(st)`, `ctx` dropped (`ailang_ctx_free`
→ `AILANG_RC_STATS` ctx readback). Symbol-fan: per-thread `(acc,n)`
vs per-symbol host reference (bit-exact, same stream order).
Time-shard: each shard's `(acc,n)` vs a single-thread host fold of
that exact window (bit-exact); host-summed `(Σacc,Σn)` vs the
identically-summed reference partials (bit-exact by construction);
whole-window single-thread total as a tolerance cross-check only.

## Error handling

- `stream_tick*` returns `None` for an unknown symbol or an empty
  window → the adapter surfaces a typed error to the harness, never a
  panic *inside* a kernel call (a half-threaded `State` must not
  leak; on the error path the current `st` is `ailang_rc_dec`'d
  before unwinding).
- The frozen-layout contract (`ailang_rc_alloc`, never raw `malloc`;
  fixed offsets) is **enforced by construction**: `make_state` /
  `make_tick` are the only box constructors and there is no public
  way to hand the kernel a raw pointer.
- `own` discipline: each `Tick` is freshly `ailang_rc_alloc`'d per
  call and consumed by the kernel; `st` is consumed and replaced by
  the return each step; only the final return is host-`dec`'d. This
  is the `tick_roundtrip.c` contract, unchanged.
- Real data absent (`/mnt/tickdata/Pepperstone` missing): the
  real-data swarm tests **skip gracefully**, mirroring
  `data-server`'s own `tests/data_server.rs::skip_if_no_data()`
  precedent — consistency with the wired crate's own convention, not
  an ad-hoc gate. The hermetic smoke test still runs and still
  proves the wiring/ABI handshake.

## Testing strategy

Three layers; the regression/leak posture mirrors the M1/M2/M3
global-leak-freedom proof (`AILANG_RC_STATS`, Σallocs == Σfrees
across all stat lines; sanitiser-clean).

1. **Hermetic smoke (always green, no `/mnt`).** A test-only writer
   emits the documented 24-byte packed `RawTickRecord` layout
   (`{time:f64 Delphi-day, ask:f64, bid:f64}`) into a ZIP'd
   `SYM_YYYY_MM.tick` in a `tempfile::tempdir()`, points the **real**
   `DataServer::new(tempdir)` at it, runs a single-thread fold, and
   asserts `(acc, n)` equals a host reference fold over the same
   synthetic ticks. Proves "adapter compiles, links, ABI handshake
   holds" independent of real data — a genuinely different failure
   mode from "real data present". This is also data-server's own
   tempdir-fixture unit discipline.
2. **Symbol-fan swarm (headline existence proof; skip-if-absent).** N
   threads, one liquid symbol each (default spine `EURUSD`; planner
   enumerates from what is present, e.g. `EURUSD`/`XAUUSD`), bounded
   window (default order-of one month per thread — enough real
   volume to surface per-tick-FFI friction, ~seconds runtime), each
   own `ailang_ctx_t`. Each thread's `(acc, n)` == single-thread host
   reference fold for that symbol/window. Globally leak-free,
   sanitiser-clean.
3. **Time-shard swarm (boundary-invisibility proof; skip-if-absent).**
   One symbol, N threads each a disjoint month-window. Assert, in
   order of strength: (a) each shard's `(acc,n)` == a single-thread
   host fold of *that exact window* in stream order, **bit-exact** —
   this is the boundary-invisibility proof (the kernel result for a
   window is independent of data-server's chunking); (b) the
   host-summed `(Σacc,Σn)` == the identically-summed reference
   partials, bit-exact by construction; (c) the whole-window
   single-thread total within an f64-reassociation tolerance only
   (cross-shard re-association is host arithmetic, not a kernel
   property). The M4-retirement claim's first actual proof is (a).

Friction harvest: the swarm runs are timed; the per-tick-FFI cost at
real tick volume is recorded in the close-out journal as the input to
the P2 batch-crossing decision (observation, not a perf gate — no
bench baseline is moved by M5).

## Acceptance criteria

- `git diff` after M5 touches **only** `ail-embed/**`, `Cargo.toml`
  (the `[workspace] exclude`/non-membership note if needed), docs,
  and the per-iter journal/stats. **Zero** diff to
  `crates/ailang-*`, `crates/ail/`, `runtime/`, `examples/*.ail`.
  Mechanically checkable; this *is* the no-language-change +
  Invariant-1 guarantee.
- The hermetic smoke test passes everywhere (no `/mnt`, no
  `../libs/data-server` data files needed beyond the crate itself).
- On this machine (real data present): the symbol-fan swarm and the
  time-shard swarm both pass — symbol-fan per-thread `(acc,n)`
  bit-exact vs the same-order single-thread reference; time-shard
  per-shard `(acc,n)` bit-exact vs a single-thread fold of that
  exact window and the host-summed partials bit-exact by
  construction (whole-window total only within f64-reassociation
  tolerance) — globally leak-free (`AILANG_RC_STATS` Σallocs ==
  Σfrees), sanitiser-clean (TSan: no data race across the per-thread
  ctxs).
- Where real data is absent, the two swarm tests skip (not fail),
  mirroring `data-server`'s own precedent.
- The architect audit at close confirms Invariant 1 (no
  finance/`data-server` symbol or dependency in any compiler crate or
  `runtime/`) and that the M3 frozen-layout SSOT is unmoved.
- The close-out journal records the measured per-tick-FFI cost at
  real volume as the P2 perf-decision input.

### Decisions (forks resolved during this brainstorm)

- **What is `data-server`** → the real external crate at
  `/home/brummel/dev/libs/data-server` (not a synthetic mock); real
  data at `/mnt/tickdata/Pepperstone` (6.3 GB).
- **Test-data strategy** → real-data swarm is the headline proof +
  friction harvest (skip-if-absent per data-server's own
  `skip_if_no_data()`); tiny hermetic synthetic-format fixture is the
  always-green floor. Real-as-opt-in was rejected: it would gut the
  harvest, which is only honest at real scale.
- **What is `ail-embed`** → a lean reusable embedding module +
  the real E2E on top (Option 2). Not "real E2E only" (the frozen
  ABI was *designed* to be bound generically — the module is the
  artefact the freeze is for); not two published crates (speculative
  infra, implies successor milestones the arc does not have).
- **Topology** → in-repo, `[workspace]`-excluded (orchestrator
  decision; Invariant-1/build-hygiene reason above).
- **Swarm shape** → both symbol-fan (headline) and time-shard
  (boundary-invisibility second assertion) — user choice "3, volle
  Möhre".

### Out of scope / known pre-existing drift (not M5's job)

DESIGN.md §"Frozen value layout" still calls a recursive typed-free
for boxed-field records "an additive M4 concern" (`docs/DESIGN.md`
~:2358-2360). M4 is retired; this is stale doc-honesty drift already
flagged by the M4-retirement journal's forward note as
reconcile-when-touched. M5 does not touch it and must not be charged
with it at audit; noted here so the architect sees it is pre-existing,
not M5-introduced.