AILang/docs/plans/0089-embedding-abi-m5.1.md

embedding-abi-m5.1 — lean ail-embed core + build.rs + hermetic smoke — Implementation Plan

Parent spec: docs/specs/0044-embedding-abi-m5.md

For agentic workers: REQUIRED SUB-SKILL: use skills/implement to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Stand up the workspace-excluded ail-embed crate: a lean, data-server-free embedding core (Rust port of the audited crates/ail/tests/embed/tick_roundtrip.c) that links the M3-frozen staticlib via a build.rs, plus a hermetic data-server smoke test proving the ABI handshake without /mnt.

Architecture: ail-embed/ is a sibling crate, not an AILang [workspace] member (Invariant 1: the compiler workspace owes nothing to data-server//mnt). Its build.rs emits + links libembed_backtest_step_tick.a + libailang_rt.a from the already-shipped kernel. src/lib.rs is the core only — extern "C" to the frozen ABI + Kernel running a price fold; zero data_server:: reference. data-server is a dev-dependency used solely by the integration smoke test, so the iter-1 library has zero dependencies. The adapter API + thread-swarm are deliberately later iterations and are NOT in this plan.

Tech Stack: Rust (edition 2021), std::ffi::c_void, std::process::Command (build.rs); dev-only: data-server (path ../libs/data-server), zip = "2", tempfile = "3". Links the M3 staticlib emitted by the in-repo ail CLI.

---

Files this plan creates or modifies:

• Create: ail-embed/Cargo.toml — manifest for the workspace-excluded crate; zero [dependencies], dev-deps only
• Create: ail-embed/.gitignore — ignore /target (per-crate, mirrors ../libs/data-server/.gitignore precedent)
• Create: ail-embed/build.rs — resolve ail, emit --emit=staticlib, emit link directives
• Create: ail-embed/src/lib.rs — the lean embedding core: extern "C" block, Ctx, Kernel, frozen-layout box helpers, in-source RED unit test
• Create: ail-embed/tests/smoke.rs — hermetic data-server smoke (synthetic Pepperstone-format fixture → real DataServer → Kernel → assertion)
• Modify: Cargo.toml:3-10 — add a 2-line comment after the members array documenting the deliberate non-membership of ail-embed/

Cross-reference anchors (read-only, do not edit):

• crates/ail/tests/embed/tick_roundtrip.c:31-75 — the C SSOT the Rust core byte-mirrors (extern decls, make_state/make_tick, fold/read-back, own discipline)
• crates/ail/tests/embed_tick_e2e.rs:61-78 — the ail build --emit=staticlib -o + lib<module>.a/libailang_rt.a link incantation ported into build.rs
• docs/DESIGN.md:2322-2360 — frozen value-layout SSOT (State 24B, Tick 16B; p-8 rc header; own/free contract)
• ../libs/data-server/src/loader.rs:45-101 — ZIP .bin read path the fixture writer must satisfy
• ../libs/data-server/src/records.rs:60-66,188-191 — RawTickRecord #[repr(C,packed)] 24-byte layout + its size_of==24 pin
• ../libs/data-server/src/lib.rs:88,166-171,236-238 — filename scanner regex, DataServer::new, stream_tick

---

Task 1: Scaffold the workspace-excluded ail-embed crate + build.rs

Stands up the crate so the novel build.rs (no in-repo precedent) is de-risked before any embedding logic: a stub lib that links the staticlib and compiles proves the emit+link path in isolation.

Files:

• Create: ail-embed/Cargo.toml

• Create: ail-embed/.gitignore

• Create: ail-embed/build.rs

• Create: ail-embed/src/lib.rs (stub, replaced in Task 2)

• Modify: Cargo.toml:3-10

• Step 1: Create ail-embed/Cargo.toml

[package]
name = "ail-embed"
version = "0.0.1"
edition = "2021"
license = "MIT"
publish = false
description = "Lean embedding of an AILang M3-frozen staticlib kernel into a Rust host. Not an AILang [workspace] member (Invariant 1)."

# Iter m5.1: the library itself has ZERO dependencies — the embedding
# core only touches the frozen C ABI. `data-server` is dev-only,
# exercised solely by the hermetic smoke test, so Invariant 1 holds
# in the dependency graph, not just on paper.
[dependencies]

[dev-dependencies]
data-server = { path = "../libs/data-server" }
zip = "2"
tempfile = "3"

• Step 2: Create ail-embed/.gitignore

/target

• Step 3: Create ail-embed/src/lib.rs as a stub

//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. Replaced with the real core in Task 2; this stub
//! exists so Task 1 can prove the build.rs emit+link path alone.

• Step 4: Create ail-embed/build.rs

//! Emits the M3-frozen kernel staticlib and links it.
//!
//! Resolution of the `ail` binary has no in-repo precedent (the
//! in-test mechanism `env!("CARGO_BIN_EXE_ail")` is unavailable to a
//! workspace-excluded crate's build script): use `$AIL_BIN` if set,
//! else `cargo build --manifest-path <repo>/Cargo.toml -p ail` and
//! use `<repo>/target/debug/ail`. The AILang workspace target dir
//! (`<repo>/target`) is distinct from this crate's
//! (`ail-embed/target`), so the nested `cargo build` cannot deadlock
//! on the outer build's lock.
use std::env;
use std::path::PathBuf;
use std::process::Command;

fn main() {
    let manifest = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap());
    let repo = manifest.parent().unwrap().to_path_buf(); // /home/brummel/dev/ailang
    let kernel = repo.join("examples/embed_backtest_step_tick.ail");
    let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());

    println!("cargo:rerun-if-changed={}", kernel.display());
    println!("cargo:rerun-if-changed=build.rs");
    println!("cargo:rerun-if-env-changed=AIL_BIN");

    let ail_bin: PathBuf = match env::var("AIL_BIN") {
        Ok(p) => PathBuf::from(p),
        Err(_) => {
            let status = Command::new("cargo")
                .args(["build", "--manifest-path"])
                .arg(repo.join("Cargo.toml"))
                .args(["-p", "ail"])
                .status()
                .expect("spawn `cargo build -p ail`");
            assert!(status.success(), "building the `ail` CLI failed");
            repo.join("target/debug/ail")
        }
    };

    let build = Command::new(&ail_bin)
        .arg("build")
        .arg(&kernel)
        .args(["--emit=staticlib", "-o"])
        .arg(&out_dir)
        .output()
        .expect("spawn `ail build --emit=staticlib`");
    assert!(
        build.status.success(),
        "ail build --emit=staticlib failed:\n{}",
        String::from_utf8_lossy(&build.stderr)
    );

    println!("cargo:rustc-link-search=native={}", out_dir.display());
    println!("cargo:rustc-link-lib=static=embed_backtest_step_tick");
    println!("cargo:rustc-link-lib=static=ailang_rt");
}

• Step 5: Document the deliberate non-membership in the root Cargo.toml

Modify Cargo.toml — insert the comment immediately after the closing ] of members (currently Cargo.toml:10). The exact existing block is:

members = [
    "crates/ailang-core",
    "crates/ailang-check",
    "crates/ailang-codegen",
    "crates/ailang-surface",
    "crates/ailang-prose",
    "crates/ail",
]

Replace it verbatim with:

members = [
    "crates/ailang-core",
    "crates/ailang-check",
    "crates/ailang-codegen",
    "crates/ailang-surface",
    "crates/ailang-prose",
    "crates/ail",
]
# `ail-embed/` is intentionally NOT a member: it depends on the
# external `../libs/data-server` and is the sole data-server↔AILang
# meeting point (Invariant 1, docs/specs/0044-embedding-abi-m5.md).
# Adding it here would couple the compiler workspace to a sibling dir.

• Step 6: Verify the crate builds and the staticlib links

Run: cargo build --manifest-path ail-embed/Cargo.toml Expected: PASS — exit 0, ending in a Finished \dev` profileline. On the first run it additionally builds theailCLI (nestedcargo build -p ail, may take a minute); subsequent runs are cached. No undefined referencelinker error (the stub lib references no ABI symbol yet, but-l static=…` resolves cleanly against the emitted archives).

• Step 7: Verify the AILang compiler workspace is untouched in its graph

Run: cargo metadata --format-version 1 --manifest-path Cargo.toml --no-deps | grep -c data-server Expected: 0 — the AILang workspace metadata contains no data-server package (Invariant 1 in the dependency graph).

---

Task 2: The lean embedding core (Ctx, Kernel, frozen-layout helpers)

Ports tick_roundtrip.c:31-75 into a safe Rust API. RED-first with an in-source unit test that needs no data-server — proving the ABI handshake with zero finance knowledge.

Files:

• Modify: ail-embed/src/lib.rs (replace the Task 1 stub)

• Step 1: Write the failing in-source unit test

Replace the entire contents of ail-embed/src/lib.rs with only the test module below (the API it calls does not exist yet — this is the RED state):

//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. The Rust analogue of the audited C host
//! `crates/ail/tests/embed/tick_roundtrip.c`: same frozen value
//! layout (DESIGN.md §"Frozen value layout", :2322-2360), same
//! `own`-mode contract. ZERO `data_server`/finance knowledge.

#[cfg(test)]
mod tests {
    use super::Kernel;

    /// The kernel folds `acc += px; n += 1` over the price stream.
    /// Same-order f64 sum ⇒ bit-exact: 3.0 + 4.0 + 5.0 == 12.0, n==3.
    #[test]
    fn kernel_run_sums_prices() {
        let k = Kernel::new();
        let (acc, n) = k.run([3.0_f64, 4.0, 5.0]);
        assert_eq!(acc, 12.0);
        assert_eq!(n, 3);
    }
}

• Step 2: Run the test to verify it fails

Run: cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices Expected: FAIL — compile error cannot find type \Kernel` in this scope(orunresolved import `super::Kernel``). The test cannot run because the API is unimplemented.

• Step 3: Write the minimal implementation

Prepend the following to ail-embed/src/lib.rs, above the #[cfg(test)] mod tests block (keep the module doc comment at the top of the file):

use std::ffi::c_void;

// The M3-frozen C ABI. Symbol names + signatures mirror
// `crates/ail/tests/embed/tick_roundtrip.c:31-36`. `backtest_step_tick`
// is the author-chosen export of `examples/embed_backtest_step_tick.ail`.
unsafe extern "C" {
    fn ailang_ctx_new() -> *mut c_void;
    fn ailang_ctx_free(c: *mut c_void);
    fn ailang_rc_alloc(n: usize) -> *mut c_void;
    fn ailang_rc_dec(p: *mut c_void);
    fn backtest_step_tick(c: *mut c_void, st: *mut c_void, tick: *mut c_void)
        -> *mut c_void;
}

/// A per-thread embedding context (M2 `ailang_ctx_t`). `!Send` by
/// construction (raw pointer) — one ctx per OS thread, as the ABI
/// requires.
struct Ctx(*mut c_void);

impl Ctx {
    fn new() -> Self {
        Self(unsafe { ailang_ctx_new() })
    }

    /// `State` box, frozen layout (DESIGN.md :2328-2336):
    /// payload `8 + 2*8` = 24 bytes — tag:i64@0, acc:f64@8, n:i64@16.
    /// `ailang_rc_alloc` sets the `p-8` rc header to 1 and zeroes the
    /// payload, so the tag write of 0 is the single-ctor tag.
    fn make_state(acc: f64, n: i64) -> *mut c_void {
        unsafe {
            let p = ailang_rc_alloc(8 + 2 * 8) as *mut u8;
            (p.add(0) as *mut i64).write_unaligned(0);
            (p.add(8) as *mut f64).write_unaligned(acc);
            (p.add(16) as *mut i64).write_unaligned(n);
            p as *mut c_void
        }
    }

    /// `Tick` box, frozen layout: payload `8 + 1*8` = 16 bytes —
    /// tag:i64@0, px:f64@8.
    fn make_tick(px: f64) -> *mut c_void {
        unsafe {
            let p = ailang_rc_alloc(8 + 1 * 8) as *mut u8;
            (p.add(0) as *mut i64).write_unaligned(0);
            (p.add(8) as *mut f64).write_unaligned(px);
            p as *mut c_void
        }
    }

    /// Reads back `(acc, n)` from a `State` payload pointer.
    fn read_state(st: *mut c_void) -> (f64, i64) {
        unsafe {
            let p = st as *mut u8;
            let acc = (p.add(8) as *const f64).read_unaligned();
            let n = (p.add(16) as *const i64).read_unaligned();
            (acc, n)
        }
    }
}

impl Drop for Ctx {
    fn drop(&mut self) {
        unsafe { ailang_ctx_free(self.0) }
    }
}

/// Embeds the M3-frozen `(State, Tick) -> State` kernel and folds a
/// price stream through it. Raw ABI pointers never escape this type
/// (spec §"Error handling": no public way to hand the kernel a raw
/// pointer).
pub struct Kernel {
    ctx: Ctx,
}

impl Kernel {
    pub fn new() -> Self {
        Self { ctx: Ctx::new() }
    }

    /// Folds `prices` via the kernel and returns `(Σ px, count)`.
    /// `own` discipline (DESIGN.md :2346-2352): each `Tick` is freshly
    /// `ailang_rc_alloc`'d and consumed by the kernel; `st` is
    /// consumed and replaced by the return each step; only the final
    /// return is host-`dec`'d.
    pub fn run<I: IntoIterator<Item = f64>>(&self, prices: I) -> (f64, i64) {
        let mut st = Ctx::make_state(0.0, 0);
        for px in prices {
            let tick = Ctx::make_tick(px);
            st = unsafe { backtest_step_tick(self.ctx.0, st, tick) };
        }
        let out = Ctx::read_state(st);
        unsafe { ailang_rc_dec(st) };
        out
    }
}

impl Default for Kernel {
    fn default() -> Self {
        Self::new()
    }
}

• Step 4: Run the test to verify it passes

Run: cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices Expected: PASS — test tests::kernel_run_sums_prices ... ok, 1 passed; 0 failed.

---

Task 3: Hermetic data-server smoke (synthetic Pepperstone fixture)

Integration coverage proving the real data-server wiring + ABI handshake without /mnt. Adds no production code (Task 2 already ships the API); a build+run gate, not a contrived RED.

Files:

• Create: ail-embed/tests/smoke.rs

• Step 1: Write ail-embed/tests/smoke.rs

//! Hermetic smoke (Testing strategy §1): synthesize a
//! Pepperstone-format `.tick` ZIP, point the REAL `DataServer` at it,
//! fold the stream through the embedding `Kernel`, assert the result
//! equals a host reference fold. Proves "adapter compiles, links,
//! ABI handshake holds" with no `/mnt` dependency. The fixture writer
//! dumps `data_server::records::RawTickRecord` bytes (the crate's own
//! public on-disk type) into a `.bin` ZIP entry — correct by
//! construction, mirroring `data-server/src/loader.rs:110-124`.

use std::io::Write;
use std::sync::Arc;

use ail_embed::Kernel;
use data_server::records::RawTickRecord;
use data_server::DataServer;

/// Writes `recs` as a Pepperstone tick file `<sym>_2017_01.tick`
/// (ZIP holding one packed-`RawTickRecord` `.bin`) into `dir`.
/// Filename satisfies the scanner regex `^(.+)_(\d{4})_(\d{2})\.tick$`
/// (`data-server/src/lib.rs:88`); the `.bin` entry name is arbitrary
/// as long as it ends in `.bin` (`loader.rs:50-55`).
fn write_tick_fixture(dir: &std::path::Path, sym: &str, recs: &[RawTickRecord]) {
    let path = dir.join(format!("{sym}_2017_01.tick"));
    let f = std::fs::File::create(&path).unwrap();
    let mut zip = zip::ZipWriter::new(f);
    zip.start_file::<String, ()>("TICK.bin".into(), Default::default())
        .unwrap();
    for r in recs {
        // SAFETY: RawTickRecord is #[repr(C, packed)], 24 bytes; this
        // is exactly data-server's own test serialisation
        // (loader.rs:117-120).
        let bytes: &[u8] = unsafe {
            std::slice::from_raw_parts(r as *const RawTickRecord as *const u8, 24)
        };
        zip.write_all(bytes).unwrap();
    }
    zip.finish().unwrap();
}

#[test]
fn hermetic_smoke_data_server_roundtrip() {
    // 10 deterministic ticks. mid = (ask+bid)/2; with ask==bid==i+1,
    // mid == i+1. Σ_{i=0..9}(i+1) == 55.0 exactly in f64; n == 10.
    // `time` is any monotonic Delphi-day value (no window ⇒ unused by
    // the fold; only the filename drives symbol/format scanning).
    let recs: Vec<RawTickRecord> = (0..10)
        .map(|i| RawTickRecord {
            time: 42795.0 + i as f64 * 1e-4,
            ask: (i + 1) as f64,
            bid: (i + 1) as f64,
        })
        .collect();

    let dir = tempfile::tempdir().unwrap();
    write_tick_fixture(dir.path(), "TEST", &recs);

    let server = Arc::new(DataServer::new(dir.path()));
    assert!(server.has_symbol("TEST"), "fixture symbol not scanned");

    // Collect the mid-price stream in data-server order.
    let mut it = server.stream_tick("TEST").expect("TEST tick stream");
    let mut prices: Vec<f64> = Vec::new();
    while let Some(chunk) = it.next_chunk() {
        for r in chunk.iter() {
            prices.push((r.ask + r.bid) / 2.0);
        }
    }
    assert_eq!(prices.len(), 10, "all ticks streamed");

    // Host reference fold, same order, same ops ⇒ bit-exact.
    let ref_acc: f64 = prices.iter().copied().fold(0.0, |a, p| a + p);
    let ref_n = prices.len() as i64;

    let (acc, n) = Kernel::new().run(prices.iter().copied());

    assert_eq!(acc, ref_acc, "kernel acc bit-exact vs host reference");
    assert_eq!(acc, 55.0, "deterministic fixture sum");
    assert_eq!(n, ref_n);
    assert_eq!(n, 10);
}

• Step 2: Run the smoke test

Run: cargo test --manifest-path ail-embed/Cargo.toml --test smoke hermetic_smoke_data_server_roundtrip Expected: PASS — test hermetic_smoke_data_server_roundtrip ... ok, 1 passed; 0 failed. (Builds data-server as a dev-dependency on first run.)

• Step 3: Run the full crate test suite

Run: cargo test --manifest-path ail-embed/Cargo.toml Expected: PASS — kernel_run_sums_prices (unit) + hermetic_smoke_data_server_roundtrip (integration) both ok; 2 passed; 0 failed total across the two binaries; 0 failed.

• Step 4: Mechanically verify Invariant 1 / no-language-change

Run: git status --porcelain | grep -E '^\?\?|^ ?M' | grep -vE '^.. (ail-embed/|Cargo\.toml|docs/)' Expected: empty output — the working tree adds/modifies only ail-embed/**, the root Cargo.toml comment, and docs/. Zero diff to crates/ailang-*, crates/ail/, runtime/, examples/*.ail (the spec's no-language-change + Invariant-1 guarantee, mechanically checked).

---

Self-review (planner Step 5)

1. Spec coverage. Arch §1 (lean core) → Task 2. Concrete code shapes: implementation-shape + frozen box layout → Task 2 (box helpers cite DESIGN.md:2328-2336); build/link incantation → Task 1 build.rs (ports embed_tick_e2e.rs:61-78). Testing §1 hermetic smoke → Task 3. No-language-change / Invariant-1 acceptance → Task 1 (zero-dep lib, dev-only data-server, root-Cargo.toml comment) + Task 1 Step 7 (cargo metadata graph check) + Task 3 Step 4 (git status path check). Adapter/swarm (Arch §2/§3, Testing §2/§3) explicitly deferred per the iteration scope — not a coverage gap.
2. Placeholder scan. No "TBD/TODO/implement later/similar to/add appropriate". The Task 1 stub src/lib.rs is a named, shown artefact explicitly replaced in Task 2 (not a placeholder).
3. Type/name consistency. Kernel, Kernel::new, Kernel::run, Ctx, make_state/make_tick/read_state, backtest_step_tick, crate ail-embed / import ail_embed, test names kernel_run_sums_prices + hermetic_smoke_data_server_roundtrip — identical across Tasks 2-3 and the verification filters.
4. Step granularity. Each step is one file write or one command; all 2-5 min. The largest (Task 2 Step 3) is a single verbatim paste.
5. No commit steps. None. Work stays in the working tree; the Boss commits the iter.
6. Pin/replacement substring contiguity. No contains(...) / grep-of-a-verbatim-body pairing. The only greps (Task 1 Step 7 grep -c data-server; Task 3 Step 4 path grep) assert over command output / git status, not over a verbatim text body this plan also writes — N/A, no contiguity hazard.
7. Compile-gate vs deferred-caller. No signature change with deferred callers. Task 1's build gate uses a stub lib that references no ABI symbol (links cleanly); Task 2 introduces the symbol references and its own gate in the same task. No cross-task compile-gate contradiction.
8. Verification filter strings resolve. kernel_run_sums_prices, --test smoke, hermetic_smoke_data_server_roundtrip are all defined by this plan in the same tasks that filter on them — resolve by construction. Task 1 Step 6/7 and Task 3 Step 3 use unfiltered builds / explicit-count assertions (2 passed; 0 failed, grep -c … == 0), so "nothing ran" cannot masquerade as success.