# embedding-abi-m5.1 — lean `ail-embed` core + build.rs + hermetic smoke — Implementation Plan

> **Parent spec:** `docs/specs/0044-embedding-abi-m5.md`
>
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
> to run this plan. Steps use `- [ ]` checkboxes for tracking.

**Goal:** Stand up the workspace-excluded `ail-embed` crate: a lean,
data-server-free embedding core (Rust port of the audited
`crates/ail/tests/embed/tick_roundtrip.c`) that links the
M3-frozen staticlib via a `build.rs`, plus a hermetic data-server
smoke test proving the ABI handshake without `/mnt`.

**Architecture:** `ail-embed/` is a sibling crate, *not* an AILang
`[workspace]` member (Invariant 1: the compiler workspace owes
nothing to `data-server`/`/mnt`). Its `build.rs` emits + links
`libembed_backtest_step_tick.a` + `libailang_rt.a` from the
already-shipped kernel. `src/lib.rs` is the core only — `extern "C"`
to the frozen ABI + `Kernel` running a price fold; **zero**
`data_server::` reference. `data-server` is a **dev-dependency** used
solely by the integration smoke test, so the iter-1 library has zero
dependencies. The adapter API + thread-swarm are deliberately
later iterations and are NOT in this plan.

**Tech Stack:** Rust (edition 2021), `std::ffi::c_void`,
`std::process::Command` (build.rs); dev-only: `data-server` (path
`../libs/data-server`), `zip = "2"`, `tempfile = "3"`. Links the
M3 staticlib emitted by the in-repo `ail` CLI.

---

**Files this plan creates or modifies:**

- Create: `ail-embed/Cargo.toml` — manifest for the workspace-excluded crate; zero `[dependencies]`, dev-deps only
- Create: `ail-embed/.gitignore` — ignore `/target` (per-crate, mirrors `../libs/data-server/.gitignore` precedent)
- Create: `ail-embed/build.rs` — resolve `ail`, emit `--emit=staticlib`, emit link directives
- Create: `ail-embed/src/lib.rs` — the lean embedding core: `extern "C"` block, `Ctx`, `Kernel`, frozen-layout box helpers, in-source RED unit test
- Create: `ail-embed/tests/smoke.rs` — hermetic data-server smoke (synthetic Pepperstone-format fixture → real `DataServer` → `Kernel` → assertion)
- Modify: `Cargo.toml:3-10` — add a 2-line comment after the `members` array documenting the deliberate non-membership of `ail-embed/`

Cross-reference anchors (read-only, do not edit):
- `crates/ail/tests/embed/tick_roundtrip.c:31-75` — the C SSOT the Rust core byte-mirrors (extern decls, `make_state`/`make_tick`, fold/read-back, `own` discipline)
- `crates/ail/tests/embed_tick_e2e.rs:61-78` — the `ail build --emit=staticlib -o` + `lib<module>.a`/`libailang_rt.a` link incantation ported into `build.rs`
- `docs/DESIGN.md:2322-2360` — frozen value-layout SSOT (State 24B, Tick 16B; `p-8` rc header; `own`/free contract)
- `../libs/data-server/src/loader.rs:45-101` — ZIP `.bin` read path the fixture writer must satisfy
- `../libs/data-server/src/records.rs:60-66,188-191` — `RawTickRecord` `#[repr(C,packed)]` 24-byte layout + its `size_of==24` pin
- `../libs/data-server/src/lib.rs:88,166-171,236-238` — filename scanner regex, `DataServer::new`, `stream_tick`

---

### Task 1: Scaffold the workspace-excluded `ail-embed` crate + build.rs

Stands up the crate so the novel `build.rs` (no in-repo precedent)
is de-risked *before* any embedding logic: a stub lib that links the
staticlib and compiles proves the emit+link path in isolation.

**Files:**
- Create: `ail-embed/Cargo.toml`
- Create: `ail-embed/.gitignore`
- Create: `ail-embed/build.rs`
- Create: `ail-embed/src/lib.rs` (stub, replaced in Task 2)
- Modify: `Cargo.toml:3-10`

- [ ] **Step 1: Create `ail-embed/Cargo.toml`**

```toml
[package]
name = "ail-embed"
version = "0.0.1"
edition = "2021"
license = "MIT"
publish = false
description = "Lean embedding of an AILang M3-frozen staticlib kernel into a Rust host. Not an AILang [workspace] member (Invariant 1)."

# Iter m5.1: the library itself has ZERO dependencies — the embedding
# core only touches the frozen C ABI. `data-server` is dev-only,
# exercised solely by the hermetic smoke test, so Invariant 1 holds
# in the dependency graph, not just on paper.
[dependencies]

[dev-dependencies]
data-server = { path = "../libs/data-server" }
zip = "2"
tempfile = "3"
```

- [ ] **Step 2: Create `ail-embed/.gitignore`**

```gitignore
/target
```

- [ ] **Step 3: Create `ail-embed/src/lib.rs` as a stub**

```rust
//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. Replaced with the real core in Task 2; this stub
//! exists so Task 1 can prove the build.rs emit+link path alone.
```

- [ ] **Step 4: Create `ail-embed/build.rs`**

```rust
//! Emits the M3-frozen kernel staticlib and links it.
//!
//! Resolution of the `ail` binary has no in-repo precedent (the
//! in-test mechanism `env!("CARGO_BIN_EXE_ail")` is unavailable to a
//! workspace-excluded crate's build script): use `$AIL_BIN` if set,
//! else `cargo build --manifest-path <repo>/Cargo.toml -p ail` and
//! use `<repo>/target/debug/ail`. The AILang workspace target dir
//! (`<repo>/target`) is distinct from this crate's
//! (`ail-embed/target`), so the nested `cargo build` cannot deadlock
//! on the outer build's lock.
use std::env;
use std::path::PathBuf;
use std::process::Command;

fn main() {
    let manifest = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap());
    let repo = manifest.parent().unwrap().to_path_buf(); // /home/brummel/dev/ailang
    let kernel = repo.join("examples/embed_backtest_step_tick.ail");
    let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());

    println!("cargo:rerun-if-changed={}", kernel.display());
    println!("cargo:rerun-if-changed=build.rs");
    println!("cargo:rerun-if-env-changed=AIL_BIN");

    let ail_bin: PathBuf = match env::var("AIL_BIN") {
        Ok(p) => PathBuf::from(p),
        Err(_) => {
            let status = Command::new("cargo")
                .args(["build", "--manifest-path"])
                .arg(repo.join("Cargo.toml"))
                .args(["-p", "ail"])
                .status()
                .expect("spawn `cargo build -p ail`");
            assert!(status.success(), "building the `ail` CLI failed");
            repo.join("target/debug/ail")
        }
    };

    let build = Command::new(&ail_bin)
        .arg("build")
        .arg(&kernel)
        .args(["--emit=staticlib", "-o"])
        .arg(&out_dir)
        .output()
        .expect("spawn `ail build --emit=staticlib`");
    assert!(
        build.status.success(),
        "ail build --emit=staticlib failed:\n{}",
        String::from_utf8_lossy(&build.stderr)
    );

    println!("cargo:rustc-link-search=native={}", out_dir.display());
    println!("cargo:rustc-link-lib=static=embed_backtest_step_tick");
    println!("cargo:rustc-link-lib=static=ailang_rt");
}
```

- [ ] **Step 5: Document the deliberate non-membership in the root `Cargo.toml`**

Modify `Cargo.toml` — insert the comment immediately after the
closing `]` of `members` (currently `Cargo.toml:10`). The exact
existing block is:

```toml
members = [
    "crates/ailang-core",
    "crates/ailang-check",
    "crates/ailang-codegen",
    "crates/ailang-surface",
    "crates/ailang-prose",
    "crates/ail",
]
```

Replace it verbatim with:

```toml
members = [
    "crates/ailang-core",
    "crates/ailang-check",
    "crates/ailang-codegen",
    "crates/ailang-surface",
    "crates/ailang-prose",
    "crates/ail",
]
# `ail-embed/` is intentionally NOT a member: it depends on the
# external `../libs/data-server` and is the sole data-server↔AILang
# meeting point (Invariant 1, docs/specs/0044-embedding-abi-m5.md).
# Adding it here would couple the compiler workspace to a sibling dir.
```

- [ ] **Step 6: Verify the crate builds and the staticlib links**

Run: `cargo build --manifest-path ail-embed/Cargo.toml`
Expected: PASS — exit 0, ending in a `Finished \`dev\` profile`
line. On the first run it additionally builds the `ail` CLI (nested
`cargo build -p ail`, may take a minute); subsequent runs are cached.
No `undefined reference` linker error (the stub lib references no ABI
symbol yet, but `-l static=…` resolves cleanly against the emitted
archives).

- [ ] **Step 7: Verify the AILang compiler workspace is untouched in its graph**

Run: `cargo metadata --format-version 1 --manifest-path Cargo.toml --no-deps | grep -c data-server`
Expected: `0` — the AILang workspace metadata contains no
`data-server` package (Invariant 1 in the dependency graph).

---

### Task 2: The lean embedding core (`Ctx`, `Kernel`, frozen-layout helpers)

Ports `tick_roundtrip.c:31-75` into a safe Rust API. RED-first with
an in-source unit test that needs **no** data-server — proving the
ABI handshake with zero finance knowledge.

**Files:**
- Modify: `ail-embed/src/lib.rs` (replace the Task 1 stub)

- [ ] **Step 1: Write the failing in-source unit test**

Replace the entire contents of `ail-embed/src/lib.rs` with *only*
the test module below (the API it calls does not exist yet — this is
the RED state):

```rust
//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. The Rust analogue of the audited C host
//! `crates/ail/tests/embed/tick_roundtrip.c`: same frozen value
//! layout (DESIGN.md §"Frozen value layout", :2322-2360), same
//! `own`-mode contract. ZERO `data_server`/finance knowledge.

#[cfg(test)]
mod tests {
    use super::Kernel;

    /// The kernel folds `acc += px; n += 1` over the price stream.
    /// Same-order f64 sum ⇒ bit-exact: 3.0 + 4.0 + 5.0 == 12.0, n==3.
    #[test]
    fn kernel_run_sums_prices() {
        let k = Kernel::new();
        let (acc, n) = k.run([3.0_f64, 4.0, 5.0]);
        assert_eq!(acc, 12.0);
        assert_eq!(n, 3);
    }
}
```

- [ ] **Step 2: Run the test to verify it fails**

Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices`
Expected: FAIL — compile error `cannot find type \`Kernel\` in this
scope` (or `unresolved import \`super::Kernel\``). The test cannot
run because the API is unimplemented.

- [ ] **Step 3: Write the minimal implementation**

Prepend the following to `ail-embed/src/lib.rs`, *above* the
`#[cfg(test)] mod tests` block (keep the module doc comment at the
top of the file):

```rust
use std::ffi::c_void;

// The M3-frozen C ABI. Symbol names + signatures mirror
// `crates/ail/tests/embed/tick_roundtrip.c:31-36`. `backtest_step_tick`
// is the author-chosen export of `examples/embed_backtest_step_tick.ail`.
unsafe extern "C" {
    fn ailang_ctx_new() -> *mut c_void;
    fn ailang_ctx_free(c: *mut c_void);
    fn ailang_rc_alloc(n: usize) -> *mut c_void;
    fn ailang_rc_dec(p: *mut c_void);
    fn backtest_step_tick(c: *mut c_void, st: *mut c_void, tick: *mut c_void)
        -> *mut c_void;
}

/// A per-thread embedding context (M2 `ailang_ctx_t`). `!Send` by
/// construction (raw pointer) — one ctx per OS thread, as the ABI
/// requires.
struct Ctx(*mut c_void);

impl Ctx {
    fn new() -> Self {
        Self(unsafe { ailang_ctx_new() })
    }

    /// `State` box, frozen layout (DESIGN.md :2328-2336):
    /// payload `8 + 2*8` = 24 bytes — tag:i64@0, acc:f64@8, n:i64@16.
    /// `ailang_rc_alloc` sets the `p-8` rc header to 1 and zeroes the
    /// payload, so the tag write of 0 is the single-ctor tag.
    fn make_state(acc: f64, n: i64) -> *mut c_void {
        unsafe {
            let p = ailang_rc_alloc(8 + 2 * 8) as *mut u8;
            (p.add(0) as *mut i64).write_unaligned(0);
            (p.add(8) as *mut f64).write_unaligned(acc);
            (p.add(16) as *mut i64).write_unaligned(n);
            p as *mut c_void
        }
    }

    /// `Tick` box, frozen layout: payload `8 + 1*8` = 16 bytes —
    /// tag:i64@0, px:f64@8.
    fn make_tick(px: f64) -> *mut c_void {
        unsafe {
            let p = ailang_rc_alloc(8 + 1 * 8) as *mut u8;
            (p.add(0) as *mut i64).write_unaligned(0);
            (p.add(8) as *mut f64).write_unaligned(px);
            p as *mut c_void
        }
    }

    /// Reads back `(acc, n)` from a `State` payload pointer.
    fn read_state(st: *mut c_void) -> (f64, i64) {
        unsafe {
            let p = st as *mut u8;
            let acc = (p.add(8) as *const f64).read_unaligned();
            let n = (p.add(16) as *const i64).read_unaligned();
            (acc, n)
        }
    }
}

impl Drop for Ctx {
    fn drop(&mut self) {
        unsafe { ailang_ctx_free(self.0) }
    }
}

/// Embeds the M3-frozen `(State, Tick) -> State` kernel and folds a
/// price stream through it. Raw ABI pointers never escape this type
/// (spec §"Error handling": no public way to hand the kernel a raw
/// pointer).
pub struct Kernel {
    ctx: Ctx,
}

impl Kernel {
    pub fn new() -> Self {
        Self { ctx: Ctx::new() }
    }

    /// Folds `prices` via the kernel and returns `(Σ px, count)`.
    /// `own` discipline (DESIGN.md :2346-2352): each `Tick` is freshly
    /// `ailang_rc_alloc`'d and consumed by the kernel; `st` is
    /// consumed and replaced by the return each step; only the final
    /// return is host-`dec`'d.
    pub fn run<I: IntoIterator<Item = f64>>(&self, prices: I) -> (f64, i64) {
        let mut st = Ctx::make_state(0.0, 0);
        for px in prices {
            let tick = Ctx::make_tick(px);
            st = unsafe { backtest_step_tick(self.ctx.0, st, tick) };
        }
        let out = Ctx::read_state(st);
        unsafe { ailang_rc_dec(st) };
        out
    }
}

impl Default for Kernel {
    fn default() -> Self {
        Self::new()
    }
}
```

- [ ] **Step 4: Run the test to verify it passes**

Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices`
Expected: PASS — `test tests::kernel_run_sums_prices ... ok`,
`1 passed; 0 failed`.

---

### Task 3: Hermetic data-server smoke (synthetic Pepperstone fixture)

Integration coverage proving the real `data-server` wiring + ABI
handshake without `/mnt`. Adds no production code (Task 2 already
ships the API); a build+run gate, not a contrived RED.

**Files:**
- Create: `ail-embed/tests/smoke.rs`

- [ ] **Step 1: Write `ail-embed/tests/smoke.rs`**

```rust
//! Hermetic smoke (Testing strategy §1): synthesize a
//! Pepperstone-format `.tick` ZIP, point the REAL `DataServer` at it,
//! fold the stream through the embedding `Kernel`, assert the result
//! equals a host reference fold. Proves "adapter compiles, links,
//! ABI handshake holds" with no `/mnt` dependency. The fixture writer
//! dumps `data_server::records::RawTickRecord` bytes (the crate's own
//! public on-disk type) into a `.bin` ZIP entry — correct by
//! construction, mirroring `data-server/src/loader.rs:110-124`.

use std::io::Write;
use std::sync::Arc;

use ail_embed::Kernel;
use data_server::records::RawTickRecord;
use data_server::DataServer;

/// Writes `recs` as a Pepperstone tick file `<sym>_2017_01.tick`
/// (ZIP holding one packed-`RawTickRecord` `.bin`) into `dir`.
/// Filename satisfies the scanner regex `^(.+)_(\d{4})_(\d{2})\.tick$`
/// (`data-server/src/lib.rs:88`); the `.bin` entry name is arbitrary
/// as long as it ends in `.bin` (`loader.rs:50-55`).
fn write_tick_fixture(dir: &std::path::Path, sym: &str, recs: &[RawTickRecord]) {
    let path = dir.join(format!("{sym}_2017_01.tick"));
    let f = std::fs::File::create(&path).unwrap();
    let mut zip = zip::ZipWriter::new(f);
    zip.start_file::<String, ()>("TICK.bin".into(), Default::default())
        .unwrap();
    for r in recs {
        // SAFETY: RawTickRecord is #[repr(C, packed)], 24 bytes; this
        // is exactly data-server's own test serialisation
        // (loader.rs:117-120).
        let bytes: &[u8] = unsafe {
            std::slice::from_raw_parts(r as *const RawTickRecord as *const u8, 24)
        };
        zip.write_all(bytes).unwrap();
    }
    zip.finish().unwrap();
}

#[test]
fn hermetic_smoke_data_server_roundtrip() {
    // 10 deterministic ticks. mid = (ask+bid)/2; with ask==bid==i+1,
    // mid == i+1. Σ_{i=0..9}(i+1) == 55.0 exactly in f64; n == 10.
    // `time` is any monotonic Delphi-day value (no window ⇒ unused by
    // the fold; only the filename drives symbol/format scanning).
    let recs: Vec<RawTickRecord> = (0..10)
        .map(|i| RawTickRecord {
            time: 42795.0 + i as f64 * 1e-4,
            ask: (i + 1) as f64,
            bid: (i + 1) as f64,
        })
        .collect();

    let dir = tempfile::tempdir().unwrap();
    write_tick_fixture(dir.path(), "TEST", &recs);

    let server = Arc::new(DataServer::new(dir.path()));
    assert!(server.has_symbol("TEST"), "fixture symbol not scanned");

    // Collect the mid-price stream in data-server order.
    let mut it = server.stream_tick("TEST").expect("TEST tick stream");
    let mut prices: Vec<f64> = Vec::new();
    while let Some(chunk) = it.next_chunk() {
        for r in chunk.iter() {
            prices.push((r.ask + r.bid) / 2.0);
        }
    }
    assert_eq!(prices.len(), 10, "all ticks streamed");

    // Host reference fold, same order, same ops ⇒ bit-exact.
    let ref_acc: f64 = prices.iter().copied().fold(0.0, |a, p| a + p);
    let ref_n = prices.len() as i64;

    let (acc, n) = Kernel::new().run(prices.iter().copied());

    assert_eq!(acc, ref_acc, "kernel acc bit-exact vs host reference");
    assert_eq!(acc, 55.0, "deterministic fixture sum");
    assert_eq!(n, ref_n);
    assert_eq!(n, 10);
}
```

- [ ] **Step 2: Run the smoke test**

Run: `cargo test --manifest-path ail-embed/Cargo.toml --test smoke hermetic_smoke_data_server_roundtrip`
Expected: PASS — `test hermetic_smoke_data_server_roundtrip ... ok`,
`1 passed; 0 failed`. (Builds `data-server` as a dev-dependency on
first run.)

- [ ] **Step 3: Run the full crate test suite**

Run: `cargo test --manifest-path ail-embed/Cargo.toml`
Expected: PASS — `kernel_run_sums_prices` (unit) +
`hermetic_smoke_data_server_roundtrip` (integration) both `ok`;
`2 passed; 0 failed` total across the two binaries; `0 failed`.

- [ ] **Step 4: Mechanically verify Invariant 1 / no-language-change**

Run: `git status --porcelain | grep -E '^\?\?|^ ?M' | grep -vE '^.. (ail-embed/|Cargo\.toml|docs/)'`
Expected: empty output — the working tree adds/modifies **only**
`ail-embed/**`, the root `Cargo.toml` comment, and `docs/`. **Zero**
diff to `crates/ailang-*`, `crates/ail/`, `runtime/`,
`examples/*.ail` (the spec's no-language-change + Invariant-1
guarantee, mechanically checked).

---

## Self-review (planner Step 5)

1. **Spec coverage.** Arch §1 (lean core) → Task 2. Concrete code
   shapes: implementation-shape + frozen box layout → Task 2 (box
   helpers cite DESIGN.md:2328-2336); build/link incantation → Task 1
   build.rs (ports embed_tick_e2e.rs:61-78). Testing §1 hermetic
   smoke → Task 3. No-language-change / Invariant-1 acceptance →
   Task 1 (zero-dep lib, dev-only data-server, root-Cargo.toml
   comment) + Task 1 Step 7 (`cargo metadata` graph check) + Task 3
   Step 4 (`git status` path check). Adapter/swarm (Arch §2/§3,
   Testing §2/§3) explicitly deferred per the iteration scope — not
   a coverage gap.
2. **Placeholder scan.** No "TBD/TODO/implement later/similar
   to/add appropriate". The Task 1 stub `src/lib.rs` is a named,
   shown artefact explicitly replaced in Task 2 (not a placeholder).
3. **Type/name consistency.** `Kernel`, `Kernel::new`, `Kernel::run`,
   `Ctx`, `make_state`/`make_tick`/`read_state`, `backtest_step_tick`,
   crate `ail-embed` / import `ail_embed`, test names
   `kernel_run_sums_prices` + `hermetic_smoke_data_server_roundtrip`
   — identical across Tasks 2-3 and the verification filters.
4. **Step granularity.** Each step is one file write or one command;
   all 2-5 min. The largest (Task 2 Step 3) is a single verbatim
   paste.
5. **No commit steps.** None. Work stays in the working tree; the
   Boss commits the iter.
6. **Pin/replacement substring contiguity.** No `contains(...)` /
   grep-of-a-verbatim-body pairing. The only greps (Task 1 Step 7
   `grep -c data-server`; Task 3 Step 4 path grep) assert over
   command output / `git status`, not over a verbatim text body this
   plan also writes — N/A, no contiguity hazard.
7. **Compile-gate vs deferred-caller.** No signature change with
   deferred callers. Task 1's build gate uses a stub lib that
   references no ABI symbol (links cleanly); Task 2 introduces the
   symbol references and its own gate in the same task. No
   cross-task compile-gate contradiction.
8. **Verification filter strings resolve.** `kernel_run_sums_prices`,
   `--test smoke`, `hermetic_smoke_data_server_roundtrip` are all
   defined *by this plan* in the same tasks that filter on them —
   resolve by construction. Task 1 Step 6/7 and Task 3 Step 3 use
   unfiltered builds / explicit-count assertions
   (`2 passed; 0 failed`, `grep -c … == 0`), so "nothing ran" cannot
   masquerade as success.