Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

plan: embedding-abi-m5.1 — lean ail-embed core + build.rs + hermetic smoke

Browse Source 
First iteration of M5 (spec ae905de). 3 tasks, RED-first on the
core: (1) scaffold the workspace-EXCLUDED ail-embed crate +
build.rs (de-risks the no-in-repo-precedent ail-binary resolution
in isolation with a stub lib), (2) the lean embedding core
(Ctx/Kernel/frozen-layout helpers, Rust port of the audited
tick_roundtrip.c:31-75, zero data_server use) with an in-source
data-server-free RED unit test, (3) hermetic data-server smoke
(synthetic Pepperstone-format ZIP fixture via the public
RawTickRecord type → real DataServer → Kernel, bit-exact vs host
reference).

Boss decomposition calls recorded in-plan: data-server is dev-only
(iter-1 library has ZERO deps — Invariant 1 maximally honest);
build.rs resolves ail via AIL_BIN-else-cargo-build-parent-workspace
(separate target dir, no nested-cargo lock); fixture writer mirrors
data-server/src/loader.rs:110-124. Invariant-1/no-language-change
mechanically gated (cargo metadata graph check + git status path
check). Adapter + swarm explicitly deferred to iter 2+.
This commit is contained in:
Brummel
2026-05-19 01:01:09 +02:00
parent ae905de00c
commit 22f02aa26b
1 changed files with 536 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/embedding-abi-m5.1.md
+536
View File
@@ -0,0 +1,536 @@
# embedding-abi-m5.1 — lean `ail-embed` core + build.rs + hermetic smoke — Implementation Plan
> **Parent spec:** `docs/specs/2026-05-19-embedding-abi-m5.md`
>
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
> to run this plan. Steps use `- [ ]` checkboxes for tracking.
**Goal:** Stand up the workspace-excluded `ail-embed` crate: a lean,
data-server-free embedding core (Rust port of the audited
`crates/ail/tests/embed/tick_roundtrip.c`) that links the
M3-frozen staticlib via a `build.rs`, plus a hermetic data-server
smoke test proving the ABI handshake without `/mnt`.
**Architecture:** `ail-embed/` is a sibling crate, *not* an AILang
`[workspace]` member (Invariant 1: the compiler workspace owes
nothing to `data-server`/`/mnt`). Its `build.rs` emits + links
`libembed_backtest_step_tick.a` + `libailang_rt.a` from the
already-shipped kernel. `src/lib.rs` is the core only — `extern "C"`
to the frozen ABI + `Kernel` running a price fold; **zero**
`data_server::` reference. `data-server` is a **dev-dependency** used
solely by the integration smoke test, so the iter-1 library has zero
dependencies. The adapter API + thread-swarm are deliberately
later iterations and are NOT in this plan.
**Tech Stack:** Rust (edition 2021), `std::ffi::c_void`,
`std::process::Command` (build.rs); dev-only: `data-server` (path
`../libs/data-server`), `zip = "2"`, `tempfile = "3"`. Links the
M3 staticlib emitted by the in-repo `ail` CLI.
---
**Files this plan creates or modifies:**
- Create: `ail-embed/Cargo.toml` — manifest for the workspace-excluded crate; zero `[dependencies]`, dev-deps only
- Create: `ail-embed/.gitignore` — ignore `/target` (per-crate, mirrors `../libs/data-server/.gitignore` precedent)
- Create: `ail-embed/build.rs` — resolve `ail`, emit `--emit=staticlib`, emit link directives
- Create: `ail-embed/src/lib.rs` — the lean embedding core: `extern "C"` block, `Ctx`, `Kernel`, frozen-layout box helpers, in-source RED unit test
- Create: `ail-embed/tests/smoke.rs` — hermetic data-server smoke (synthetic Pepperstone-format fixture → real `DataServer` → `Kernel` → assertion)
- Modify: `Cargo.toml:3-10` — add a 2-line comment after the `members` array documenting the deliberate non-membership of `ail-embed/`
Cross-reference anchors (read-only, do not edit):
- `crates/ail/tests/embed/tick_roundtrip.c:31-75` — the C SSOT the Rust core byte-mirrors (extern decls, `make_state`/`make_tick`, fold/read-back, `own` discipline)
- `crates/ail/tests/embed_tick_e2e.rs:61-78` — the `ail build --emit=staticlib -o` + `lib<module>.a`/`libailang_rt.a` link incantation ported into `build.rs`
- `docs/DESIGN.md:2322-2360` — frozen value-layout SSOT (State 24B, Tick 16B; `p-8` rc header; `own`/free contract)
- `../libs/data-server/src/loader.rs:45-101` — ZIP `.bin` read path the fixture writer must satisfy
- `../libs/data-server/src/records.rs:60-66,188-191` — `RawTickRecord` `#[repr(C,packed)]` 24-byte layout + its `size_of==24` pin
- `../libs/data-server/src/lib.rs:88,166-171,236-238` — filename scanner regex, `DataServer::new`, `stream_tick`
---
### Task 1: Scaffold the workspace-excluded `ail-embed` crate + build.rs
Stands up the crate so the novel `build.rs` (no in-repo precedent)
is de-risked *before* any embedding logic: a stub lib that links the
staticlib and compiles proves the emit+link path in isolation.
**Files:**
- Create: `ail-embed/Cargo.toml`
- Create: `ail-embed/.gitignore`
- Create: `ail-embed/build.rs`
- Create: `ail-embed/src/lib.rs` (stub, replaced in Task 2)
- Modify: `Cargo.toml:3-10`
- [ ] **Step 1: Create `ail-embed/Cargo.toml`**
```toml
[package]
name = "ail-embed"
version = "0.0.1"
edition = "2021"
license = "MIT"
publish = false
description = "Lean embedding of an AILang M3-frozen staticlib kernel into a Rust host. Not an AILang [workspace] member (Invariant 1)."
# Iter m5.1: the library itself has ZERO dependencies — the embedding
# core only touches the frozen C ABI. `data-server` is dev-only,
# exercised solely by the hermetic smoke test, so Invariant 1 holds
# in the dependency graph, not just on paper.
[dependencies]
[dev-dependencies]
data-server = { path = "../libs/data-server" }
zip = "2"
tempfile = "3"
```
- [ ] **Step 2: Create `ail-embed/.gitignore`**
```gitignore
/target
```
- [ ] **Step 3: Create `ail-embed/src/lib.rs` as a stub**
```rust
//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. Replaced with the real core in Task 2; this stub
//! exists so Task 1 can prove the build.rs emit+link path alone.
```
- [ ] **Step 4: Create `ail-embed/build.rs`**
```rust
//! Emits the M3-frozen kernel staticlib and links it.
//!
//! Resolution of the `ail` binary has no in-repo precedent (the
//! in-test mechanism `env!("CARGO_BIN_EXE_ail")` is unavailable to a
//! workspace-excluded crate's build script): use `$AIL_BIN` if set,
//! else `cargo build --manifest-path <repo>/Cargo.toml -p ail` and
//! use `<repo>/target/debug/ail`. The AILang workspace target dir
//! (`<repo>/target`) is distinct from this crate's
//! (`ail-embed/target`), so the nested `cargo build` cannot deadlock
//! on the outer build's lock.
use std::env;
use std::path::PathBuf;
use std::process::Command;
fn main() {
let manifest = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap());
let repo = manifest.parent().unwrap().to_path_buf(); // /home/brummel/dev/ailang
let kernel = repo.join("examples/embed_backtest_step_tick.ail");
let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
println!("cargo:rerun-if-changed={}", kernel.display());
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rerun-if-env-changed=AIL_BIN");
let ail_bin: PathBuf = match env::var("AIL_BIN") {
Ok(p) => PathBuf::from(p),
Err(_) => {
let status = Command::new("cargo")
.args(["build", "--manifest-path"])
.arg(repo.join("Cargo.toml"))
.args(["-p", "ail"])
.status()
.expect("spawn `cargo build -p ail`");
assert!(status.success(), "building the `ail` CLI failed");
repo.join("target/debug/ail")
}
};
let build = Command::new(&ail_bin)
.arg("build")
.arg(&kernel)
.args(["--emit=staticlib", "-o"])
.arg(&out_dir)
.output()
.expect("spawn `ail build --emit=staticlib`");
assert!(
build.status.success(),
"ail build --emit=staticlib failed:\n{}",
String::from_utf8_lossy(&build.stderr)
);
println!("cargo:rustc-link-search=native={}", out_dir.display());
println!("cargo:rustc-link-lib=static=embed_backtest_step_tick");
println!("cargo:rustc-link-lib=static=ailang_rt");
}
```
- [ ] **Step 5: Document the deliberate non-membership in the root `Cargo.toml`**
Modify `Cargo.toml` — insert the comment immediately after the
closing `]` of `members` (currently `Cargo.toml:10`). The exact
existing block is:
```toml
members = [
"crates/ailang-core",
"crates/ailang-check",
"crates/ailang-codegen",
"crates/ailang-surface",
"crates/ailang-prose",
"crates/ail",
]
```
Replace it verbatim with:
```toml
members = [
"crates/ailang-core",
"crates/ailang-check",
"crates/ailang-codegen",
"crates/ailang-surface",
"crates/ailang-prose",
"crates/ail",
]
# `ail-embed/` is intentionally NOT a member: it depends on the
# external `../libs/data-server` and is the sole data-server↔AILang
# meeting point (Invariant 1, docs/specs/2026-05-19-embedding-abi-m5.md).
# Adding it here would couple the compiler workspace to a sibling dir.
```
- [ ] **Step 6: Verify the crate builds and the staticlib links**
Run: `cargo build --manifest-path ail-embed/Cargo.toml`
Expected: PASS — exit 0, ending in a `Finished \`dev\` profile`
line. On the first run it additionally builds the `ail` CLI (nested
`cargo build -p ail`, may take a minute); subsequent runs are cached.
No `undefined reference` linker error (the stub lib references no ABI
symbol yet, but `-l static=…` resolves cleanly against the emitted
archives).
- [ ] **Step 7: Verify the AILang compiler workspace is untouched in its graph**
Run: `cargo metadata --format-version 1 --manifest-path Cargo.toml --no-deps | grep -c data-server`
Expected: `0` — the AILang workspace metadata contains no
`data-server` package (Invariant 1 in the dependency graph).
---
### Task 2: The lean embedding core (`Ctx`, `Kernel`, frozen-layout helpers)
Ports `tick_roundtrip.c:31-75` into a safe Rust API. RED-first with
an in-source unit test that needs **no** data-server — proving the
ABI handshake with zero finance knowledge.
**Files:**
- Modify: `ail-embed/src/lib.rs` (replace the Task 1 stub)
- [ ] **Step 1: Write the failing in-source unit test**
Replace the entire contents of `ail-embed/src/lib.rs` with *only*
the test module below (the API it calls does not exist yet — this is
the RED state):
```rust
//! Lean embedding of an AILang M3-frozen staticlib kernel into a
//! Rust host. The Rust analogue of the audited C host
//! `crates/ail/tests/embed/tick_roundtrip.c`: same frozen value
//! layout (DESIGN.md §"Frozen value layout", :2322-2360), same
//! `own`-mode contract. ZERO `data_server`/finance knowledge.
#[cfg(test)]
mod tests {
use super::Kernel;
/// The kernel folds `acc += px; n += 1` over the price stream.
/// Same-order f64 sum ⇒ bit-exact: 3.0 + 4.0 + 5.0 == 12.0, n==3.
#[test]
fn kernel_run_sums_prices() {
let k = Kernel::new();
let (acc, n) = k.run([3.0_f64, 4.0, 5.0]);
assert_eq!(acc, 12.0);
assert_eq!(n, 3);
}
}
```
- [ ] **Step 2: Run the test to verify it fails**
Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices`
Expected: FAIL — compile error `cannot find type \`Kernel\` in this
scope` (or `unresolved import \`super::Kernel\``). The test cannot
run because the API is unimplemented.
- [ ] **Step 3: Write the minimal implementation**
Prepend the following to `ail-embed/src/lib.rs`, *above* the
`#[cfg(test)] mod tests` block (keep the module doc comment at the
top of the file):
```rust
use std::ffi::c_void;
// The M3-frozen C ABI. Symbol names + signatures mirror
// `crates/ail/tests/embed/tick_roundtrip.c:31-36`. `backtest_step_tick`
// is the author-chosen export of `examples/embed_backtest_step_tick.ail`.
unsafe extern "C" {
fn ailang_ctx_new() -> *mut c_void;
fn ailang_ctx_free(c: *mut c_void);
fn ailang_rc_alloc(n: usize) -> *mut c_void;
fn ailang_rc_dec(p: *mut c_void);
fn backtest_step_tick(c: *mut c_void, st: *mut c_void, tick: *mut c_void)
-> *mut c_void;
}
/// A per-thread embedding context (M2 `ailang_ctx_t`). `!Send` by
/// construction (raw pointer) — one ctx per OS thread, as the ABI
/// requires.
struct Ctx(*mut c_void);
impl Ctx {
fn new() -> Self {
Self(unsafe { ailang_ctx_new() })
}
/// `State` box, frozen layout (DESIGN.md :2328-2336):
/// payload `8 + 2*8` = 24 bytes — tag:i64@0, acc:f64@8, n:i64@16.
/// `ailang_rc_alloc` sets the `p-8` rc header to 1 and zeroes the
/// payload, so the tag write of 0 is the single-ctor tag.
fn make_state(acc: f64, n: i64) -> *mut c_void {
unsafe {
let p = ailang_rc_alloc(8 + 2 * 8) as *mut u8;
(p.add(0) as *mut i64).write_unaligned(0);
(p.add(8) as *mut f64).write_unaligned(acc);
(p.add(16) as *mut i64).write_unaligned(n);
p as *mut c_void
}
}
/// `Tick` box, frozen layout: payload `8 + 1*8` = 16 bytes —
/// tag:i64@0, px:f64@8.
fn make_tick(px: f64) -> *mut c_void {
unsafe {
let p = ailang_rc_alloc(8 + 1 * 8) as *mut u8;
(p.add(0) as *mut i64).write_unaligned(0);
(p.add(8) as *mut f64).write_unaligned(px);
p as *mut c_void
}
}
/// Reads back `(acc, n)` from a `State` payload pointer.
fn read_state(st: *mut c_void) -> (f64, i64) {
unsafe {
let p = st as *mut u8;
let acc = (p.add(8) as *const f64).read_unaligned();
let n = (p.add(16) as *const i64).read_unaligned();
(acc, n)
}
}
}
impl Drop for Ctx {
fn drop(&mut self) {
unsafe { ailang_ctx_free(self.0) }
}
}
/// Embeds the M3-frozen `(State, Tick) -> State` kernel and folds a
/// price stream through it. Raw ABI pointers never escape this type
/// (spec §"Error handling": no public way to hand the kernel a raw
/// pointer).
pub struct Kernel {
ctx: Ctx,
}
impl Kernel {
pub fn new() -> Self {
Self { ctx: Ctx::new() }
}
/// Folds `prices` via the kernel and returns `(Σ px, count)`.
/// `own` discipline (DESIGN.md :2346-2352): each `Tick` is freshly
/// `ailang_rc_alloc`'d and consumed by the kernel; `st` is
/// consumed and replaced by the return each step; only the final
/// return is host-`dec`'d.
pub fn run<I: IntoIterator<Item = f64>>(&self, prices: I) -> (f64, i64) {
let mut st = Ctx::make_state(0.0, 0);
for px in prices {
let tick = Ctx::make_tick(px);
st = unsafe { backtest_step_tick(self.ctx.0, st, tick) };
}
let out = Ctx::read_state(st);
unsafe { ailang_rc_dec(st) };
out
}
}
impl Default for Kernel {
fn default() -> Self {
Self::new()
}
}
```
- [ ] **Step 4: Run the test to verify it passes**
Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices`
Expected: PASS — `test tests::kernel_run_sums_prices ... ok`,
`1 passed; 0 failed`.
---
### Task 3: Hermetic data-server smoke (synthetic Pepperstone fixture)
Integration coverage proving the real `data-server` wiring + ABI
handshake without `/mnt`. Adds no production code (Task 2 already
ships the API); a build+run gate, not a contrived RED.
**Files:**
- Create: `ail-embed/tests/smoke.rs`
- [ ] **Step 1: Write `ail-embed/tests/smoke.rs`**
```rust
//! Hermetic smoke (Testing strategy §1): synthesize a
//! Pepperstone-format `.tick` ZIP, point the REAL `DataServer` at it,
//! fold the stream through the embedding `Kernel`, assert the result
//! equals a host reference fold. Proves "adapter compiles, links,
//! ABI handshake holds" with no `/mnt` dependency. The fixture writer
//! dumps `data_server::records::RawTickRecord` bytes (the crate's own
//! public on-disk type) into a `.bin` ZIP entry — correct by
//! construction, mirroring `data-server/src/loader.rs:110-124`.
use std::io::Write;
use std::sync::Arc;
use ail_embed::Kernel;
use data_server::records::RawTickRecord;
use data_server::DataServer;
/// Writes `recs` as a Pepperstone tick file `<sym>_2017_01.tick`
/// (ZIP holding one packed-`RawTickRecord` `.bin`) into `dir`.
/// Filename satisfies the scanner regex `^(.+)_(\d{4})_(\d{2})\.tick$`
/// (`data-server/src/lib.rs:88`); the `.bin` entry name is arbitrary
/// as long as it ends in `.bin` (`loader.rs:50-55`).
fn write_tick_fixture(dir: &std::path::Path, sym: &str, recs: &[RawTickRecord]) {
let path = dir.join(format!("{sym}_2017_01.tick"));
let f = std::fs::File::create(&path).unwrap();
let mut zip = zip::ZipWriter::new(f);
zip.start_file::<String, ()>("TICK.bin".into(), Default::default())
.unwrap();
for r in recs {
// SAFETY: RawTickRecord is #[repr(C, packed)], 24 bytes; this
// is exactly data-server's own test serialisation
// (loader.rs:117-120).
let bytes: &[u8] = unsafe {
std::slice::from_raw_parts(r as *const RawTickRecord as *const u8, 24)
};
zip.write_all(bytes).unwrap();
}
zip.finish().unwrap();
}
#[test]
fn hermetic_smoke_data_server_roundtrip() {
// 10 deterministic ticks. mid = (ask+bid)/2; with ask==bid==i+1,
// mid == i+1. Σ_{i=0..9}(i+1) == 55.0 exactly in f64; n == 10.
// `time` is any monotonic Delphi-day value (no window ⇒ unused by
// the fold; only the filename drives symbol/format scanning).
let recs: Vec<RawTickRecord> = (0..10)
.map(|i| RawTickRecord {
time: 42795.0 + i as f64 * 1e-4,
ask: (i + 1) as f64,
bid: (i + 1) as f64,
})
.collect();
let dir = tempfile::tempdir().unwrap();
write_tick_fixture(dir.path(), "TEST", &recs);
let server = Arc::new(DataServer::new(dir.path()));
assert!(server.has_symbol("TEST"), "fixture symbol not scanned");
// Collect the mid-price stream in data-server order.
let mut it = server.stream_tick("TEST").expect("TEST tick stream");
let mut prices: Vec<f64> = Vec::new();
while let Some(chunk) = it.next_chunk() {
for r in chunk.iter() {
prices.push((r.ask + r.bid) / 2.0);
}
}
assert_eq!(prices.len(), 10, "all ticks streamed");
// Host reference fold, same order, same ops ⇒ bit-exact.
let ref_acc: f64 = prices.iter().copied().fold(0.0, |a, p| a + p);
let ref_n = prices.len() as i64;
let (acc, n) = Kernel::new().run(prices.iter().copied());
assert_eq!(acc, ref_acc, "kernel acc bit-exact vs host reference");
assert_eq!(acc, 55.0, "deterministic fixture sum");
assert_eq!(n, ref_n);
assert_eq!(n, 10);
}
```
- [ ] **Step 2: Run the smoke test**
Run: `cargo test --manifest-path ail-embed/Cargo.toml --test smoke hermetic_smoke_data_server_roundtrip`
Expected: PASS — `test hermetic_smoke_data_server_roundtrip ... ok`,
`1 passed; 0 failed`. (Builds `data-server` as a dev-dependency on
first run.)
- [ ] **Step 3: Run the full crate test suite**
Run: `cargo test --manifest-path ail-embed/Cargo.toml`
Expected: PASS — `kernel_run_sums_prices` (unit) +
`hermetic_smoke_data_server_roundtrip` (integration) both `ok`;
`2 passed; 0 failed` total across the two binaries; `0 failed`.
- [ ] **Step 4: Mechanically verify Invariant 1 / no-language-change**
Run: `git status --porcelain | grep -E '^\?\?|^ ?M' | grep -vE '^.. (ail-embed/|Cargo\.toml|docs/)'`
Expected: empty output — the working tree adds/modifies **only**
`ail-embed/**`, the root `Cargo.toml` comment, and `docs/`. **Zero**
diff to `crates/ailang-*`, `crates/ail/`, `runtime/`,
`examples/*.ail` (the spec's no-language-change + Invariant-1
guarantee, mechanically checked).
---
## Self-review (planner Step 5)
1. **Spec coverage.** Arch §1 (lean core) → Task 2. Concrete code
shapes: implementation-shape + frozen box layout → Task 2 (box
helpers cite DESIGN.md:2328-2336); build/link incantation → Task 1
build.rs (ports embed_tick_e2e.rs:61-78). Testing §1 hermetic
smoke → Task 3. No-language-change / Invariant-1 acceptance →
Task 1 (zero-dep lib, dev-only data-server, root-Cargo.toml
comment) + Task 1 Step 7 (`cargo metadata` graph check) + Task 3
Step 4 (`git status` path check). Adapter/swarm (Arch §2/§3,
Testing §2/§3) explicitly deferred per the iteration scope — not
a coverage gap.
2. **Placeholder scan.** No "TBD/TODO/implement later/similar
to/add appropriate". The Task 1 stub `src/lib.rs` is a named,
shown artefact explicitly replaced in Task 2 (not a placeholder).
3. **Type/name consistency.** `Kernel`, `Kernel::new`, `Kernel::run`,
`Ctx`, `make_state`/`make_tick`/`read_state`, `backtest_step_tick`,
crate `ail-embed` / import `ail_embed`, test names
`kernel_run_sums_prices` + `hermetic_smoke_data_server_roundtrip`
— identical across Tasks 2-3 and the verification filters.
4. **Step granularity.** Each step is one file write or one command;
all 2-5 min. The largest (Task 2 Step 3) is a single verbatim
paste.
5. **No commit steps.** None. Work stays in the working tree; the
Boss commits the iter.
6. **Pin/replacement substring contiguity.** No `contains(...)` /
grep-of-a-verbatim-body pairing. The only greps (Task 1 Step 7
`grep -c data-server`; Task 3 Step 4 path grep) assert over
command output / `git status`, not over a verbatim text body this
plan also writes — N/A, no contiguity hazard.
7. **Compile-gate vs deferred-caller.** No signature change with
deferred callers. Task 1's build gate uses a stub lib that
references no ABI symbol (links cleanly); Task 2 introduces the
symbol references and its own gate in the same task. No
cross-task compile-gate contradiction.
8. **Verification filter strings resolve.** `kernel_run_sums_prices`,
`--test smoke`, `hermetic_smoke_data_server_roundtrip` are all
defined *by this plan* in the same tasks that filter on them —
resolve by construction. Task 1 Step 6/7 and Task 3 Step 3 use
unfiltered builds / explicit-count assertions
(`2 passed; 0 failed`, `grep -c … == 0`), so "nothing ran" cannot
masquerade as success.