diff --git a/docs/plans/embedding-abi-m5.1.md b/docs/plans/embedding-abi-m5.1.md new file mode 100644 index 0000000..24e988d --- /dev/null +++ b/docs/plans/embedding-abi-m5.1.md @@ -0,0 +1,536 @@ +# embedding-abi-m5.1 — lean `ail-embed` core + build.rs + hermetic smoke — Implementation Plan + +> **Parent spec:** `docs/specs/2026-05-19-embedding-abi-m5.md` +> +> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement` +> to run this plan. Steps use `- [ ]` checkboxes for tracking. + +**Goal:** Stand up the workspace-excluded `ail-embed` crate: a lean, +data-server-free embedding core (Rust port of the audited +`crates/ail/tests/embed/tick_roundtrip.c`) that links the +M3-frozen staticlib via a `build.rs`, plus a hermetic data-server +smoke test proving the ABI handshake without `/mnt`. + +**Architecture:** `ail-embed/` is a sibling crate, *not* an AILang +`[workspace]` member (Invariant 1: the compiler workspace owes +nothing to `data-server`/`/mnt`). Its `build.rs` emits + links +`libembed_backtest_step_tick.a` + `libailang_rt.a` from the +already-shipped kernel. `src/lib.rs` is the core only — `extern "C"` +to the frozen ABI + `Kernel` running a price fold; **zero** +`data_server::` reference. `data-server` is a **dev-dependency** used +solely by the integration smoke test, so the iter-1 library has zero +dependencies. The adapter API + thread-swarm are deliberately +later iterations and are NOT in this plan. + +**Tech Stack:** Rust (edition 2021), `std::ffi::c_void`, +`std::process::Command` (build.rs); dev-only: `data-server` (path +`../libs/data-server`), `zip = "2"`, `tempfile = "3"`. Links the +M3 staticlib emitted by the in-repo `ail` CLI. + +--- + +**Files this plan creates or modifies:** + +- Create: `ail-embed/Cargo.toml` — manifest for the workspace-excluded crate; zero `[dependencies]`, dev-deps only +- Create: `ail-embed/.gitignore` — ignore `/target` (per-crate, mirrors `../libs/data-server/.gitignore` precedent) +- Create: `ail-embed/build.rs` — resolve `ail`, emit `--emit=staticlib`, emit link directives +- Create: `ail-embed/src/lib.rs` — the lean embedding core: `extern "C"` block, `Ctx`, `Kernel`, frozen-layout box helpers, in-source RED unit test +- Create: `ail-embed/tests/smoke.rs` — hermetic data-server smoke (synthetic Pepperstone-format fixture → real `DataServer` → `Kernel` → assertion) +- Modify: `Cargo.toml:3-10` — add a 2-line comment after the `members` array documenting the deliberate non-membership of `ail-embed/` + +Cross-reference anchors (read-only, do not edit): +- `crates/ail/tests/embed/tick_roundtrip.c:31-75` — the C SSOT the Rust core byte-mirrors (extern decls, `make_state`/`make_tick`, fold/read-back, `own` discipline) +- `crates/ail/tests/embed_tick_e2e.rs:61-78` — the `ail build --emit=staticlib -o` + `lib.a`/`libailang_rt.a` link incantation ported into `build.rs` +- `docs/DESIGN.md:2322-2360` — frozen value-layout SSOT (State 24B, Tick 16B; `p-8` rc header; `own`/free contract) +- `../libs/data-server/src/loader.rs:45-101` — ZIP `.bin` read path the fixture writer must satisfy +- `../libs/data-server/src/records.rs:60-66,188-191` — `RawTickRecord` `#[repr(C,packed)]` 24-byte layout + its `size_of==24` pin +- `../libs/data-server/src/lib.rs:88,166-171,236-238` — filename scanner regex, `DataServer::new`, `stream_tick` + +--- + +### Task 1: Scaffold the workspace-excluded `ail-embed` crate + build.rs + +Stands up the crate so the novel `build.rs` (no in-repo precedent) +is de-risked *before* any embedding logic: a stub lib that links the +staticlib and compiles proves the emit+link path in isolation. + +**Files:** +- Create: `ail-embed/Cargo.toml` +- Create: `ail-embed/.gitignore` +- Create: `ail-embed/build.rs` +- Create: `ail-embed/src/lib.rs` (stub, replaced in Task 2) +- Modify: `Cargo.toml:3-10` + +- [ ] **Step 1: Create `ail-embed/Cargo.toml`** + +```toml +[package] +name = "ail-embed" +version = "0.0.1" +edition = "2021" +license = "MIT" +publish = false +description = "Lean embedding of an AILang M3-frozen staticlib kernel into a Rust host. Not an AILang [workspace] member (Invariant 1)." + +# Iter m5.1: the library itself has ZERO dependencies — the embedding +# core only touches the frozen C ABI. `data-server` is dev-only, +# exercised solely by the hermetic smoke test, so Invariant 1 holds +# in the dependency graph, not just on paper. +[dependencies] + +[dev-dependencies] +data-server = { path = "../libs/data-server" } +zip = "2" +tempfile = "3" +``` + +- [ ] **Step 2: Create `ail-embed/.gitignore`** + +```gitignore +/target +``` + +- [ ] **Step 3: Create `ail-embed/src/lib.rs` as a stub** + +```rust +//! Lean embedding of an AILang M3-frozen staticlib kernel into a +//! Rust host. Replaced with the real core in Task 2; this stub +//! exists so Task 1 can prove the build.rs emit+link path alone. +``` + +- [ ] **Step 4: Create `ail-embed/build.rs`** + +```rust +//! Emits the M3-frozen kernel staticlib and links it. +//! +//! Resolution of the `ail` binary has no in-repo precedent (the +//! in-test mechanism `env!("CARGO_BIN_EXE_ail")` is unavailable to a +//! workspace-excluded crate's build script): use `$AIL_BIN` if set, +//! else `cargo build --manifest-path /Cargo.toml -p ail` and +//! use `/target/debug/ail`. The AILang workspace target dir +//! (`/target`) is distinct from this crate's +//! (`ail-embed/target`), so the nested `cargo build` cannot deadlock +//! on the outer build's lock. +use std::env; +use std::path::PathBuf; +use std::process::Command; + +fn main() { + let manifest = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap()); + let repo = manifest.parent().unwrap().to_path_buf(); // /home/brummel/dev/ailang + let kernel = repo.join("examples/embed_backtest_step_tick.ail"); + let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); + + println!("cargo:rerun-if-changed={}", kernel.display()); + println!("cargo:rerun-if-changed=build.rs"); + println!("cargo:rerun-if-env-changed=AIL_BIN"); + + let ail_bin: PathBuf = match env::var("AIL_BIN") { + Ok(p) => PathBuf::from(p), + Err(_) => { + let status = Command::new("cargo") + .args(["build", "--manifest-path"]) + .arg(repo.join("Cargo.toml")) + .args(["-p", "ail"]) + .status() + .expect("spawn `cargo build -p ail`"); + assert!(status.success(), "building the `ail` CLI failed"); + repo.join("target/debug/ail") + } + }; + + let build = Command::new(&ail_bin) + .arg("build") + .arg(&kernel) + .args(["--emit=staticlib", "-o"]) + .arg(&out_dir) + .output() + .expect("spawn `ail build --emit=staticlib`"); + assert!( + build.status.success(), + "ail build --emit=staticlib failed:\n{}", + String::from_utf8_lossy(&build.stderr) + ); + + println!("cargo:rustc-link-search=native={}", out_dir.display()); + println!("cargo:rustc-link-lib=static=embed_backtest_step_tick"); + println!("cargo:rustc-link-lib=static=ailang_rt"); +} +``` + +- [ ] **Step 5: Document the deliberate non-membership in the root `Cargo.toml`** + +Modify `Cargo.toml` — insert the comment immediately after the +closing `]` of `members` (currently `Cargo.toml:10`). The exact +existing block is: + +```toml +members = [ + "crates/ailang-core", + "crates/ailang-check", + "crates/ailang-codegen", + "crates/ailang-surface", + "crates/ailang-prose", + "crates/ail", +] +``` + +Replace it verbatim with: + +```toml +members = [ + "crates/ailang-core", + "crates/ailang-check", + "crates/ailang-codegen", + "crates/ailang-surface", + "crates/ailang-prose", + "crates/ail", +] +# `ail-embed/` is intentionally NOT a member: it depends on the +# external `../libs/data-server` and is the sole data-server↔AILang +# meeting point (Invariant 1, docs/specs/2026-05-19-embedding-abi-m5.md). +# Adding it here would couple the compiler workspace to a sibling dir. +``` + +- [ ] **Step 6: Verify the crate builds and the staticlib links** + +Run: `cargo build --manifest-path ail-embed/Cargo.toml` +Expected: PASS — exit 0, ending in a `Finished \`dev\` profile` +line. On the first run it additionally builds the `ail` CLI (nested +`cargo build -p ail`, may take a minute); subsequent runs are cached. +No `undefined reference` linker error (the stub lib references no ABI +symbol yet, but `-l static=…` resolves cleanly against the emitted +archives). + +- [ ] **Step 7: Verify the AILang compiler workspace is untouched in its graph** + +Run: `cargo metadata --format-version 1 --manifest-path Cargo.toml --no-deps | grep -c data-server` +Expected: `0` — the AILang workspace metadata contains no +`data-server` package (Invariant 1 in the dependency graph). + +--- + +### Task 2: The lean embedding core (`Ctx`, `Kernel`, frozen-layout helpers) + +Ports `tick_roundtrip.c:31-75` into a safe Rust API. RED-first with +an in-source unit test that needs **no** data-server — proving the +ABI handshake with zero finance knowledge. + +**Files:** +- Modify: `ail-embed/src/lib.rs` (replace the Task 1 stub) + +- [ ] **Step 1: Write the failing in-source unit test** + +Replace the entire contents of `ail-embed/src/lib.rs` with *only* +the test module below (the API it calls does not exist yet — this is +the RED state): + +```rust +//! Lean embedding of an AILang M3-frozen staticlib kernel into a +//! Rust host. The Rust analogue of the audited C host +//! `crates/ail/tests/embed/tick_roundtrip.c`: same frozen value +//! layout (DESIGN.md §"Frozen value layout", :2322-2360), same +//! `own`-mode contract. ZERO `data_server`/finance knowledge. + +#[cfg(test)] +mod tests { + use super::Kernel; + + /// The kernel folds `acc += px; n += 1` over the price stream. + /// Same-order f64 sum ⇒ bit-exact: 3.0 + 4.0 + 5.0 == 12.0, n==3. + #[test] + fn kernel_run_sums_prices() { + let k = Kernel::new(); + let (acc, n) = k.run([3.0_f64, 4.0, 5.0]); + assert_eq!(acc, 12.0); + assert_eq!(n, 3); + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices` +Expected: FAIL — compile error `cannot find type \`Kernel\` in this +scope` (or `unresolved import \`super::Kernel\``). The test cannot +run because the API is unimplemented. + +- [ ] **Step 3: Write the minimal implementation** + +Prepend the following to `ail-embed/src/lib.rs`, *above* the +`#[cfg(test)] mod tests` block (keep the module doc comment at the +top of the file): + +```rust +use std::ffi::c_void; + +// The M3-frozen C ABI. Symbol names + signatures mirror +// `crates/ail/tests/embed/tick_roundtrip.c:31-36`. `backtest_step_tick` +// is the author-chosen export of `examples/embed_backtest_step_tick.ail`. +unsafe extern "C" { + fn ailang_ctx_new() -> *mut c_void; + fn ailang_ctx_free(c: *mut c_void); + fn ailang_rc_alloc(n: usize) -> *mut c_void; + fn ailang_rc_dec(p: *mut c_void); + fn backtest_step_tick(c: *mut c_void, st: *mut c_void, tick: *mut c_void) + -> *mut c_void; +} + +/// A per-thread embedding context (M2 `ailang_ctx_t`). `!Send` by +/// construction (raw pointer) — one ctx per OS thread, as the ABI +/// requires. +struct Ctx(*mut c_void); + +impl Ctx { + fn new() -> Self { + Self(unsafe { ailang_ctx_new() }) + } + + /// `State` box, frozen layout (DESIGN.md :2328-2336): + /// payload `8 + 2*8` = 24 bytes — tag:i64@0, acc:f64@8, n:i64@16. + /// `ailang_rc_alloc` sets the `p-8` rc header to 1 and zeroes the + /// payload, so the tag write of 0 is the single-ctor tag. + fn make_state(acc: f64, n: i64) -> *mut c_void { + unsafe { + let p = ailang_rc_alloc(8 + 2 * 8) as *mut u8; + (p.add(0) as *mut i64).write_unaligned(0); + (p.add(8) as *mut f64).write_unaligned(acc); + (p.add(16) as *mut i64).write_unaligned(n); + p as *mut c_void + } + } + + /// `Tick` box, frozen layout: payload `8 + 1*8` = 16 bytes — + /// tag:i64@0, px:f64@8. + fn make_tick(px: f64) -> *mut c_void { + unsafe { + let p = ailang_rc_alloc(8 + 1 * 8) as *mut u8; + (p.add(0) as *mut i64).write_unaligned(0); + (p.add(8) as *mut f64).write_unaligned(px); + p as *mut c_void + } + } + + /// Reads back `(acc, n)` from a `State` payload pointer. + fn read_state(st: *mut c_void) -> (f64, i64) { + unsafe { + let p = st as *mut u8; + let acc = (p.add(8) as *const f64).read_unaligned(); + let n = (p.add(16) as *const i64).read_unaligned(); + (acc, n) + } + } +} + +impl Drop for Ctx { + fn drop(&mut self) { + unsafe { ailang_ctx_free(self.0) } + } +} + +/// Embeds the M3-frozen `(State, Tick) -> State` kernel and folds a +/// price stream through it. Raw ABI pointers never escape this type +/// (spec §"Error handling": no public way to hand the kernel a raw +/// pointer). +pub struct Kernel { + ctx: Ctx, +} + +impl Kernel { + pub fn new() -> Self { + Self { ctx: Ctx::new() } + } + + /// Folds `prices` via the kernel and returns `(Σ px, count)`. + /// `own` discipline (DESIGN.md :2346-2352): each `Tick` is freshly + /// `ailang_rc_alloc`'d and consumed by the kernel; `st` is + /// consumed and replaced by the return each step; only the final + /// return is host-`dec`'d. + pub fn run>(&self, prices: I) -> (f64, i64) { + let mut st = Ctx::make_state(0.0, 0); + for px in prices { + let tick = Ctx::make_tick(px); + st = unsafe { backtest_step_tick(self.ctx.0, st, tick) }; + } + let out = Ctx::read_state(st); + unsafe { ailang_rc_dec(st) }; + out + } +} + +impl Default for Kernel { + fn default() -> Self { + Self::new() + } +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cargo test --manifest-path ail-embed/Cargo.toml kernel_run_sums_prices` +Expected: PASS — `test tests::kernel_run_sums_prices ... ok`, +`1 passed; 0 failed`. + +--- + +### Task 3: Hermetic data-server smoke (synthetic Pepperstone fixture) + +Integration coverage proving the real `data-server` wiring + ABI +handshake without `/mnt`. Adds no production code (Task 2 already +ships the API); a build+run gate, not a contrived RED. + +**Files:** +- Create: `ail-embed/tests/smoke.rs` + +- [ ] **Step 1: Write `ail-embed/tests/smoke.rs`** + +```rust +//! Hermetic smoke (Testing strategy §1): synthesize a +//! Pepperstone-format `.tick` ZIP, point the REAL `DataServer` at it, +//! fold the stream through the embedding `Kernel`, assert the result +//! equals a host reference fold. Proves "adapter compiles, links, +//! ABI handshake holds" with no `/mnt` dependency. The fixture writer +//! dumps `data_server::records::RawTickRecord` bytes (the crate's own +//! public on-disk type) into a `.bin` ZIP entry — correct by +//! construction, mirroring `data-server/src/loader.rs:110-124`. + +use std::io::Write; +use std::sync::Arc; + +use ail_embed::Kernel; +use data_server::records::RawTickRecord; +use data_server::DataServer; + +/// Writes `recs` as a Pepperstone tick file `_2017_01.tick` +/// (ZIP holding one packed-`RawTickRecord` `.bin`) into `dir`. +/// Filename satisfies the scanner regex `^(.+)_(\d{4})_(\d{2})\.tick$` +/// (`data-server/src/lib.rs:88`); the `.bin` entry name is arbitrary +/// as long as it ends in `.bin` (`loader.rs:50-55`). +fn write_tick_fixture(dir: &std::path::Path, sym: &str, recs: &[RawTickRecord]) { + let path = dir.join(format!("{sym}_2017_01.tick")); + let f = std::fs::File::create(&path).unwrap(); + let mut zip = zip::ZipWriter::new(f); + zip.start_file::("TICK.bin".into(), Default::default()) + .unwrap(); + for r in recs { + // SAFETY: RawTickRecord is #[repr(C, packed)], 24 bytes; this + // is exactly data-server's own test serialisation + // (loader.rs:117-120). + let bytes: &[u8] = unsafe { + std::slice::from_raw_parts(r as *const RawTickRecord as *const u8, 24) + }; + zip.write_all(bytes).unwrap(); + } + zip.finish().unwrap(); +} + +#[test] +fn hermetic_smoke_data_server_roundtrip() { + // 10 deterministic ticks. mid = (ask+bid)/2; with ask==bid==i+1, + // mid == i+1. Σ_{i=0..9}(i+1) == 55.0 exactly in f64; n == 10. + // `time` is any monotonic Delphi-day value (no window ⇒ unused by + // the fold; only the filename drives symbol/format scanning). + let recs: Vec = (0..10) + .map(|i| RawTickRecord { + time: 42795.0 + i as f64 * 1e-4, + ask: (i + 1) as f64, + bid: (i + 1) as f64, + }) + .collect(); + + let dir = tempfile::tempdir().unwrap(); + write_tick_fixture(dir.path(), "TEST", &recs); + + let server = Arc::new(DataServer::new(dir.path())); + assert!(server.has_symbol("TEST"), "fixture symbol not scanned"); + + // Collect the mid-price stream in data-server order. + let mut it = server.stream_tick("TEST").expect("TEST tick stream"); + let mut prices: Vec = Vec::new(); + while let Some(chunk) = it.next_chunk() { + for r in chunk.iter() { + prices.push((r.ask + r.bid) / 2.0); + } + } + assert_eq!(prices.len(), 10, "all ticks streamed"); + + // Host reference fold, same order, same ops ⇒ bit-exact. + let ref_acc: f64 = prices.iter().copied().fold(0.0, |a, p| a + p); + let ref_n = prices.len() as i64; + + let (acc, n) = Kernel::new().run(prices.iter().copied()); + + assert_eq!(acc, ref_acc, "kernel acc bit-exact vs host reference"); + assert_eq!(acc, 55.0, "deterministic fixture sum"); + assert_eq!(n, ref_n); + assert_eq!(n, 10); +} +``` + +- [ ] **Step 2: Run the smoke test** + +Run: `cargo test --manifest-path ail-embed/Cargo.toml --test smoke hermetic_smoke_data_server_roundtrip` +Expected: PASS — `test hermetic_smoke_data_server_roundtrip ... ok`, +`1 passed; 0 failed`. (Builds `data-server` as a dev-dependency on +first run.) + +- [ ] **Step 3: Run the full crate test suite** + +Run: `cargo test --manifest-path ail-embed/Cargo.toml` +Expected: PASS — `kernel_run_sums_prices` (unit) + +`hermetic_smoke_data_server_roundtrip` (integration) both `ok`; +`2 passed; 0 failed` total across the two binaries; `0 failed`. + +- [ ] **Step 4: Mechanically verify Invariant 1 / no-language-change** + +Run: `git status --porcelain | grep -E '^\?\?|^ ?M' | grep -vE '^.. (ail-embed/|Cargo\.toml|docs/)'` +Expected: empty output — the working tree adds/modifies **only** +`ail-embed/**`, the root `Cargo.toml` comment, and `docs/`. **Zero** +diff to `crates/ailang-*`, `crates/ail/`, `runtime/`, +`examples/*.ail` (the spec's no-language-change + Invariant-1 +guarantee, mechanically checked). + +--- + +## Self-review (planner Step 5) + +1. **Spec coverage.** Arch §1 (lean core) → Task 2. Concrete code + shapes: implementation-shape + frozen box layout → Task 2 (box + helpers cite DESIGN.md:2328-2336); build/link incantation → Task 1 + build.rs (ports embed_tick_e2e.rs:61-78). Testing §1 hermetic + smoke → Task 3. No-language-change / Invariant-1 acceptance → + Task 1 (zero-dep lib, dev-only data-server, root-Cargo.toml + comment) + Task 1 Step 7 (`cargo metadata` graph check) + Task 3 + Step 4 (`git status` path check). Adapter/swarm (Arch §2/§3, + Testing §2/§3) explicitly deferred per the iteration scope — not + a coverage gap. +2. **Placeholder scan.** No "TBD/TODO/implement later/similar + to/add appropriate". The Task 1 stub `src/lib.rs` is a named, + shown artefact explicitly replaced in Task 2 (not a placeholder). +3. **Type/name consistency.** `Kernel`, `Kernel::new`, `Kernel::run`, + `Ctx`, `make_state`/`make_tick`/`read_state`, `backtest_step_tick`, + crate `ail-embed` / import `ail_embed`, test names + `kernel_run_sums_prices` + `hermetic_smoke_data_server_roundtrip` + — identical across Tasks 2-3 and the verification filters. +4. **Step granularity.** Each step is one file write or one command; + all 2-5 min. The largest (Task 2 Step 3) is a single verbatim + paste. +5. **No commit steps.** None. Work stays in the working tree; the + Boss commits the iter. +6. **Pin/replacement substring contiguity.** No `contains(...)` / + grep-of-a-verbatim-body pairing. The only greps (Task 1 Step 7 + `grep -c data-server`; Task 3 Step 4 path grep) assert over + command output / `git status`, not over a verbatim text body this + plan also writes — N/A, no contiguity hazard. +7. **Compile-gate vs deferred-caller.** No signature change with + deferred callers. Task 1's build gate uses a stub lib that + references no ABI symbol (links cleanly); Task 2 introduces the + symbol references and its own gate in the same task. No + cross-task compile-gate contradiction. +8. **Verification filter strings resolve.** `kernel_run_sums_prices`, + `--test smoke`, `hermetic_smoke_data_server_roundtrip` are all + defined *by this plan* in the same tasks that filter on them — + resolve by construction. Task 1 Step 6/7 and Task 3 Step 3 use + unfiltered builds / explicit-count assertions + (`2 passed; 0 failed`, `grep -c … == 0`), so "nothing ran" cannot + masquerade as success.