iter embedding-abi-m5.1 (DONE 3/3): lean ail-embed core + build.rs + hermetic smoke

M5 iteration 1 (spec ae905de, plan 22f02aa). Stands up the
workspace-excluded `ail-embed` crate:

- zero-dependency embedding core (`ail-embed/src/lib.rs`): extern "C"
  to the M3-frozen ABI + frozen-layout State/Tick box helpers + a
  Kernel price fold; the Rust port of the audited
  crates/ail/tests/embed/tick_roundtrip.c. Raw pointers never escape
  the type.
- build.rs (no in-repo precedent): AIL_BIN env override else nested
  `cargo build -p ail` against the parent workspace (separate target
  dir → no cargo-lock deadlock), `ail build --emit=staticlib`, link
  directives.
- hermetic data-server smoke (ail-embed/tests/smoke.rs): synthetic
  Pepperstone-format ZIP fixture via data-server's own public
  RawTickRecord type → real DataServer → Kernel, bit-exact vs a
  same-order host reference fold; runs with no /mnt.
- `ail-embed` is its own cargo workspace root (empty [workspace]
  table); data-server is a dev-dependency only. Root Cargo.toml gains
  only a 4-line non-membership comment.

Invariant 1 Boss-verified independently: full+no-deps cargo metadata
on the AILang workspace shows data-server count 0; git status path
filter empty (zero diff to crates/ailang-*, crates/ail/, runtime/,
examples/*.ail); src/lib.rs zero code-level data_server; AILang
cargo build --workspace still clean. ail-embed suite 2/2 green
(kernel_run_sums_prices unit RED-first + hermetic_smoke integration),
verified by me, not just the agent report.

Two toolchain-forced corrections to the plan's verbatim
ail-embed/Cargo.toml (added empty [workspace] table; sibling dev-dep
path ../libs -> ../../libs, manifest-relative) — confined to the
plan-created manifest, no acceptance gate altered, 0 review re-loops.
Journal Concerns records the planner-recon implication for the next
workspace-excluded-nested-crate plan. Adapter API + thread-swarm
deferred to M5 iter 2+ per spec/plan.

Includes the per-iter journal, stats, and the INDEX.md line.

docs/journals/2026-05-19-iter-embedding-abi-m5.1.md

@@ -0,0 +1,112 @@
+# iter embedding-abi-m5.1 — lean `ail-embed` core + build.rs + hermetic smoke
+
+**Date:** 2026-05-19
+**Started from:** 22f02aa26b57c478760265dfc6468b54d35c7cdc
+**Status:** DONE
+**Tasks completed:** 3 of 3
+
+## Summary
+
+Stood up the workspace-excluded `ail-embed` crate: a zero-dependency
+embedding core that links the M3-frozen staticlib via a `build.rs`
+and folds a price stream through the frozen `(State, Tick) -> State`
+C ABI, plus a hermetic `data-server` smoke test proving the real
+adapter wiring without `/mnt`. The crate is its own cargo workspace
+root (not an AILang `[workspace]` member); `data-server` is a
+dev-dependency only, so Invariant 1 holds in the dependency graph,
+mechanically verified by both the `cargo metadata` grep (Task 1
+Step 7 → `0`) and the `git status` path filter (Task 3 Step 4 →
+empty). Zero diff to `crates/ailang-*`, `crates/ail/`, `runtime/`,
+`examples/*.ail`. Both ratifying tests are green
+(`kernel_run_sums_prices` unit + `hermetic_smoke_data_server_roundtrip`
+integration; 2/0). Two minimal, toolchain-forced corrections to the
+plan's verbatim `ail-embed/Cargo.toml` were required (see Concerns) —
+neither alters the iter's intent or any acceptance gate.
+
+## Per-task notes
+
+- iter embedding-abi-m5.1.1: Scaffold the workspace-excluded crate +
+  build.rs. Created `ail-embed/{Cargo.toml,.gitignore,src/lib.rs
+  (stub),build.rs}`; appended the deliberate-non-membership comment
+  to the root `Cargo.toml` members block (verbatim). Build gate
+  PASS, `cargo metadata … grep -c data-server` == 0. RED-first N/A
+  (build-infra de-risking task; verification is the build+metadata
+  gate, which the plan scripts).
+- iter embedding-abi-m5.1.2: The lean embedding core. RED-first
+  honored: wrote the test-only `src/lib.rs` first, observed RED
+  (`unresolved import \`super::Kernel\``, exactly the plan's
+  expected reason), then prepended the verbatim impl block
+  (`extern "C"` ABI, `Ctx` + frozen-layout box helpers + `Drop`,
+  `Kernel::new`/`run`, `Default`). GREEN: `kernel_run_sums_prices`
+  1/0.
+- iter embedding-abi-m5.1.3: Hermetic data-server smoke. Created
+  `ail-embed/tests/smoke.rs` verbatim (synthetic Pepperstone ZIP
+  fixture → real `DataServer` → `Kernel` fold → bit-exact assertion
+  vs host reference and against the closed-form `55.0`). Test-only
+  task (no separate RED). Full suite 2/0; Invariant-1 path filter
+  empty.
+
+## Concerns
+
+- iter .1 (DONE_WITH_CONCERNS): the plan's verbatim
+  `ail-embed/Cargo.toml` could not satisfy its own Step 6 build gate
+  as written. Two minimal toolchain-forced corrections, both confined
+  to the in-scope plan-created `ail-embed/Cargo.toml`, neither
+  changing iter intent or any acceptance gate:
+  1. Added an empty `[workspace]` table. Cargo hard-errors on a
+     nested package that "believes it's in a workspace when it's
+     not"; the empty table is cargo's documented mechanism for a
+     deliberately workspace-excluded nested crate (and is what makes
+     the Step 7 metadata grep return `0` — the AILang root manifest
+     no longer traverses into `ail-embed`). A comment at the table
+     records the rationale.
+  2. Corrected the `data-server` dev-dep path from
+     `../libs/data-server` to `../../libs/data-server`. The crate
+     lives at `/home/brummel/dev/libs/data-server` — a sibling of
+     the repo *directory*, one level above the repo root. From
+     `ail-embed/Cargo.toml`, `../` is the repo root, so the plan's
+     literal resolved to the non-existent
+     `repo/libs/data-server`. The plan's read-only cross-reference
+     anchors (`../libs/data-server/...`) are written from the
+     repo-root cwd and are correct *there*; only the manifest path
+     literal, resolved relative to `ail-embed/`, needed the extra
+     `../`. `build.rs` is unaffected (it never references
+     data-server; it computes `repo = manifest.parent()`).
+
+  Plan-template implication for the orchestrator: planner path-recon
+  for a workspace-excluded nested crate should (a) emit the empty
+  `[workspace]` table in the manifest template and (b) resolve
+  sibling-dependency paths relative to the *manifest's* directory,
+  not the repo-root cwd the anchors are written from.
+
+## Known debt
+
+- No `examples/*.ail.json` + `crates/ail/tests/e2e.rs` E2E fixture
+  added. Deliberate, not a gap: this iter ships zero compiler-surface
+  change (mechanically enforced by Task 3 Step 4 — adding a fixture
+  under `examples/` or `crates/ail/tests/` would itself break the
+  iter's Invariant-1 acceptance). The milestone invariant
+  ("M3-frozen staticlib embeds + folds bit-exactly via the C ABI;
+  data-server is a dev-only meeting point") is already protected by
+  the two tests this iter ships. Standard E2E protects compiler
+  invariants; there is none to protect here.
+- Adapter API + thread-swarm are explicitly deferred to embedding-abi
+  iter 2+ per the plan and spec — not touched, not debt of this iter.
+
+## Blocked detail
+
+N/A — DONE.
+
+## Files touched
+
+- Modified: `Cargo.toml` (root — 4-line non-membership comment after
+  the `members` array; no other change)
+- Created: `ail-embed/Cargo.toml`, `ail-embed/.gitignore`,
+  `ail-embed/build.rs`, `ail-embed/src/lib.rs`,
+  `ail-embed/tests/smoke.rs`
+- Untracked build artefacts under `ail-embed/target/` are
+  `.gitignore`d by `ail-embed/.gitignore`.
+
+## Stats
+
+bench/orchestrator-stats/2026-05-19-iter-embedding-abi-m5.1.json

docs/journals/INDEX.md

@@ -106,3 +106,4 @@
 - 2026-05-18 — iter embedding-abi-m3.tidy (M3 audit [medium]+[low] doc-honesty fix, DONE 3/3, pin-safe): closed the two DRIFT items the M3 milestone-close audit routed here (M2.tidy `[medium]+[low] doc-honesty → tidy` precedent). [medium] docs/DESIGN.md §"Embedding ABI" — surgically replaced ONLY the contradicted M1-era parenthetical "(modes apply only to heap-shaped types, which the scalar-only rule above forbids at an export boundary anyway)" with the present-tense truth "(a single-constructor record export parameter, by contrast, carries `own`/`borrow` — the ownership contract the frozen value layout below specifies)"; the parenthetical shared physical line :2300 with the docs_honesty_pin.rs:135 pinned bare-scalar sentence ("Export parameters are written **bare**: a scalar type carries no `own`/`borrow` mode", norm()-whitespace-collapsed, fn form_a_scalar_param_carveout_present_and_old_rule_absent) — the edit kept every pinned word (line :2299 + the `own`/`borrow` mode` continuation untouched), the planner Step-5 item-6 presence-pin-vs-verbatim-edit collision the M2.tidy precedent ran. [low] crates/ailang-codegen/src/lib.rs:608-610 — comment-only honesty fix ("gate guarantees Int/Float; map Int→i64,Float→double" → "...Int/Float or a single-constructor record of those (M3); map Int→i64, Float→double, a record → ptr"); tree-wide-grep-confirmed no test pins the comment text; the byte-pin (embed_record_layout_pin) + forwarder-IR pin (embed_staticlib_lowering) assert generated IR not source comments → byte-identical before/after (the guard that no codegen moved). Boss-verified independently: both stale fragments grep-ABSENT; the 4 standing pins green at the exact recon baseline (docs_honesty_pin 5/0 ⇒ pin-safety held, design_schema_drift 8/0, embed_record_layout_pin 1/0, embed_staticlib_lowering 3/0); workspace 639/77 byte-unchanged from the M3-DONE baseline (docs/comment tidy, zero behaviour/test delta); diff exactly 2 files (DESIGN.md 5±, codegen/src/lib.rs 7±). No language/checker/codegen behaviour change; no audit/fieldtest gate (the 4 pins + 639/77 ARE the regression coverage, M2.tidy precedent). One non-gating planner-quality defect recorded in Concerns: Task-3 Step-1's `guarantees every param` verification grep is a substring of the plan's own Task-2 replacement text (non-discriminating) — the orchestrator correctly verified the substantive intent via discriminating fragments instead of bending code (same family as planner Step-5 item-8). bench: already carry-on / NO ratify at the M3 audit (causally exonerated by byte-identical generated IR; this tidy touches no executable path). audit source docs/journals/2026-05-18-audit-embedding-abi-m3.md (b8a60b1) → plan docs/plans/embedding-abi-m3.tidy.md (44ced51) → iter this commit. M3 milestone substantively closed + sound; roadmap [~]→[x] follows. → 2026-05-18-iter-embedding-abi-m3.tidy.md
 - 2026-05-18 — brainstorm embedding-abi-m4 → RETIRED, never speced (premise collapsed under its own feature-acceptance gate during Step-2/3 Q&A; no spec, no grounding-check, no planner handoff — the "problem mis-framed → don't ratify a known-unneeded shape" brainstorm path): `/boss` picked top-P0 "Embedding ABI — M4: sequence crossing via `List`"; user green-lit a continue-here brainstorm; recon (`ailang-plan-recon`) returned a full fact sheet; two user forks resolved in Q&A (own-only `List` param; `List Record`-only element) and Approach A (structural list-shaped `is_c_abi_type` arm, no name-match; B name-anchored / C `std_list`-SSOT-first rejected on language/scope grounds) recommended — all now moot. Struck on **feature-acceptance clause 2**: the shipped M3 gate `is_c_abi_type` (`crates/ailang-check/src/lib.rs:1934-1953`) is a per-parameter loop accepting a C scalar OR a single-ctor all-scalar record *independently per param*, and the forwarder's `llvm_scalar` maps every non-scalar `Type::Con`→`ptr` (M3-frozen), so `(State, Tick) -> State` (both single-ctor all-scalar records) is **already gate-accepted + forwarder-supported today** — the minimal data-server binding is M3 (shipped) + a host-side per-tick loop; cons-list crossing would *add* a 2N+1-box-per-chunk host builder + the deferred flat-array perf debt and removes no redundancy, with no named consumer (M5's adapter unrolls each chunk host-side — a clean adapter, the sole data-server↔AILang meeting point per Invariant 1; whole-chunk in-kernel visibility is semantically void since `State` threads across calls regardless of chunk boundaries). Honest mid-Q&A correction recorded: I had asserted "M5 cannot wire data-server without M4" — false (M5 wires it on M3 + `for tick in chunk`); re-deriving against the code rather than defending the roadmap I wrote is what surfaced the clause-2 failure ("user suggestions ≠ directives, form own judgment"). Outcome: M4 retired in `docs/roadmap.md` (struck entry kept one cycle, never `[x]`); M5 reconciled (`depends on:` M4→M3 + Tick-coverage todo; adapter unrolls host-side; friction feeds the host-per-tick-FFI-vs-batch P2 perf decision); residual = a new `[todo]` "Tick-coverage on M3" (E2E+fixture pinning the two-record-param per-tick `(State, Tick) -> State` shape — capability present today but only E2E-proven for a single record param `State`; every shipped M3 fixture pushes a scalar `Float` sample, none a record `Tick`; test backfill, no brainstorm — the actual "minimal data-server binding"). Forward note: the P2 flat-array item's "1024 cons-cells/chunk" framing is now partly stale (cons-list path dropped) — reconcile when picked up, not now. → 2026-05-18-brainstorm-embedding-abi-m4-retired.md
 - 2026-05-18 — iter bugfix-over-strict-mode-ctor-rebuild-consume (RED→GREEN, debug→implement mini, DONE 1/1): fixed a conservative `[over-strict-mode]` false-positive surfaced by the Tick-coverage fixtures. The lint's consume-detection (`any_sub_binder_consumed_for`/`pattern_has_consumed_heap_binder`, `crates/ailang-check/src/linearity.rs`) only recognised a consume of an `(own (con T))` param when a *heap-typed* pattern-binder was moved out of `match p`; when `p` was destructured into purely *primitive* fields fed into a `Term::Ctor` rebuilding `p`'s own ctor, that genuine dismantle+rebuild consume was invisible, so the lint spuriously advised `(borrow ...)`. Real harm: an LLM author "fixing" the spurious warning by flipping an export's declared mode `own`→`borrow` would silently invert the ABI ownership contract. RED-first: debugger disproved the carrier's initial nested-`match` hypothesis (the M3 `embed_backtest_step_record.ail` is silent only because its implicit-mode scalar `Float` param disables the lint via the activation gate, linearity.rs:327 — NOT because it handles the rebuild; the defect reproduces single-param, no nesting), wrote the synthetic RED unit `over_strict_mode_silent_when_ctor_rebuilt_from_primitive_fields` committed as its own audit-trail commit `a11cb7c`. GREEN (implement mini): added a 2nd recognition path to `any_sub_binder_consumed_for` — a `match p` arm that destructures binders out of `p`'s ctor and references any of them (primitive or not) inside a `Term::Ctor`'s args in the arm body genuinely consumes `p`; two pure helpers (`ctor_uses_any_binder` + deep `term_mentions_any_binder`, so `(+ acc px)`-mediated flow counts), conservative toward NOT suppressing (a ctor ignoring `p`'s payload still warns — negative-control proven), over-strict-only (by-name shadowing imprecision is extra-silence, never under-strict; recorded as Known debt). Both stale doc comments that mis-attributed the FP to nested `match` corrected for doc-honesty (debugger concern #2, same code region — in-scope, not opportunistic). +166/−20 in linearity.rs only; check-only, zero codegen/runtime/ABI/schema/DESIGN.md change. Boss-verified independently: RED→GREEN, full `cargo test -p ailang-check` 108/0 lib + every binary 0-failed with NO existing test modified, `ail check embed_backtest_step_tick.ail` no longer over-strict on `st`/`tick` (exit 0), `_tick_borrow.ail` + M3 `embed_backtest_step_record.ail` still clean, the already-green `embed_tick_e2e` + bench posture untouched. No audit/fieldtest gate (lint-precision bugfix; the RED test + the green check-suite ARE the regression coverage). RED `a11cb7c` → GREEN this commit. → 2026-05-18-iter-bugfix-over-strict-mode-ctor-rebuild-consume.md
+- 2026-05-19 — iter embedding-abi-m5.1 (DONE 3/3): M5 iter 1 — stood up the workspace-excluded `ail-embed` crate. A zero-dependency embedding core (Rust port of the audited `crates/ail/tests/embed/tick_roundtrip.c`: `extern "C"` to the M3-frozen ABI + frozen-layout `State`/`Tick` box helpers + a `Kernel` price fold, raw pointers never escaping the type) links the M3 staticlib via a new `build.rs` (no in-repo precedent — `AIL_BIN` env override else nested `cargo build -p ail` against the parent workspace, separate target dir so no cargo-lock deadlock). Plus a hermetic `data-server` smoke: a synthetic Pepperstone-format ZIP fixture written via the crate's own public `RawTickRecord` type (correct-by-construction, mirrors `data-server/src/loader.rs:110-124`) → real `DataServer` → `Kernel`, asserting bit-exact vs a same-order host reference fold and the closed-form `55.0`; runs with no `/mnt`. `ail-embed` is its own cargo workspace root (empty `[workspace]` table) so the AILang compiler workspace owes it nothing; `data-server` is a dev-dependency only ⇒ Invariant 1 holds in the dependency graph, not just on paper. Boss-verified independently: ail-embed suite 2/2 green (`kernel_run_sums_prices` unit RED-first + `hermetic_smoke_data_server_roundtrip` integration), full+no-deps `cargo metadata` on the AILang workspace shows `data-server` count 0, `git status` path filter empty (zero diff to `crates/ailang-*`/`crates/ail/`/`runtime/`/`examples/*.ail`), `src/lib.rs` zero code-level `data_server`, AILang `cargo build --workspace` still clean. Two toolchain-forced corrections to the plan's verbatim `ail-embed/Cargo.toml` (added the empty `[workspace]` table; sibling dev-dep path `../libs`→`../../libs` manifest-relative) — confined to the plan-created manifest, no acceptance gate altered, 0 review re-loops; planner-recon implication recorded in the journal Concerns. Adapter API + thread-swarm explicitly deferred to M5 iter 2+. → 2026-05-19-iter-embedding-abi-m5.1.md