From 4bc7eb2e0ac3f82d9269dbd6e66ec545ba0e03cb Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 18 May 2026 15:28:48 +0200 Subject: [PATCH] =?UTF-8?q?roadmap:=20close=20Embedding=20ABI=20=E2=80=94?= =?UTF-8?q?=20M1=20([~]->[x]);=20route=203=20fieldtest=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M1 fully ratified and CLOSED: spec (grounding-check PASS) + plan (Boss-Repaired for the module==stem fixture defect) + iter (818177d partial+Repair / e406d07 DONE 7/7) + audit 425c4eb CLEAN + fieldtest 6a4e866 thesis-substantiated (0 bugs; clause-1 + clause-3 empirically confirmed first-try). 3 non-blocking fieldtest findings routed to P1, ahead of M2: - [todo] form_a.md scalar-parameter mode carve-out — docs-honesty tidy (friction + spec_gap#1, shared root: form_a.md's unconditional 'every param needs own/borrow' misdirects the scalar export headline; scalars require bare (con Int)). Pre-existing form_a.md defect M1 surfaced. Settled behaviour -> tidy, no brainstorm. - [feature] ail emit-ir --emit=staticlib — restore the Decision-5 kernel-IR-readability affordance (add it; not narrow DESIGN.md). INDEX line added. Next milestone (Embedding ABI — M2) has no spec -> /boss new-milestone bounce-back. --- docs/journals/INDEX.md | 1 + docs/roadmap.md | 134 ++++++++++++++++++++++------------------- 2 files changed, 74 insertions(+), 61 deletions(-) diff --git a/docs/journals/INDEX.md b/docs/journals/INDEX.md index 0e5385e..8c4968a 100644 --- a/docs/journals/INDEX.md +++ b/docs/journals/INDEX.md @@ -95,3 +95,4 @@ - 2026-05-18 — audit docs-honesty-lint (milestone close, CLEAN — carry-on, NO ratify): scope `928e1c0..e50b400`. Architect `clean`: scope claim exact (zero `crates/**/src/**` + `runtime/`; sole crates/ change the additive RED-first-now-GREEN `docs_honesty_pin.rs`; lockstep pairs untouched), all 14 DESIGN.md corrections genuinely present-tense with the 4 protected KEEPs (`Diverge`-reserved/`Regions`-rejected/`tiebreaker`/`No deriving`) un-stripped, agent-def "four→five" lockstep complete, sweep residue exactly the 3 documented deliberate KEEPs. One `[medium]` standing advisory wart — Sweep 5's regex self-matches the discriminator subsection's own forbidden-phrasing catalogue (DESIGN.md L62) forever (architect-flagged; orchestrator-adjudicated → filed as a P3 todo, NOT a fix iteration — not a new failure mode since the script is advisory-with-residue by design). Bench: `compile_check.py` 0/24 + `cross_lang.py` 0/25 EXIT 0 (cross_lang independently reads the same `bench_list_sum` bump path +5.95% within tol); `check.py` EXIT 1, 3 firings (`bench_list_sum.bump_s` +13.42%, `latency.implicit_at_rc.{p99_9,max}_us` +37.65%/+134.70%). Bencher causal exoneration **decisive**: HEAD vs `5bb7211` vs `de66eb7` — all 5 firing bench binaries `cmp`-byte-identical, `ail` sha256 identical at all three commits ⇒ zero causal mechanism; `*.max_us` = tracked-P2 `-n 5` tail-jitter (4th occurrence on a zero-runtime-change milestone, median/p99 rock-steady), `*.bump_s` = tracked-P2 environmental-anchor-staleness (persistent ~+15%, stale 0.046s anchor). **EXONERATED, NO baseline ratify** (Iron Law forbids ratifying noise; spec forecloses a zero-codegen baseline bump). Two pre-existing standing P2 items kept orchestrator-owned, explicitly NOT bundled into this close (bencher recommendation honoured). Milestone audit **CLEAN**; no fieldtest (zero authoring-surface change — the wrap-robust pin + Sweep 5 ARE the regression coverage). **docs-honesty-lint fully ratified and CLOSED**: spec (grounding-check PASS 10/10) + plan + single iter + audit clean; roadmap P0 flipped `[~]`→`[x]`, WhatsNew.md user-facing entry appended → 2026-05-18-audit-docs-honesty-lint.md - 2026-05-18 — iter embedding-abi-m1.1 (Embedding-ABI M1, DONE across 2 dispatches + a Boss plan Repair): a compiled AILang scalar `fn` is now callable in-process from a C/Rust host. Five pipeline layers shipped: additive `FnDef.export: Option` (serde `default`+`skip_serializing_if`, modelled on `FnDef.doc`, ~85-site compile-driven thread incl. all in-source `#[cfg(test)]`/drift/hash literals, hash-stable golden `b4662aa70839f60b`); Form-A `(export "")` modifier (parse/print/`form_a.md` lockstep, byte-identical round-trip); check-side scalar-only+effect-free export gate — two new `CheckError` Error variants `export-non-scalar-signature`/`export-has-effects`, params→ret→effects first-violation-wins, unconditional on `f.export.is_some()` (the feature-acceptance clause-3 discriminator *in code*); codegen `Target::StaticLib` (suppresses `@main`+`MissingEntryMain`, emits one external `@` forwarder per export to `@ail__`); CLI `ail build --emit=staticlib` (`build_staticlib`: `libembed_backtest_step.a` program-only + separate `libailang_rt.a`). Coherent stop PROVEN: `embed_e2e.rs` `cc`-links both archives, `backtest_step(0,3)=9`/`(9,4)=25`, `s==25`, exit 0. Task 1 (fixed-first CLI `MissingEntryMain` baseline pin) green before any staticlib code. Process: Task 4 BLOCKED on the first dispatch — a Boss plan-defect (gate/headline fixtures declared `module ≠ file-stem`; the loader hard-enforces `module==stem` at `workspace.rs:438`, so `ail check` panicked on load before the gate). Boss verified, chose **Option 2 over the orchestrator-recommended Option 1** (keep descriptive `embed_*` filenames, rename fixture *modules* to their stems — the C symbol `backtest_step` stays via `(export …)`, *demonstrating* spec Decision 2's mangling-decoupling rather than violating it; archive→`libembed_backtest_step.a`, internal→`@ail_embed_backtest_step_step`, `host.c` unchanged), Repaired the plan (Design-decision 6 + Task-4 module==stem + Task-5/6/7 dependent strings + folded the no-`*.`-Float-head and `content_hash_fn`→`def_hash` concerns), committed Tasks 1–3 as a verified known-good partial (`818177d`), re-dispatched [4,7]. Independent Boss verification post-completion: E2E 1/1, gate 6/6, lowering 2/2, CLI 2/2, full `cargo test --workspace` 0 failures, `roundtrip_cli`+`design_schema_drift`+`spec_drift` green, Invariant 1 holds (no new dep in `ailang-core`/`ailang-codegen`/`runtime`). 3 behaviour-neutral plan pseudo-vs-reality concerns recorded (`emit_ir` routes via `lower_workspace`; dead `Path` import dropped; `##` vs `###` heading). Bench: `compile_check.py`/`cross_lang.py` exit 0; `check.py` exit 1 = tracked-P2 `*.bump_s`/`*.max_us` known-noise (default-exe codegen/runtime byte-unchanged — audit-adjudicated, not a plan gate). spec `docs/specs/2026-05-18-embedding-abi-m1.md` (grounding-check PASS) → plan `docs/plans/embedding-abi-m1.1.md` (Repaired) → commits `818177d` (partial 1–3) + this completion. Milestone-close `audit` then `fieldtest` (surface-touching — `(export …)` is new authoring surface) are the next pipeline steps. → 2026-05-18-iter-embedding-abi-m1.1.md - 2026-05-18 — audit embedding-abi-m1 (milestone close, CLEAN — carry-on, NO ratify): scope `064599e..e406d07`. Architect `clean` — Invariant 1 (clean core) holds semantically (no finance/`data-server` vocab/logic/dep in `ailang-core`/`ailang-codegen`/`runtime`; `backtest_step` is an opaque author string through generic `FnDef.export`; AILang↔host meeting point correctly M5-deferred), DESIGN.md honesty satisfied (§"Embedding ABI (M1)" + fn-JSON `"export"` + field rustdoc present-tense; "provisional until M3"/"M2 changes the C sig" correctly *labelled* reserved-unimplemented), lockstep intact (`FnDef`↔fn-JSON↔drift/spec-drift carry the field; `@` forwarder additive raw-IR *beside* unchanged `@ail__` mangling — not through `lower_app`; schema exhaustively-constructed, no `_=>`/`..Default`; in-source `missing_entry_main_is_error` byte-unchanged; `_adapter`/`_clos` untouched), Design-decision 6 confirmed correct (fixture module==stem differs from the spec's *illustrative* `(module backtest)`/`(module bad)` by the loader invariant + spec Decision 2 — the fixtures *demonstrate* the C-symbol decoupling, not violate the spec). One `[low]` — ~85 `export: None,` lines brace-column-indented (rustfmt-divergent) — **orchestrator-adjudicated carry-on on evidence** (not pending, not ignored): `cargo fmt --all --check` shows **1137 fmt-divergent locations tree-wide predating M1**, so rustfmt is demonstrably not a project convention and the architect's "cargo fmt would churn later" premise is moot; selectively aligning 85/1137 would impose an absent standard (noise, not tidy). Bench: `compile_check.py` 0/24 + `cross_lang.py` 0/25 EXIT 0; `check.py` EXIT 1, 7 firings (`*.bump_s` trio +12–14%; `explicit_at_rc`/`implicit_at_rc` `p99_9`/`max_us` +36–124%). Bencher causal exoneration **decisive**: 21/21 bench binaries `cmp`-byte-identical `064599e`↔`e406d07` (`bench_list_sum_bump` sha `dce18c288904c9f0` both) — M1's `Target::Executable` default leaves every bench fixture's emitted machine code unchanged ⇒ zero causal mechanism; RC-latency tails reproduce *on the byte-identical oracle itself* (3× run-to-run swing, median/p99 steady = `-n 5` single-sample jitter); fresh re-run collapsed 6/7 firings to ok. Both families map (confirmed) onto the two pre-existing roadmap-P2 tracked items (~4th zero-runtime-change milestone they fire+collapse on). **EXONERATED, NO baseline ratify** (Iron Law forbids ratifying noise against a zero-default-path-byte iter; spec forecloses a zero-codegen baseline bump); the two P2 bench-harness-recalibration items stay orchestrator-owned, NOT bundled (bencher recommendation honoured). M1 milestone audit **CLEAN**; `fieldtest` (surface-touching — `(export "")` is new authoring surface) is the next pipeline step before close → 2026-05-18-audit-embedding-abi-m1.md +- 2026-05-18 — fieldtest embedding-abi-m1 (milestone CLOSE, surface-touch — thesis substantiated, 3 non-blocking follow-ups routed): post-audit downstream-LLM-author field test of M1's `(export "")` + `ail build --emit=staticlib` + scalar/effect-free gate, DESIGN.md + public examples only (never compiler source). 4 fresh programs: Int fixed-point EMA `(State,Sample)->State` fold + Float leaky integrator (both author→`--emit=staticlib`→C-host link→typed scalar return, `s==67`/`s==1.875`, exit 0 — **clause-1 confirmed first-try**), IntList-param export (correctly rejected `export-non-scalar-signature`), `!IO` scalar-clean export (correctly rejected `export-has-effects`) — **clause-3 confirmed, both diagnostics precise+self-correcting**. **0 bugs**; the M1 capability + gate are sound and the milestone thesis ("an LLM author makes a scalar fold callable") is empirically substantiated. 3 working + 1 friction + 2 spec_gap. The friction + spec_gap#1 share a root: `crates/ailang-core/specs/form_a.md` "Schema invariants" item 1 + Pitfalls state an *unconditional* "every fn param MUST be `(own/borrow T)`" but scalars require **bare** `(con Int)` (`(own (con Int))`→`use-after-consume`, `(borrow)`→`consume-while-borrowed`); the doc the LLM reads first misdirects M1's headline task into a body-pointing linearity diagnostic — a **pre-existing form_a.md docs-honesty defect M1 made acute, not introduced** (the positive examples only succeeded because the fieldtester imitated the public `embed_*` corpus over the form_a.md prose rule). spec_gap#2: no public `emit-ir --emit=staticlib` though Decision 5 makes kernel-IR readability load-bearing. Boss independently re-verified via the public CLI (positive E2E builds both archives; both rejections fire the documented codes). Orchestrator routing (own judgement, not the fieldtester's recommended-action column): friction+spec_gap#1 → ONE P1 `[todo]` docs-honesty tidy (form_a.md scalar-param carve-out symmetric with the existing return-type carve-out + DESIGN.md mirror; behaviour settled → tidy, no brainstorm; ahead of M2 because it undermines the clause-1 ergonomics of the surface M2–M5 extend); spec_gap#2 → P1 `[feature]` add `--emit=staticlib` to `ail emit-ir` (restore the affordance, NOT narrow DESIGN.md). None are bugs (no debug); none M1-blocking. **embedding-abi-m1 fully ratified and CLOSED**: spec (grounding-check PASS) + plan (Boss-Repaired) + iter (`818177d` partial+Repair / `e406d07` DONE) + audit `425c4eb` CLEAN + fieldtest thesis-substantiated; roadmap P1 `[~]`→`[x]`, 2 follow-ups added P1-ahead-of-M2. Next *milestone* is Embedding ABI — M2 (no spec) = a `/boss` new-milestone bounce-back (not auto-started); the 2 P1 follow-ups are autonomous-eligible tidies/feature but the session is context-deep — surfaced to the user for the session-shape/next-item call. → 2026-05-18-fieldtest-embedding-abi-m1.md diff --git a/docs/roadmap.md b/docs/roadmap.md index c3bfe2b..f4a81d5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -129,69 +129,81 @@ work progresses. ## P1 — Next -- [~] **\[milestone\]** Embedding ABI — M1: linkable artifact + - scalar C entrypoint + `main`-free lifecycle. First of a - five-milestone arc (M1–M5) that makes a compiled AILang kernel - callable in-process from a concurrent Rust host, so a swarm of - backtests over `data-server`-supplied market data runs each - backtest as an independent single-threaded AILang computation. +- [x] **\[milestone\]** Embedding ABI — M1: linkable artifact + + scalar C entrypoint + `main`-free lifecycle — CLOSED 2026-05-18. + First of the five-milestone arc (M1–M5) making a compiled AILang + kernel callable in-process from a concurrent Rust host. Shipped: + additive `FnDef.export: Option` (hash-stable, ~85-site + compile-driven thread); Form-A `(export "")` modifier + (round-trip-gated); check-side scalar-only+effect-free export gate + (`export-non-scalar-signature` / `export-has-effects` — the + feature-acceptance clause-3 discriminator *in code*); codegen + `Target::StaticLib` (suppresses `@main`+`MissingEntryMain`, emits + one external `@` forwarder per export, decoupled from + `ail__` mangling per Decision 2); CLI `ail build + --emit=staticlib` (`lib.a` + separate `libailang_rt.a`). + Coherent stop PROVEN: C host links both archives, calls the + scalar kernel, typed return holds. spec `51160e9` (grounding-check + PASS) → plan `b0bd7ef` (Repaired) → iter `818177d` (partial 1–3 + + Boss plan Repair: a module≠stem fixture defect, resolved Option 2) + + `e406d07` (DONE 4–7). Milestone-close `audit` `425c4eb` CLEAN + (architect clean — Invariant 1 holds semantically, lockstep + intact; sole `[low]` rustfmt-divergence evidence-adjudicated + carry-on, 1137 pre-existing tree-wide divergences prove rustfmt + is not a project convention; bench `check.py` exit 1 decisively + causally exonerated — 21/21 bench binaries `cmp`-byte-identical + vs pre-milestone, the two tracked-P2 noise families, NO ratify). + `fieldtest` `6a4e866` substantiated the thesis (0 bugs; Int+Float + positive E2E first-try clean = clause-1; both clause-3 rejections + precise+self-correcting; 3 working) — 3 non-blocking findings + routed to the two follow-up items directly below. Stays here + briefly for context; remove once stale (full record in + `docs/journals/INDEX.md`). + - context: `docs/specs/2026-05-18-embedding-abi-m1.md`; + `docs/specs/2026-05-18-fieldtest-embedding-abi-m1.md`; + `docs/journals/2026-05-18-iter-embedding-abi-m1.1.md`; + `docs/journals/2026-05-18-audit-embedding-abi-m1.md` - **Motivation (shared by M1–M5).** The target deployment of - AILang is financial-data analysis: a host (Rust, owning - `data-server`'s shared zero-copy cache) drives a swarm of - backtests; each backtest is the pure per-chunk fold AILang is - built for (`(State, Chunk) -> State`, the `sum_sq` loop/recur - shape). Today AILang is a whole-program compiler whose only - output channel is `io/print_str` — there is no way to invoke a - compiled AILang function from a host with structured arguments - and a typed return. That missing call surface, not the math, is - the entire blocker. This is a backend/packaging capability, not - a language feature: the language semantics do not change. Its - justification is strategic (this is where the language is meant - to run), not the standard authoring-utility gate — a deliberate, - user-made direction call (2026-05-18 chat). +- [ ] **\[todo\]** form_a.md scalar-parameter mode carve-out — docs- + honesty tidy (M1 fieldtest friction + spec_gap, shared root). + `crates/ailang-core/specs/form_a.md` "Schema invariants" item 1 + + the Pitfalls bullet state an *unconditional* "every `(fn …)` param + MUST be wrapped in `(own T)`/`(borrow T)`; implicit mode is + rejected" — but scalars require **bare** `(con Int)` (`(own (con + Int))` → `use-after-consume`, `(borrow …)` → + `consume-while-borrowed`; bare checks `ok`). The doc the LLM reads + first actively misdirects M1's headline export task into a + body-pointing linearity diagnostic that never names the real cause. + Pre-existing form_a.md defect M1 made acute (the export task is the + first where a careful fresh author follows the mode prose for a new + program). Fix: state the scalar-parameter carve-out symmetrically + with the existing return-type carve-out (form_a.md item 1 + + Pitfalls), and mirror the canonical bare-scalar export-param + spelling in DESIGN.md §"Embedding ABI (M1)". Behaviour is settled + (no language/checker change) → a docs-honesty tidy in the + `docs-honesty-lint` class (anti-regression presence/absence pin), + not a brainstorm milestone. P1 — it undermines the clause-1 + authoring ergonomics of the surface the rest of the M2–M5 arc + extends, so it precedes M2. + - context: `docs/specs/2026-05-18-fieldtest-embedding-abi-m1.md` + findings [friction] + [spec_gap]#1. - **Invariants (audited at every close in this arc).** - (1) *Clean core* — `ailang-core` / `ailang-codegen` / the - runtime gain no `data-server`/finance knowledge and no - dependency on either; the only place AILang and `data-server` - meet is a separate host-side adapter crate (`ail-embed`, M5) - that depends on both while neither depends on it. The `audit` - architect review verifies this each close. - (2) *Single-threaded per computation* — all concurrency is - structural and owned by the host; no AILang value ever crosses - a thread boundary, so non-atomic RC and "no shared mutable - refs" stay intact. - (3) *Native-AOT only* — no interpreter/playground is introduced - (the "emit LLVM → clang -O2" performance pillar is load-bearing - for the swarm). - - **M1 scope.** CLI emit-mode producing a relocatable archive - instead of linking an executable; codegen exports an - author-designated `fn` under a stable C symbol with a - C-compatible signature instead of wrapping `main` (today - hard-coded `define i32 @main() { call i8 @ail__main() }`, - `crates/ailang-codegen/src/lib.rs:556`); a minimal explicit - lifecycle so the runtime is usable with no `main`-time setup. - Scalar signatures only (`(Int)->Int`, `(Float,Float)->Float`) - — no ADT crosses the boundary yet. Coherent stop: AILang - arithmetic callable from C/Rust, proven by a host harness that - links the `.a`, calls the symbol, asserts the return. - - context: 2026-05-18 chat (`data-server` analysis → - embedding-ABI decomposition). Spec landed + user-approved - 2026-05-18: `docs/specs/2026-05-18-embedding-abi-m1.md` - (grounding-check PASS on 3rd dispatch; 7 design decisions - pinned — additive `FnDef.export: Option` / - `(export "")` mangling-decoupled symbol / `lib.a` - + separate `libailang_rt.a` / no runtime lifecycle API / - `--emit=staticlib` suppresses `@main`+`MissingEntryMain` / - scalar+effect-free export gate / Task 1 fixed = - `MissingEntryMain` build-path baseline pin). Plan landed - 2026-05-18 `docs/plans/embedding-abi-m1.1.md` (7 tasks; recon - file-map Boss-verified against source; both compile-break - tasks — `FnDef.export` ~95-site thread, `lower_workspace_inner` - `Target` param — scrubbed for self-review-item-7 ordering). - Next: `implement`. +- [ ] **\[feature\]** `ail emit-ir --emit=staticlib` — restore the + Decision-5 IR-readability affordance for kernels (M1 fieldtest + spec_gap). DESIGN.md Decision 5 makes "the compiler emits LLVM IR + as text so the LLM can read what it generated" a load-bearing + author affordance, but `ail emit-ir` has no `--emit=staticlib` + (only `ail build` does), and `emit-ir` on a `main`-free kernel + hits the executable-path `MissingEntryMain` rejection — so the + author cannot read the generated `@` forwarder for an + exported kernel, the exact thing M1 introduced. Resolution chosen + on the merits: **add** `--emit=staticlib` to `ail emit-ir` + (symmetric with `ail build`, reusing the `Target::StaticLib` + path) — NOT narrow DESIGN.md (narrowing a load-bearing affordance + to dodge a gap is the wrong direction). Scoped feature, no design + fork. P1. + - context: `docs/specs/2026-05-18-fieldtest-embedding-abi-m1.md` + finding [spec_gap]#2. - [ ] **\[milestone\]** Embedding ABI — M2: per-thread runtime context + concurrency safety. `ailang_ctx_new()/_free` threaded