From 3ec406e687c3bf8d3441a5445eab5ceefabd9a95 Mon Sep 17 00:00:00 2001 From: Brummel Date: Fri, 29 May 2026 18:26:59 +0200 Subject: [PATCH] =?UTF-8?q?spec:=20raw-buf=20=E2=80=94=20re-carve=20.2/.3/?= =?UTF-8?q?.4=20post=20intrinsic-bodies=20(refs=20#7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit raw-buf.1 (intercept registry) shipped (140a0c0). The intrinsic-bodies milestone (#9) then landed and pulled the foundation out from under the original raw-buf.2/.3 split, so this revises the spec. Two foundation changes from intrinsic-bodies: 1. The placeholder-body convention the original spec leaned on ((body x) round-trip stubs) is gone. prelude.ail now uses (intrinsic) markers, and check_fn (lib.rs:2173) typechecks every non-intrinsic body including kernel-tier ones. RawBuf.get with a placeholder body (declared (ret a), body constructs a RawBuf) is now a hard type error — get/new/set/size MUST be (intrinsic). 2. An (intrinsic) marker now requires a matching INTERCEPTS entry as a lockstep pair (intercepts_bijection_with_intrinsic_markers). The "manifest visible / codegen deferred" intermediate state the original raw-buf.2 was designed to ship is forbidden by that invariant — the marker and its codegen entry must land in the same iteration. Re-carve (3 remaining iterations, retirement isolated): raw-buf.2 — kernel family-crate rename ailang-kernel-stub → ailang-kernel. Pure refactor, zero behavioural change. raw-buf.3 — RawBuf end-to-end: raw_buf module ((intrinsic) markers) + 12 INTERCEPTS entries + Term::New desugar + drop + ailang-surface wiring + E2E. Manifest+codegen land together. Workspace count 4 → 5. raw-buf.4 — kernel_stub retirement: RawBuf subsumes every ratification role (Term::New, param-in, kernel-tier auto-import, intrinsic); answer entry + stub removed, count 5 → 4. Preserved unchanged: Goal, slab layout, feature-acceptance argument. The raw_buf module Form-A source in § Concrete code shapes is gate-verified: ail check → ok (35 symbols across 3 modules); the four (intrinsic) markers are legal because the module is (kernel). Grounding-check: 8/9 load-bearing current-behaviour assumptions ratified by named green tests. The single override is the Term::New codegen-deferral arm (lib.rs:2094 + sibling at lib.rs:3296, the latter surfaced by the grounding agent) — a transient prep.2 marker that raw-buf.3 deletes, not a relied-upon invariant; forward-ratified by raw-buf.3's build-and-run E2E. Override logged, not a discard. --- docs/specs/0054-raw-buf.md | 718 +++++++++++++++++++------------------ 1 file changed, 374 insertions(+), 344 deletions(-) diff --git a/docs/specs/0054-raw-buf.md b/docs/specs/0054-raw-buf.md index 9ce391b..05b2d5f 100644 --- a/docs/specs/0054-raw-buf.md +++ b/docs/specs/0054-raw-buf.md @@ -1,9 +1,36 @@ # raw-buf — Design Spec -**Date:** 2026-05-29 +**Date:** 2026-05-29 (revised 2026-05-29) **Status:** Draft — awaiting user spec review **Authors:** orchestrator + Claude +> **Revision note.** raw-buf.1 (intercept registry) shipped +> (`140a0c0`). This spec was then re-carved because the +> `intrinsic-bodies` milestone (#9) landed afterwards and removed +> the foundation the original raw-buf.2/.3 split stood on: +> +> 1. The **placeholder-body convention** the original spec relied +> on (`(body x)` round-trip stubs, "the existing convention from +> `examples/prelude.ail`") no longer exists. `prelude.ail` now +> uses `(intrinsic)` markers, and the checker (`check_fn`, +> `crates/ailang-check/src/lib.rs:2173`) typechecks every +> non-intrinsic body — including kernel-tier bodies. A +> placeholder body for `RawBuf.get` (declared `(ret a)`, body +> constructing a `RawBuf`) is now a hard type error. `get` (and +> `new`/`set`/`size`) MUST be `(intrinsic)`. +> 2. An `(intrinsic)` marker now requires a matching `INTERCEPTS` +> codegen entry as a **lockstep pair**, enforced by +> `intercepts::tests::intercepts_bijection_with_intrinsic_markers`. +> The "manifest visible, codegen deferred" intermediate state +> the original raw-buf.2 was designed to ship is therefore +> forbidden by an invariant. The marker and its codegen entry +> must land in the same iteration. +> +> Consequence: the kernel-manifest and the codegen now land +> together in one iteration. The remaining work is re-carved into +> three iterations (raw-buf.2/.3/.4) below. The Goal, slab layout, +> and feature-acceptance argument are unchanged. + ## Goal Ship `RawBuf a` — the first kernel-tier *base* extension — as a @@ -22,97 +49,170 @@ backlog: RawBuf is that primitive. The milestone also fulfils the architectural commitment from the -kernel-extensions whitepaper § "Plugin contract": - -> *The migration of existing hardcoded primitive-instance -> intercepts (`try_emit_primitive_instance_body`) into the plugin -> registry is triggered by the first real base extension shipping -> (RawBuf). One registry mechanism for all intercepts.* +kernel-extensions whitepaper § "Plugin contract": the migration +of the hardcoded primitive-instance intercepts into one registry, +triggered by the first real base extension shipping. raw-buf.1 +already discharged that migration (the registry is live and the +`intrinsic-bodies` bijection pin locks marker ↔ entry); RawBuf is +now the first *real-payload* consumer of that registry. This makes raw-buf both a feature ship (a new type LLM authors -can reach for) and a foundation move (the *one* plumbing path -for all current and future codegen intercepts). +can reach for) and the proof that the registry path carries a +genuine domain type, not just the migrated primitive intercepts. ## Architecture -Three layered moves, one per iteration: +Four iterations. raw-buf.1 is done; .2/.3/.4 remain. -1. **Intercept registry** (raw-buf.1) — Lift the existing - hard-coded `try_emit_primitive_instance_body` match - (`eq__Str`, `compare__Int|Bool|Str`, plus the `float_*` - family that lowers to `fcmp`) into a registry table - `&[Intercept { name, sig, emit }]`. The single existing - dispatch site in `lower_fn_body` (the `emit_fn` caller - that today reaches `try_emit_primitive_instance_body`) - becomes a table lookup. Zero behavioural change; every - existing test stays green. This iter is a pure refactor - — its ratification is the workspace test suite as-is. +1. **Intercept registry** (raw-buf.1) — **DONE** (`140a0c0`). + Lifted the hard-coded `try_emit_primitive_instance_body` match + into the registry table `INTERCEPTS` in + `crates/ailang-codegen/src/intercepts.rs`; the dispatch site is + a `intercepts::lookup` call. Pure refactor, zero behavioural + change. (The subsequent `intrinsic-bodies` milestone added the + `(intrinsic)` marker, the `Term::Intrinsic` leaf, and the + marker ↔ entry bijection pin on top of this registry.) -2. **RawBuf kernel-tier manifest + checker side** (raw-buf.2) — - Rename `crates/ailang-kernel-stub/` to `crates/ailang-kernel/` - and re-shape it as a family-crate: one Cargo crate hosting - *all* kernel-tier modules as sibling Rust submodules under - `src/`, one submodule per `.ail` source. The renamed crate - carries `src/kernel_stub/{mod.rs, source.ail}` (the existing - stub, unchanged) and gains `src/raw_buf/{mod.rs, source.ail}` - (the new RawBuf module). `src/lib.rs` re-exports each - submodule's `SOURCE` const under a stable name - (`pub use kernel_stub::SOURCE as STUB_AIL;`, - `pub use raw_buf::SOURCE as RAW_BUF_AIL;`), preserving the - public-symbol surface every consumer already binds to. The - parse hops stay in `ailang-surface` (`parse_kernel_stub` - joined by `parse_raw_buf`); the acyclic dependency chain - `ailang-surface → ailang-kernel → ailang-core` is preserved. - The raw_buf module declares: - - `(data RawBuf (vars a) (ctor B (storage-tag)) (param-in - (a Int Float Bool)))` — opaque single-ctor TypeDef with - element-type restriction. - - `(fn new ...)`, `(fn get ...)`, `(fn set ...)`, - `(fn size ...)` with type signatures from the whitepaper - and placeholder bodies (same convention as the existing - `eq__Int` / `compare__Int` intercept stubs in - `examples/prelude.ail`). - Checker accepts a consumer-fixture `(con RawBuf (con Int))` - in a type slot; `ail check` passes. `ail build` fails - precisely with `intercept-not-registered: new__RawBuf__Int` - (or the existing Term::New-deferral diagnostic), because the - registry has no entry yet — that diagnostic IS the - ratification. +2. **Kernel family-crate rename** (raw-buf.2) — Pure refactor, + zero behavioural change. Rename `crates/ailang-kernel-stub/` → + `crates/ailang-kernel/` and reshape it as a *family-crate*: one + Cargo crate hosting all kernel-tier modules as sibling Rust + submodules under `src/`, one submodule per `.ail` source. The + renamed crate carries `src/kernel_stub/{mod.rs, source.ail}` + (the existing stub, **verbatim including its current `answer` + intrinsic fn** — the source has grown since the original spec + was written). `src/lib.rs` re-exports each submodule's `SOURCE` + const under the stable name every consumer already binds + (`pub use kernel_stub::SOURCE as STUB_AIL;`). The acyclic chain + `ailang-surface → ailang-kernel → ailang-core` is preserved; + the parse hop `parse_kernel_stub` stays in `ailang-surface`. No + `raw_buf` submodule yet — this iteration is *only* the rename, + so its failure mode is "existing tests break", the cleanest + bisection target. Ratified by the existing suite green. -3. **RawBuf codegen intercepts + Term::New desugar + stub - retirement** (raw-buf.3) — Register the 12 element-type- - specialised entries (`new__RawBuf__Int|Float|Bool`, - `RawBuf_get__Int|Float|Bool`, `RawBuf_set__Int|Float|Bool`, - `RawBuf_size__Int|Float|Bool`) into the registry. Lower - each via `@ailang_rc_alloc` + getelementptr + load/store — - no new C code. Add a desugar pass `(new T args)` → - `(app T.new args)` so Term::New is eliminated before - codegen — Term::New becomes purely a checker-side construct - (the desugar is the codegen completion the prep.2 deferral - pointed at). Worked consumer program runs end-to-end. Retire - the `kernel_stub` submodule from `crates/ailang-kernel/` - since RawBuf is now the second-and-real kernel-tier - consumer that CLAUDE.md names as the stub's retirement - trigger: delete `src/kernel_stub/` and its `lib.rs` - re-export, drop `parse_kernel_stub` and its surface - injection in `ailang-surface`, and update INDEX + - whitepaper to reflect. +3. **RawBuf end-to-end** (raw-buf.3) — The feature ship, landing + the manifest and the codegen together (the lockstep pair the + bijection invariant requires). Adds: + - The `raw_buf` submodule (`src/raw_buf/{mod.rs, source.ail}`) + with the `RawBuf` TypeDef (`param-in (a Int Float Bool)`) and + the four operations `new`/`get`/`set`/`size`, each an + `(intrinsic)` marker (NOT a placeholder body). + - The 12 element-specialised `INTERCEPTS` entries (4 ops × 3 + element types) lowering each via `@ailang_rc_alloc` + + getelementptr + load/store — no new C code. The entry `name` + of each MUST equal the mono-mangled symbol of its + `(intrinsic)` marker; the bijection test is the pin that + forces this and surfaces the exact expected strings. + - A desugar pass `(new T )` → an instantiated + `(app T.new )` that consumes the `NewArg::Type` + element type to drive monomorphisation to `new__RawBuf__`, + so `Term::New` is eliminated before codegen. `Term::New` + becomes a checker-side-only construct (the codegen completion + the prep.2 deferral pointed at). raw-buf.3 removes **both** + `Term::New` deferral arms codegen currently carries — the + `lower_term` arm (`crates/ailang-codegen/src/lib.rs:2094`, + "milestone raw-buf" internal error) and the sibling + synth-path arm (`lib.rs:3296`). These arms are the deferral + marker prep.2 left; raw-buf.3 discharges them. They are not a + relied-upon current invariant — the milestone's correctness + depends on *removing* them, not on their persisting. + - Drop for primitive-element RawBuf: a single + `@ailang_rc_release` on the slab pointer (no element-walking; + elements are primitives). + - `ailang-surface` wiring: `parse_raw_buf` + the workspace + injection that makes `raw_buf` a kernel-tier auto-imported + module (workspace module count 4 → 5). + The worked consumer program runs end-to-end and prints `60`. + `kernel_stub` stays alongside this iteration (count 5). -The ordering is not arbitrary: raw-buf.1 is the only iter with -zero behavioural change, so its failure mode is "existing tests -break" — the cleanest possible bisection target. raw-buf.2 ships -the checker-visible surface on an unchanged codegen substrate, -so its failure mode is checker-isolated. raw-buf.3 lands codegen -on a registry that has already absorbed every legacy intercept, -so the RawBuf entries do not co-mingle with a registry move. +4. **kernel_stub retirement** (raw-buf.4) — Focused removal. + RawBuf now subsumes every ratification role kernel_stub held: + `Term::New` end-to-end (the `(new RawBuf …)` consumer), + `param-in` reject (the Str-element reject fixture), kernel-tier + auto-import (RawBuf visible without `(import raw_buf)`), and the + `(intrinsic)` mechanism (RawBuf's four ops + their registry + entries). The stub is therefore pure redundancy. Delete + `src/kernel_stub/` + its re-export, drop `parse_kernel_stub` and + its surface injection, remove the `answer` `INTERCEPTS` entry + + `emit_answer` (its `(intrinsic)` marker leaves with the stub — + the bijection lockstep holds), delete the + `kernel_stub_module_round_trips` test, re-baseline `workspace_pin` + to 4 (raw_buf in, kernel_stub out — net unchanged), and update + `design/INDEX.md` + `design/models/0007-kernel-extensions.md`. + The `ailang-kernel` crate itself stays as the family-crate home + for future kernel-tier modules (series, matrix, …). Isolating + the retirement keeps each bijection-registry change (12 entries + added in .3, the `answer` entry removed in .4) in its own diff. + +The ordering rationale: .2 is the only zero-behavioural-change +iteration (rename), the cleanest bisection target; .3 lands the +feature on a registry that has already absorbed every legacy +intercept (raw-buf.1) and on the renamed family-crate (raw-buf.2), +so the RawBuf manifest+codegen do not co-mingle with a registry +move or a crate move; .4 removes the now-redundant stub without +touching the new RawBuf code. ## Concrete code shapes -### Primary — the user-facing program raw-buf delivers +### Primary — the raw_buf kernel-tier module (raw-buf.3, gate-verified) + +This is the exact Form-A source the `raw_buf` submodule embeds. It +parses and type-checks against the current tool (`ail check` → +`ok`); the four `(intrinsic)` markers are legal because the module +is `(kernel)`. The element type is restricted to `{Int, Float, +Bool}` via `param-in`. `RawBuf.get` returns the bare element type +`a`; `set` is linear (`own` in, `own` out). + +```ail +(module raw_buf + (kernel) + (data RawBuf (vars a) + (doc "Mutable, indexed, fixed-size flat buffer of primitive elements. Opaque single-ctor handle; codegen intercepts emit raw alloc/load/store over an @ailang_rc_alloc slab. Element type restricted to {Int, Float, Bool} via param-in.") + (ctor B a) + (param-in (a Int Float Bool))) + (fn new + (doc "Allocate an uninitialised RawBuf of capacity n. Compiler-supplied (intrinsic) body; codegen intercept new__RawBuf__ emits @ailang_rc_alloc plus the i64 size header. Element bytes are uninitialised; caller must set before get.") + (type (forall (vars a) (fn-type (params (con Int)) (ret (own (con RawBuf a)))))) + (params n) + (intrinsic)) + (fn get + (doc "Indexed read. UB if i >= (size b); caller checks bounds via size. Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus load.") + (type (forall (vars a) (fn-type (params (borrow (con RawBuf a)) (con Int)) (ret a)))) + (params b i) + (intrinsic)) + (fn set + (doc "Indexed write. Linear: own in, own out. Under uniqueness in-place; under shared copy-on-write (Issue #22). Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus store.") + (type (forall (vars a) (fn-type (params (own (con RawBuf a)) (con Int) a) (ret (own (con RawBuf a)))))) + (params b i v) + (intrinsic)) + (fn size + (doc "Element count. Compiler-supplied (intrinsic) body; codegen intercept emits a single i64 load from the slab header.") + (type (forall (vars a) (fn-type (params (borrow (con RawBuf a))) (ret (con Int))))) + (params b) + (intrinsic))) +``` + +The `(ctor B a)` is a nominal fiction for the type system: it +gives `RawBuf` an identity and keeps `a` non-phantom. Real code +never constructs it — every operation is codegen-intercepted, and +`(new RawBuf …)` desugars to `(app RawBuf.new …)`, not to a +`(term-ctor RawBuf B …)`. The four `(intrinsic)` markers carry no +runnable body; codegen supplies the implementation via the +registry (raw-buf.3). + +### Primary — the worked consumer program (raw-buf.3 acceptance) + +This is the program an LLM author naturally writes for "I need a +small mutable indexed buffer to score N values and read them +back." It is the feature-acceptance criterion's empirical +evidence. It references the `raw_buf` module, which is only +injected once raw-buf.3 wires it through `ailang-surface` — so it +does **not** check against the current tool (RawBuf is not yet a +workspace module). Left unlabeled deliberately: it describes +post-raw-buf.3 surface, not current behaviour. ``` (module raw_buf_demo - (import raw_buf) (fn main (type (fn-type (params) (ret (con Int)))) (params) @@ -127,109 +227,79 @@ so the RawBuf entries do not co-mingle with a registry move. ; ail check && ail build && ail run → prints 60 ``` -This is the iter raw-buf.3 acceptance fixture (and the -feature-acceptance criterion's empirical evidence — see § Feature -acceptance below). It is exactly the program an LLM author -naturally writes to express "I need a small mutable indexed -buffer to score N values and read them back." +No `(import raw_buf)` needed: `RawBuf` is visible via the +kernel-tier auto-import path (prep.3). `RawBuf.set` / `.get` +resolve via type-scoped namespacing (prep.1). `(new RawBuf (con +Int) 3)` is a `Term::New` whose `NewArg::Type` is `(con Int)`; the +raw-buf.3 desugar threads that into `new__RawBuf__Int`. -### Primary — the param-in reject fixture +### Primary — the param-in reject fixture (raw-buf.3 must-fail) ``` -(module raw_buf_reject_string - (import raw_buf) - (fn main - (type (fn-type (params) (ret (con Unit)))) - (params) - (body (do io/print_str "unreached")) - (signature-hole (new RawBuf (con Str) 3)))) ; rejected at check +(module raw_buf_reject_str + (fn cant_str + (type (fn-type (params (borrow (con RawBuf (con Str)))) (ret (con Str)))) + (params buf) + (body (app RawBuf.get buf 0)))) ; ail check → param-not-in-restricted-set: type Str not in {Int, Float, Bool} -; at TypeDef raw_buf.RawBuf, var a ``` -This must fail at the checker, with the diagnostic produced by -the existing `ParamNotInRestrictedSet` arm shipped in -prep.3 (per fieldtest F8). It is the must-fail evidence that the -`param-in` restriction is alive on a real-payload extension, not -just on the prep.3 stub. +Must fail at the checker with the `ParamNotInRestrictedSet` +diagnostic shipped in prep.3 (fieldtest F8). It is the must-fail +evidence that the `param-in` restriction is alive on a +real-payload extension, not just the prep.3 `StubT` fixture. Like +the worked program, it depends on `raw_buf` being injected, so it +is not gate-checkable against the current tool (unlabeled). -### Primary — the raw-buf kernel-tier module +### Secondary — INTERCEPTS entry shape (raw-buf.3, implementation) -``` -(module raw_buf - (kernel) - (data RawBuf (vars a) - (ctor B (storage-tag)) - (param-in (a Int Float Bool))) - - (fn new - (doc "Allocate uninitialised RawBuf of capacity n. Codegen - intercept new__RawBuf__ emits @ailang_rc_alloc + the - size header. Element bytes are uninitialised — caller - must (set …) before (get …).") - (type (fn-type (params (con Int)) (ret (own (RawBuf a))))) - (params n) - (body (term-ctor RawBuf B n))) ; placeholder - - (fn get - (doc "Indexed read. UB if i >= (size b); caller checks - bounds. Codegen intercept RawBuf_get__ emits - getelementptr + load.") - (type (fn-type - (params (borrow (RawBuf a)) (con Int)) - (ret (con a)))) - (params b i) - (body (term-ctor RawBuf B 0))) ; placeholder - - (fn set - (doc "Indexed write. Linear: own in, own out. Under - uniqueness, in-place; under shared, copy-on-write - (Issue #22 territory). Codegen intercept - RawBuf_set__ emits getelementptr + store.") - (type (fn-type - (params (own (RawBuf a)) (con Int) (con a)) - (ret (own (RawBuf a))))) - (params b i v) - (body b)) ; placeholder - - (fn size - (doc "Element count. Codegen intercept RawBuf_size__ - emits a single i64 load from the slab header.") - (type (fn-type (params (borrow (RawBuf a))) (ret (con Int)))) - (params b) - (body 0))) ; placeholder -``` - -The placeholder bodies are the existing convention from -`examples/prelude.ail` (e.g. `eq__Int` ships with `(body x)` -that the round-trip preserves but codegen replaces). They exist -to satisfy the round-trip invariant; codegen never executes them. - -### Secondary — registry shape (implementation only) +The real registry struct (raw-buf.1) is `Intercept { name, +expected_params, expected_ret, wants_alwaysinline, emit }`. The 12 +RawBuf entries follow the existing rows' shape. One representative +(the exact symbol strings are derived by the implementer from the +bijection test, which mangles each `(intrinsic)` marker and +reports the expected name): ```rust -// crates/ailang-codegen/src/intercepts.rs (new file) - -pub struct IntrinsicSig { - pub params: Vec, // LLVM types as text, e.g. "i64", "ptr" - pub ret: String, -} - -pub struct Intercept { - pub name: &'static str, - pub sig_check: fn(params: &[String], ret: &str) -> Result<()>, - pub emit: fn(cg: &mut Lowering) -> Result<()>, -} - -pub fn lookup(name: &str) -> Option<&'static Intercept> { ... } - -// Wired into crates/ailang-codegen/src/lib.rs:1321 (the existing -// try_emit dispatch site) — that match becomes one lookup call. +// crates/ailang-codegen/src/intercepts.rs — appended to INTERCEPTS +Intercept { + name: "new__RawBuf__Int", // == mono-mangled name of raw_buf.new @ a=Int + expected_params: &["i64"], // capacity n + expected_ret: "ptr", // slab base pointer + wants_alwaysinline: false, + emit: emit_new_raw_buf_int, // @ailang_rc_alloc(8 + n*8); store i64 n at offset 0 +}, +// + new__RawBuf__{Float,Bool}, RawBuf_get__{Int,Float,Bool}, +// RawBuf_set__{Int,Float,Bool}, RawBuf_size__{Int,Float,Bool} ``` -### Secondary — slab layout +### Secondary — Term::New desugar (raw-buf.3, implementation) -RawBuf payload (post RC header) is: +`Term::New { type_name, args }` where `args: Vec` +(`NewArg::Type | NewArg::Value`, both already parsed + round-trip +ratified). The new desugar pass rewrites it to an application of +the type's `new` fn, instantiated at the element type carried by +the leading `NewArg::Type`: + +``` +(new RawBuf (con Int) 3) ; Term::New { type_name: "RawBuf", + ; args: [Type(con Int), Value(3)] } + ──desugar──▶ +(app RawBuf.new 3) @ a = Int ; instantiation pins mono → new__RawBuf__Int +``` + +The desugar consumes the `NewArg::Type` for the instantiation and +passes only the `NewArg::Value` args as call arguments. After this +pass no `Term::New` survives into codegen; both deferral arms +(`lib.rs:2094` in `lower_term`, "milestone raw-buf" internal +error, and the sibling at `lib.rs:3296` in the synth path) become +unreachable and are removed. Whether +the prep.2 checker already accepts the `NewArg::Type`-bearing +`Term::New` semantically, or needs a small extension, is a +raw-buf.3 task to verify first (the form parses + round-trips +today; the lift pass passes `NewArg::Type` through untouched). + +### Secondary — slab layout (raw-buf.3) ``` [ size : i64 ][ elem_0 ][ elem_1 ] ... [ elem_{size-1} ] @@ -237,217 +307,177 @@ RawBuf payload (post RC header) is: offset 0 offset 8 offset 8 + size*sizeof(T) ``` -Element widths: `Int = 8`, `Float = 8`, `Bool = 1`. Alignment to -8 bytes for header; element payload directly follows. `@ailang_rc_alloc` -is called with `8 + size * sizeof(T)` and gets back a `ptr` that -the codegen treats as the slab base. The size header is *self- -describing* so drop and `RawBuf.size` need no side-table. - -Drop for RawBuf with primitive elements is trivial: a single -`@ailang_rc_release` on the slab pointer. No element-walking -because element types are primitives (no recursive drops). The -existing per-TypeDef drop generation gets a "primitive-element- -RawBuf" arm that emits the trivial drop, dispatched via the same -registry. +Element widths: `Int = 8`, `Float = 8`, `Bool = 1`. Header is +i64, 8-byte aligned; element payload follows directly. +`@ailang_rc_alloc` is called with `8 + size * sizeof(T)` and +returns the slab base `ptr`. The size header is self-describing, +so drop and `RawBuf.size` need no side-table. Drop is a single +`@ailang_rc_release` on the slab pointer — primitive elements +carry no recursive drops. The per-TypeDef drop generation gets a +"primitive-element-RawBuf" arm dispatched via the same registry. ## Components | Component | Iter | Files | |-----------|------|-------| -| Intercept registry | raw-buf.1 | new file `crates/ailang-codegen/src/intercepts.rs`; edits to `crates/ailang-codegen/src/lib.rs` at the single dispatch site that today calls `try_emit_primitive_instance_body`, and removal of each absorbed match arm in that fn | -| Kernel family-crate (rename + reshape) + RawBuf submodule | raw-buf.2 | rename `crates/ailang-kernel-stub/` → `crates/ailang-kernel/`; restructure as `src/lib.rs` + `src/kernel_stub/{mod.rs, source.ail}` (existing stub re-housed) + `src/raw_buf/{mod.rs, source.ail}` (new); update `Cargo.toml` package name `ailang-kernel-stub` → `ailang-kernel`; update workspace member entry + every dependent crate's `Cargo.toml` (`ailang-surface` is the only consumer today); add `parse_raw_buf` in `crates/ailang-surface/` next to existing `parse_kernel_stub`; extend the workspace injection in `ailang-surface` to inject both modules | -| RawBuf intercept entries | raw-buf.3 | additions to `crates/ailang-codegen/src/intercepts.rs` (12 entries); desugar pass `Term::New` → `(app T.new args)` in `crates/ailang-codegen/src/synth.rs` or `crates/ailang-check/src/desugar.rs` | -| Stub retirement (submodule level) | raw-buf.3 (close) | delete `crates/ailang-kernel/src/kernel_stub/` + its re-export in `src/lib.rs`; drop `parse_kernel_stub` and its surface injection in `ailang-surface` (defn at `crates/ailang-surface/src/loader.rs:54`, re-export at `crates/ailang-surface/src/lib.rs:39`); delete the `kernel_stub_module_round_trips` test at `crates/ailang-core/tests/design_schema_drift.rs:743` (its `parse_kernel_stub` callee is gone — the analogous `raw_buf_module_round_trips` test added in raw-buf.2 carries the round-trip invariant forward); update `crates/ailang-core/tests/workspace_pin.rs` (was bumped to 4 in iter-tidy `a844de3`; raw_buf joins as the stub leaves so the count stays at 4 — see Data flow); update `design/INDEX.md` § Models row + `design/models/0007-kernel-extensions.md` § "ratifying fixture" sentence. The `ailang-kernel` crate itself stays — it is the family-crate that future kernel-tier modules (series, matrix, …) will plug into. | +| Intercept registry | raw-buf.1 (DONE) | `crates/ailang-codegen/src/intercepts.rs` + dispatch site in `lib.rs` | +| Kernel family-crate rename | raw-buf.2 | `git mv crates/ailang-kernel-stub crates/ailang-kernel`; reshape to `src/lib.rs` + `src/kernel_stub/{mod.rs, source.ail}` (existing stub source incl. `answer` re-housed verbatim); `Cargo.toml` package + workspace member + dep-alias renames; `crates/ailang-surface/Cargo.toml` dep rename + `loader.rs` use-path renames; `design/INDEX.md` + `CLAUDE.md` code-layout row | +| RawBuf module + codegen + desugar + drop | raw-buf.3 | new `crates/ailang-kernel/src/raw_buf/{mod.rs, source.ail}` + `lib.rs` re-export; `parse_raw_buf` + workspace injection in `crates/ailang-surface/src/loader.rs` + re-export in `lib.rs`; 12 entries + emit fns in `crates/ailang-codegen/src/intercepts.rs`; `Term::New` desugar in `crates/ailang-core/src/desugar.rs` (and removal of the codegen `Term::New` error arm in `crates/ailang-codegen/src/lib.rs`); primitive-element-RawBuf drop arm; `workspace_pin` 4 → 5; round-trip + E2E tests | +| kernel_stub retirement | raw-buf.4 | delete `crates/ailang-kernel/src/kernel_stub/` + its `lib.rs` re-export; drop `parse_kernel_stub` + injection in `ailang-surface`; remove `answer` entry + `emit_answer` in `intercepts.rs`; delete `kernel_stub_module_round_trips` test; `workspace_pin` 5 → 4; `design/INDEX.md` + `design/models/0007-kernel-extensions.md` updates | ## Data flow -**Build-time** (raw-buf.2 onwards): -1. CLI loads workspace → `ailang-surface` injects `raw_buf` as - a kernel-tier module (`kernel: true`) just like `prelude` - today. -2. `check` walks consumer code; `(con RawBuf (con Int))` - resolves to `raw_buf.RawBuf` via the kernel-tier auto-import - path that prep.3 shipped. -3. `param-in` restriction on RawBuf's `a` parameter rejects - non-{Int,Float,Bool} instantiations at the consumer - typecheck site, using the existing - `ParamNotInRestrictedSet` diagnostic from prep.3. +**Build-time** (raw-buf.3 onwards): +1. CLI loads workspace → `ailang-surface` injects `raw_buf` as a + kernel-tier module (`kernel: true`) just like `prelude`. +2. `check` walks consumer code; `(con RawBuf (con Int))` resolves + to `raw_buf.RawBuf` via the kernel-tier auto-import path + (prep.3). +3. `param-in` on RawBuf's `a` rejects non-{Int,Float,Bool} + instantiations at the consumer typecheck site, using the + existing `ParamNotInRestrictedSet` diagnostic. **Codegen-time** (raw-buf.3 onwards): -1. Desugar pass replaces every `Term::New {type_name, args}` - with `Term::App { callee: .new, args }` — same - form that already lowers cleanly today. -2. For each fn in the workspace, codegen reaches the dispatch - site at `lib.rs:1321`. If the fn's mono name has a registry - entry, the entry's `emit` runs; the `.ail` placeholder body - is discarded. Otherwise, the normal AILang lowering runs. -3. For RawBuf consumers, `new__RawBuf__Int` emits the alloc - call; `RawBuf_set__Int` emits a `store i64`; - `RawBuf_get__Int` emits a `load i64`; `RawBuf_size__Int` - emits a single header load. +1. The desugar pass replaces every `Term::New { type_name, args }` + with an instantiated `(app .new )`, pinning + the element type from the leading `NewArg::Type`. +2. For each fn, codegen reaches the registry dispatch site. If the + fn's mono name resolves via `intercepts::lookup`, the entry's + `emit` runs; the `(intrinsic)` marker carries no body to fall + back to. Otherwise the normal AILang lowering runs. +3. For RawBuf consumers, `new__RawBuf__Int` emits the alloc; + `RawBuf_set__Int` a `store i64`; `RawBuf_get__Int` a `load + i64`; `RawBuf_size__Int` a single header load. -**Workspace-module-count interaction** (raw-buf.3 close): -The current count of 4 (`prelude` + `kernel_stub` + two -test-fixture modules per the iter-tidy commit). When -`kernel_stub` retires and `raw_buf` joins, the count stays 4. -If `raw_buf` lands *before* the stub retires, the count is 5 -mid-iter — caught by the existing `workspace_pin` test, which -the iter explicitly re-baselines. +**Bijection interaction** (raw-buf.3 / .4): +Each RawBuf `(intrinsic)` marker reachable from the workspace is +collected by `workspace_intrinsic_markers()` and must resolve to +an `INTERCEPTS` entry (and vice versa, minus the +`OPTIMISATION_ONLY` allowlist). raw-buf.3 adds 12 markers + 12 +entries together. raw-buf.4 removes the `answer` marker (with the +stub) and the `answer` entry together. A half-landed change on +either side is a hard unit-test failure +(`intercepts_bijection_with_intrinsic_markers`). + +**Workspace module count:** prelude + kernel_stub + 2 test +fixtures = 4 today. raw-buf.3 adds raw_buf → 5. raw-buf.4 retires +kernel_stub → 4. The `workspace_pin` test pins each transition. ## Error handling -Two diagnostics; both already shipped, no new arms needed in -this milestone: +Two consumer-facing diagnostics; both already shipped, no new arms +needed: -- **`param-not-in-restricted-set`** (prep.3, fieldtest F8). - Fires when a consumer writes `(con RawBuf (con Str))` or - any non-{Int,Float,Bool} element. Names the type, the - variable, the TypeDef, and the restricted set. +- **`param-not-in-restricted-set`** (prep.3, fieldtest F8). Fires + when a consumer writes `(con RawBuf (con Str))` or any + non-{Int,Float,Bool} element. Names the type, the variable, the + TypeDef, and the restricted set. +- **`new-type-not-constructible`** (prep.3, fieldtest F5). Fires + when a consumer writes `(new SomeType …)` and `SomeType` has no + `(fn new …)` in its home module. raw_buf ships `(fn new …)`, so + the worked program passes; the diagnostic still protects future + authors who write `(new Counter 0)` against a `new`-less type. -- **`new-type-not-constructible`** (prep.3, fieldtest F5). - Fires when a consumer writes `(new SomeType …)` and - `SomeType` has no `(fn new …)` in its home module. Since - raw-buf ships `(fn new …)` in the kernel-tier `raw_buf` - module, the consumer program above passes — but if a - future LLM author writes `(new Counter 0)` against a - type that lacks `new`, this diagnostic still fires - precisely. +The codegen `Term::New` deferral arms (`lib.rs:2094` in +`lower_term`, message "… milestone raw-buf"; and the sibling at +`lib.rs:3296` in the synth path) are **removed** in raw-buf.3 once +the desugar eliminates `Term::New` before codegen. Neither is a +user-facing diagnostic — both are the deferral marker prep.2 left, +which raw-buf.3 discharges. They have no test pinning them today +(they are an internal `CodegenError::Internal`, not an asserted +behaviour); raw-buf.3's end-to-end build-and-run E2E (the worked +program printing `60`) is the forward ratification — it exercises +the `Term::New` → desugar → codegen path that replaces them. -Bounds checks are *not* in scope. Per whitepaper § "RawBuf — -the base extension": `RawBuf.get / .set` are UB if `i >= -size(b)`; caller checks. Bounds-check insertion is a future -axis tracked separately (no backlog item yet — the discipline -is "caller checks via `RawBuf.size`"). - -A future `intercept-not-registered` diagnostic surfaces at -raw-buf.2 — the iter raw-buf.2 acceptance is precisely that -this fires (consumer code checks clean but does not build), -demonstrating the manifest/codegen separation works. raw-buf.3 -makes it stop firing for the 12 RawBuf entries. +Bounds checks are **not** in scope (whitepaper § "RawBuf — the +base extension"): `RawBuf.get/.set` are UB if `i >= size(b)`; +caller checks via `RawBuf.size`. Bounds-check insertion is a +future axis, no backlog item yet. ## Testing strategy -Each iter ratified independently: +**raw-buf.2 (rename):** +- All existing workspace tests pass unchanged — the rename is a + pure refactor. Zero test-count change. The crate-rename's + failure mode is "the workspace does not build / existing tests + break", caught by `cargo build --workspace` + `cargo test + --workspace`. -**raw-buf.1 (registry refactor):** -- All ~668 existing workspace tests pass unchanged. The - refactor is *primarily* ratified by zero behavioural - change in the existing suite — the legacy intercepts - (`eq__Str`, `compare__Int|Bool|Str`, the `float_*` family) - are exercised end-to-end by the following build-and-run - E2E tests: - - `eq_primitives_smoke_compiles_and_runs` and - `compare_primitives_smoke_prints_1_2_3_thrice` in - `crates/ail/tests/e2e.rs` — pin the `eq__Str` and - `compare__Int|Bool|Str` arms (Str path included). - - `float_compare_smoke_prints_true_true_false` in - `crates/ail/tests/float_compare_smoke_e2e.rs` — pins - the `float_*` family via `float_eq` + `float_lt`. - - `eq_ord_polymorphic_runs_end_to_end` in - `crates/ail/tests/eq_ord_e2e.rs` — secondary Str-path - ratifier via polymorphic Eq/Ord dispatch. - If the registry rewrite breaks any intercept, those tests - catch it. -- One new unit test in the new `intercepts` module: - `registry_contains_all_legacy_arms` enumerates the names - that were in the legacy match (`eq__Str`, `compare__Int`, - `compare__Bool`, `compare__Str`, and each `float_*` name) - and asserts each resolves to a registered entry. This pins - the migration as complete (a half-finished migration where - the old match still has an arm becomes a hard test fail). +**raw-buf.3 (RawBuf end-to-end):** +- Round-trip: `raw_buf_module_round_trips` (mirror of + `kernel_stub_module_round_trips` in `design_schema_drift.rs`) — + parse → serialise → round-trip; assert `kernel == true`, name + `raw_buf`, and `RawBuf` TypeDef `param_in` for `a` contains all + of Int/Float/Bool. +- E2E `raw_buf_int_e2e`: the worked consumer program — `ail check + && ail build && ail run` → prints `60`. +- E2E `raw_buf_float_e2e` / `raw_buf_bool_e2e`: Float and Bool + element variants (Bool catches any i1/i8 packing question). +- E2E `raw_buf_param_in_reject_e2e`: the Str-element reject + fixture — `ail check` exits non-zero with + `param-not-in-restricted-set` in stderr. +- Drop ratification: a consumer that allocates a RawBuf and lets + it go out of scope without a leak — re-uses the existing + leak-check harness, no new infra. +- Bijection: `intercepts_bijection_with_intrinsic_markers` stays + green with the 12 new markers + 12 new entries (the test forces + the exact mangled-name match). +- `workspace_pin` count assertion 4 → 5 (+ `contains_key("raw_buf")`). -**raw-buf.2 (manifest + checker):** -- New E2E test `crates/ail/tests/raw_buf_check_e2e.rs`: - consumer fixture with `(con RawBuf (con Int))` in a type - signature, `ail check` exits 0. -- New E2E test `crates/ail/tests/raw_buf_param_in_reject_e2e.rs`: - consumer fixture with `(con RawBuf (con Str))`, `ail check` - exits non-zero with `param-not-in-restricted-set` in stderr. -- New E2E test `crates/ail/tests/raw_buf_build_intercept_missing_e2e.rs`: - consumer fixture with `(new RawBuf (con Int) 3)`, `ail - build` exits non-zero with `intercept-not-registered` (or - the Term::New deferral message) — pins the manifest/codegen - separation. (This test gets deleted or inverted to PASS in - raw-buf.3.) -- New round-trip test for the embedded `RAW_BUF_AIL` — - parse → print round-trips byte-isomorphically (mirrors - `kernel_stub_module_round_trips` in - `crates/ailang-core/tests/design_schema_drift.rs`). -- Workspace module-count test bumped: 4 → 5 (mid-iter state - before stub retires). - -**raw-buf.3 (codegen + stub retirement):** -- The intercept-missing E2E test from raw-buf.2 inverted: - same consumer fixture, `ail build` now exits 0 and the - produced binary, when run, prints `60`. -- New E2E `crates/ail/tests/raw_buf_float_e2e.rs`: Float - variant exercise. -- New E2E `crates/ail/tests/raw_buf_bool_e2e.rs`: Bool - variant exercise (catches the i1/i8 packing question - if any). -- Drop ratification: a consumer that allocates and lets - the RawBuf go out of scope without a leak. Re-uses the - existing leak-check harness; no new infra. -- Stub-retirement ratification: deleted-submodule diff hits - `crates/ailang-kernel/src/kernel_stub/` (folder removal) + - `crates/ailang-kernel/src/lib.rs` (re-export drop) + - `crates/ailang-surface/` (parse_kernel_stub + injection - drop) + `crates/ailang-core/tests/workspace_pin.rs` + - `design/INDEX.md` + `design/models/0007-kernel-extensions.md`. - Test `workspace_pin` re-baselines to 4 (raw_buf in, - kernel_stub out — net unchanged). The `ailang-kernel` crate - itself stays as the family-crate home for future kernel-tier - modules (series, matrix, …). +**raw-buf.4 (retirement):** +- `workspace_pin` re-baselines 5 → 4 (raw_buf in, kernel_stub + out); `contains_key("kernel_stub")` assertion removed, + `contains_key("raw_buf")` retained. +- `kernel_stub_module_round_trips` deleted (its `parse_kernel_stub` + callee is gone; `raw_buf_module_round_trips` carries the + round-trip invariant forward). +- Bijection stays green after the `answer` marker + entry leave + together. +- The full suite stays green; the diff is a removal, ratified by + "nothing references the stub and everything still builds". ## Acceptance criteria The milestone ships when: -1. The worked consumer program in § Concrete code shapes - end-to-end: `ail check && ail build && ail run` prints - `60`. +1. The worked consumer program runs end-to-end: `ail check && ail + build && ail run` prints `60` (raw-buf.3). 2. The param-in reject fixture exits at `ail check` with the - prep.3 diagnostic. -3. Every existing primitive-instance intercept (`eq__Str`, - `compare__Int|Bool|Str`, the `float_*` family) is dispatched - via the registry — *no remaining match-arm* in - `try_emit_primitive_instance_body` outside the registry path. -4. `ailang-kernel-stub` is retired from the workspace; `raw_buf` - takes its slot as the canonical kernel-tier base-extension - anchor. -5. Workspace test suite green, plus the new tests enumerated - in § Testing strategy. + prep.3 `param-not-in-restricted-set` diagnostic (raw-buf.3). +3. RawBuf's four operations are `(intrinsic)` markers, each paired + with an `INTERCEPTS` entry; `intercepts_bijection_with_intrinsic_markers` + is green (raw-buf.3). +4. `Term::New` no longer reaches codegen — the desugar eliminates + it and both deferral arms (`lib.rs:2094` + `lib.rs:3296`) are + removed (raw-buf.3). +5. `ailang-kernel-stub` is renamed to the `ailang-kernel` + family-crate (raw-buf.2); `kernel_stub` is retired and `raw_buf` + takes over its ratification roles (raw-buf.4). +6. Workspace test suite green at each iteration boundary, plus the + new tests enumerated in § Testing strategy. ## Feature-acceptance argument Applied per `design/contracts/0004-feature-acceptance.md`: -1. **LLM author reaches for it naturally.** RawBuf is the - only path to mutable indexed storage in AILang — the - language has no other "I want to fill N slots and read - them back" primitive. The worked consumer program above is - what an LLM author writes when expressing that need; - there is no alternative shape to reach for. +1. **LLM author reaches for it naturally.** RawBuf is the only + path to mutable indexed storage in AILang — the language has no + other "fill N slots and read them back" primitive. The worked + consumer program above is what an LLM author writes for that + need; there is no alternative shape to reach for. -2. **Measurable improvement.** Embedding-ABI M5 measured - per-tick FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks). - RawBuf as the batch-FFI primitive amortises that per-tick - cost across one crossing. The series milestone (#8) is also - blocked on RawBuf; shipping raw-buf removes that block. +2. **Measurable improvement.** Embedding-ABI M5 measured per-tick + FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks). RawBuf as + the batch-FFI primitive amortises that per-tick cost across one + crossing. The Series milestone (#8) is also blocked on RawBuf; + shipping raw-buf removes that block. 3. **No reintroduced bug class.** Mutation discipline stays mode-tracked (whitepaper § "RawBuf — the base extension"): - `RawBuf.set` is `own → own`, not effect-tracked. No - `RawBuf` effect declared. Uniqueness inference rewrites it - in-place; shared ownership falls back to copy-on-write - (Issue #22). The pure-core invariant — code without a - declared effect is pure — survives intact. The local- - reasoning invariant survives too: a fn that takes - `borrow (RawBuf Int)` and returns `Int` *is* pure from the - caller's view, because the buffer is not mutated. - -The registry-migration aspect of the milestone (raw-buf.1) -removes redundancy structurally: the growth point of -"every new primitive intercept needs another match arm" is -collapsed into a single dispatch table. That is criterion-2 -evidence for the migration as a piece. + `RawBuf.set` is `own → own`, not effect-tracked. No `RawBuf` + effect is declared. Uniqueness inference rewrites in-place; + shared ownership falls back to copy-on-write (Issue #22). The + pure-core invariant survives: code without a declared effect is + pure. Local reasoning survives: a fn taking `borrow (RawBuf + Int)` and returning `Int` is pure from the caller's view, + because the buffer is not mutated.