# raw-buf — Design Spec **Date:** 2026-05-29 (revised 2026-05-29) **Status:** Draft — awaiting user spec review **Authors:** orchestrator + Claude > **Revision note.** raw-buf.1 (intercept registry) shipped > (`140a0c0`). This spec was then re-carved because the > `intrinsic-bodies` milestone (#9) landed afterwards and removed > the foundation the original raw-buf.2/.3 split stood on: > > 1. The **placeholder-body convention** the original spec relied > on (`(body x)` round-trip stubs, "the existing convention from > `examples/prelude.ail`") no longer exists. `prelude.ail` now > uses `(intrinsic)` markers, and the checker (`check_fn`, > `crates/ailang-check/src/lib.rs:2173`) typechecks every > non-intrinsic body — including kernel-tier bodies. A > placeholder body for `RawBuf.get` (declared `(ret a)`, body > constructing a `RawBuf`) is now a hard type error. `get` (and > `new`/`set`/`size`) MUST be `(intrinsic)`. > 2. An `(intrinsic)` marker now requires a matching `INTERCEPTS` > codegen entry as a **lockstep pair**, enforced by > `intercepts::tests::intercepts_bijection_with_intrinsic_markers`. > The "manifest visible, codegen deferred" intermediate state > the original raw-buf.2 was designed to ship is therefore > forbidden by an invariant. The marker and its codegen entry > must land in the same iteration. > > Consequence: the kernel-manifest and the codegen now land > together in one iteration. The remaining work is re-carved into > three iterations (raw-buf.2/.3/.4) below. The Goal, slab layout, > and feature-acceptance argument are unchanged. ## Goal Ship `RawBuf a` — the first kernel-tier *base* extension — as a mutable, indexed, bounded-size flat buffer of primitive elements (`{Int, Float, Bool}`). RawBuf is the storage primitive that unblocks two downstream needs both already captured in the backlog: - **Series (milestone #8)** — a bounded ring buffer with financial-style indexing, implemented as a *library extension* (pure AILang) wrapping `RawBuf a` plus bookkeeping fields. - **Embedding-ABI batch FFI (subsumes closed #2)** — the M5 friction-harvest measured per-tick FFI at ~206 ns/tick on real EURUSD volume (~658 ms / 3.19M ticks). A contiguous primitive slice amortises that per-tick cost via a batch crossing. RawBuf is that primitive. The milestone also fulfils the architectural commitment from the kernel-extensions whitepaper § "Plugin contract": the migration of the hardcoded primitive-instance intercepts into one registry, triggered by the first real base extension shipping. raw-buf.1 already discharged that migration (the registry is live and the `intrinsic-bodies` bijection pin locks marker ↔ entry); RawBuf is now the first *real-payload* consumer of that registry. This makes raw-buf both a feature ship (a new type LLM authors can reach for) and the proof that the registry path carries a genuine domain type, not just the migrated primitive intercepts. ## Architecture Four iterations. raw-buf.1 is done; .2/.3/.4 remain. 1. **Intercept registry** (raw-buf.1) — **DONE** (`140a0c0`). Lifted the hard-coded `try_emit_primitive_instance_body` match into the registry table `INTERCEPTS` in `crates/ailang-codegen/src/intercepts.rs`; the dispatch site is a `intercepts::lookup` call. Pure refactor, zero behavioural change. (The subsequent `intrinsic-bodies` milestone added the `(intrinsic)` marker, the `Term::Intrinsic` leaf, and the marker ↔ entry bijection pin on top of this registry.) 2. **Kernel family-crate rename** (raw-buf.2) — Pure refactor, zero behavioural change. Rename `crates/ailang-kernel-stub/` → `crates/ailang-kernel/` and reshape it as a *family-crate*: one Cargo crate hosting all kernel-tier modules as sibling Rust submodules under `src/`, one submodule per `.ail` source. The renamed crate carries `src/kernel_stub/{mod.rs, source.ail}` (the existing stub, **verbatim including its current `answer` intrinsic fn** — the source has grown since the original spec was written). `src/lib.rs` re-exports each submodule's `SOURCE` const under the stable name every consumer already binds (`pub use kernel_stub::SOURCE as STUB_AIL;`). The acyclic chain `ailang-surface → ailang-kernel → ailang-core` is preserved; the parse hop `parse_kernel_stub` stays in `ailang-surface`. No `raw_buf` submodule yet — this iteration is *only* the rename, so its failure mode is "existing tests break", the cleanest bisection target. Ratified by the existing suite green. 3. **RawBuf end-to-end** (raw-buf.3) — The feature ship, landing the manifest and the codegen together (the lockstep pair the bijection invariant requires). Adds: - The `raw_buf` submodule (`src/raw_buf/{mod.rs, source.ail}`) with the `RawBuf` TypeDef (`param-in (a Int Float Bool)`) and the four operations `new`/`get`/`set`/`size`, each an `(intrinsic)` marker (NOT a placeholder body). - The 12 element-specialised `INTERCEPTS` entries (4 ops × 3 element types) lowering each via `@ailang_rc_alloc` + getelementptr + load/store — no new C code. The entry `name` of each MUST equal the mono-mangled symbol of its `(intrinsic)` marker; the bijection test is the pin that forces this and surfaces the exact expected strings. - A desugar pass `(new T )` → an instantiated `(app T.new )` that consumes the `NewArg::Type` element type to drive monomorphisation to `new__RawBuf__`, so `Term::New` is eliminated before codegen. `Term::New` becomes a checker-side-only construct (the codegen completion the prep.2 deferral pointed at). raw-buf.3 removes **both** `Term::New` deferral arms codegen currently carries — the `lower_term` arm (`crates/ailang-codegen/src/lib.rs:2094`, "milestone raw-buf" internal error) and the sibling synth-path arm (`lib.rs:3296`). These arms are the deferral marker prep.2 left; raw-buf.3 discharges them. They are not a relied-upon current invariant — the milestone's correctness depends on *removing* them, not on their persisting. - Drop for primitive-element RawBuf: a single `@ailang_rc_release` on the slab pointer (no element-walking; elements are primitives). - `ailang-surface` wiring: `parse_raw_buf` + the workspace injection that makes `raw_buf` a kernel-tier auto-imported module (workspace module count 4 → 5). The worked consumer program runs end-to-end and prints `60`. `kernel_stub` stays alongside this iteration (count 5). 4. **kernel_stub retirement** (raw-buf.4) — Focused removal. RawBuf now subsumes every ratification role kernel_stub held: `Term::New` end-to-end (the `(new RawBuf …)` consumer), `param-in` reject (the Str-element reject fixture), kernel-tier auto-import (RawBuf visible without `(import raw_buf)`), and the `(intrinsic)` mechanism (RawBuf's four ops + their registry entries). The stub is therefore pure redundancy. Delete `src/kernel_stub/` + its re-export, drop `parse_kernel_stub` and its surface injection, remove the `answer` `INTERCEPTS` entry + `emit_answer` (its `(intrinsic)` marker leaves with the stub — the bijection lockstep holds), delete the `kernel_stub_module_round_trips` test, re-baseline `workspace_pin` to 4 (raw_buf in, kernel_stub out — net unchanged), and update `design/INDEX.md` + `design/models/0007-kernel-extensions.md`. The `ailang-kernel` crate itself stays as the family-crate home for future kernel-tier modules (series, matrix, …). Isolating the retirement keeps each bijection-registry change (12 entries added in .3, the `answer` entry removed in .4) in its own diff. The ordering rationale: .2 is the only zero-behavioural-change iteration (rename), the cleanest bisection target; .3 lands the feature on a registry that has already absorbed every legacy intercept (raw-buf.1) and on the renamed family-crate (raw-buf.2), so the RawBuf manifest+codegen do not co-mingle with a registry move or a crate move; .4 removes the now-redundant stub without touching the new RawBuf code. ## Concrete code shapes ### Primary — the raw_buf kernel-tier module (raw-buf.3, gate-verified) This is the exact Form-A source the `raw_buf` submodule embeds. It parses and type-checks against the current tool (`ail check` → `ok`); the four `(intrinsic)` markers are legal because the module is `(kernel)`. The element type is restricted to `{Int, Float, Bool}` via `param-in`. `RawBuf.get` returns the bare element type `a`; `set` is linear (`own` in, `own` out). ```ail (module raw_buf (kernel) (data RawBuf (vars a) (doc "Mutable, indexed, fixed-size flat buffer of primitive elements. Opaque single-ctor handle; codegen intercepts emit raw alloc/load/store over an @ailang_rc_alloc slab. Element type restricted to {Int, Float, Bool} via param-in.") (ctor B a) (param-in (a Int Float Bool))) (fn new (doc "Allocate an uninitialised RawBuf of capacity n. Compiler-supplied (intrinsic) body; codegen intercept new__RawBuf__ emits @ailang_rc_alloc plus the i64 size header. Element bytes are uninitialised; caller must set before get.") (type (forall (vars a) (fn-type (params (con Int)) (ret (own (con RawBuf a)))))) (params n) (intrinsic)) (fn get (doc "Indexed read. UB if i >= (size b); caller checks bounds via size. Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus load.") (type (forall (vars a) (fn-type (params (borrow (con RawBuf a)) (con Int)) (ret a)))) (params b i) (intrinsic)) (fn set (doc "Indexed write. Linear: own in, own out. Under uniqueness in-place; under shared copy-on-write (Issue #22). Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus store.") (type (forall (vars a) (fn-type (params (own (con RawBuf a)) (con Int) a) (ret (own (con RawBuf a)))))) (params b i v) (intrinsic)) (fn size (doc "Element count. Compiler-supplied (intrinsic) body; codegen intercept emits a single i64 load from the slab header.") (type (forall (vars a) (fn-type (params (borrow (con RawBuf a))) (ret (con Int))))) (params b) (intrinsic))) ``` The `(ctor B a)` is a nominal fiction for the type system: it gives `RawBuf` an identity and keeps `a` non-phantom. Real code never constructs it — every operation is codegen-intercepted, and `(new RawBuf …)` desugars to `(app RawBuf.new …)`, not to a `(term-ctor RawBuf B …)`. The four `(intrinsic)` markers carry no runnable body; codegen supplies the implementation via the registry (raw-buf.3). ### Primary — the worked consumer program (raw-buf.3 acceptance) This is the program an LLM author naturally writes for "I need a small mutable indexed buffer to score N values and read them back." It is the feature-acceptance criterion's empirical evidence. It references the `raw_buf` module, which is only injected once raw-buf.3 wires it through `ailang-surface` — so it does **not** check against the current tool (RawBuf is not yet a workspace module). Left unlabeled deliberately: it describes post-raw-buf.3 surface, not current behaviour. ``` (module raw_buf_demo (fn main (type (fn-type (params) (ret (con Int)))) (params) (body (let buf (new RawBuf (con Int) 3) (let buf (app RawBuf.set buf 0 10) (let buf (app RawBuf.set buf 1 20) (let buf (app RawBuf.set buf 2 30) (app + (app RawBuf.get buf 0) (app + (app RawBuf.get buf 1) (app RawBuf.get buf 2)))))))))) ; ail check && ail build && ail run → prints 60 ``` No `(import raw_buf)` needed: `RawBuf` is visible via the kernel-tier auto-import path (prep.3). `RawBuf.set` / `.get` resolve via type-scoped namespacing (prep.1). `(new RawBuf (con Int) 3)` is a `Term::New` whose `NewArg::Type` is `(con Int)`; the raw-buf.3 desugar threads that into `new__RawBuf__Int`. ### Primary — the param-in reject fixture (raw-buf.3 must-fail) ``` (module raw_buf_reject_str (fn cant_str (type (fn-type (params (borrow (con RawBuf (con Str)))) (ret (con Str)))) (params buf) (body (app RawBuf.get buf 0)))) ; ail check → param-not-in-restricted-set: type Str not in {Int, Float, Bool} ``` Must fail at the checker with the `ParamNotInRestrictedSet` diagnostic shipped in prep.3 (fieldtest F8). It is the must-fail evidence that the `param-in` restriction is alive on a real-payload extension, not just the prep.3 `StubT` fixture. Like the worked program, it depends on `raw_buf` being injected, so it is not gate-checkable against the current tool (unlabeled). ### Secondary — INTERCEPTS entry shape (raw-buf.3, implementation) The real registry struct (raw-buf.1) is `Intercept { name, expected_params, expected_ret, wants_alwaysinline, emit }`. The 12 RawBuf entries follow the existing rows' shape. One representative (the exact symbol strings are derived by the implementer from the bijection test, which mangles each `(intrinsic)` marker and reports the expected name): ```rust // crates/ailang-codegen/src/intercepts.rs — appended to INTERCEPTS Intercept { name: "new__RawBuf__Int", // == mono-mangled name of raw_buf.new @ a=Int expected_params: &["i64"], // capacity n expected_ret: "ptr", // slab base pointer wants_alwaysinline: false, emit: emit_new_raw_buf_int, // @ailang_rc_alloc(8 + n*8); store i64 n at offset 0 }, // + new__RawBuf__{Float,Bool}, RawBuf_get__{Int,Float,Bool}, // RawBuf_set__{Int,Float,Bool}, RawBuf_size__{Int,Float,Bool} ``` ### Secondary — Term::New desugar (raw-buf.3, implementation) `Term::New { type_name, args }` where `args: Vec` (`NewArg::Type | NewArg::Value`, both already parsed + round-trip ratified). The new desugar pass rewrites it to an application of the type's `new` fn, instantiated at the element type carried by the leading `NewArg::Type`: ``` (new RawBuf (con Int) 3) ; Term::New { type_name: "RawBuf", ; args: [Type(con Int), Value(3)] } ──desugar──▶ (app RawBuf.new 3) @ a = Int ; instantiation pins mono → new__RawBuf__Int ``` The desugar consumes the `NewArg::Type` for the instantiation and passes only the `NewArg::Value` args as call arguments. After this pass no `Term::New` survives into codegen; both deferral arms (`lib.rs:2094` in `lower_term`, "milestone raw-buf" internal error, and the sibling at `lib.rs:3296` in the synth path) become unreachable and are removed. Whether the prep.2 checker already accepts the `NewArg::Type`-bearing `Term::New` semantically, or needs a small extension, is a raw-buf.3 task to verify first (the form parses + round-trips today; the lift pass passes `NewArg::Type` through untouched). ### Secondary — slab layout (raw-buf.3) ``` [ size : i64 ][ elem_0 ][ elem_1 ] ... [ elem_{size-1} ] ^ ^ ^ offset 0 offset 8 offset 8 + size*sizeof(T) ``` Element widths: `Int = 8`, `Float = 8`, `Bool = 1`. Header is i64, 8-byte aligned; element payload follows directly. `@ailang_rc_alloc` is called with `8 + size * sizeof(T)` and returns the slab base `ptr`. The size header is self-describing, so drop and `RawBuf.size` need no side-table. Drop is a single `@ailang_rc_release` on the slab pointer — primitive elements carry no recursive drops. The per-TypeDef drop generation gets a "primitive-element-RawBuf" arm dispatched via the same registry. ## Components | Component | Iter | Files | |-----------|------|-------| | Intercept registry | raw-buf.1 (DONE) | `crates/ailang-codegen/src/intercepts.rs` + dispatch site in `lib.rs` | | Kernel family-crate rename | raw-buf.2 | `git mv crates/ailang-kernel-stub crates/ailang-kernel`; reshape to `src/lib.rs` + `src/kernel_stub/{mod.rs, source.ail}` (existing stub source incl. `answer` re-housed verbatim); `Cargo.toml` package + workspace member + dep-alias renames; `crates/ailang-surface/Cargo.toml` dep rename + `loader.rs` use-path renames; `design/INDEX.md` + `CLAUDE.md` code-layout row | | RawBuf module + codegen + desugar + drop | raw-buf.3 | new `crates/ailang-kernel/src/raw_buf/{mod.rs, source.ail}` + `lib.rs` re-export; `parse_raw_buf` + workspace injection in `crates/ailang-surface/src/loader.rs` + re-export in `lib.rs`; 12 entries + emit fns in `crates/ailang-codegen/src/intercepts.rs`; `Term::New` desugar in `crates/ailang-core/src/desugar.rs` (and removal of the codegen `Term::New` error arm in `crates/ailang-codegen/src/lib.rs`); primitive-element-RawBuf drop arm; `workspace_pin` 4 → 5; round-trip + E2E tests | | kernel_stub retirement | raw-buf.4 | delete `crates/ailang-kernel/src/kernel_stub/` + its `lib.rs` re-export; drop `parse_kernel_stub` + injection in `ailang-surface`; remove `answer` entry + `emit_answer` in `intercepts.rs`; delete `kernel_stub_module_round_trips` test; `workspace_pin` 5 → 4; `design/INDEX.md` + `design/models/0007-kernel-extensions.md` updates | ## Data flow **Build-time** (raw-buf.3 onwards): 1. CLI loads workspace → `ailang-surface` injects `raw_buf` as a kernel-tier module (`kernel: true`) just like `prelude`. 2. `check` walks consumer code; `(con RawBuf (con Int))` resolves to `raw_buf.RawBuf` via the kernel-tier auto-import path (prep.3). 3. `param-in` on RawBuf's `a` rejects non-{Int,Float,Bool} instantiations at the consumer typecheck site, using the existing `ParamNotInRestrictedSet` diagnostic. **Codegen-time** (raw-buf.3 onwards): 1. The desugar pass replaces every `Term::New { type_name, args }` with an instantiated `(app .new )`, pinning the element type from the leading `NewArg::Type`. 2. For each fn, codegen reaches the registry dispatch site. If the fn's mono name resolves via `intercepts::lookup`, the entry's `emit` runs; the `(intrinsic)` marker carries no body to fall back to. Otherwise the normal AILang lowering runs. 3. For RawBuf consumers, `new__RawBuf__Int` emits the alloc; `RawBuf_set__Int` a `store i64`; `RawBuf_get__Int` a `load i64`; `RawBuf_size__Int` a single header load. **Bijection interaction** (raw-buf.3 / .4): Each RawBuf `(intrinsic)` marker reachable from the workspace is collected by `workspace_intrinsic_markers()` and must resolve to an `INTERCEPTS` entry (and vice versa, minus the `OPTIMISATION_ONLY` allowlist). raw-buf.3 adds 12 markers + 12 entries together. raw-buf.4 removes the `answer` marker (with the stub) and the `answer` entry together. A half-landed change on either side is a hard unit-test failure (`intercepts_bijection_with_intrinsic_markers`). **Workspace module count:** prelude + kernel_stub + 2 test fixtures = 4 today. raw-buf.3 adds raw_buf → 5. raw-buf.4 retires kernel_stub → 4. The `workspace_pin` test pins each transition. ## Error handling Two consumer-facing diagnostics; both already shipped, no new arms needed: - **`param-not-in-restricted-set`** (prep.3, fieldtest F8). Fires when a consumer writes `(con RawBuf (con Str))` or any non-{Int,Float,Bool} element. Names the type, the variable, the TypeDef, and the restricted set. - **`new-type-not-constructible`** (prep.3, fieldtest F5). Fires when a consumer writes `(new SomeType …)` and `SomeType` has no `(fn new …)` in its home module. raw_buf ships `(fn new …)`, so the worked program passes; the diagnostic still protects future authors who write `(new Counter 0)` against a `new`-less type. The codegen `Term::New` deferral arms (`lib.rs:2094` in `lower_term`, message "… milestone raw-buf"; and the sibling at `lib.rs:3296` in the synth path) are **removed** in raw-buf.3 once the desugar eliminates `Term::New` before codegen. Neither is a user-facing diagnostic — both are the deferral marker prep.2 left, which raw-buf.3 discharges. They have no test pinning them today (they are an internal `CodegenError::Internal`, not an asserted behaviour); raw-buf.3's end-to-end build-and-run E2E (the worked program printing `60`) is the forward ratification — it exercises the `Term::New` → desugar → codegen path that replaces them. Bounds checks are **not** in scope (whitepaper § "RawBuf — the base extension"): `RawBuf.get/.set` are UB if `i >= size(b)`; caller checks via `RawBuf.size`. Bounds-check insertion is a future axis, no backlog item yet. ## Testing strategy **raw-buf.2 (rename):** - All existing workspace tests pass unchanged — the rename is a pure refactor. Zero test-count change. The crate-rename's failure mode is "the workspace does not build / existing tests break", caught by `cargo build --workspace` + `cargo test --workspace`. **raw-buf.3 (RawBuf end-to-end):** - Round-trip: `raw_buf_module_round_trips` (mirror of `kernel_stub_module_round_trips` in `design_schema_drift.rs`) — parse → serialise → round-trip; assert `kernel == true`, name `raw_buf`, and `RawBuf` TypeDef `param_in` for `a` contains all of Int/Float/Bool. - E2E `raw_buf_int_e2e`: the worked consumer program — `ail check && ail build && ail run` → prints `60`. - E2E `raw_buf_float_e2e` / `raw_buf_bool_e2e`: Float and Bool element variants (Bool catches any i1/i8 packing question). - E2E `raw_buf_param_in_reject_e2e`: the Str-element reject fixture — `ail check` exits non-zero with `param-not-in-restricted-set` in stderr. - Drop ratification: a consumer that allocates a RawBuf and lets it go out of scope without a leak — re-uses the existing leak-check harness, no new infra. - Bijection: `intercepts_bijection_with_intrinsic_markers` stays green with the 12 new markers + 12 new entries (the test forces the exact mangled-name match). - `workspace_pin` count assertion 4 → 5 (+ `contains_key("raw_buf")`). **raw-buf.4 (retirement):** - `workspace_pin` re-baselines 5 → 4 (raw_buf in, kernel_stub out); `contains_key("kernel_stub")` assertion removed, `contains_key("raw_buf")` retained. - `kernel_stub_module_round_trips` deleted (its `parse_kernel_stub` callee is gone; `raw_buf_module_round_trips` carries the round-trip invariant forward). - Bijection stays green after the `answer` marker + entry leave together. - The full suite stays green; the diff is a removal, ratified by "nothing references the stub and everything still builds". ## Acceptance criteria The milestone ships when: 1. The worked consumer program runs end-to-end: `ail check && ail build && ail run` prints `60` (raw-buf.3). 2. The param-in reject fixture exits at `ail check` with the prep.3 `param-not-in-restricted-set` diagnostic (raw-buf.3). 3. RawBuf's four operations are `(intrinsic)` markers, each paired with an `INTERCEPTS` entry; `intercepts_bijection_with_intrinsic_markers` is green (raw-buf.3). 4. `Term::New` no longer reaches codegen — the desugar eliminates it and both deferral arms (`lib.rs:2094` + `lib.rs:3296`) are removed (raw-buf.3). 5. `ailang-kernel-stub` is renamed to the `ailang-kernel` family-crate (raw-buf.2); `kernel_stub` is retired and `raw_buf` takes over its ratification roles (raw-buf.4). 6. Workspace test suite green at each iteration boundary, plus the new tests enumerated in § Testing strategy. ## Feature-acceptance argument Applied per `design/contracts/0004-feature-acceptance.md`: 1. **LLM author reaches for it naturally.** RawBuf is the only path to mutable indexed storage in AILang — the language has no other "fill N slots and read them back" primitive. The worked consumer program above is what an LLM author writes for that need; there is no alternative shape to reach for. 2. **Measurable improvement.** Embedding-ABI M5 measured per-tick FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks). RawBuf as the batch-FFI primitive amortises that per-tick cost across one crossing. The Series milestone (#8) is also blocked on RawBuf; shipping raw-buf removes that block. 3. **No reintroduced bug class.** Mutation discipline stays mode-tracked (whitepaper § "RawBuf — the base extension"): `RawBuf.set` is `own → own`, not effect-tracked. No `RawBuf` effect is declared. Uniqueness inference rewrites in-place; shared ownership falls back to copy-on-write (Issue #22). The pure-core invariant survives: code without a declared effect is pure. Local reasoning survives: a fn taking `borrow (RawBuf Int)` and returning `Int` is pure from the caller's view, because the buffer is not mutated.