# raw-buf — Design Spec

**Date:** 2026-05-29
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

## Goal

Ship `RawBuf a` — the first kernel-tier *base* extension — as a
mutable, indexed, bounded-size flat buffer of primitive elements
(`{Int, Float, Bool}`). RawBuf is the storage primitive that
unblocks two downstream needs both already captured in the
backlog:

- **Series (milestone #8)** — a bounded ring buffer with
  financial-style indexing, implemented as a *library extension*
  (pure AILang) wrapping `RawBuf a` plus bookkeeping fields.
- **Embedding-ABI batch FFI (subsumes closed #2)** — the M5
  friction-harvest measured per-tick FFI at ~206 ns/tick on real
  EURUSD volume (~658 ms / 3.19M ticks). A contiguous primitive
  slice amortises that per-tick cost via a batch crossing.
  RawBuf is that primitive.

The milestone also fulfils the architectural commitment from the
kernel-extensions whitepaper § "Plugin contract":

> *The migration of existing hardcoded primitive-instance
> intercepts (`try_emit_primitive_instance_body`) into the plugin
> registry is triggered by the first real base extension shipping
> (RawBuf). One registry mechanism for all intercepts.*

This makes raw-buf both a feature ship (a new type LLM authors
can reach for) and a foundation move (the *one* plumbing path
for all current and future codegen intercepts).

## Architecture

Three layered moves, one per iteration:

1. **Intercept registry** (raw-buf.1) — Lift the existing
   hard-coded `try_emit_primitive_instance_body` match
   (`eq__Str`, `compare__Int|Bool|Str`, plus the `float_*`
   family that lowers to `fcmp`) into a registry table
   `&[Intercept { name, sig, emit }]`. The single existing
   dispatch site in `lower_fn_body` (the `emit_fn` caller
   that today reaches `try_emit_primitive_instance_body`)
   becomes a table lookup. Zero behavioural change; every
   existing test stays green. This iter is a pure refactor
   — its ratification is the workspace test suite as-is.

2. **RawBuf kernel-tier manifest + checker side** (raw-buf.2) —
   Rename `crates/ailang-kernel-stub/` to `crates/ailang-kernel/`
   and re-shape it as a family-crate: one Cargo crate hosting
   *all* kernel-tier modules as sibling Rust submodules under
   `src/`, one submodule per `.ail` source. The renamed crate
   carries `src/kernel_stub/{mod.rs, source.ail}` (the existing
   stub, unchanged) and gains `src/raw_buf/{mod.rs, source.ail}`
   (the new RawBuf module). `src/lib.rs` re-exports each
   submodule's `SOURCE` const under a stable name
   (`pub use kernel_stub::SOURCE as STUB_AIL;`,
   `pub use raw_buf::SOURCE as RAW_BUF_AIL;`), preserving the
   public-symbol surface every consumer already binds to. The
   parse hops stay in `ailang-surface` (`parse_kernel_stub`
   joined by `parse_raw_buf`); the acyclic dependency chain
   `ailang-surface → ailang-kernel → ailang-core` is preserved.
   The raw_buf module declares:
   - `(data RawBuf (vars a) (ctor B (storage-tag)) (param-in
     (a Int Float Bool)))` — opaque single-ctor TypeDef with
     element-type restriction.
   - `(fn new ...)`, `(fn get ...)`, `(fn set ...)`,
     `(fn size ...)` with type signatures from the whitepaper
     and placeholder bodies (same convention as the existing
     `eq__Int` / `compare__Int` intercept stubs in
     `examples/prelude.ail`).
   Checker accepts a consumer-fixture `(con RawBuf (con Int))`
   in a type slot; `ail check` passes. `ail build` fails
   precisely with `intercept-not-registered: new__RawBuf__Int`
   (or the existing Term::New-deferral diagnostic), because the
   registry has no entry yet — that diagnostic IS the
   ratification.

3. **RawBuf codegen intercepts + Term::New desugar + stub
   retirement** (raw-buf.3) — Register the 12 element-type-
   specialised entries (`new__RawBuf__Int|Float|Bool`,
   `RawBuf_get__Int|Float|Bool`, `RawBuf_set__Int|Float|Bool`,
   `RawBuf_size__Int|Float|Bool`) into the registry. Lower
   each via `@ailang_rc_alloc` + getelementptr + load/store —
   no new C code. Add a desugar pass `(new T args)` →
   `(app T.new args)` so Term::New is eliminated before
   codegen — Term::New becomes purely a checker-side construct
   (the desugar is the codegen completion the prep.2 deferral
   pointed at). Worked consumer program runs end-to-end. Retire
   the `kernel_stub` submodule from `crates/ailang-kernel/`
   since RawBuf is now the second-and-real kernel-tier
   consumer that CLAUDE.md names as the stub's retirement
   trigger: delete `src/kernel_stub/` and its `lib.rs`
   re-export, drop `parse_kernel_stub` and its surface
   injection in `ailang-surface`, and update INDEX +
   whitepaper to reflect.

The ordering is not arbitrary: raw-buf.1 is the only iter with
zero behavioural change, so its failure mode is "existing tests
break" — the cleanest possible bisection target. raw-buf.2 ships
the checker-visible surface on an unchanged codegen substrate,
so its failure mode is checker-isolated. raw-buf.3 lands codegen
on a registry that has already absorbed every legacy intercept,
so the RawBuf entries do not co-mingle with a registry move.

## Concrete code shapes

### Primary — the user-facing program raw-buf delivers

```
(module raw_buf_demo
  (import raw_buf)
  (fn main
    (type (fn-type (params) (ret (con Int))))
    (params)
    (body
      (let buf (new RawBuf (con Int) 3)
        (let buf (app RawBuf.set buf 0 10)
          (let buf (app RawBuf.set buf 1 20)
            (let buf (app RawBuf.set buf 2 30)
              (app + (app RawBuf.get buf 0)
                (app + (app RawBuf.get buf 1)
                  (app RawBuf.get buf 2))))))))))
; ail check && ail build && ail run → prints 60
```

This is the iter raw-buf.3 acceptance fixture (and the
feature-acceptance criterion's empirical evidence — see § Feature
acceptance below). It is exactly the program an LLM author
naturally writes to express "I need a small mutable indexed
buffer to score N values and read them back."

### Primary — the param-in reject fixture

```
(module raw_buf_reject_string
  (import raw_buf)
  (fn main
    (type (fn-type (params) (ret (con Unit))))
    (params)
    (body (do io/print_str "unreached"))
    (signature-hole (new RawBuf (con Str) 3))))   ; rejected at check
; ail check → param-not-in-restricted-set: type Str not in {Int, Float, Bool}
;             at TypeDef raw_buf.RawBuf, var a
```

This must fail at the checker, with the diagnostic produced by
the existing `ParamNotInRestrictedSet` arm shipped in
prep.3 (per fieldtest F8). It is the must-fail evidence that the
`param-in` restriction is alive on a real-payload extension, not
just on the prep.3 stub.

### Primary — the raw-buf kernel-tier module

```
(module raw_buf
  (kernel)
  (data RawBuf (vars a)
    (ctor B (storage-tag))
    (param-in (a Int Float Bool)))

  (fn new
    (doc "Allocate uninitialised RawBuf of capacity n. Codegen
          intercept new__RawBuf__<T> emits @ailang_rc_alloc + the
          size header. Element bytes are uninitialised — caller
          must (set …) before (get …).")
    (type (fn-type (params (con Int)) (ret (own (RawBuf a)))))
    (params n)
    (body (term-ctor RawBuf B n)))                ; placeholder

  (fn get
    (doc "Indexed read. UB if i >= (size b); caller checks
          bounds. Codegen intercept RawBuf_get__<T> emits
          getelementptr + load.")
    (type (fn-type
            (params (borrow (RawBuf a)) (con Int))
            (ret (con a))))
    (params b i)
    (body (term-ctor RawBuf B 0)))                ; placeholder

  (fn set
    (doc "Indexed write. Linear: own in, own out. Under
          uniqueness, in-place; under shared, copy-on-write
          (Issue #22 territory). Codegen intercept
          RawBuf_set__<T> emits getelementptr + store.")
    (type (fn-type
            (params (own (RawBuf a)) (con Int) (con a))
            (ret (own (RawBuf a)))))
    (params b i v)
    (body b))                                     ; placeholder

  (fn size
    (doc "Element count. Codegen intercept RawBuf_size__<T>
          emits a single i64 load from the slab header.")
    (type (fn-type (params (borrow (RawBuf a))) (ret (con Int))))
    (params b)
    (body 0)))                                    ; placeholder
```

The placeholder bodies are the existing convention from
`examples/prelude.ail` (e.g. `eq__Int` ships with `(body x)`
that the round-trip preserves but codegen replaces). They exist
to satisfy the round-trip invariant; codegen never executes them.

### Secondary — registry shape (implementation only)

```rust
// crates/ailang-codegen/src/intercepts.rs (new file)

pub struct IntrinsicSig {
    pub params: Vec<String>,   // LLVM types as text, e.g. "i64", "ptr"
    pub ret: String,
}

pub struct Intercept {
    pub name: &'static str,
    pub sig_check: fn(params: &[String], ret: &str) -> Result<()>,
    pub emit: fn(cg: &mut Lowering) -> Result<()>,
}

pub fn lookup(name: &str) -> Option<&'static Intercept> { ... }

// Wired into crates/ailang-codegen/src/lib.rs:1321 (the existing
// try_emit dispatch site) — that match becomes one lookup call.
```

### Secondary — slab layout

RawBuf payload (post RC header) is:

```
  [ size : i64 ][ elem_0 ][ elem_1 ] ... [ elem_{size-1} ]
  ^             ^                                          ^
  offset 0      offset 8                                   offset 8 + size*sizeof(T)
```

Element widths: `Int = 8`, `Float = 8`, `Bool = 1`. Alignment to
8 bytes for header; element payload directly follows. `@ailang_rc_alloc`
is called with `8 + size * sizeof(T)` and gets back a `ptr` that
the codegen treats as the slab base. The size header is *self-
describing* so drop and `RawBuf.size` need no side-table.

Drop for RawBuf with primitive elements is trivial: a single
`@ailang_rc_release` on the slab pointer. No element-walking
because element types are primitives (no recursive drops). The
existing per-TypeDef drop generation gets a "primitive-element-
RawBuf" arm that emits the trivial drop, dispatched via the same
registry.

## Components

| Component | Iter | Files |
|-----------|------|-------|
| Intercept registry | raw-buf.1 | new file `crates/ailang-codegen/src/intercepts.rs`; edits to `crates/ailang-codegen/src/lib.rs` at the single dispatch site that today calls `try_emit_primitive_instance_body`, and removal of each absorbed match arm in that fn |
| Kernel family-crate (rename + reshape) + RawBuf submodule | raw-buf.2 | rename `crates/ailang-kernel-stub/` → `crates/ailang-kernel/`; restructure as `src/lib.rs` + `src/kernel_stub/{mod.rs, source.ail}` (existing stub re-housed) + `src/raw_buf/{mod.rs, source.ail}` (new); update `Cargo.toml` package name `ailang-kernel-stub` → `ailang-kernel`; update workspace member entry + every dependent crate's `Cargo.toml` (`ailang-surface` is the only consumer today); add `parse_raw_buf` in `crates/ailang-surface/` next to existing `parse_kernel_stub`; extend the workspace injection in `ailang-surface` to inject both modules |
| RawBuf intercept entries | raw-buf.3 | additions to `crates/ailang-codegen/src/intercepts.rs` (12 entries); desugar pass `Term::New` → `(app T.new args)` in `crates/ailang-codegen/src/synth.rs` or `crates/ailang-check/src/desugar.rs` |
| Stub retirement (submodule level) | raw-buf.3 (close) | delete `crates/ailang-kernel/src/kernel_stub/` + its re-export in `src/lib.rs`; drop `parse_kernel_stub` and its surface injection in `ailang-surface` (defn at `crates/ailang-surface/src/loader.rs:54`, re-export at `crates/ailang-surface/src/lib.rs:39`); delete the `kernel_stub_module_round_trips` test at `crates/ailang-core/tests/design_schema_drift.rs:743` (its `parse_kernel_stub` callee is gone — the analogous `raw_buf_module_round_trips` test added in raw-buf.2 carries the round-trip invariant forward); update `crates/ailang-core/tests/workspace_pin.rs` (was bumped to 4 in iter-tidy `a844de3`; raw_buf joins as the stub leaves so the count stays at 4 — see Data flow); update `design/INDEX.md` § Models row + `design/models/0007-kernel-extensions.md` § "ratifying fixture" sentence. The `ailang-kernel` crate itself stays — it is the family-crate that future kernel-tier modules (series, matrix, …) will plug into. |

## Data flow

**Build-time** (raw-buf.2 onwards):
1. CLI loads workspace → `ailang-surface` injects `raw_buf` as
   a kernel-tier module (`kernel: true`) just like `prelude`
   today.
2. `check` walks consumer code; `(con RawBuf (con Int))`
   resolves to `raw_buf.RawBuf` via the kernel-tier auto-import
   path that prep.3 shipped.
3. `param-in` restriction on RawBuf's `a` parameter rejects
   non-{Int,Float,Bool} instantiations at the consumer
   typecheck site, using the existing
   `ParamNotInRestrictedSet` diagnostic from prep.3.

**Codegen-time** (raw-buf.3 onwards):
1. Desugar pass replaces every `Term::New {type_name, args}`
   with `Term::App { callee: <type_name>.new, args }` — same
   form that already lowers cleanly today.
2. For each fn in the workspace, codegen reaches the dispatch
   site at `lib.rs:1321`. If the fn's mono name has a registry
   entry, the entry's `emit` runs; the `.ail` placeholder body
   is discarded. Otherwise, the normal AILang lowering runs.
3. For RawBuf consumers, `new__RawBuf__Int` emits the alloc
   call; `RawBuf_set__Int` emits a `store i64`;
   `RawBuf_get__Int` emits a `load i64`; `RawBuf_size__Int`
   emits a single header load.

**Workspace-module-count interaction** (raw-buf.3 close):
The current count of 4 (`prelude` + `kernel_stub` + two
test-fixture modules per the iter-tidy commit). When
`kernel_stub` retires and `raw_buf` joins, the count stays 4.
If `raw_buf` lands *before* the stub retires, the count is 5
mid-iter — caught by the existing `workspace_pin` test, which
the iter explicitly re-baselines.

## Error handling

Two diagnostics; both already shipped, no new arms needed in
this milestone:

- **`param-not-in-restricted-set`** (prep.3, fieldtest F8).
  Fires when a consumer writes `(con RawBuf (con Str))` or
  any non-{Int,Float,Bool} element. Names the type, the
  variable, the TypeDef, and the restricted set.

- **`new-type-not-constructible`** (prep.3, fieldtest F5).
  Fires when a consumer writes `(new SomeType …)` and
  `SomeType` has no `(fn new …)` in its home module. Since
  raw-buf ships `(fn new …)` in the kernel-tier `raw_buf`
  module, the consumer program above passes — but if a
  future LLM author writes `(new Counter 0)` against a
  type that lacks `new`, this diagnostic still fires
  precisely.

Bounds checks are *not* in scope. Per whitepaper § "RawBuf —
the base extension": `RawBuf.get / .set` are UB if `i >=
size(b)`; caller checks. Bounds-check insertion is a future
axis tracked separately (no backlog item yet — the discipline
is "caller checks via `RawBuf.size`").

A future `intercept-not-registered` diagnostic surfaces at
raw-buf.2 — the iter raw-buf.2 acceptance is precisely that
this fires (consumer code checks clean but does not build),
demonstrating the manifest/codegen separation works. raw-buf.3
makes it stop firing for the 12 RawBuf entries.

## Testing strategy

Each iter ratified independently:

**raw-buf.1 (registry refactor):**
- All ~668 existing workspace tests pass unchanged. The
  refactor is *primarily* ratified by zero behavioural
  change in the existing suite — the legacy intercepts
  (`eq__Str`, `compare__Int|Bool|Str`, the `float_*` family)
  are exercised end-to-end by the following build-and-run
  E2E tests:
  - `eq_primitives_smoke_compiles_and_runs` and
    `compare_primitives_smoke_prints_1_2_3_thrice` in
    `crates/ail/tests/e2e.rs` — pin the `eq__Str` and
    `compare__Int|Bool|Str` arms (Str path included).
  - `float_compare_smoke_prints_true_true_false` in
    `crates/ail/tests/float_compare_smoke_e2e.rs` — pins
    the `float_*` family via `float_eq` + `float_lt`.
  - `eq_ord_polymorphic_runs_end_to_end` in
    `crates/ail/tests/eq_ord_e2e.rs` — secondary Str-path
    ratifier via polymorphic Eq/Ord dispatch.
  If the registry rewrite breaks any intercept, those tests
  catch it.
- One new unit test in the new `intercepts` module:
  `registry_contains_all_legacy_arms` enumerates the names
  that were in the legacy match (`eq__Str`, `compare__Int`,
  `compare__Bool`, `compare__Str`, and each `float_*` name)
  and asserts each resolves to a registered entry. This pins
  the migration as complete (a half-finished migration where
  the old match still has an arm becomes a hard test fail).

**raw-buf.2 (manifest + checker):**
- New E2E test `crates/ail/tests/raw_buf_check_e2e.rs`:
  consumer fixture with `(con RawBuf (con Int))` in a type
  signature, `ail check` exits 0.
- New E2E test `crates/ail/tests/raw_buf_param_in_reject_e2e.rs`:
  consumer fixture with `(con RawBuf (con Str))`, `ail check`
  exits non-zero with `param-not-in-restricted-set` in stderr.
- New E2E test `crates/ail/tests/raw_buf_build_intercept_missing_e2e.rs`:
  consumer fixture with `(new RawBuf (con Int) 3)`, `ail
  build` exits non-zero with `intercept-not-registered` (or
  the Term::New deferral message) — pins the manifest/codegen
  separation. (This test gets deleted or inverted to PASS in
  raw-buf.3.)
- New round-trip test for the embedded `RAW_BUF_AIL` —
  parse → print round-trips byte-isomorphically (mirrors
  `kernel_stub_module_round_trips` in
  `crates/ailang-core/tests/design_schema_drift.rs`).
- Workspace module-count test bumped: 4 → 5 (mid-iter state
  before stub retires).

**raw-buf.3 (codegen + stub retirement):**
- The intercept-missing E2E test from raw-buf.2 inverted:
  same consumer fixture, `ail build` now exits 0 and the
  produced binary, when run, prints `60`.
- New E2E `crates/ail/tests/raw_buf_float_e2e.rs`: Float
  variant exercise.
- New E2E `crates/ail/tests/raw_buf_bool_e2e.rs`: Bool
  variant exercise (catches the i1/i8 packing question
  if any).
- Drop ratification: a consumer that allocates and lets
  the RawBuf go out of scope without a leak. Re-uses the
  existing leak-check harness; no new infra.
- Stub-retirement ratification: deleted-submodule diff hits
  `crates/ailang-kernel/src/kernel_stub/` (folder removal) +
  `crates/ailang-kernel/src/lib.rs` (re-export drop) +
  `crates/ailang-surface/` (parse_kernel_stub + injection
  drop) + `crates/ailang-core/tests/workspace_pin.rs` +
  `design/INDEX.md` + `design/models/0007-kernel-extensions.md`.
  Test `workspace_pin` re-baselines to 4 (raw_buf in,
  kernel_stub out — net unchanged). The `ailang-kernel` crate
  itself stays as the family-crate home for future kernel-tier
  modules (series, matrix, …).

## Acceptance criteria

The milestone ships when:

1. The worked consumer program in § Concrete code shapes
   end-to-end: `ail check && ail build && ail run` prints
   `60`.
2. The param-in reject fixture exits at `ail check` with the
   prep.3 diagnostic.
3. Every existing primitive-instance intercept (`eq__Str`,
   `compare__Int|Bool|Str`, the `float_*` family) is dispatched
   via the registry — *no remaining match-arm* in
   `try_emit_primitive_instance_body` outside the registry path.
4. `ailang-kernel-stub` is retired from the workspace; `raw_buf`
   takes its slot as the canonical kernel-tier base-extension
   anchor.
5. Workspace test suite green, plus the new tests enumerated
   in § Testing strategy.

## Feature-acceptance argument

Applied per `design/contracts/0004-feature-acceptance.md`:

1. **LLM author reaches for it naturally.** RawBuf is the
   only path to mutable indexed storage in AILang — the
   language has no other "I want to fill N slots and read
   them back" primitive. The worked consumer program above is
   what an LLM author writes when expressing that need;
   there is no alternative shape to reach for.

2. **Measurable improvement.** Embedding-ABI M5 measured
   per-tick FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks).
   RawBuf as the batch-FFI primitive amortises that per-tick
   cost across one crossing. The series milestone (#8) is also
   blocked on RawBuf; shipping raw-buf removes that block.

3. **No reintroduced bug class.** Mutation discipline stays
   mode-tracked (whitepaper § "RawBuf — the base extension"):
   `RawBuf.set` is `own → own`, not effect-tracked. No
   `RawBuf` effect declared. Uniqueness inference rewrites it
   in-place; shared ownership falls back to copy-on-write
   (Issue #22). The pure-core invariant — code without a
   declared effect is pure — survives intact. The local-
   reasoning invariant survives too: a fn that takes
   `borrow (RawBuf Int)` and returns `Int` *is* pure from the
   caller's view, because the buffer is not mutated.

The registry-migration aspect of the milestone (raw-buf.1)
removes redundancy structurally: the growth point of
"every new primitive intercept needs another match arm" is
collapsed into a single dispatch table. That is criterion-2
evidence for the migration as a piece.