AILang/docs/specs/0054-raw-buf.md

raw-buf — Design Spec

Date: 2026-05-29 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Goal

Ship RawBuf a — the first kernel-tier base extension — as a mutable, indexed, bounded-size flat buffer of primitive elements ({Int, Float, Bool}). RawBuf is the storage primitive that unblocks two downstream needs both already captured in the backlog:

• Series (milestone #8) — a bounded ring buffer with financial-style indexing, implemented as a library extension (pure AILang) wrapping RawBuf a plus bookkeeping fields.
• Embedding-ABI batch FFI (subsumes closed #2) — the M5 friction-harvest measured per-tick FFI at ~206 ns/tick on real EURUSD volume (~658 ms / 3.19M ticks). A contiguous primitive slice amortises that per-tick cost via a batch crossing. RawBuf is that primitive.

The milestone also fulfils the architectural commitment from the kernel-extensions whitepaper § "Plugin contract":

The migration of existing hardcoded primitive-instance intercepts (try_emit_primitive_instance_body) into the plugin registry is triggered by the first real base extension shipping (RawBuf). One registry mechanism for all intercepts.

This makes raw-buf both a feature ship (a new type LLM authors can reach for) and a foundation move (the one plumbing path for all current and future codegen intercepts).

Architecture

Three layered moves, one per iteration:

1. Intercept registry (raw-buf.1) — Lift the existing hard-coded try_emit_primitive_instance_body match (eq__Str, compare__Int|Bool|Str, plus the float_* family that lowers to fcmp) into a registry table &[Intercept { name, sig, emit }]. The single existing dispatch site in lower_fn_body (the emit_fn caller that today reaches try_emit_primitive_instance_body) becomes a table lookup. Zero behavioural change; every existing test stays green. This iter is a pure refactor — its ratification is the workspace test suite as-is.

2. RawBuf kernel-tier manifest + checker side (raw-buf.2) — Rename crates/ailang-kernel-stub/ to crates/ailang-kernel/ and re-shape it as a family-crate: one Cargo crate hosting all kernel-tier modules as sibling Rust submodules under src/, one submodule per .ail source. The renamed crate carries src/kernel_stub/{mod.rs, source.ail} (the existing stub, unchanged) and gains src/raw_buf/{mod.rs, source.ail} (the new RawBuf module). src/lib.rs re-exports each submodule's SOURCE const under a stable name (pub use kernel_stub::SOURCE as STUB_AIL;, pub use raw_buf::SOURCE as RAW_BUF_AIL;), preserving the public-symbol surface every consumer already binds to. The parse hops stay in ailang-surface (parse_kernel_stub joined by parse_raw_buf); the acyclic dependency chain ailang-surface → ailang-kernel → ailang-core is preserved. The raw_buf module declares:

   • (data RawBuf (vars a) (ctor B (storage-tag)) (param-in (a Int Float Bool))) — opaque single-ctor TypeDef with element-type restriction.
   • (fn new ...), (fn get ...), (fn set ...), (fn size ...) with type signatures from the whitepaper and placeholder bodies (same convention as the existing eq__Int / compare__Int intercept stubs in examples/prelude.ail). Checker accepts a consumer-fixture (con RawBuf (con Int)) in a type slot; ail check passes. ail build fails precisely with intercept-not-registered: new__RawBuf__Int (or the existing Term::New-deferral diagnostic), because the registry has no entry yet — that diagnostic IS the ratification.

3. RawBuf codegen intercepts + Term::New desugar + stub retirement (raw-buf.3) — Register the 12 element-type- specialised entries (new__RawBuf__Int|Float|Bool, RawBuf_get__Int|Float|Bool, RawBuf_set__Int|Float|Bool, RawBuf_size__Int|Float|Bool) into the registry. Lower each via @ailang_rc_alloc + getelementptr + load/store — no new C code. Add a desugar pass (new T args) → (app T.new args) so Term::New is eliminated before codegen — Term::New becomes purely a checker-side construct (the desugar is the codegen completion the prep.2 deferral pointed at). Worked consumer program runs end-to-end. Retire the kernel_stub submodule from crates/ailang-kernel/ since RawBuf is now the second-and-real kernel-tier consumer that CLAUDE.md names as the stub's retirement trigger: delete src/kernel_stub/ and its lib.rs re-export, drop parse_kernel_stub and its surface injection in ailang-surface, and update INDEX + whitepaper to reflect.

The ordering is not arbitrary: raw-buf.1 is the only iter with zero behavioural change, so its failure mode is "existing tests break" — the cleanest possible bisection target. raw-buf.2 ships the checker-visible surface on an unchanged codegen substrate, so its failure mode is checker-isolated. raw-buf.3 lands codegen on a registry that has already absorbed every legacy intercept, so the RawBuf entries do not co-mingle with a registry move.

Concrete code shapes

Primary — the user-facing program raw-buf delivers

(module raw_buf_demo
  (import raw_buf)
  (fn main
    (type (fn-type (params) (ret (con Int))))
    (params)
    (body
      (let buf (new RawBuf (con Int) 3)
        (let buf (app RawBuf.set buf 0 10)
          (let buf (app RawBuf.set buf 1 20)
            (let buf (app RawBuf.set buf 2 30)
              (app + (app RawBuf.get buf 0)
                (app + (app RawBuf.get buf 1)
                  (app RawBuf.get buf 2))))))))))
; ail check && ail build && ail run → prints 60

This is the iter raw-buf.3 acceptance fixture (and the feature-acceptance criterion's empirical evidence — see § Feature acceptance below). It is exactly the program an LLM author naturally writes to express "I need a small mutable indexed buffer to score N values and read them back."

Primary — the param-in reject fixture

(module raw_buf_reject_string
  (import raw_buf)
  (fn main
    (type (fn-type (params) (ret (con Unit))))
    (params)
    (body (do io/print_str "unreached"))
    (signature-hole (new RawBuf (con Str) 3))))   ; rejected at check
; ail check → param-not-in-restricted-set: type Str not in {Int, Float, Bool}
;             at TypeDef raw_buf.RawBuf, var a

This must fail at the checker, with the diagnostic produced by the existing ParamNotInRestrictedSet arm shipped in prep.3 (per fieldtest F8). It is the must-fail evidence that the param-in restriction is alive on a real-payload extension, not just on the prep.3 stub.

Primary — the raw-buf kernel-tier module

(module raw_buf
  (kernel)
  (data RawBuf (vars a)
    (ctor B (storage-tag))
    (param-in (a Int Float Bool)))

  (fn new
    (doc "Allocate uninitialised RawBuf of capacity n. Codegen
          intercept new__RawBuf__<T> emits @ailang_rc_alloc + the
          size header. Element bytes are uninitialised — caller
          must (set …) before (get …).")
    (type (fn-type (params (con Int)) (ret (own (RawBuf a)))))
    (params n)
    (body (term-ctor RawBuf B n)))                ; placeholder

  (fn get
    (doc "Indexed read. UB if i >= (size b); caller checks
          bounds. Codegen intercept RawBuf_get__<T> emits
          getelementptr + load.")
    (type (fn-type
            (params (borrow (RawBuf a)) (con Int))
            (ret (con a))))
    (params b i)
    (body (term-ctor RawBuf B 0)))                ; placeholder

  (fn set
    (doc "Indexed write. Linear: own in, own out. Under
          uniqueness, in-place; under shared, copy-on-write
          (Issue #22 territory). Codegen intercept
          RawBuf_set__<T> emits getelementptr + store.")
    (type (fn-type
            (params (own (RawBuf a)) (con Int) (con a))
            (ret (own (RawBuf a)))))
    (params b i v)
    (body b))                                     ; placeholder

  (fn size
    (doc "Element count. Codegen intercept RawBuf_size__<T>
          emits a single i64 load from the slab header.")
    (type (fn-type (params (borrow (RawBuf a))) (ret (con Int))))
    (params b)
    (body 0)))                                    ; placeholder

The placeholder bodies are the existing convention from examples/prelude.ail (e.g. eq__Int ships with (body x) that the round-trip preserves but codegen replaces). They exist to satisfy the round-trip invariant; codegen never executes them.

Secondary — registry shape (implementation only)

// crates/ailang-codegen/src/intercepts.rs (new file)

pub struct IntrinsicSig {
    pub params: Vec<String>,   // LLVM types as text, e.g. "i64", "ptr"
    pub ret: String,
}

pub struct Intercept {
    pub name: &'static str,
    pub sig_check: fn(params: &[String], ret: &str) -> Result<()>,
    pub emit: fn(cg: &mut Lowering) -> Result<()>,
}

pub fn lookup(name: &str) -> Option<&'static Intercept> { ... }

// Wired into crates/ailang-codegen/src/lib.rs:1321 (the existing
// try_emit dispatch site) — that match becomes one lookup call.

Secondary — slab layout

RawBuf payload (post RC header) is:

  [ size : i64 ][ elem_0 ][ elem_1 ] ... [ elem_{size-1} ]
  ^             ^                                          ^
  offset 0      offset 8                                   offset 8 + size*sizeof(T)

Element widths: Int = 8, Float = 8, Bool = 1. Alignment to 8 bytes for header; element payload directly follows. @ailang_rc_alloc is called with 8 + size * sizeof(T) and gets back a ptr that the codegen treats as the slab base. The size header is self- describing so drop and RawBuf.size need no side-table.

Drop for RawBuf with primitive elements is trivial: a single @ailang_rc_release on the slab pointer. No element-walking because element types are primitives (no recursive drops). The existing per-TypeDef drop generation gets a "primitive-element- RawBuf" arm that emits the trivial drop, dispatched via the same registry.

Components

Component	Iter	Files
Intercept registry	raw-buf.1	new file crates/ailang-codegen/src/intercepts.rs; edits to crates/ailang-codegen/src/lib.rs at the single dispatch site that today calls try_emit_primitive_instance_body, and removal of each absorbed match arm in that fn
Kernel family-crate (rename + reshape) + RawBuf submodule	raw-buf.2	rename crates/ailang-kernel-stub/ → crates/ailang-kernel/; restructure as src/lib.rs + src/kernel_stub/{mod.rs, source.ail} (existing stub re-housed) + src/raw_buf/{mod.rs, source.ail} (new); update Cargo.toml package name ailang-kernel-stub → ailang-kernel; update workspace member entry + every dependent crate's Cargo.toml (ailang-surface is the only consumer today); add parse_raw_buf in crates/ailang-surface/ next to existing parse_kernel_stub; extend the workspace injection in ailang-surface to inject both modules
RawBuf intercept entries	raw-buf.3	additions to crates/ailang-codegen/src/intercepts.rs (12 entries); desugar pass Term::New → (app T.new args) in crates/ailang-codegen/src/synth.rs or crates/ailang-check/src/desugar.rs
Stub retirement (submodule level)	raw-buf.3 (close)	delete crates/ailang-kernel/src/kernel_stub/ + its re-export in src/lib.rs; drop parse_kernel_stub and its surface injection in ailang-surface (defn at crates/ailang-surface/src/loader.rs:54, re-export at crates/ailang-surface/src/lib.rs:39); delete the kernel_stub_module_round_trips test at crates/ailang-core/tests/design_schema_drift.rs:743 (its parse_kernel_stub callee is gone — the analogous raw_buf_module_round_trips test added in raw-buf.2 carries the round-trip invariant forward); update crates/ailang-core/tests/workspace_pin.rs (was bumped to 4 in iter-tidy a844de3; raw_buf joins as the stub leaves so the count stays at 4 — see Data flow); update design/INDEX.md § Models row + design/models/0007-kernel-extensions.md § "ratifying fixture" sentence. The ailang-kernel crate itself stays — it is the family-crate that future kernel-tier modules (series, matrix, …) will plug into.

Data flow

Build-time (raw-buf.2 onwards):

1. CLI loads workspace → ailang-surface injects raw_buf as a kernel-tier module (kernel: true) just like prelude today.
2. check walks consumer code; (con RawBuf (con Int)) resolves to raw_buf.RawBuf via the kernel-tier auto-import path that prep.3 shipped.
3. param-in restriction on RawBuf's a parameter rejects non-{Int,Float,Bool} instantiations at the consumer typecheck site, using the existing ParamNotInRestrictedSet diagnostic from prep.3.

Codegen-time (raw-buf.3 onwards):

1. Desugar pass replaces every Term::New {type_name, args} with Term::App { callee: <type_name>.new, args } — same form that already lowers cleanly today.
2. For each fn in the workspace, codegen reaches the dispatch site at lib.rs:1321. If the fn's mono name has a registry entry, the entry's emit runs; the .ail placeholder body is discarded. Otherwise, the normal AILang lowering runs.
3. For RawBuf consumers, new__RawBuf__Int emits the alloc call; RawBuf_set__Int emits a store i64; RawBuf_get__Int emits a load i64; RawBuf_size__Int emits a single header load.

Workspace-module-count interaction (raw-buf.3 close): The current count of 4 (prelude + kernel_stub + two test-fixture modules per the iter-tidy commit). When kernel_stub retires and raw_buf joins, the count stays 4. If raw_buf lands before the stub retires, the count is 5 mid-iter — caught by the existing workspace_pin test, which the iter explicitly re-baselines.

Error handling

Two diagnostics; both already shipped, no new arms needed in this milestone:

• param-not-in-restricted-set (prep.3, fieldtest F8). Fires when a consumer writes (con RawBuf (con Str)) or any non-{Int,Float,Bool} element. Names the type, the variable, the TypeDef, and the restricted set.

• new-type-not-constructible (prep.3, fieldtest F5). Fires when a consumer writes (new SomeType …) and SomeType has no (fn new …) in its home module. Since raw-buf ships (fn new …) in the kernel-tier raw_buf module, the consumer program above passes — but if a future LLM author writes (new Counter 0) against a type that lacks new, this diagnostic still fires precisely.

Bounds checks are not in scope. Per whitepaper § "RawBuf — the base extension": RawBuf.get / .set are UB if i >= size(b); caller checks. Bounds-check insertion is a future axis tracked separately (no backlog item yet — the discipline is "caller checks via RawBuf.size").

A future intercept-not-registered diagnostic surfaces at raw-buf.2 — the iter raw-buf.2 acceptance is precisely that this fires (consumer code checks clean but does not build), demonstrating the manifest/codegen separation works. raw-buf.3 makes it stop firing for the 12 RawBuf entries.

Testing strategy

Each iter ratified independently:

raw-buf.1 (registry refactor):

• All ~668 existing workspace tests pass unchanged. The refactor is primarily ratified by zero behavioural change in the existing suite — the legacy intercepts (eq__Str, compare__Int|Bool|Str, the float_* family) are exercised end-to-end by the following build-and-run E2E tests:
  • eq_primitives_smoke_compiles_and_runs and compare_primitives_smoke_prints_1_2_3_thrice in crates/ail/tests/e2e.rs — pin the eq__Str and compare__Int|Bool|Str arms (Str path included).
  • float_compare_smoke_prints_true_true_false in crates/ail/tests/float_compare_smoke_e2e.rs — pins the float_* family via float_eq + float_lt.
  • eq_ord_polymorphic_runs_end_to_end in crates/ail/tests/eq_ord_e2e.rs — secondary Str-path ratifier via polymorphic Eq/Ord dispatch. If the registry rewrite breaks any intercept, those tests catch it.
• One new unit test in the new intercepts module: registry_contains_all_legacy_arms enumerates the names that were in the legacy match (eq__Str, compare__Int, compare__Bool, compare__Str, and each float_* name) and asserts each resolves to a registered entry. This pins the migration as complete (a half-finished migration where the old match still has an arm becomes a hard test fail).

raw-buf.2 (manifest + checker):

• New E2E test crates/ail/tests/raw_buf_check_e2e.rs: consumer fixture with (con RawBuf (con Int)) in a type signature, ail check exits 0.
• New E2E test crates/ail/tests/raw_buf_param_in_reject_e2e.rs: consumer fixture with (con RawBuf (con Str)), ail check exits non-zero with param-not-in-restricted-set in stderr.
• New E2E test crates/ail/tests/raw_buf_build_intercept_missing_e2e.rs: consumer fixture with (new RawBuf (con Int) 3), ail build exits non-zero with intercept-not-registered (or the Term::New deferral message) — pins the manifest/codegen separation. (This test gets deleted or inverted to PASS in raw-buf.3.)
• New round-trip test for the embedded RAW_BUF_AIL — parse → print round-trips byte-isomorphically (mirrors kernel_stub_module_round_trips in crates/ailang-core/tests/design_schema_drift.rs).
• Workspace module-count test bumped: 4 → 5 (mid-iter state before stub retires).

raw-buf.3 (codegen + stub retirement):

• The intercept-missing E2E test from raw-buf.2 inverted: same consumer fixture, ail build now exits 0 and the produced binary, when run, prints 60.
• New E2E crates/ail/tests/raw_buf_float_e2e.rs: Float variant exercise.
• New E2E crates/ail/tests/raw_buf_bool_e2e.rs: Bool variant exercise (catches the i1/i8 packing question if any).
• Drop ratification: a consumer that allocates and lets the RawBuf go out of scope without a leak. Re-uses the existing leak-check harness; no new infra.
• Stub-retirement ratification: deleted-submodule diff hits crates/ailang-kernel/src/kernel_stub/ (folder removal) + crates/ailang-kernel/src/lib.rs (re-export drop) + crates/ailang-surface/ (parse_kernel_stub + injection drop) + crates/ailang-core/tests/workspace_pin.rs + design/INDEX.md + design/models/0007-kernel-extensions.md. Test workspace_pin re-baselines to 4 (raw_buf in, kernel_stub out — net unchanged). The ailang-kernel crate itself stays as the family-crate home for future kernel-tier modules (series, matrix, …).

Acceptance criteria

The milestone ships when:

1. The worked consumer program in § Concrete code shapes end-to-end: ail check && ail build && ail run prints 60.
2. The param-in reject fixture exits at ail check with the prep.3 diagnostic.
3. Every existing primitive-instance intercept (eq__Str, compare__Int|Bool|Str, the float_* family) is dispatched via the registry — no remaining match-arm in try_emit_primitive_instance_body outside the registry path.
4. ailang-kernel-stub is retired from the workspace; raw_buf takes its slot as the canonical kernel-tier base-extension anchor.
5. Workspace test suite green, plus the new tests enumerated in § Testing strategy.

Feature-acceptance argument

Applied per design/contracts/0004-feature-acceptance.md:

1. LLM author reaches for it naturally. RawBuf is the only path to mutable indexed storage in AILang — the language has no other "I want to fill N slots and read them back" primitive. The worked consumer program above is what an LLM author writes when expressing that need; there is no alternative shape to reach for.

2. Measurable improvement. Embedding-ABI M5 measured per-tick FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks). RawBuf as the batch-FFI primitive amortises that per-tick cost across one crossing. The series milestone (#8) is also blocked on RawBuf; shipping raw-buf removes that block.

3. No reintroduced bug class. Mutation discipline stays mode-tracked (whitepaper § "RawBuf — the base extension"): RawBuf.set is own → own, not effect-tracked. No RawBuf effect declared. Uniqueness inference rewrites it in-place; shared ownership falls back to copy-on-write (Issue #22). The pure-core invariant — code without a declared effect is pure — survives intact. The local- reasoning invariant survives too: a fn that takes borrow (RawBuf Int) and returns Int is pure from the caller's view, because the buffer is not mutated.

The registry-migration aspect of the milestone (raw-buf.1) removes redundancy structurally: the growth point of "every new primitive intercept needs another match arm" is collapsed into a single dispatch table. That is criterion-2 evidence for the migration as a piece.