AILang/docs/specs/0054-raw-buf.md

raw-buf — Design Spec

Date: 2026-05-29 (revised 2026-05-29) Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Revision note. raw-buf.1 (intercept registry) shipped (140a0c0). This spec was then re-carved because the intrinsic-bodies milestone (#9) landed afterwards and removed the foundation the original raw-buf.2/.3 split stood on:

1. The placeholder-body convention the original spec relied on ((body x) round-trip stubs, "the existing convention from examples/prelude.ail") no longer exists. prelude.ail now uses (intrinsic) markers, and the checker (check_fn, crates/ailang-check/src/lib.rs:2173) typechecks every non-intrinsic body — including kernel-tier bodies. A placeholder body for RawBuf.get (declared (ret a), body constructing a RawBuf) is now a hard type error. get (and new/set/size) MUST be (intrinsic).
2. An (intrinsic) marker now requires a matching INTERCEPTS codegen entry as a lockstep pair, enforced by intercepts::tests::intercepts_bijection_with_intrinsic_markers. The "manifest visible, codegen deferred" intermediate state the original raw-buf.2 was designed to ship is therefore forbidden by an invariant. The marker and its codegen entry must land in the same iteration.

Consequence: the kernel-manifest and the codegen now land together in one iteration. The remaining work is re-carved into three iterations (raw-buf.2/.3/.4) below. The Goal, slab layout, and feature-acceptance argument are unchanged.

Goal

Ship RawBuf a — the first kernel-tier base extension — as a mutable, indexed, bounded-size flat buffer of primitive elements ({Int, Float, Bool}). RawBuf is the storage primitive that unblocks two downstream needs both already captured in the backlog:

• Series (milestone #8) — a bounded ring buffer with financial-style indexing, implemented as a library extension (pure AILang) wrapping RawBuf a plus bookkeeping fields.
• Embedding-ABI batch FFI (subsumes closed #2) — the M5 friction-harvest measured per-tick FFI at ~206 ns/tick on real EURUSD volume (~658 ms / 3.19M ticks). A contiguous primitive slice amortises that per-tick cost via a batch crossing. RawBuf is that primitive.

The milestone also fulfils the architectural commitment from the kernel-extensions whitepaper § "Plugin contract": the migration of the hardcoded primitive-instance intercepts into one registry, triggered by the first real base extension shipping. raw-buf.1 already discharged that migration (the registry is live and the intrinsic-bodies bijection pin locks marker ↔ entry); RawBuf is now the first real-payload consumer of that registry.

This makes raw-buf both a feature ship (a new type LLM authors can reach for) and the proof that the registry path carries a genuine domain type, not just the migrated primitive intercepts.

Architecture

Four iterations. raw-buf.1 is done; .2/.3/.4 remain.

1. Intercept registry (raw-buf.1) — DONE (140a0c0). Lifted the hard-coded try_emit_primitive_instance_body match into the registry table INTERCEPTS in crates/ailang-codegen/src/intercepts.rs; the dispatch site is a intercepts::lookup call. Pure refactor, zero behavioural change. (The subsequent intrinsic-bodies milestone added the (intrinsic) marker, the Term::Intrinsic leaf, and the marker ↔ entry bijection pin on top of this registry.)

2. Kernel family-crate rename (raw-buf.2) — Pure refactor, zero behavioural change. Rename crates/ailang-kernel-stub/ → crates/ailang-kernel/ and reshape it as a family-crate: one Cargo crate hosting all kernel-tier modules as sibling Rust submodules under src/, one submodule per .ail source. The renamed crate carries src/kernel_stub/{mod.rs, source.ail} (the existing stub, verbatim including its current answer intrinsic fn — the source has grown since the original spec was written). src/lib.rs re-exports each submodule's SOURCE const under the stable name every consumer already binds (pub use kernel_stub::SOURCE as STUB_AIL;). The acyclic chain ailang-surface → ailang-kernel → ailang-core is preserved; the parse hop parse_kernel_stub stays in ailang-surface. No raw_buf submodule yet — this iteration is only the rename, so its failure mode is "existing tests break", the cleanest bisection target. Ratified by the existing suite green.

3. RawBuf end-to-end (raw-buf.3) — The feature ship, landing the manifest and the codegen together (the lockstep pair the bijection invariant requires). Adds:

   • The raw_buf submodule (src/raw_buf/{mod.rs, source.ail}) with the RawBuf TypeDef (param-in (a Int Float Bool)) and the four operations new/get/set/size, each an (intrinsic) marker (NOT a placeholder body).
   • The 12 element-specialised INTERCEPTS entries (4 ops × 3 element types) lowering each via @ailang_rc_alloc + getelementptr + load/store — no new C code. The entry name of each MUST equal the mono-mangled symbol of its (intrinsic) marker; the bijection test is the pin that forces this and surfaces the exact expected strings.
   • A desugar pass (new T <types…> <values…>) → an instantiated (app T.new <values…>) that consumes the NewArg::Type element type to drive monomorphisation to new__RawBuf__<T>, so Term::New is eliminated before codegen. Term::New becomes a checker-side-only construct (the codegen completion the prep.2 deferral pointed at). raw-buf.3 removes both Term::New deferral arms codegen currently carries — the lower_term arm (crates/ailang-codegen/src/lib.rs:2094, "milestone raw-buf" internal error) and the sibling synth-path arm (lib.rs:3296). These arms are the deferral marker prep.2 left; raw-buf.3 discharges them. They are not a relied-upon current invariant — the milestone's correctness depends on removing them, not on their persisting.
   • Drop for primitive-element RawBuf: a single @ailang_rc_release on the slab pointer (no element-walking; elements are primitives).
   • ailang-surface wiring: parse_raw_buf + the workspace injection that makes raw_buf a kernel-tier auto-imported module (workspace module count 4 → 5). The worked consumer program runs end-to-end and prints 60. kernel_stub stays alongside this iteration (count 5).

4. kernel_stub retirement (raw-buf.4) — Focused removal. RawBuf now subsumes every ratification role kernel_stub held: Term::New end-to-end (the (new RawBuf …) consumer), param-in reject (the Str-element reject fixture), kernel-tier auto-import (RawBuf visible without (import raw_buf)), and the (intrinsic) mechanism (RawBuf's four ops + their registry entries). The stub is therefore pure redundancy. Delete src/kernel_stub/ + its re-export, drop parse_kernel_stub and its surface injection, remove the answer INTERCEPTS entry + emit_answer (its (intrinsic) marker leaves with the stub — the bijection lockstep holds), delete the kernel_stub_module_round_trips test, re-baseline workspace_pin to 4 (raw_buf in, kernel_stub out — net unchanged), and update design/INDEX.md + design/models/0007-kernel-extensions.md. The ailang-kernel crate itself stays as the family-crate home for future kernel-tier modules (series, matrix, …). Isolating the retirement keeps each bijection-registry change (12 entries added in .3, the answer entry removed in .4) in its own diff.

The ordering rationale: .2 is the only zero-behavioural-change iteration (rename), the cleanest bisection target; .3 lands the feature on a registry that has already absorbed every legacy intercept (raw-buf.1) and on the renamed family-crate (raw-buf.2), so the RawBuf manifest+codegen do not co-mingle with a registry move or a crate move; .4 removes the now-redundant stub without touching the new RawBuf code.

Concrete code shapes

Primary — the raw_buf kernel-tier module (raw-buf.3, gate-verified)

This is the exact Form-A source the raw_buf submodule embeds. It parses and type-checks against the current tool (ail check → ok); the four (intrinsic) markers are legal because the module is (kernel). The element type is restricted to {Int, Float, Bool} via param-in. RawBuf.get returns the bare element type a; set is linear (own in, own out).

(module raw_buf
  (kernel)
  (data RawBuf (vars a)
    (doc "Mutable, indexed, fixed-size flat buffer of primitive elements. Opaque single-ctor handle; codegen intercepts emit raw alloc/load/store over an @ailang_rc_alloc slab. Element type restricted to {Int, Float, Bool} via param-in.")
    (ctor B a)
    (param-in (a Int Float Bool)))
  (fn new
    (doc "Allocate an uninitialised RawBuf of capacity n. Compiler-supplied (intrinsic) body; codegen intercept new__RawBuf__<T> emits @ailang_rc_alloc plus the i64 size header. Element bytes are uninitialised; caller must set before get.")
    (type (forall (vars a) (fn-type (params (con Int)) (ret (own (con RawBuf a))))))
    (params n)
    (intrinsic))
  (fn get
    (doc "Indexed read. UB if i >= (size b); caller checks bounds via size. Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus load.")
    (type (forall (vars a) (fn-type (params (borrow (con RawBuf a)) (con Int)) (ret a))))
    (params b i)
    (intrinsic))
  (fn set
    (doc "Indexed write. Linear: own in, own out. Under uniqueness in-place; under shared copy-on-write (Issue #22). Compiler-supplied (intrinsic) body; codegen intercept emits getelementptr plus store.")
    (type (forall (vars a) (fn-type (params (own (con RawBuf a)) (con Int) a) (ret (own (con RawBuf a))))))
    (params b i v)
    (intrinsic))
  (fn size
    (doc "Element count. Compiler-supplied (intrinsic) body; codegen intercept emits a single i64 load from the slab header.")
    (type (forall (vars a) (fn-type (params (borrow (con RawBuf a))) (ret (con Int)))))
    (params b)
    (intrinsic)))

The (ctor B a) is a nominal fiction for the type system: it gives RawBuf an identity and keeps a non-phantom. Real code never constructs it — every operation is codegen-intercepted, and (new RawBuf …) desugars to (app RawBuf.new …), not to a (term-ctor RawBuf B …). The four (intrinsic) markers carry no runnable body; codegen supplies the implementation via the registry (raw-buf.3).

Primary — the worked consumer program (raw-buf.3 acceptance)

This is the program an LLM author naturally writes for "I need a small mutable indexed buffer to score N values and read them back." It is the feature-acceptance criterion's empirical evidence. It references the raw_buf module, which is only injected once raw-buf.3 wires it through ailang-surface — so it does not check against the current tool (RawBuf is not yet a workspace module). Left unlabeled deliberately: it describes post-raw-buf.3 surface, not current behaviour.

(module raw_buf_demo
  (fn main
    (type (fn-type (params) (ret (con Int))))
    (params)
    (body
      (let buf (new RawBuf (con Int) 3)
        (let buf (app RawBuf.set buf 0 10)
          (let buf (app RawBuf.set buf 1 20)
            (let buf (app RawBuf.set buf 2 30)
              (app + (app RawBuf.get buf 0)
                (app + (app RawBuf.get buf 1)
                  (app RawBuf.get buf 2))))))))))
; ail check && ail build && ail run → prints 60

No (import raw_buf) needed: RawBuf is visible via the kernel-tier auto-import path (prep.3). RawBuf.set / .get resolve via type-scoped namespacing (prep.1). (new RawBuf (con Int) 3) is a Term::New whose NewArg::Type is (con Int); the raw-buf.3 desugar threads that into new__RawBuf__Int.

Primary — the param-in reject fixture (raw-buf.3 must-fail)

(module raw_buf_reject_str
  (fn cant_str
    (type (fn-type (params (borrow (con RawBuf (con Str)))) (ret (con Str))))
    (params buf)
    (body (app RawBuf.get buf 0))))
; ail check → param-not-in-restricted-set: type Str not in {Int, Float, Bool}

Must fail at the checker with the ParamNotInRestrictedSet diagnostic shipped in prep.3 (fieldtest F8). It is the must-fail evidence that the param-in restriction is alive on a real-payload extension, not just the prep.3 StubT fixture. Like the worked program, it depends on raw_buf being injected, so it is not gate-checkable against the current tool (unlabeled).

Secondary — INTERCEPTS entry shape (raw-buf.3, implementation)

The real registry struct (raw-buf.1) is Intercept { name, expected_params, expected_ret, wants_alwaysinline, emit }. The 12 RawBuf entries follow the existing rows' shape. One representative (the exact symbol strings are derived by the implementer from the bijection test, which mangles each (intrinsic) marker and reports the expected name):

// crates/ailang-codegen/src/intercepts.rs — appended to INTERCEPTS
Intercept {
    name: "new__RawBuf__Int",          // == mono-mangled name of raw_buf.new @ a=Int
    expected_params: &["i64"],          // capacity n
    expected_ret: "ptr",               // slab base pointer
    wants_alwaysinline: false,
    emit: emit_new_raw_buf_int,        // @ailang_rc_alloc(8 + n*8); store i64 n at offset 0
},
// + new__RawBuf__{Float,Bool}, RawBuf_get__{Int,Float,Bool},
//   RawBuf_set__{Int,Float,Bool}, RawBuf_size__{Int,Float,Bool}

Secondary — Term::New desugar (raw-buf.3, implementation)

Term::New { type_name, args } where args: Vec<NewArg> (NewArg::Type | NewArg::Value, both already parsed + round-trip ratified). The new desugar pass rewrites it to an application of the type's new fn, instantiated at the element type carried by the leading NewArg::Type:

(new RawBuf (con Int) 3)        ; Term::New { type_name: "RawBuf",
                                ;   args: [Type(con Int), Value(3)] }
  ──desugar──▶
(app RawBuf.new 3)  @ a = Int   ; instantiation pins mono → new__RawBuf__Int

The desugar consumes the NewArg::Type for the instantiation and passes only the NewArg::Value args as call arguments. After this pass no Term::New survives into codegen; both deferral arms (lib.rs:2094 in lower_term, "milestone raw-buf" internal error, and the sibling at lib.rs:3296 in the synth path) become unreachable and are removed. Whether the prep.2 checker already accepts the NewArg::Type-bearing Term::New semantically, or needs a small extension, is a raw-buf.3 task to verify first (the form parses + round-trips today; the lift pass passes NewArg::Type through untouched).

Secondary — slab layout (raw-buf.3)

  [ size : i64 ][ elem_0 ][ elem_1 ] ... [ elem_{size-1} ]
  ^             ^                                          ^
  offset 0      offset 8                                   offset 8 + size*sizeof(T)

Element widths: Int = 8, Float = 8, Bool = 1. Header is i64, 8-byte aligned; element payload follows directly. @ailang_rc_alloc is called with 8 + size * sizeof(T) and returns the slab base ptr. The size header is self-describing, so drop and RawBuf.size need no side-table. Drop is a single @ailang_rc_release on the slab pointer — primitive elements carry no recursive drops. The per-TypeDef drop generation gets a "primitive-element-RawBuf" arm dispatched via the same registry.

Components

Component	Iter	Files
Intercept registry	raw-buf.1 (DONE)	crates/ailang-codegen/src/intercepts.rs + dispatch site in lib.rs
Kernel family-crate rename	raw-buf.2	git mv crates/ailang-kernel-stub crates/ailang-kernel; reshape to src/lib.rs + src/kernel_stub/{mod.rs, source.ail} (existing stub source incl. answer re-housed verbatim); Cargo.toml package + workspace member + dep-alias renames; crates/ailang-surface/Cargo.toml dep rename + loader.rs use-path renames; design/INDEX.md + CLAUDE.md code-layout row
RawBuf module + codegen + desugar + drop	raw-buf.3	new crates/ailang-kernel/src/raw_buf/{mod.rs, source.ail} + lib.rs re-export; parse_raw_buf + workspace injection in crates/ailang-surface/src/loader.rs + re-export in lib.rs; 12 entries + emit fns in crates/ailang-codegen/src/intercepts.rs; Term::New desugar in crates/ailang-core/src/desugar.rs (and removal of the codegen Term::New error arm in crates/ailang-codegen/src/lib.rs); primitive-element-RawBuf drop arm; workspace_pin 4 → 5; round-trip + E2E tests
kernel_stub retirement	raw-buf.4	delete crates/ailang-kernel/src/kernel_stub/ + its lib.rs re-export; drop parse_kernel_stub + injection in ailang-surface; remove answer entry + emit_answer in intercepts.rs; delete kernel_stub_module_round_trips test; workspace_pin 5 → 4; design/INDEX.md + design/models/0007-kernel-extensions.md updates

Data flow

Build-time (raw-buf.3 onwards):

1. CLI loads workspace → ailang-surface injects raw_buf as a kernel-tier module (kernel: true) just like prelude.
2. check walks consumer code; (con RawBuf (con Int)) resolves to raw_buf.RawBuf via the kernel-tier auto-import path (prep.3).
3. param-in on RawBuf's a rejects non-{Int,Float,Bool} instantiations at the consumer typecheck site, using the existing ParamNotInRestrictedSet diagnostic.

Codegen-time (raw-buf.3 onwards):

1. The desugar pass replaces every Term::New { type_name, args } with an instantiated (app <type_name>.new <values>), pinning the element type from the leading NewArg::Type.
2. For each fn, codegen reaches the registry dispatch site. If the fn's mono name resolves via intercepts::lookup, the entry's emit runs; the (intrinsic) marker carries no body to fall back to. Otherwise the normal AILang lowering runs.
3. For RawBuf consumers, new__RawBuf__Int emits the alloc; RawBuf_set__Int a store i64; RawBuf_get__Int a load i64; RawBuf_size__Int a single header load.

Bijection interaction (raw-buf.3 / .4): Each RawBuf (intrinsic) marker reachable from the workspace is collected by workspace_intrinsic_markers() and must resolve to an INTERCEPTS entry (and vice versa, minus the OPTIMISATION_ONLY allowlist). raw-buf.3 adds 12 markers + 12 entries together. raw-buf.4 removes the answer marker (with the stub) and the answer entry together. A half-landed change on either side is a hard unit-test failure (intercepts_bijection_with_intrinsic_markers).

Workspace module count: prelude + kernel_stub + 2 test fixtures = 4 today. raw-buf.3 adds raw_buf → 5. raw-buf.4 retires kernel_stub → 4. The workspace_pin test pins each transition.

Error handling

Two consumer-facing diagnostics; both already shipped, no new arms needed:

• param-not-in-restricted-set (prep.3, fieldtest F8). Fires when a consumer writes (con RawBuf (con Str)) or any non-{Int,Float,Bool} element. Names the type, the variable, the TypeDef, and the restricted set.
• new-type-not-constructible (prep.3, fieldtest F5). Fires when a consumer writes (new SomeType …) and SomeType has no (fn new …) in its home module. raw_buf ships (fn new …), so the worked program passes; the diagnostic still protects future authors who write (new Counter 0) against a new-less type.

The codegen Term::New deferral arms (lib.rs:2094 in lower_term, message "… milestone raw-buf"; and the sibling at lib.rs:3296 in the synth path) are removed in raw-buf.3 once the desugar eliminates Term::New before codegen. Neither is a user-facing diagnostic — both are the deferral marker prep.2 left, which raw-buf.3 discharges. They have no test pinning them today (they are an internal CodegenError::Internal, not an asserted behaviour); raw-buf.3's end-to-end build-and-run E2E (the worked program printing 60) is the forward ratification — it exercises the Term::New → desugar → codegen path that replaces them.

Bounds checks are not in scope (whitepaper § "RawBuf — the base extension"): RawBuf.get/.set are UB if i >= size(b); caller checks via RawBuf.size. Bounds-check insertion is a future axis, no backlog item yet.

Testing strategy

raw-buf.2 (rename):

• All existing workspace tests pass unchanged — the rename is a pure refactor. Zero test-count change. The crate-rename's failure mode is "the workspace does not build / existing tests break", caught by cargo build --workspace + cargo test --workspace.

raw-buf.3 (RawBuf end-to-end):

• Round-trip: raw_buf_module_round_trips (mirror of kernel_stub_module_round_trips in design_schema_drift.rs) — parse → serialise → round-trip; assert kernel == true, name raw_buf, and RawBuf TypeDef param_in for a contains all of Int/Float/Bool.
• E2E raw_buf_int_e2e: the worked consumer program — ail check && ail build && ail run → prints 60.
• E2E raw_buf_float_e2e / raw_buf_bool_e2e: Float and Bool element variants (Bool catches any i1/i8 packing question).
• E2E raw_buf_param_in_reject_e2e: the Str-element reject fixture — ail check exits non-zero with param-not-in-restricted-set in stderr.
• Drop ratification: a consumer that allocates a RawBuf and lets it go out of scope without a leak — re-uses the existing leak-check harness, no new infra.
• Bijection: intercepts_bijection_with_intrinsic_markers stays green with the 12 new markers + 12 new entries (the test forces the exact mangled-name match).
• workspace_pin count assertion 4 → 5 (+ contains_key("raw_buf")).

raw-buf.4 (retirement):

• workspace_pin re-baselines 5 → 4 (raw_buf in, kernel_stub out); contains_key("kernel_stub") assertion removed, contains_key("raw_buf") retained.
• kernel_stub_module_round_trips deleted (its parse_kernel_stub callee is gone; raw_buf_module_round_trips carries the round-trip invariant forward).
• Bijection stays green after the answer marker + entry leave together.
• The full suite stays green; the diff is a removal, ratified by "nothing references the stub and everything still builds".

Acceptance criteria

The milestone ships when:

1. The worked consumer program runs end-to-end: ail check && ail build && ail run prints 60 (raw-buf.3).
2. The param-in reject fixture exits at ail check with the prep.3 param-not-in-restricted-set diagnostic (raw-buf.3).
3. RawBuf's four operations are (intrinsic) markers, each paired with an INTERCEPTS entry; intercepts_bijection_with_intrinsic_markers is green (raw-buf.3).
4. Term::New no longer reaches codegen — the desugar eliminates it and both deferral arms (lib.rs:2094 + lib.rs:3296) are removed (raw-buf.3).
5. ailang-kernel-stub is renamed to the ailang-kernel family-crate (raw-buf.2); kernel_stub is retired and raw_buf takes over its ratification roles (raw-buf.4).
6. Workspace test suite green at each iteration boundary, plus the new tests enumerated in § Testing strategy.

Feature-acceptance argument

Applied per design/contracts/0004-feature-acceptance.md:

1. LLM author reaches for it naturally. RawBuf is the only path to mutable indexed storage in AILang — the language has no other "fill N slots and read them back" primitive. The worked consumer program above is what an LLM author writes for that need; there is no alternative shape to reach for.

2. Measurable improvement. Embedding-ABI M5 measured per-tick FFI at ~206 ns/tick (~658 ms / 3.19M EURUSD ticks). RawBuf as the batch-FFI primitive amortises that per-tick cost across one crossing. The Series milestone (#8) is also blocked on RawBuf; shipping raw-buf removes that block.

3. No reintroduced bug class. Mutation discipline stays mode-tracked (whitepaper § "RawBuf — the base extension"): RawBuf.set is own → own, not effect-tracked. No RawBuf effect is declared. Uniqueness inference rewrites in-place; shared ownership falls back to copy-on-write (Issue #22). The pure-core invariant survives: code without a declared effect is pure. Local reasoning survives: a fn taking borrow (RawBuf Int) and returning Int is pure from the caller's view, because the buffer is not mutated.