AILang/design/models/0007-kernel-extensions.md

Kernel extensions — plugin-style domain types whitepaper

STATUS. Design accepted 2026-05-28. Implementation in progress across two Gitea milestones: kernel-extension-mechanics (the language-level mechanisms) and series (the first concrete consumer). This whitepaper describes the design as a coherent whole; the per-milestone specs in docs/specs/ carry the implementation-level detail. As each milestone closes, sections of this whitepaper transition from forward-looking design to present-state description, per the honesty-rule.

The problem

AILang's core needs to stay small. Five primitive types (Int, Bool, Unit, Str, Float), algebraic data types, typeclasses, algebraic effects, RC + uniqueness — that is the language. New primitive types are a tax: they inflate the schema, the checker, the codegen, the documentation surface, and the LLM-author's mental model of what the language is.

But: AILang authors will work in specific domains. Streaming analytics wants a bounded ring buffer. Numerical work wants flat arrays and matrices. Financial work wants fixed-precision decimals and timestamps with arithmetic. None of these belong in ailang-core — yet without them, the LLM author has to hand-roll linked-list approximations that are algorithmically wrong (O(N²) where O(N) is the natural complexity).

The resolution: kernel extensions — a plugin contract that lets a domain-specific type live in its own crate, register with the compiler at boot time, and present itself to user code as if it were a built-in primitive. Core has no domain knowledge of any specific extension; extensions have no core dependency beyond the plugin contract.

The first consumer is Series — a bounded ring buffer for streaming data. Future consumers are not specified here but the shape is intentional: each new domain type is one new crate, zero core changes.

Worked example — Series for streaming analytics

What an LLM-author writes when given the task "compute simple moving average over a stream of float values, window size 3":

(module sma_demo

  (data FloatList
    (ctor FNil)
    (ctor FCons (con Float) (con FloatList)))

  (fn sum_window_step
    (type (fn-type
      (params (borrow (Series (con Float))) (con Int) (con Int) (con Float))
      (ret (con Float))))
    (params s n i acc)
    (body
      (if (app eq i n)
        acc
        (tail-app sum_window_step s n
          (app + i 1)
          (app + acc (app Series.at s i))))))

  (fn emit_if_full
    (type (fn-type
      (params (borrow (Series (con Float))) (con Int))
      (ret (con Unit)) (effects IO)))
    (params s n)
    (body
      (if (app >= (app Series.total_count s) n)
        (app print (app /
          (app sum_window_step s n 0 0.0)
          (app int_to_float n)))
        unit)))

  (fn run_stream
    (type (fn-type
      (params (own (Series (con Float))) (con Int) (con FloatList))
      (ret (con Unit)) (effects IO)))
    (params s n input)
    (body
      (match input
        (case (pat-ctor FNil) unit)
        (case (pat-ctor FCons v rest)
          (let s_new (app Series.push s v)
            (seq (app emit_if_full s_new n)
              (tail-app run_stream s_new n rest)))))))

  (fn main
    (type (fn-type (params) (ret (con Unit)) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (app run_stream s 3
          (term-ctor FloatList FCons 1.0
          (term-ctor FloatList FCons 5.0
          (term-ctor FloatList FCons 3.0
          (term-ctor FloatList FCons 8.0
          (term-ctor FloatList FCons 6.0
          (term-ctor FloatList FCons 2.0
          (term-ctor FloatList FNil)))))))))))

Four constructs in this program are not in today's AILang. Each maps onto one of the four mechanisms this whitepaper defines:

• Series appears in type position with no module qualifier ((con Series (con Float))) — kernel-tier modules.
• (new Series (con Float) 3) constructs the series — new term construct.
• (app Series.at s i), (app Series.total_count s), (app Series.push s v) — type-scoped namespacing.
• The (own (Series ...)) mode on run_stream plus the linear state-threading via (let s_new ...) is the uniqueness mode discipline that already exists in AILang. Series carries no separate algebraic effect; its mutation is mode-tracked, not effect-tracked. Under uniqueness inference (Gitea #22), the rebuild-on-push compiles to in-place mutation.

A fifth mechanism, param-in, is not visible at the call site but enforces that (con Series (con Str)) would be rejected.

The four mechanisms

1. Type-scoped namespacing

Form. <TypeName>.<member> resolves to the <member> definition in the home module of <TypeName>. The home module of a type T is the module whose defs contains a TypeDef named T.

Canonical form. Type-scoped access is the canonical form for operations associated with a type. The pre-existing module-scoped form (<module>.<member>) remains a valid surface only for free-standing definitions not bound to a specific type receiver, and as a collision disambiguator if two types in the workspace ever shared a name (which is not the case today).

Examples.

(app Series.at s i)               ; series_at via type-scope
(app Maybe.from_maybe 0 m)        ; from_maybe via type-scope  (was: std_maybe.from_maybe)
(con Series (con Float))          ; type itself accessible bare under kernel-tier
(con Maybe (con Int))             ; type itself, if Maybe is in scope via import

Why this is LLM-natural. When asked "what can I do with a Series?", an LLM produces Series.<x> queries. When asked "what can I do with a Maybe?", it produces Maybe.<x> queries. The type, not the module, is the LLM's mental anchor. Module-scoped access requires knowing where a function lives — extra cognitive load with no expressive benefit.

Schema impact. Zero. The dotted form already parses today as Term::Var { name: "Maybe.from_maybe" } (the . is a permitted identifier character in the existing lexer). What changes is the resolver: it tries the type-namespace branch first, falls back to the module-namespace branch.

Diagnostics. TypeScopedMemberNotFound (the receiver is a known type, but the member is not defined in its home module); TypeScopedReceiverNotAType (the receiver name is neither a module nor a type).

2. The new term construct

Form. (new T arg1 arg2 ...) where T is a TypeName and argN is either a Type expression (e.g. (con Float)) or a Term expression. The construct desugars to: look up new in the home module of T, call it with the given args.

AST. New variant Term::New { type_name: TypeName, args: Vec<NewArg> } where NewArg = NewArg::Type(Type) | NewArg::Value(Term). Each arg carries its kind in serialised JSON.

Why complementary to term-ctor, not replacing it. term-ctor constructs an ADT value via a named data constructor (a tag in the TypeDef's ctors list). new calls a function in the type's home module. They serve different purposes:

• (term-ctor IntList ICons 5 INil) builds an IntList from its ICons ctor. This is structural — the AST node carries the ctor name and field args directly.
• (new Series (con Float) 3) calls Series.new : (Type, Int) -> Series a. This is functional — there is no Series data ctor; the type is opaque and construction is a defined operation.

A type can have both: an ADT could expose term-ctor for direct ctor access plus a new def for builder-style construction. The two paths are not in tension; they answer different design needs.

Diagnostics. NewTypeNotConstructible (no new def in the type's home module); NewArgKindMismatch (a Type arg where the signature expects a Value, or vice versa).

3. Kernel-tier modules with auto-import

Form. A new boolean field kernel on Module (omitted when false, so existing modules hash-stable). When kernel: true:

• The module's top-level defs are accessible bare in every other module of the workspace, with no (import ...) declaration required.
• The module's types are accessible bare in type position.

Why. The architectural property the user wants is: the domain type lives outside core, but at the call site it feels like a primitive. Bare Series in (con Series (con Float)) without seeing (import series) somewhere is the ergonomic property.

Single mechanism. Auto-injection is not a new behaviour introduced by this design — it exists today, hardcoded to one module name. crates/ailang-surface/src/loader.rs:98-108 unconditionally injects parse_prelude() and threads &["prelude"] into workspace::build_workspace as the implicit-imports list. Prelude's 12 free fns (see examples/prelude.ail:85-148) are already callable bare in every consumer module by virtue of this hardcoded path, ratified by crates/ail/tests/prelude_free_fns.rs.

The kernel-tier flag generalises this existing single-name auto-injection into a flag-driven multi-module mechanism. After the kernel-extensions design lands, prelude carries kernel: true and the implicit-imports list is derived from the set of all kernel-flagged modules. Consumer-observable behaviour for prelude is unchanged; the code path is rewritten from "hardcoded one name" to "flag-filtered all modules". Other kernel-tier modules (the stub, future Series, future Matrix) become auto-injected through the same path.

Class-method dispatch (a separate mechanism — see method dispatch) is orthogonal: the dispatch is about how method calls are resolved; the auto-import is about which names are in scope.

4. param-in — closed-set type parameter restriction

Form. Optional field on TypeDef: param-in: Map<VarName, Set<TypeName>>. Skipped when absent. Each entry constrains a type variable to a fixed set of named types.

Example.

// TypeDef Series
{ "kind": "type",
  "name": "Series",
  "vars": ["a"],
  "ctors": [],
  "param-in": { "a": ["Int", "Float", "Bool"] }
}

When the checker sees Type::Con { name: "Series", args: [Type::Con { name: "Str", ... }] }, it looks up Series's param-in, finds that a must be in {Int, Float, Bool}, and rejects Str.

Why not a marker class. class Primitive a with three instances (Int, Float, Bool) and no methods would express the same restriction via existing typeclass machinery. Reason it is not chosen: class constraints model behaviour (a type satisfies some interface). param-in models structural identity (the type's storage layout). Series's element-type restriction is structural — the C-runtime ring buffer has a data[] whose layout depends on whether the element is 8 bytes (Int/Float) or 1 byte (Bool). A marker class would model this as "Int is a Primitive" which is true but indirect. param-in says directly: "Series's element variable must be one of these three named types". The typecheck site is a set-membership lookup, not a class-instance discharge.

Diagnostic. ParamNotInRestrictedSet names the offending type and the allowed set.

Two-tier extension architecture

Kernel extensions split into two tiers, distinguished by what they need to provide:

Base extensions

Provide primitives that are not expressible in AILang itself — typically because they require mutable indexed storage, hardware/OS interaction, or library-wrapping that has no AILang surface. A base extension ships:

1. A kernel-tier module manifest with the TypeDef (and param-in if applicable) plus the operation signatures.
2. Codegen intercepts — Rust code registered with the compiler that, at each call site of an operation, emits the appropriate LLVM IR text (using existing runtime symbols like @ailang_rc_alloc for allocation). This follows the try_emit_primitive_instance_body precedent: Rust returns IR text, no separate C glue.
3. Optional C runtime support in runtime/<extension>.c — only when the operations truly cannot be expressed as IR over existing runtime primitives (e.g. wrapping PCRE2 for a future Regex extension). For the foreseeable base extensions (RawBuf, Matrix), no C is needed.

The first base extension is RawBuf T — a mutable, indexed, bounded-size flat buffer of primitive elements. Element type is restricted to {Int, Float, Bool} via param-in for the MVP.

Library extensions

Provide domain-specific types that are expressible in AILang once a base extension has been added. A library extension ships:

1. A kernel-tier .ail module (or .ail.json) — plain AILang code, no Rust intercepts, no C. The module declares an ADT and a set of fns; the fns call into a base extension's API.
2. Nothing else. No codegen intercepts (regular AILang codegen handles it), no C runtime (regular AILang RC handles it), no compiler-side plug-ins.

The first library extension is Series T — a bounded ring buffer with financial-style indexing. Implemented as an ADT wrapping a RawBuf T plus four bookkeeping Int fields (lookback, head, count, total). All operations are AILang fns.

Why split

• AILang-in-AILang where possible. Domain types are written in the language they are for. A LLM author who needs to understand Series's eviction logic reads series.ail, not a Rust intercept registry. The implementation language matches the authoring language.
• Composability. Multiple library extensions can share one base extension. Series, Matrix, Hashmap, and future Vector all build on RawBuf. Each is a few hundred lines of AILang, not a new Rust intercept.
• Cost localisation. The Rust-intercept layer is small and centralised at the base. Domain growth (more library extensions) does not enlarge the Rust footprint.
• Forward axis: SoA for records. When the day comes that Series should hold records (e.g. {price: Float, volume: Int}), the change is internal to RawBuf — its intercept dispatches on whether the element type is primitive (flat buffer) or record (struct-of-arrays). Series's AILang code does not change.

The plugin contract (consolidated)

Base extension:

• Kernel-tier module manifest.
• Rust codegen intercepts emitting LLVM IR for each operation.
• Optional C runtime — only when LLVM IR over existing runtime symbols cannot express the operation.

Library extension:

• Kernel-tier .ail module.
• Depends on one or more base extensions for its primitives.

Both register with the compiler via the kernel-tier auto-injection mechanism described above. Core has no per-extension code. The four mechanisms (type-scoped namespacing, new, kernel-tier modules, param-in) are all general — the same checker logic handles RawBuf, Series, and any future extension.

The migration of existing hardcoded primitive-instance intercepts (try_emit_primitive_instance_body) into the plugin registry is triggered by the first real base extension shipping (RawBuf). One registry mechanism for all intercepts, replacing the current single-case hardcoded list.

CLI / discovery (ail describe <T>) consults the workspace TypeDef registry and surfaces the kernel: true flag on the home module — uniform for base and library extensions; the user does not need to know which tier hosts the type.

Series as the first library consumer (built on RawBuf)

RawBuf — the base extension underneath

RawBuf a is a mutable, indexed, fixed-size buffer. Element type a restricted to {Int, Float, Bool} for the MVP via param-in. API:

• (new RawBuf (con T) (size: Int)) : own (RawBuf T) — allocates an uninitialised buffer.
• RawBuf.get : forall a. (borrow (RawBuf a), Int) -> a — indexed read. UB if index >= RawBuf.size(b); caller checks bounds.
• RawBuf.set : forall a. (own (RawBuf a), Int, a) -> own (RawBuf a) — indexed write. Linear signature: caller hands over the buffer, callee returns it after mutation. Under uniqueness inference, the codegen rewrites this into in-place mutation; under shared ownership it falls back to a full buffer copy (Issue #22 territory).
• RawBuf.size : forall a. (borrow (RawBuf a)) -> Int.

Mutation discipline: ownership-mode-tracked, not effect-tracked. The own/borrow signatures are the mutation contract. No RawBuf effect is declared. This is consistent with AILang's existing memory-model — RC + uniqueness as the canonical mutation story.

Storage at the LLVM-IR level is type-specialised: the Rust intercept registered for RawBuf dispatches on the element type at each call site and emits the appropriate getelementptr / load / store instructions over an opaque ptr allocated via @ailang_rc_alloc. No new C code; the existing RC machinery handles allocation, drop, and copy-on-share.

Series — the library extension

Series a is a bounded ring buffer with financial-style indexing (index 0 = newest). It is a plain AILang ADT in a kernel-tier .ail module:

(module series (kernel)

  (data Series (vars a)
    (param-in (a Int Float Bool))
    (ctor S (own (RawBuf a))     ; the storage
            (con Int)              ; lookback
            (con Int)              ; head index
            (con Int)              ; current count
            (con Int)))            ; total pushes ever

  (fn new
    (type (fn-type
      (params (con Int))
      (ret (own (Series a)))))
    (params lookback)
    (body
      (term-ctor Series S
        (new RawBuf lookback) lookback 0 0 0)))

  (fn push
    (type (fn-type
      (params (own (Series a)) (con a))
      (ret (own (Series a)))))
    (params s v)
    (body
      (match s
        (case (pat-ctor S buf lookback head count total)
          (term-ctor Series S
            (app RawBuf.set buf head v)
            lookback
            (app % (app + head 1) lookback)
            (if (app < count lookback) (app + count 1) count)
            (app + total 1))))))

  (fn at
    (type (fn-type
      (params (borrow (Series a)) (con Int))
      (ret (con a))))
    (params s i)
    (body
      (match s
        (case (pat-ctor S buf lookback head count total)
          (app RawBuf.get buf
            (app % (app + (app - head 1) (app + (app * lookback 2) i)) lookback))))))

  (fn len
    (params s)
    (body (match s (case (pat-ctor S buf lookback head count total) count))))

  (fn total_count
    (params s)
    (body (match s (case (pat-ctor S buf lookback head count total) total)))))

Push semantics: take Series by own, return a new Series with the buffer mutated (via RawBuf.set) and the bookkeeping fields updated. The match-and-rebuild pattern is the explicit linear state-threading style. Under uniqueness inference, both the RawBuf mutation and the Series ADT rebuild become in-place; the caller-side (let s_new (app Series.push s v) ...) pattern optimises to single-allocation, two-counter-update cost. Under shared ownership, push allocates a new Series wrapper (~40 bytes RC overhead) per call — the performance fallback that Issue #22 will narrow.

No Series effect. The mutation is mode-tracked. Caller signatures do not gain a [Series] or [RawBuf] effect entry; they only need the linear threading discipline.

Series.at uses financial-style indexing (0 = newest): buf_index = (head - 1 - i + 2*lookback) mod lookback. UB if i >= count; the caller checks via Series.len or Series.total_count. The + 2*lookback is a non-negative adjustment so the modulo arithmetic stays well-defined for small head values.

Forward axis: SoA for records (already accommodated)

Extending Series to hold records (Series {price: Float, volume: Int}) is purely a RawBuf change. The RawBuf intercept gains a dispatch path for record element types — primitive types stay flat (AoS), record types switch to struct-of-arrays internally with per-field parallel storage. RawBuf.get for a record-typed buffer materialises the record on demand from the parallel slots; RawBuf.set distributes the record's fields across the parallel slots. Series's AILang code does not change; the API surface is identical.

The Series milestone scope does not include record element types; that is a future axis. But the architecture accommodates it without revisiting Series itself.

Feature-acceptance argument

Series's three-clause feature-acceptance check is in the spec; the summary: LLM authors naturally reach for a bounded ring buffer for streaming workloads (clause 1 — the SMA worked example above is the evidence); the type structurally eliminates a class of off-by-one and warmup-handling bugs (clause 2); the bounded push-only mutation surface — gated by the Series effect — does not reintroduce the iterated-mutable-state bug class that the mut/var/assign removal addressed (clause 3 — see docs/specs/0052-kernel-extension-mechanics.md's clause-3 discussion).

The same three-clause check applies to each future kernel extension. The whitepaper does not pre-justify Matrix or Decimal or Time — each gets its own feature-acceptance gate when its spec is written.

Migration policy (pre-production frame)

AILang is pre-production. No external author depends on a particular surface; no compiled artifact in the wild. The implication for kernel-extensions: when a new mechanism is strictly better than an existing one for the use case it covers, the existing one is retired or repositioned, not preserved in parallel for compatibility.

Concrete consequences for the prep milestone:

• Type-scoped namespacing makes <module>.<TypeOrFn> non-canonical for type-associated operations. Existing examples and fixtures using std_maybe.from_maybe, std_pair.from_pair, etc., are rewritten to type-scoped form. Hash pins are refreshed in lockstep — each test crate (crates/ailang-core/tests/hash_pin.rs, crates/ailang-surface/tests/prelude_module_hash_pin.rs, and any other hash-pin file the prep recon enumerates) is walked, per the "hash-pin blast-radius audit" practice.

• The existing BareCrossModuleTypeRef / BadCrossModuleTypeRef diagnostics are repurposed: what they considered "bad" before (a bare type ref to a foreign module's type) becomes "bad" still, but the correct form is now type-scoped (when feasible) rather than module-qualified.

• The prelude module gains kernel: true. The hardcoded prelude-name paths in crates/ailang-surface/src/loader.rs:98-108 and crates/ailang-core/src/workspace.rs:308-311, 467, 2655 migrate to the generic flag-driven mechanism. Consumer- observable behaviour for prelude is unchanged (the 12 free fns remain callable bare); the code path no longer hardcodes any module name as special.

• Codegen intercepts: the existing try_emit_primitive_instance_body hardcoded list is migrated into the plugin registry as part of the Series milestone — when there is the first real external consumer, the mechanism graduates from "hardcoded for one case" to "registry for many cases".

The reason migration cost is not a decision driver: there is nothing to break externally, and rewrites inside the workspace are cheap. Choosing the right design now is the priority; cost of refactoring tests and fixtures is the project's own problem and is amortised over zero external consumers.

Coexistence with existing mechanisms

The mechanisms in this whitepaper interact with several pre-existing parts of the language. Each interaction is named here so the spec can quote the position:

• Classes / method dispatch. Orthogonal. Type-scoped namespacing accesses top-level defs in a type's home module; class-method dispatch (see method dispatch) resolves class method calls by type-driven instance lookup. Both can coexist on the same type — e.g. Series could someday have a Show instance (class method show dispatches via the instance) and a Series.dump free-fn def (accessed via type-scope). Both mechanisms remain.

• Algebraic effects. Reused unchanged. The Series effect is a new name in the flat effect-set; the (do effect/op args) syntax is the existing one (io/print_str is the precedent).

• RC + uniqueness. Series values are RC-managed like any heap value; drop is via a kernel-supplied @ailang_series_drop. Future Uniqueness inference (Gitea #22) can in principle recognise unique Series and elide RC ops on push — but the baseline does not depend on that.

• Heap-Str ABI. The Series runtime borrows the heap-Str RC-pointer-to-opaque-struct pattern (see str-abi). Same pattern: opaque ptr from the compiler's view; dedicated _new, _drop, and operation symbols in C; RC header in the struct prefix.

• Term::Ctor + pattern matching. Unchanged. term-ctor and pat-ctor keep their roles for named-ctor ADTs.

• term-ctor vs new. Different constructs for different needs. The choice point for a hypothetical hybrid (e.g. "every named ctor implicitly defines a new fn") is out of scope for this whitepaper — the two coexist by design.

Forward axes

This whitepaper does not commit to any future kernel extension. The shape it enables — domain-specific types as plugins — is intended to make a future Matrix, Decimal, Time, etc., a self-contained crate-level proposal each gated by its own feature-acceptance and brainstorm pass. The plugin contract above is the only commitment.

Cross-references

• Feature-acceptance gate applied to each kernel extension: feature-acceptance.
• Method dispatch (orthogonal to type-scoped namespacing): method-dispatch.
• Honesty rule (this whitepaper's STATUS contract): honesty-rule.
• Reference design for Series storage layout: /home/brummel/dev/RustAst/src/ast/rtl/series/ (external Rust project; not in this repo).
• Per-milestone specs: docs/specs/0052-kernel-extension-mechanics.md (prep milestone); future docs/specs/YYYY-MM-DD-series.md (Series milestone, written after prep closes).