From 91b9bf0581425f1ac4a45ebd4e5465052566caaf Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 7 May 2026 13:05:46 +0200 Subject: [PATCH] Iter 8c: docs (DESIGN.md + JOURNAL.md) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DESIGN.md: - Term schema: add `ctor`, `match`, `lam` rows. The schema fragment is now exhaustive for the supported language; prior versions silently dropped Iter 3/Iter 8 additions. - "What is not (yet) supported": closures-with-capture moves out of pending. New positive bullet for anonymous lambdas. Polymorphism added as an explicit pending bullet (Forall is parseable but not inferred). Smoke-test list extended with closure.ail.json. JOURNAL: append Iter 8 entry covering both 8a (closure-pair ABI) and 8b (Term::Lam + capture). Includes the rationale for skipping TIR (KISS won — `synth` already provides enough type info), the closure- conversion sketch, hash stability check (existing examples produce the same fn hashes pre vs post Iter 8), and the Iter 9 plan with two candidates ranked. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/DESIGN.md | 38 +++++++++---- docs/JOURNAL.md | 138 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+), 11 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index a455db0..dc0a628 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -178,9 +178,20 @@ hashes stay bit-identical. { "t": "let", "name": "", "value": Term, "body": Term } { "t": "if", "cond": Term, "then": Term, "else": Term } { "t": "do", "op": "/", "args": [Term...] } +{ "t": "ctor", "type": "", "ctor": "", "args": [Term...] } +{ "t": "match", "scrutinee": Term, "arms": [Arm...] } +{ "t": "lam", + "params": [""...], + "paramTypes": [Type...], + "retType": Type, + "effects": [""...], + "body": Term } ``` In the MVP, `do` is only a direct call to a built-in effect op (no handler). +A `lam` term constructs an anonymous function value; free variables of +its body are captured from the enclosing scope (see Iter 8 closure +conversion in JOURNAL). ### Type @@ -229,21 +240,22 @@ ail build — full pipeline → binary ## What is not (yet) supported -Snapshot of the boundary at the end of Iter 7. Items move out of this list +Snapshot of the boundary at the end of Iter 8. Items move out of this list as iterations land; the JOURNAL records the exact iteration. -- No closures with capture / no anonymous lambdas. Both will require a - typed IR stage (TIR) and closure conversion in lowering. First-class - function references (top-level fns as values, fn-typed parameters, - indirect calls) **are** supported as of Iter 7 — see below. - No effect handlers — only the built-in IO and Diverge ops. - No refinements / SMT escalation. +- No polymorphism in inference. `Type::Forall` is parseable but the + typechecker rejects polymorphic uses inside a body (`PolymorphicNot + Supported`). Generic functions must be monomorphised by the author + via separate top-level defs. - No cross-module ADTs. ADTs are local to a module; ctor names must be unique within their module but may collide across modules. - No visibility rules in imports. Every top-level def of an imported module is reachable; there is no `pub` / `priv`. -- No GC. The `Ctor` heap layout leaks. Acceptable for current example - programs; required before any longer-running program. +- No GC. ADT boxes, lambda envs, and closure pairs all leak. Acceptable + for current example programs; required before any longer-running + program. What **is** supported (and used as the smoke test for the pipeline): @@ -256,13 +268,17 @@ What **is** supported (and used as the smoke test for the pipeline): - **Imports + qualified cross-module references** via dotted names (Iter 5). - **First-class function references** (Iter 7). A top-level fn name (or - qualified `prefix.def`) used as a `Term::Var` is a fn-pointer value of - AILang type `Type::Fn { ... }` and LLVM type `ptr`. Fn-typed parameters - may be called directly as `f(args)`. No capture, no lambdas — the only - way to construct a fn-value is to refer to a top-level def. + qualified `prefix.def`) used as a `Term::Var` is a fn-value. +- **Anonymous lambdas with capture** (Iter 8). `Term::Lam` constructs a + closure that captures any free variables of its body from the + enclosing scope. All fn-values share a single ABI: a `ptr` to a + closure pair `{ thunk_ptr, env_ptr }`. Top-level fns get an auto- + generated adapter and a static closure pair (env = null) so they + remain passable as values without heap overhead. Pipeline regression smoke tests: - `examples/sum.ail.json` → prints 55 (recursion, arithmetic). - `examples/list.ail.json` → prints 42 (ADTs + match). - `examples/hof.ail.json` → prints 42 (first-class fn-refs, indirect call). +- `examples/closure.ail.json` → prints 42 (lambda capturing a let-bound var). diff --git a/docs/JOURNAL.md b/docs/JOURNAL.md index c6090a3..871a738 100644 --- a/docs/JOURNAL.md +++ b/docs/JOURNAL.md @@ -519,3 +519,141 @@ working, the main delta is: a fn-value also needs an environment pointer, the sidetable becomes per-value (heap-allocated), and the calling convention shifts to `(env_ptr, args...)`. Touches every existing call path — that's why it gets its own iteration. + +## 2026-05-07 — Iter 8 done: closures with capture (no TIR needed) + +Iter 7's plan named TIR as the prerequisite for closures. On +inspection that bundling was wrong — TIR is one possible +implementation strategy, not a structural requirement. The +typechecker already attaches enough type information through `synth` +that the codegen can read capture types out of `self.locals` +directly. So Iter 8 ships closures **without** introducing TIR. KISS +won. + +The work split into two commits: + +**Iter 8a — closure-pair ABI flip.** Every fn-value is now a `ptr` to +a heap or static closure pair `{ thunk_ptr, env_ptr }`, regardless of +whether it came from a lambda or a top-level def reference. To keep +top-level-fn references cheap, every top-level fn auto-emits: + +```llvm +define @ail___adapter(ptr %_env, ) { + %r = call @ail__() + ret %r +} +@ail___clos = constant { ptr, ptr } { @adapter, null } +``` + +`Term::Var` resolving to a top-level fn returns the address of +`_clos`, never the bare fn pointer. `emit_indirect_call` was +rewritten to GEP+load both halves and call `thunk(env, args...)`. +Direct calls (statically-known callees in `Term::App`) bypass the +adapter and stay at the original speed. + +The Iter 7 hof example (`apply(inc, 41)`) continues to print 42 +unchanged — only the IR shape changed, not the source. IR snapshot +files for sum/list/max3/hello/ws_main were refreshed. + +**Iter 8b — Term::Lam + capture + lambda lifting.** New AST node: + +```jsonc +{ "t": "lam", + "params": ["x"...], + "paramTypes": [Type...], + "retType": Type, + "effects": ["..."], + "body": Term } +``` + +Param/return types are explicit. The typechecker accepts the +declared `Type::Fn` shape, checks the body's type against `retType`, +and verifies that body effects are a subset of the declared lambda +effects (no row polymorphism in the MVP). Constructing a lambda is +pure; the act of *calling* picks up the declared effects, via the +existing App branch. + +Codegen does textbook closure conversion: + +1. **Free-variable analysis.** `collect_captures` walks the body + skipping builtins (`+`, `==`, ...), the current module's top- + level fns, and qualified `prefix.def` names. The remainder are + captures. Inner lambdas contribute their own free vars upward. +2. **Lift to thunk.** For each lambda, generate a fresh + `@ail___lam(ptr %env, params...)`. State the body + into a side buffer (the emitter's `body`/`locals`/`counter` are + saved and reset, then restored). Captures and lambda params are + pushed as named locals so the body lowering finds them. The + thunk text goes into a `deferred_thunks` queue and is appended + after the parent fn's `}` — LLVM IR doesn't care about fn order. +3. **Pack at the use site.** In the OUTER body emit: + + ```llvm + %env = call ptr @malloc(i64 <8 * captures>) + ; for each capture i: store at offset 8*i + %clos = call ptr @malloc(i64 16) + ; store thunk_ptr at offset 0, env at offset 8 + ``` + + `%clos` is the value returned by the Lam term. Its sig is + registered in the sidetable so subsequent indirect calls work. + +4. **Capture sigs propagate.** A fn-typed capture (e.g. capturing a + fn-typed param of an outer scope) keeps its FnSig in the thunk's + sidetable, so the captured fn can still be indirect-called from + inside the lambda. + +Capture layout uses 8-byte slots regardless of LLVM type. Typed +load/store reads only the bytes it needs — wasted padding for `i1` +and `i8` is fine at this scale. + +**Architecture self-check:** + +- *Would I use this language now?* Yes for substantially more cases. + `let n = 3 in apply(\\x. x + n, 39)` is the example I would have + reached for in Iter 6 and bounced off. It now compiles and runs. + `map`/`fold`/`filter` over user-supplied predicates are within + reach — only the absence of polymorphism still forces author-side + monomorphisation. +- *Did I think of everything?* Hash stability checked manually: + `examples/sum.ail.json` produced the same fn hashes + (`db33f57cb329935e`, `d9a916a0ed10a3d3`) before and after Iter 8. + Existing modules without `Term::Lam` serialise bit-identically. ✓ +- *Consistency:* DESIGN.md "What is not (yet) supported" rewritten + in the same edit. The Term schema gained `lam`, `ctor`, `match` + rows that were already supported but had been omitted from the + schema fragment. Now the doc is exhaustive for the supported + language. +- *Visualisation:* `ail describe` already renders Lam terms (added + pretty-printer rule), and the codegen IR for `closure.ail.json` + reads as a textbook closure-conversion lowering. +- *KISS check:* I considered three alternatives and all were + strictly worse — fat-pointer ABI (aggregate-passing concerns), full + TIR layer (large rewrite), uniform heap pair without static-closure + optimisation (regressed Iter 7 to one malloc per fn-value escape). + +**Tests:** 49 green (was 48 after Iter 7). One new e2e: +`closure_captures_let_n` builds and runs `examples/closure.ail.json` +asserting "42". IR snapshot files refreshed for the per-fn adapter + +static-closure scaffold — only structural delta. + +**Plan iteration 9:** + +Two candidates, both real pain points: + +1. **Polymorphic inference.** Make `Type::Forall` actually work in + `synth` — instantiate fresh type variables at each use site, allow + `let id = \\x. x in (id 1, id true)`. This unblocks generic + `map`/`fold`/etc. without per-type clones. Probably small (~150 + LOC in the typechecker; codegen already monomorphises by + instantiation when it lowers the call). + +2. **GC / region reclamation.** Right now ADT boxes, lambda envs, + and closure pairs all leak through the program's lifetime. A + minimal mark-and-sweep over a tagged heap would let us run real + programs. Bigger lift, ~400-600 LOC plus runtime support. + +Leaning toward (1) for the next iteration: it's the smaller bite +*and* the bigger expressivity unlock. (2) becomes acute only when +someone tries to run an unbounded loop, which the current examples +don't.