Iter 16b.3: post-typecheck lift for LetRec with Let-bound captures

Adds path-2 from the 16b.2 planning entry. Desugar now defers LetRecs whose captures include Term::Let-bound names; a new ailang-check::lift_letrecs pass runs after typecheck and uses the elaborated env to resolve capture types. ailang-check learns a real Term::LetRec typing rule in synth and verify_tail_positions. - desugar: Term::LetRec arm gains a defer-arm; helpers promoted to pub for reuse by the lift pass; find_non_callee_use moved before classification. - ailang-check::synth/verify_tail_positions: real LetRec rules (effect-subset, locals install, recursive name in body+in_term). - ailang-check::lift.rs (new, 720 LOC): post-typecheck lift with post-order traversal, env-walk for capture-type resolution, fast-path skip when no LetRec is present. - ail::main.rs: build path now does load → check → desugar → lift_letrecs → codegen. - examples/local_rec_let_capture.{ailx,ail.json}: new fixture capturing a let-bound `threshold` in a recursive helper. - e2e + check unit tests: 106 → 110 (+4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 22:07:05 +02:00
parent d5f63bc3e5
commit ca30606aec
9 changed files with 1552 additions and 68 deletions
@@ -701,6 +701,7 @@ conversion in JOURNAL). A `seq` term evaluates `lhs` for its effects
            ├─ resolve names + assign hashes
            ├─ desugar (AST → AST, Iter 16a)
            ├─ typecheck (HM, effect rows)
+            ├─ lift_letrecs (post-typecheck AST → AST, Iter 16b.3)
            ├─ lower to MIR (SSA-like, named SSA values)
            ├─ emit LLVM IR (.ll)
            └─ clang -O2 *.ll -o binary  (links libgc for @GC_malloc)
@@ -716,6 +717,17 @@ in the `check` entry point continues to hash from the *original*
 on-disk module, not the desugared one, so `ail diff` and `ail manifest`
 report identities that match the canonical JSON the user is editing.

+The **lift_letrecs** pass (`ailang-check::lift_letrecs`, Iter 16b.3)
+runs **after** typecheck and **before** codegen, but only on the
+`build` / `run` paths — the `check` subcommand stops at typecheck
+and never sees a lifted module. It eliminates every `Term::LetRec`
+that the desugar pass left in place (the case where at least one
+capture is `Term::Let`-bound, so its type is only knowable after
+inference). The output is a module with synthetic `<hint>$lr_N`
+top-level fns appended, ready for codegen. Synthetic FnDefs added
+by this pass do **not** appear in `CheckedModule.symbols` — same
+invariant as the 16b.2 lifts in desugar.
+
 ## CLI

 ```
@@ -3697,3 +3697,193 @@ accumulates captures across nesting). 16d (chain-machinery
 exhaustiveness or `__unreachable__` — planning needed), 16e
 (`==` extension to Bool/Str/Unit), 17a (per-fn arena, gated)
 unchanged.
+
+## Iter 16b.3 — LetRec captures of Let-bound names
+
+**Goal.** Lift 16b.2's "fn/Lam-param captures only" restriction. A
+`(let-rec ...)` may now capture names bound by a `Term::Let` in the
+enclosing scope. Match-arm captures (16b.4), name-as-value (16b.5),
+Forall enclosing fn (16b.6), and nested LetRec mutual capture
+(16b.7) remain rejected at desugar time.
+
+**Architectural choice (path-2: post-typecheck lift).** The desugar
+pass cannot resolve a `Term::Let`-bound name's type — `Term::Let`
+carries no annotation; the value's type is inferred at typecheck.
+Three options were on the table (path-1 = stay-at-desugar with
+restrictions, path-2 = post-typecheck lift, path-3 = run a private
+inference inside desugar). Path-2 is the cleanest: typechecker is
+the single source of truth, and the lift now becomes a small
+AST-→-AST pass with O(1) type lookups against the typechecker's
+env. `Term::LetRec` reaches the typechecker only when the desugar
+pass deferred it — for the 16b.2 fast-path (KnownType captures
+only) the desugar still does the lift in one hop.
+
+**What shipped.**
+
+- `crates/ailang-core/src/desugar.rs` (1853 → 1931 LOC, +78). The
+  `Term::LetRec` arm now has three exits:
+  (a) `KnownType`-only captures → existing 16b.2 lift (unchanged).
+  (b) Any `LetBound` capture → reconstruct the `Term::LetRec`
+      with desugared sub-terms and return it (defer to
+      post-typecheck pass).
+  (c) `MatchArm` / `EnclosingLetRec` → panic with the same
+      follow-up-iter pointers as 16b.2.
+  The `find_non_callee_use` (16b.5) check moved to run before
+  classification so the diagnostic fires consistently regardless
+  of which exit is taken. Four helpers (`free_vars_in_term`,
+  `subst_var`, `subst_call_with_extras`, `find_non_callee_use`,
+  `pattern_binds`) were promoted from `fn` to `pub fn` so the
+  post-typecheck pass can reuse them.
+- `crates/ailang-check/src/lib.rs` (2887 → 3281 LOC, +394
+  including tests).
+  - `verify_tail_positions` and `synth` arms for `Term::LetRec`
+    replaced. `verify_tail_positions`: body is NOT in tail
+    position (it's a fn body — the LetRec name is what gets
+    tail-called); in_term inherits the enclosing context. `synth`:
+    peel any `Forall` defensively, validate param count, install
+    `name + params` in locals for the body, synth the body, unify
+    against `ret_ty`, check the effect-subset rule (same as
+    `Term::Lam`); restore locals; install `name` for the
+    in-clause; synth, return.
+  - New `pub fn check_and_lift(m) -> Result<(CheckedModule,
+    Module)>` runs check + desugar + `lift_letrecs` and returns
+    both the original-symbols `CheckedModule` and the lifted
+    module ready for codegen.
+- `crates/ailang-check/src/lift.rs` (new file, 720 LOC). The
+  `pub fn lift_letrecs(m: &Module) -> Result<Module>` post-pass.
+  - Fast-path: `contains_any_letrec(m)` returns `false` →
+    return input unchanged. Skips env construction so cross-
+    module modules (which we don't fully wire up here) are not
+    touched unless they actually contain a deferred LetRec.
+  - Builds a single-module env (builtins + module type defs +
+    module globals + imports + current_module) and walks every
+    `Def::Fn`'s body, threading a parallel `IndexMap<String,
+    Type>` of locals as scope. At each `Term::Let`, synth the
+    value's type to populate locals; at each `Term::Match`
+    arm, run a minimal `type_check_pattern_for_lift` to infer
+    pattern-arm bindings.
+  - At every `Term::LetRec`: post-order recurse first (so
+    nested LetRecs are lifted before their enclosing one),
+    re-run `find_non_callee_use` defensively, recompute
+    captures via `free_vars_in_term`, look up each capture's
+    type from `locals`, build the lifted `FnDef` (capture types
+    appended to params), append to a `lifted: Vec<Def>`
+    accumulator, rewrite call sites in body and in_term via
+    `subst_call_with_extras`, then `subst_var` for any
+    leftover bare references. Synthetic name `<hint>$lr_N`
+    seeded past the highest existing `*$lr_N` suffix in
+    `m.defs` so it cannot collide with a 16b.2 lift. Synthetic
+    FnDefs carry a `doc` of the form
+    `"Lifted by 16b.3 from let-rec '<name>' inside '<enclosing>'."`.
+  - The `subst_call_with_extras` and `subst_var` helpers come
+    straight from `ailang-core::desugar` — re-used rather than
+    duplicated (the brief asked for one or the other; re-use
+    via `pub` keeps the rewrite logic single-sourced).
+- `crates/ail/src/main.rs` (+25 LOC). `build_to` now goes
+  `load → check → per-module (desugar + lift_letrecs) → codegen`.
+  The `check` subcommand stays typecheck-only (no lift needed
+  for type-checking). Codegen's internal `desugar_module`
+  call is harmless on a lifted module — no `Term::LetRec`
+  remains, so the LetRec arm is never invoked.
+- `examples/local_rec_let_capture.ailx` + `.ail.json` (48 LOC
+  source). `count_below(n)` returns how many `i` in `1..=n` are
+  strictly less than a `let threshold = (app + 5 5)` computed
+  inside the enclosing fn. The recursive helper `loop` captures
+  `threshold` (Let-bound; type unknown until typecheck) and
+  recurses on its own counter `i`. The lift produces
+  `loop$lr_0(i: Int, n: Int, threshold: Int) -> Int` and
+  rewrites every `(app loop X)` to `(app loop$lr_0 X n threshold)`.
+  The let-value is `(app + 5 5)` rather than a literal so the
+  type-synthesis path is exercised. Drives at 0, 5, 15 →
+  output `0\n5\n9\n`.
+- `crates/ail/tests/e2e.rs::local_rec_let_capture_demo` (+18):
+  e2e count 39 → 40.
+- `crates/ailang-core/src/desugar.rs::tests`. The 16b.2-era
+  `let_rec_capture_let_binding_panics` test was repurposed
+  into a positive test
+  `let_rec_capture_let_binding_is_deferred_to_post_typecheck`
+  that asserts the desugar leaves the LetRec in place (no
+  lifted fn appended; original LetRec still reachable). Net
+  test count unchanged.
+- `crates/ailang-check/src/lib.rs::tests` (24 → 27, +3):
+  `letrec_with_let_binding_capture_typechecks` (positive),
+  `letrec_body_wrong_return_type_is_rejected` (negative — body
+  returns Bool but declared Int), and
+  `lift_letrecs_on_let_capture_produces_synthetic_fn` (asserts
+  synthetic FnDef added with augmented signature and call sites
+  rewritten).
+- `docs/DESIGN.md` (+12 LOC). Pipeline section gains the
+  `lift_letrecs` stage between `check` and `codegen`, with a
+  paragraph clarifying it runs only on the `build` / `run`
+  paths and that synthetic FnDefs do not appear in
+  `CheckedModule.symbols`.
+
+**The deferral mechanism.** Given
+`(let y (app + 5 5) (let-rec helper (params x) (type Fn(Int) -> Int)
+   (body (app + x y)) (in (app helper 1))))`
+inside an enclosing fn:
+
+- Desugar runs. The LetRec's outer scope is `{n: Int (fn-param),
+  y: LetBound}`. Free vars of `body` minus `{name, params}` =
+  `{+, y}`; intersection with scope = `{y}`. `y` is `LetBound`,
+  so the all-or-nothing classifier sees `has_let_bound = true`
+  and reconstructs the LetRec with desugared sub-terms instead
+  of lifting.
+- Typecheck runs. The new `Term::LetRec` arm in `synth` extends
+  locals with `helper: Fn(Int) -> Int` and `x: Int`, synths the
+  body to `Int`, unifies with `ret_ty`, and accepts.
+- `lift_letrecs` runs. Walking outer's body, it threads locals.
+  At the `Term::Let`, `synth_type` resolves `(app + 5 5) → Int`
+  and inserts `y: Int` into locals. At the `Term::LetRec`, the
+  capture analyser collects `{y}` and reads its type as `Int`.
+  Builds `helper$lr_0(x: Int, y: Int) -> Int`, rewrites
+  `(app helper 1)` → `(app helper$lr_0 1 y)`, appends the
+  lifted `FnDef`. The Term::LetRec node is replaced by its
+  rewritten in-clause.
+- Codegen runs on the lifted module. No `Term::LetRec` remains;
+  the four `unreachable!("Term::LetRec eliminated by desugar")`
+  arms in codegen stay valid (the message is a slight
+  misnomer post-16b.3 — by the time codegen runs, every LetRec
+  has been eliminated by desugar OR lift_letrecs — but the
+  invariant holds).
+
+**Tests: 106 → 110 (+4).**
+
+- e2e: 39 → 40 (`local_rec_let_capture_demo`).
+- `ailang-check::tests`: 24 → 27 (the three new LetRec tests).
+- `ailang-core::desugar::tests`: 9 → 9 (one `#[should_panic]`
+  test repurposed to a positive defer-check test — net 0).
+
+**Cumulative state, post-16b.3.**
+
+- Stdlib unchanged (5 modules, 29 combinators).
+- `Term` enum: 11 variants (unchanged). All pre-16b.3 fixtures
+  hash bit-identically — additive at the typechecker / new-pass
+  level only.
+- Compiler stages: load → desugar → typecheck →
+  **lift_letrecs (16b.3)** → codegen. The `check` subcommand
+  stops at typecheck and skips the lift; `build` / `run` go all
+  the way through.
+- The four `unreachable!("Term::LetRec eliminated by desugar")`
+  arms in `ailang-codegen` remain correct: by the time codegen
+  runs, no LetRec survives — desugar lifts the 16b.1/16b.2
+  cases, lift_letrecs catches the 16b.3 case. The two
+  `unreachable!` arms in `ailang-check` were replaced with the
+  new typing rule (`verify_tail_positions` and `synth`).
+- Compiler bugs surfaced and fixed in dogfood since 14a: 5/5
+  (unchanged).
+
+**Queue update post-16b.3.** 16b.3 done. Open: **16b.4** (LetRec
+captures of match-arm pattern bindings — needs ADT-def lookup +
+constructor-field substitution; would slot into the same
+`lift_letrecs` pass with extended pattern-arm walk), **16b.5**
+(closure conversion — lifts the "name-as-value-only" restriction
+for both LetRec and Lam; out of scope for this iter line),
+**16b.6** (LetRec inside a `Type::Forall`-quantified fn — needs
+synthesised `Forall` for the lifted signature), **16b.7**
+(nested LetRec mutual capture — generalised lifting that
+accumulates captures across nesting; partly already handled by
+the post-order traversal in `lift_letrecs` but the mutual case
+is genuinely harder). 16d (chain-machinery exhaustiveness or
+`__unreachable__`), 16e (`==` extension to Bool/Str/Unit),
+17a (per-fn arena, gated) unchanged.