Iter 14e: explicit, verified tail calls

Decision 8 ships. Term::App and Term::Do gain tail: bool with
serde-default false and skip-when-false serialisation. New
typecheck pass verify_tail_positions enforces tail-position rules
(Scheme-style propagation through match arms, seq.rhs, let body,
lam body). Codegen emits musttail call for marked App calls.

Hash invariance verified: only the two migrated print_list defs
(list_map_poly.print_list, sort.print_list) changed hashes; all
other defs across all 18 fixtures kept bit-identical hashes —
confirms the skip-when-false serialisation rule works.

Tests 76 -> 79: tail_call_in_non_tail_position_is_rejected,
tail_call_in_tail_position_is_accepted, plus an IR-grep e2e test
asserting that print_list's recursive call site emits musttail
in the lowered IR. Existing 25 e2e tests unchanged in behaviour
(map -> [2,3,4], sort -> sorted list).

IR evidence at the recursive site:
  %v7 = musttail call i8 @ail_list_map_poly_print_list(ptr %v6)
  ret i8 %v7

Two deviations called out in the implementer report and JOURNAL:
1. tail-do uses tail call, not musttail. Cross-type return
   (runtime helpers return i32, AILang Unit is i8) would have
   LLVM reject musttail. Path is implemented but not exercised
   by any current fixture; proper fix is runtime-helper signature
   change, punted.
2. block_terminated flag in codegen so tail-call emit
   (musttail call + ret) doesn't get a duplicate trailing ret
   from surrounding code (match-arm phi, fn-body, lambda thunk).
   Internal plumbing; required for IR well-formedness.

Form (A) productions now at ~30, exactly the constraint-1
budget. Future surface additions need to retire something or
explicit-budget-rebalance in DESIGN.md.

GC notes from implementer survey land in JOURNAL:
- Allocations cluster in lower_ctor; every term-ctor does
  malloc(8+8n).
- Tail recursion does not reduce alloc pressure, only stack.
  For map-style ctor-blocked recursions, allocation IS the
  bottleneck.
- Per-fn arena is sound only when fn return type contains no
  boxed ADT. Most current fixtures violate this.

Plan 14f: Boehm conservative GC (GC_malloc, -lgc) as a first
cut. Single-iter integration, no AST/schema change. Stress
test: build a 100k Cons list, observe RSS doesn't blow up.

After 14f the language is feature-complete enough for stdlib
work (15a).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/DESIGN.md

@@ -429,6 +429,21 @@ fixtures round-trip identically through `print → parse → canonical
 JSON`; the three hand-written `.ailx` exhibits parse to canonical
 JSON identical to their corresponding `.ail.json` files.
 
+3. **Tail-call surface (Iter 14e).** Decision 8 ships two new
+   productions, both positional analogues of their non-tail
+   counterparts. Their result terms set `Term::App.tail = true` /
+   `Term::Do.tail = true`; the typechecker's `verify_tail_positions`
+   pass enforces that the marker is only used in tail position.
+
+   ```
+   tail-app-term ::= "(" "tail-app" term term+ ")"
+   tail-do-term  ::= "(" "tail-do" ident term* ")"
+   ```
+
+   Production count after Iter 14e: ~30, still inside the 30-rule
+   constraint-1 budget. No new lexical rule (`tail-app` / `tail-do`
+   are bare ident tokens; no special casing).
+
 ## Decision 7: redundancy removal — `Term::If` is not a primitive
 
 `Term::If { cond, then, else_ }` is semantically a subset of
@@ -454,6 +469,63 @@ No schema version bump (no third-party consumes `ailang/v0`).
 Hash invalidation for the three migrated fixtures (`sum`, `sort`,
 `max3`) is intentional; the new hashes become the new identity.
 
+## Decision 8: explicit, verified tail calls
+
+For an LLM author, recursion is the natural iteration form
+(`\n. if n == 0 then () else loop(n-1)` is what I reach for, not
+a `for`-loop). Without a tail-call guarantee, every recursive
+program has a silent stack-depth ceiling that no compile-time
+diagnostic warns about. That is exactly the class of correctness
+hazard the language exists to eliminate.
+
+Solution: explicit, verified tail calls.
+
+- **AST.** `Term::App { fn, args, tail: bool }` and
+  `Term::Do { op, args, tail: bool }` gain a `tail` flag,
+  serde-defaulting to `false` so existing fixtures load with
+  `tail: false` and their hashes stay bit-identical.
+- **Typecheck.** A new pass `verify_tail_positions(fn_body)`
+  runs after the main type-check. It walks the body with an
+  `is_tail_context: bool` threaded down. The flag is `true` at
+  the start, `true` for the body of every `Term::Match` arm,
+  `true` for the right operand of `Term::Seq`, `true` for the
+  body of `Term::Let`, `true` for the body of `Term::Lam` (each
+  Lam opens its own tail scope). The flag is `false` for: args
+  of any `App`/`Do`/`Ctor`, scrutinee of `Match`, left of
+  `Seq`, condition of `Let`-bound expression. When the walker
+  visits an `App { tail: true }` or `Do { tail: true }`, the
+  flag must be `true` at that visit; otherwise emit diagnostic
+  `tail-call-not-in-tail-position`.
+- **Codegen.** Emit `musttail call` (LLVM IR) for marked calls
+  instead of plain `call`. LLVM rejects at IR-verification
+  time if the call cannot physically be a tail call (calling
+  convention mismatch, signature divergence, etc.). The reject
+  surfaces as a hard build error, not a silent runtime
+  surprise.
+- **Form (A).** Two new keywords: `tail-app`, `tail-do`.
+  Productions are positional analogues of `app`/`do` with
+  `tail: true` set on the resulting term. EBNF gains 2 lines;
+  total production count goes from ~28 to ~30, still inside
+  the constraint-1 budget.
+
+**What this does NOT promise.** Per the 14d tail-call survey,
+many existing recursive calls are *not* in tail position
+because they are arguments to constructor calls (e.g.
+`Cons (f h) (map f t)`). 14e adds annotation + verification;
+it does **not** add a CPS transform or accumulator-form
+rewrite. Programs whose recursion is constructor-blocked
+will continue to be stack-bounded by recursion depth. The
+canonical authoring pattern in such cases is to write the
+accumulator-form variant (`map_acc`, `fold_left`, etc.)
+explicitly. The stdlib (15a onward) ships both forms where
+relevant.
+
+The 14d migration of existing fixtures will be partial: only
+`print_list`-style terminal recursions get marked. The
+constructor-blocked recursions in `map`, `sort`, `insert`
+remain unmarked — they cannot benefit from `musttail`
+without a source-level rewrite.
+
 ## Mangling scheme (Iter 5c)
 
 All AILang functions are mangled to `@ail_<module>_<def>` — even in the

docs/JOURNAL.md

@@ -1874,6 +1874,134 @@ patterns; `map`/`sort` style stays stack-bounded by depth,
 which makes 14f (GC) the more important iter for handling
 long lists than 14e by itself.
 
+## Iter 14e — explicit, verified tail calls
+
+Decision 8 ships. `Term::App` and `Term::Do` carry a `tail: bool`
+flag (serde-default false, skip-when-false in serialisation).
+A new `verify_tail_positions` typecheck pass walks each fn body
+and Lam body with an `is_tail_context: bool` threaded down per
+the standard Scheme rules, rejecting any `tail: true` call that
+sits outside tail position. Codegen emits `musttail call` for
+marked App calls.
+
+**Hash deltas (intentional, only the migrated calls):**
+
+| def | hash before | hash after |
+|---|---|---|
+| `list_map_poly.print_list` | unchanged 14c value | new |
+| `sort.print_list` | unchanged 14d value | new |
+
+All other defs across all 18 fixtures kept bit-identical hashes.
+This is the canary for the `skip_serializing_if = "is_false"`
+serde rule: an unmarked `App`/`Do` serialises identically to its
+pre-14e form, so untouched defs cannot drift.
+
+**Tests: 79/79 (was 76, +3).** New tests:
+- `tail_call_in_non_tail_position_is_rejected` (check unit) —
+  asserts the diagnostic fires on a deliberately-misplaced
+  `tail: true` call (e.g. as a `Cons` arg).
+- `tail_call_in_tail_position_is_accepted` (check unit) —
+  asserts the verifier accepts the canonical `print_list` shape.
+- `iter14e_print_list_recursion_emits_musttail` (e2e IR-grep) —
+  builds `list_map_poly`, dumps IR, asserts the recursive call
+  site uses `musttail call`. This is the only direct evidence
+  that the AST flag actually reaches LLVM.
+
+**IR-snapshot evidence** at the recursive site of
+`print_list` after migration:
+
+```
+%v7 = musttail call i8 @ail_list_map_poly_print_list(ptr %v6)
+ret i8 %v7
+```
+
+`musttail` followed immediately by `ret` of the same SSA value —
+LLVM's terminator rule satisfied. Smoke: `list_map_poly` →
+`2/3/4`, `insertion_sort_orders_list` → identical sorted list.
+Behavior unchanged; only the calling shape did.
+
+**Two implementer deviations (called out, both reasonable):**
+
+1. **`tail-do` falls back to `tail call`, not `musttail`.** LLVM
+   `musttail` requires identical caller/callee return types. AILang
+   IO ops dispatch through runtime helpers (`printf`/`puts`)
+   returning `i32`, while AILang's `Unit` lowers to `i8`. Cross-type
+   `musttail` would be rejected by the verifier. So `Term::Do` with
+   `tail: true` lowers to `tail call` (the LLVM optimisation hint,
+   not the guarantee), then `ret i8 0`. No fixture currently uses
+   `tail-do`, so the path is implemented but not exercised
+   end-to-end. Proper fix: change runtime helper signatures to
+   return `i8`. Punted; not blocking.
+
+2. **`block_terminated` plumbing in codegen.** A `tail-app` /
+   `tail-do` emits `musttail call ... ret ...` directly and sets
+   `self.block_terminated = true`. Surrounding code (match-arm phi
+   construction, fn-body trailing-ret, lambda-thunk trailing-ret)
+   checks the flag and skips the fall-through emit. When every
+   match arm is a tail call, the join block is omitted entirely.
+   This was unavoidable to keep the IR well-formed — adding a
+   second `ret` after a `musttail call`+`ret` would be a verifier
+   error. The flag is a small piece of state but it's the right
+   shape for "the current basic block has been definitively
+   terminated by a sub-emission".
+
+**Form (A) at constraint ceiling.** Two new productions
+(`tail-app-term`, `tail-do-term`) bring the count to ~30, which
+is exactly the constraint-1 budget. Future productions need to
+either retire something or accept an explicit budget rebalancing
+in DESIGN.md.
+
+**GC notes from the implementer (informs 14f).**
+
+- **Allocations cluster in `lower_ctor`** (~line 850 of codegen).
+  Every `term-ctor` does `malloc(8 + 8 * n)`. In `print_list` we
+  allocate nothing per recursion (just match + read fields +
+  recurse); allocations come from `map`, from `main`'s
+  list-building Cons chain, and from any other user code that
+  builds ADTs.
+- **Lambda envs and closure pairs allocate too** (`lower_lambda`).
+  Closure pair: `malloc(16)`. Env block: `malloc(env_size)`.
+  Direct-application closures (the common case for HOF args)
+  could be arena'd cleanly because the closure dies after the
+  call returns. Stored or returned closures escape.
+- **Tail recursion does NOT reduce allocation pressure**, only
+  stack depth. For `print_list`-style recursions there's no
+  allocation to begin with, so the win is purely stack-bounded.
+  For `map`-style ctor-blocked recursions, each step allocates
+  one new `Cons` box — that's where allocation-side work pays
+  off.
+- **The "obviously safe" arena boundary** is a fn whose return
+  type contains no boxed ADT (i.e. returns `Int`/`Bool`/`Unit`/
+  `Str` only). All ADT boxes allocated inside such a fn cannot
+  escape; an arena freed at fn return is sound by construction.
+  Most current fixtures violate this — `map`, `sort` return ADTs
+  — so a per-fn-arena scheme alone won't carry. Need a heap
+  with GC for escaping allocations.
+
+This narrows 14f's design space: probably **Boehm conservative
+GC across the board** as a first cut (`GC_malloc` substituted
+for `malloc`, `-lgc` linked, no language change). Add a
+per-fn-arena optimisation later for non-escaping cases if the
+profile justifies it. Boehm is a one-iter shot; arenas would be
+a multi-iter design pass with escape analysis.
+
+**Plan 14f.** Boehm-GC integration. Concretely:
+- `GC_malloc` instead of `malloc` in lowered IR.
+- `-lgc` added to the clang link command in `ailang-codegen`'s
+  build path (probably in the CLI, since the codegen crate
+  emits IR text and clang is invoked downstream).
+- Conservative scan handles AILang's stack and globals out of
+  the box.
+- Verify on a stress test: build a list of 100k Cons cells in
+  `map`, run, observe RSS doesn't blow up. Boehm collects
+  unreachable boxes during allocation pressure.
+- No AST or schema change. No language-level change.
+
+After 14f, the language is "done enough" for stdlib (15a).
+Anything else (records as a primitive, nested patterns, local
+recursive let, type classes) can layer on later without forcing
+stdlib rewrites.
+