iter loop-recur.3: codegen — real LLVM-IR lowering + run-to-value E2E (milestone terminal)

Third and terminal iteration of the standalone loop/recur
milestone (plan eae73bf). Replaces the iter-1 lower_term
CodegenError::Internal stub for Term::Loop/Term::Recur with real
LLVM-IR lowering: loop binders as entry-block allocas (mut.3
pending_entry_allocas, reusing mut_var_allocas so the existing
Term::Var load path is byte-unchanged), a fresh loop-header block,
recur stores + back-edge br, a loop_frames stack saved/restored at
the lambda boundary. clang -O2 mem2reg promotes the allocas to
phi. Four Boss design calls implemented verbatim and journalled
(alloca-not-hand-phi; mut_var_allocas reuse; emergent loop-exit
via the if/match join; single block_terminated field + parallel
SET site). Diff confirmed surgical: codegen/lib.rs 3 hunks (field
+ init + stub->2-arms), lambda.rs 2 hunks (save+restore); zero
edits to any existing block_terminated SET/READ site, tail-app
lowering, or verify_tail_positions (Boss call 4, the spec-pinned
invariant). Three new .ail fixtures: sum_to->55,
deep-n 1e6->500000500000 (clause-2 correctness made executable),
infinite-loop build-only. Codegen-only: no schema/typecheck
change; hash pins + drift trio stay green untouched.

One DONE_WITH_CONCERNS (T3): the plan's `cargo test ... tail`
filter resolved to no tests; ran via real names + the full 619/0
which subsumes it (feedback_plan_pseudo_vs_reality class, no
behaviour change). Boss systemic fix folded in: planner SKILL.md
Step-5 gains item 8 (verification-command filter strings must
resolve) — the second planner-meta-gap this milestone surfaced.

cargo test --workspace 616 -> 619 / 0 red (Boss-reran
independently); the 3 loop/recur e2e explicitly green. All three
components shipped: the loop/recur milestone is structurally
complete. Milestone-close audit + fieldtest is the next step.
This commit is contained in:
2026-05-17 23:57:46 +02:00
parent eae73bf320
commit edd2558d35
10 changed files with 434 additions and 3 deletions
@@ -0,0 +1,188 @@
# iter loop-recur.3 — Codegen (Component 5) + run-to-value E2E
**Date:** 2026-05-17
**Started from:** eae73bf320440f6b031715e72c1f20b0ac86e024
**Status:** DONE
**Tasks completed:** 3 of 3
## Summary
The **terminal iteration** of the standalone-`loop`/`recur` milestone:
the iter-1 `lower_term` `CodegenError::Internal` stub for
`Term::Loop`/`Term::Recur` is replaced with real LLVM-IR lowering, so
a `sum_to`-class loop program builds and runs to the correct printed
value, a deep-`n` variant is safe by construction, and an infinite
loop compiles. Loop binders are loop-carried values lowered as
entry-block allocas (the mut.3 `pending_entry_allocas` mechanism)
registered in the existing `mut_var_allocas` map, so the existing
`Term::Var` load path resolves them with zero new Var code
(representation-sharing); a fresh `loop.header.<id>` block is reached
by an unconditional `br` from the pre-header, `recur` lowers each arg
to SSA *before* any store (simultaneous positional rebind), stores
into the binder allocas, back-edges `br` to the header, and sets the
single existing `block_terminated` field at its OWN new emit site so
the loop's exit value is the emergent product of the existing
`if`/`match` join (no separate loop-result phi/exit-block). A new
`loop_frames` codegen stack lets `recur` find its target; it is
saved/reset/restored at the single lambda-lowering boundary exactly
as mut.3's `mut_var_allocas` triple. `clang -O2` mem2reg promotes
the binder allocas to the phi nodes the spec's implementation-shape
describes. This is a **codegen-only iteration**: no schema/AST/serde
change, no typecheck change; hash pins (`loop_recur` + iter13a) and
the drift trio + `carve_out_inventory` stay green untouched
(confirmed, not modified — empirical proof none scans the codegen
region). `cargo test --workspace` 616 → 619 / 0 red (the 3 new e2e
tests). After this iter the milestone is structurally CLOSED;
milestone-close (audit/fieldtest) is the Boss's post-iter call.
## Per-task notes
- iter loop-recur.3.1: positive `sum_to` run-to-value E2E (RED →
GREEN). RED observed verbatim: `ail build` on `loop_sum_to_run.ail`
fails with the iter-1 stub `internal: Term::Loop/Term::Recur
lowering lands in loop-recur iter 3` (parse + iter-1/iter-2
typecheck pass; only codegen stubbed — confirmed via direct `ail
build`). GREEN: `loop_frames` field + init, lambda-boundary
save/reset/restore, the two real `Term::Loop`/`Term::Recur` arms
replacing the single stub arm. `loop_sum_to_run.ail``55`;
`cargo build -p ailang-codegen` Finished; full e2e suite 88/0.
Plan's literal arms compiled as-written — no plan-vs-reality
substitution needed (pre-grounding confirmed `mut_var_allocas:
(String, Type)` and `lower_term -> (String /*ssa*/, String
/*llvm_ty*/)` matched the plan's assumptions exactly).
- iter loop-recur.3.2: deep-`n` safe-by-construction +
infinite-loop-compiles (test+fixture only; the codegen mechanism
shipped in 3.1, so no separate RED). `loop_sum_to_deep.ail` runs
1e6 iterations via the back-edge → `500000500000` with no stack
growth (the spec clause-2 correctness claim made executable —
mem2reg promoted the binder allocas so the loop is an iterative
back-edge, not a stack-growing recursion). `loop_forever_build.ail`
(`spin`'s loop has no non-`recur` exit) compiles `ail build` exit
0 — `recur` sets `block_terminated`, propagating through
`emit_fn`'s `if !self.block_terminated` fall-through guard so
`spin` emits no `ret` and is a well-formed never-returning fn; the
binary is never executed (spec "typechecks AND compiles, no
termination claim"). Full e2e suite 90/0.
- iter loop-recur.3.3: regression gates — pure verification, ZERO
source changes (confirmed `git diff --name-only HEAD` = only the
3 T1/T2 source files). tail-app/tail-do/`verify_tail_positions`
non-regression green (the parallel `block_terminated` SET site
touched no existing SET/READ site — Boss call 4); `loop_recur` +
iter13a hash pins + drift trio + `carve_out_inventory` green
untouched (codegen-only iter, no schema change); full workspace
619 passed = exactly the loop-recur.2 baseline (616) + the 3 new
e2e tests, zero FAILED/error; `round_trip` green (the 3 new `.ail`
fixtures auto-discovered, parse→print→parse idempotent — Roundtrip
Invariant holds for runnable loop/recur programs).
## Boss design calls (mirrored from the plan header at iter close)
All four were settled in the plan header on
architectural-consistency / spec-pinned-invariant grounds (NOT
effort); each verified against actual read source before
implementation; none reopened, re-derived, or "improved" in any
phase:
1. **Loop binders → entry-block alloca + `mut_var_allocas` reuse,
NOT hand-emitted `phi`.** The spec's "phi" wording lives only in
the explicitly-*secondary* implementation-shape subsection. The
established in-repo loop-carried-value mechanism is
alloca-in-entry-block (mut.3 `pending_entry_allocas`) + `clang
-O2` mem2reg, which produces the identical optimized binary. The
linear-emit codegen architecture structurally cannot hand-emit a
header `phi` whose back-edge predecessors are discovered *during*
body lowering without a string-splice hack with no precedent (the
`if` arm sidesteps this by opening its join block *last* — a loop
header cannot be opened last). Rationale = architectural
consistency + absence of a linear-emit `phi` precedent + identical
-O2 output. Implemented verbatim; `loop_sum_to_run.ail`→55 and
`loop_sum_to_deep.ail`→500000500000 are the operational proof
mem2reg delivers the loop-carried-SSA intent.
2. **Binders ride the existing `mut_var_allocas` map + the existing
`Term::Var` load path (representation-sharing).** A loop binder's
codegen representation — a named alloca slot, loaded on each
`Term::Var` use, stored on `recur` — is structurally identical to
a mut-var's. The `Term::Loop` arm inserts each binder into
`mut_var_allocas` (value type `(String, Type)`, identical to the
`Term::Mut` precedent); the existing `lib.rs` `Term::Var` arm
resolves them via its pre-existing `mut_var_allocas.get(name)`
`load` path **byte-unchanged** (zero edits to Var-resolution; no
parallel map). Verified against real source before implementing.
mut.3's shipped+E2E-green mut-var read path is the proof the load
works for loop binders.
3. **The loop's exit value is the emergent product of the existing
`if`/`match` join once `recur` sets `block_terminated` — no
separate loop-result phi/exit-block.** The body is typically
`(if c <exit> (recur …))`; the existing
one-branch-terminated logic at `lib.rs:1606-1613` already drops
the terminated (recur) branch and returns the non-recur branch's
`(ssa,ty)` directly — *iff* `recur` sets `block_terminated`. The
`Term::Loop` arm therefore returns `lower_term(body)`'s `(ssa,ty)`
directly; no separate loop-result phi was built. An infinite loop
(body = bare `recur`) compiles because `recur` sets
`block_terminated`, propagating through the `emit_fn`
`if !self.block_terminated` fall-through guard exactly as a
tail-app-terminated function.
4. **Reuse the single `block_terminated` field; `recur` sets it at
its OWN new emit site (a parallel SET call-site), zero edits to
any existing SET or READ site.** A back-edge `br` IS a block
terminator with exactly `block_terminated`'s semantics. `recur`'s
own `self.block_terminated = true;` is a NEW call-site parallel
to (not replacing) tail-app's SET at `lib.rs:2279`. The diff
confirms tail-app's SET and every READ site
(`:1124/1581/1594/1600-1613/...`) are byte-identical; the
tail-app/`verify_tail_positions` non-regression gate (T3 Step 1)
is the operational proof.
## Concerns
- DONE_WITH_CONCERNS (iter loop-recur.3.3): the plan's literal Task-3
Step-1 command `cargo test --workspace tail` filters on the
test-name substring `tail`, which matches **no** test in the
workspace (the real tail-app guards are
`ailang-check::lib::tail_call_in_{non_,}tail_position_is_*`,
`ailang-prose::lib` `*_tail_*`, e2e
`iter14e_print_list_recursion_emits_musttail`). Ran the gate via
the real test names instead (2 + 3 + 1, all green); the
authoritative non-regression gate is Step 3's full-workspace
619/0 which subsumes them. Recurring `feedback_plan_pseudo_vs_reality`
class (a literal verification command whose filter doesn't resolve
to its evident target); resolved by preserving the gate's evident
intent (tail-app/`verify_tail_positions` byte-unchanged proven),
no behaviour change. Observation, not a correctness risk — the
full-suite count `619 = 616 + 3` is independent confirmation
nothing regressed.
## Known debt
- None. The iteration's invariants (loop/recur lowers to real IR;
run-to-value 55; deep-`n` 500000500000 safe-by-construction;
infinite loop compiles; tail-app + hash pins + drift trio
byte-unchanged; round-trip on the 3 new fixtures) are fully pinned
by the 3 new e2e tests + the unchanged regression suite.
## Files touched
- Codegen: `crates/ailang-codegen/src/lib.rs` (`loop_frames` field +
init; the two real `Term::Loop`/`Term::Recur` arms replacing the
iter-1 single stub arm), `crates/ailang-codegen/src/lambda.rs`
(lambda-boundary `loop_frames` save/reset + restore, alongside the
mut.3 triple)
- Tests: `crates/ail/tests/e2e.rs` (3 new test fns: 2 build+run, 1
build-only)
- New fixtures: `examples/loop_sum_to_run.ail`,
`examples/loop_sum_to_deep.ail`, `examples/loop_forever_build.ail`
- No edit / byte-frozen (confirmed, not modified): every existing
`block_terminated` SET/READ site + tail-app/tail-do lowering +
`verify_tail_positions`; the iter-2 typecheck (`synth`
Loop/Recur arms, `verify_loop_body`, `loop_stack`, the four
`Recur*` variants); the iter-1 `synth_with_extras`
Loop→body-type / Recur→Unit pass-through; `hash_pin.rs`
(`loop_recur` + iter13a pins) + the drift trio +
`carve_out_inventory.rs` (codegen-only iter; the 3 new `.ail`
fixtures are round-trip-auto-covered, NOT carve-outs — inventory
stays 8)
## Stats
bench/orchestrator-stats/2026-05-17-iter-loop-recur.3.json
+1
View File
@@ -83,3 +83,4 @@
- 2026-05-16 — iter effect-doc-honesty: standalone documentation-honesty tidy (split out from the retired effect-op-arg-modes bundle — user rejected bundling a real DESIGN.md fix with speculative build-ahead infra). Corrected three false effect-system claims the recon surfaced: DESIGN.md Decision 3 "row-polymorphic (`![IO | r]`)" (no `EffectRow`/row var exists — effect sets are flat closed sets unified by set-equality) + "`IO` and `Diverge` … are wired up" (`Diverge` is 100% vapour: zero code in any crate) → reconciled to IO-only/Diverge-reserved-unimplemented (modelled on Decision 4's reserved-refinements precedent); DESIGN.md §"What is not supported" "IO and Diverge ops" bullet; `ast.rs` `Term::Do` doc-comment "resolved against the effect-handler table at link time" → the real mechanism (typecheck `Env::effect_ops` lookup + `lower_effect_op` literal codegen match, no handler table). Satellite lockstep: `form_a.md:226` + rule-3, `main.rs:289` merge-prose CONTRACT example. Guarded by a new 4-test doc-presence pin `crates/ailang-core/tests/effect_doc_honesty_pin.rs` (fiction-absent + corrected-anchor-present, single-line substrings — wrap-robust). No language/checker/codegen change; `ail check`/`run` byte-unchanged by construction. `cargo test --workspace` 600 → 604; drift/coverage trio (`design_schema_drift`/`spec_drift`/`schema_coverage`) green, empirically confirming none scans the effect-prose region. One concern: the plan's Task-2 replacement body soft-wrapped a phrase the single-line pin asserts; orchestrator resolved correctly (exact wording + exact pin preserved, wrap column moved) — recurring-class planner gap, fixed forward by a Step-5 self-review tightening → 2026-05-16-iter-effect-doc-honesty.md
- 2026-05-17 — iter loop-recur.1: standalone-`loop`/`recur` milestone (1 of 3) — the strictly-additive AST-node foundation. `Term::Loop { binders: Vec<LoopBinder>, body }` / `Term::Recur { args }` / `struct LoopBinder` (mirrors `MutVar`'s `(name,type,init)` triple; `binders` no `skip_serializing_if`) added end-to-end across all six crates' no-`_`-wildcard exhaustive `Term` matches (~30 sites incl. one recon-undercount integration-test pin): Form-A `parse_loop`/`parse_recur` (binder/body boundary disambiguated by an `is_term_head_kw` guard — the plan's flagged sole parser judgement) + print + `form_a.md` grammar/notes + prose render/free-var/subst lockstep + canonical-JSON serde/round-trip + DESIGN.md §Data-model `"t":"loop"`/`"t":"recur"` blocks + design_schema_drift/spec_drift/schema_coverage anchors + a new `loop_recur` hash pin (additivity proven: iter13a + new pin both green, pre-existing canonical-JSON bytes byte-stable) + `examples/loop_sum_to.ail` round-trip fixture. NO typecheck semantics + NO real codegen by design: the two semantic dispatch points stubbed `synth``CheckError::Internal` / `lower_term``CodegenError::Internal` exactly as mut.1; pure analysis walkers (escape/lambda/`synth_with_extras`) get real structural pass-through. Two binding Boss design calls (mirrored into the journal): (1) `verify_tail_positions` "byte-unchanged" = tail-app *role* unchanged not source-frozen — it is an exhaustive no-wildcard match so two additive descent-only arms are mandatory; acceptance evidence is the tail-app non-regression test (all `tail`-filtered tests byte-identical), mut.1 set the precedent; (2) codegen IS iter-1 scope as stubs/pass-throughs (no real loop-header/phi/back-edge — that is iter 3). Three plan-pseudo-vs-reality substitutions, all intent-preserving, recurring `feedback_specs_need_concrete_code`/`feedback_plan_pseudo_vs_reality` class: serde `Type::Con` literal `{"t":"con",…,"args":[]}`→real `{"k":"con","name":"Int"}`; bare `Int``(con Int)` in binder triples (bare `Int` parses as `Type::Var`); `ailang_core::ast::LoopBinder`→unqualified inside ailang-core. Boss forward-fix folded in: Concern 2 surfaced that the committed specs themselves (`2026-05-17-loop-recur.md` headline + `bad_recur`, `2026-05-17-llm-surface-discipline.md` §5) wrote bare `Int` — corrected all three snippets to `(con Int)` (doc-honesty, same class as the mono.rs header; no design/schema/assumption change, no re-brainstorm). `cargo test --workspace` 600→608 / 0 red (Boss-reran independently). Component 4 (typecheck) = iter 2; Component 5 (codegen) + positive/deep-`n` E2E = iter 3 → 2026-05-17-iter-loop-recur.1.md
- 2026-05-17 — iter loop-recur.2: standalone-`loop`/`recur` milestone (2 of 3) — Component 4 typecheck semantics. The iter-1 `synth` `CheckError::Internal` stub for `Term::Loop`/`Term::Recur` is replaced with real binder typing (each init synthed in outer scope + already-declared binder names of THIS loop via ordinary `locals` save/restore mirroring `Term::Let` verbatim; loop's static type = body type) + positional `recur` arity/per-arg-type checking via a new `loop_stack: &mut Vec<Vec<Type>>` frame threaded exactly as mut.2's `mut_scope_stack` (every recursive + cross-module + test synth caller, compile-swept). New private `verify_loop_body(t,in_loop_tail)` pass — a SIBLING of the byte-frozen `verify_tail_positions` (0 deletions, tail-app role untouched), invoked in `check_fn` after it so synth-raised codes take precedence by pass ordering. Four `Recur*` `CheckError` variants (`RecurOutsideLoop`, `RecurArityMismatch{expected,got}`, `RecurTypeMismatch{position,expected,got}`, `RecurNotInTailPosition`) + `code()` + 2 `ctx()` arms, bracket-`[code]`-free Display (F2); four negative `.ail.json` fixtures fire their codes point-exactly (`assert_eq!`, non-vacuous) via a new `loop_recur_typecheck_pin.rs` (7 tests) + a ct1 F2 human-mode sibling; `carve_out_inventory` 13→17 with the pre-existing mut.4-tidy "Twelve"-vs-13 stale-header drift corrected in passing. iter-1's `loop_sum_to.ail` round-trip fixture now ALSO typechecks clean (one fixture, two properties — no near-duplicate); `loop_forever.ail` (loop whose only path is `recur`) typechecks clean, asserting the spec "no termination claim is made or enforced". Three binding Boss design calls (mirrored into the journal): (1) `RecurTypeMismatch` is an Assign-style `subst.apply`-both-then-structural-`!=` pre-check, NOT a `unify`-propagate (a propagate would surface generic `type-mismatch` and fail the point-exact-code acceptance); (2) `loop_stack` element type is positional `Vec<Type>` NOT name-keyed `IndexMap` (recur rebinds by position; binder names live in `locals`); (3) diagnostic-code precedence is by pass ordering (synth before verify_loop_body), no explicit logic. NO codegen (iter-1 `lower_term` `CodegenError::Internal` stub stays — iter 3); NO `Diverge`/`verify_structural_recursion`/guardedness (spec deliberate boundary). Two DONE_WITH_CONCERNS, both Task 2, both journalled: (a) a plan-ordering defect — Task 2's `cargo build` 0-errors gate is unsatisfiable while `check_fn` (deferred by the plan to Task 4) is an unthreaded caller; orchestrator pulled the byte-identical declaration+threading forward to Task 2, left only the `verify_loop_body` invocation in Task 4 (no semantic change, only sequencing); (b) recon-undercount of cross-module synth callers (`builtins.rs`×2, `lift.rs:746`, `mono.rs:720`/`:1361`), the recurring mut.2-class, resolved via the plan's own designated compile-sweep oracle. Boss systemic fix folded into this commit: planner SKILL.md Step-5 self-review gains item 7 (compile-gate vs. deferred-caller ordering) so the Concern-(a) class is scrubbed at plan time, not discovered at execution. `cargo test --workspace` 608→616 / 0 red (Boss-reran independently); `verify_tail_positions`/tail-app byte-frozen confirmed by diff-hunk inspection. Component 5 (codegen loop-header/phi/back-edge) + the positive RUN-to-value / deep-`n` E2E = iteration 3 → 2026-05-17-iter-loop-recur.2.md
- 2026-05-17 — iter loop-recur.3: standalone-`loop`/`recur` milestone (3 of 3, TERMINAL) — Component 5 codegen + run-to-value E2E. The iter-1 `lower_term` `CodegenError::Internal` stub for `Term::Loop`/`Term::Recur` is replaced with real LLVM-IR lowering: loop binders are loop-carried values lowered as entry-block allocas (the mut.3 `pending_entry_allocas` mechanism) registered in the EXISTING `mut_var_allocas` map so the EXISTING `Term::Var` load path resolves them with zero new Var code (representation-sharing only — Var-lowering byte-unchanged); a fresh `loop.header.<id>` block reached by an unconditional `br` from the pre-header; `recur` lowers ALL args to SSA BEFORE any store (simultaneous positional rebind), stores into the binder allocas, back-edges `br` to the header, and sets the SINGLE existing `block_terminated` field at its OWN new emit site (a parallel SET call-site) so the loop's exit value is the emergent product of the existing `if`/`match` join — no separate loop-result phi/exit-block. A new `loop_frames` codegen stack lets `recur` find its target header+slots; saved/reset/restored at the single lambda-lowering boundary exactly as mut.3's `mut_var_allocas` triple. `clang -O2` mem2reg promotes the binder allocas to the phi nodes the spec's secondary implementation-shape describes. Four binding Boss design calls (mirrored into the journal), all on architectural-consistency / spec-pinned-invariant grounds (NOT effort): (1) alloca+mem2reg NOT hand-emitted phi (the linear-emit architecture has no phi-with-body-discovered-back-edge-predecessor precedent; the `if` arm sidesteps by opening its join last — a loop header cannot be opened last; mem2reg yields identical optimized output; spec's "phi" is the explicitly-secondary impl-shape the spec subordinates to the planner's exact-bytes authority); (2) reuse `mut_var_allocas` + the existing `Term::Var` load path (byte-unchanged; representation-sharing, the iter-2 AST/typecheck binder distinction is post-typecheck-irrelevant to codegen); (3) emergent loop-exit via the existing if/match join once recur terminates its block, no separate result phi; (4) reuse the single `block_terminated` field, recur sets it at its own emit site, ZERO edits to any existing SET/READ site (a second field would force editing every READ site — the opposite of the spec-pinned "tail-app's use byte-unchanged"). Diff confirmed surgical by hunk-header inspection: codegen/lib.rs = exactly 3 hunks (field + init + the stub→2-arms replacement), codegen/lambda.rs = exactly 2 hunks (save + restore); NO hunk at any existing `block_terminated` SET/READ site, at tail-app/tail-do lowering, or at `verify_tail_positions`. Three new `.ail` fixtures: `loop_sum_to_run.ail``55` (run-to-value), `loop_sum_to_deep.ail``500000500000` (1e6 iterations via the back-edge, no stack growth — the spec clause-2 correctness claim made executable), `loop_forever_build.ail` (infinite loop, no non-recur exit, `ail build` exit 0 — "typechecks AND compiles, no termination claim"; never executed). Codegen-ONLY iteration: no schema/AST/serde/typecheck change; hash pins (`loop_recur` + iter13a) + drift trio + `carve_out_inventory` confirmed green UNTOUCHED (not modified); the 3 new `.ail` fixtures are round-trip-auto-covered, NOT carve-outs. One DONE_WITH_CONCERNS (T3): the plan's literal `cargo test --workspace tail` filter resolves to ZERO tests (real guards are `*_tail_position_*`/`*musttail*`); orchestrator ran the gate via real names + the authoritative full-suite 619/0 which subsumes it — recurring `feedback_plan_pseudo_vs_reality` class, no behaviour change. Boss systemic fix folded into this commit: planner SKILL.md Step-5 gains item 8 (verification-command filter strings must resolve to ≥1 named test, or use the unfiltered suite + a count assertion) — the SECOND planner-meta-gap this milestone surfaced (iter-2 added item 7). `cargo test --workspace` 616→619 / 0 red (Boss-reran independently); the 3 loop/recur e2e explicitly green (55 / 500000500000 / infinite-compiles). **All three components shipped — the loop/recur milestone is structurally complete; milestone-close (mandatory `audit`, then `fieldtest` since loop/recur is user-visible Form-A surface) is the Boss's next pipeline step.** → 2026-05-17-iter-loop-recur.3.md