Brummel
41d406bcbb
Reconsidered 14d's removal of Term::If. The decision was wrong.
"No redundancies" requires judgment; reducibility (if -> match)
is not redundancy in the strong sense. Term::If is a primitive
control-flow shape; bool branching is the second most common
shape after sequencing, and removing it cost 3x tokens on every
branch site (`(if c a b)` 4 tokens vs the match-on-Bool form
12 tokens).
Meta-pattern fixed: I had been treating user observations as
directives. User said "if is a subset of match"; I jumped to
remove it citing CLAUDE.md, with no independent conviction.
The leak appeared in 14f's JOURNAL prose ("three lines for what
if used to do in one"), which read as regret. Two feedback
memories saved (memory/feedback_user_suggestions_not_directives,
memory/feedback_no_nostalgia_for_removed_features) to head this
off.
Implementation: mechanical reverse-application of 14d's diff at
every site (AST, check including the 14e tail-position arm,
codegen 4 sites, surface parser/printer, pretty, CLI walker,
e2e test mutation). Removed lower_bool_match helper — it existed
only because 14d's migration shape needed codegen for non-ptr
match scrutinees; with Term::If back, match-on-Bool returns to
its pre-14d unsupported state. Three fixtures (sum, sort, max3)
restored to pre-14d shape. gc_stress (added in 14f) also
migrated back to (if ...) since it was authored under the wrong
constraint.
14e (musttail) and 14f (GC_malloc) verified intact in IR.
Hashes restored to pre-14d values:
- sum.sum: db33f57cb329935e
- sort.insert: 697fcb9f30f8633a
- max3.max: 65c45d6a45dd0a72
- max3.max3: 624b14429bf302f5
All other defs across all 18 fixtures keep their post-14f
hashes. Tests 80/80 green; cargo doc 0 warnings. LOC delta
+265/-295 net -30.
DESIGN.md Decision 7 preserved with a "Status: REVERTED" header
for audit trail. Form-(A) `if-term` production restored.
Plan: back to 15a (std_maybe stdlib module).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2197 lines
100 KiB
Markdown
2197 lines
100 KiB
Markdown
# JOURNAL
|
||
|
||
Chronological notes for myself. Not every change; only decisions, obstacles,
|
||
and observations that future iterations will need.
|
||
|
||
## 2026-05-07 — Day 0
|
||
|
||
- Repo initialised. Assignment in `CLAUDE.md`: LLM-native language, LLVM backend.
|
||
- Design decisions captured in `docs/DESIGN.md`.
|
||
- Toolchain: `rustc 1.94`, `llvm-config 22.1.3`, `clang` available.
|
||
- Decided against `inkwell` in favour of LLVM IR text emit. Rationale in DESIGN.md.
|
||
- Workspace layout:
|
||
- `crates/ailang-core` — AST, type, hash, JSON schema
|
||
- `crates/ailang-check` — typechecker (comes later)
|
||
- `crates/ailang-codegen` — lowering + LLVM IR emit
|
||
- `crates/ail` — CLI
|
||
- MVP goal: `examples/sum.ail.json` → binary that prints 55. **Achieved.**
|
||
|
||
## 2026-05-07 — architecture review after the MVP
|
||
|
||
Still on track? Broadly yes. Concrete observations:
|
||
|
||
**What holds:**
|
||
|
||
- JSON AST + canonical form + content hash are all lego bricks that later
|
||
tools can build on without a refactor (`ail deps`, `ail diff`).
|
||
- The LLVM IR text pipeline works as planned. No libllvm version pain.
|
||
- The effect set is wired into the type system from the start. Extensible to
|
||
row-poly without touching the core.
|
||
|
||
**Debt that accrues interest:**
|
||
|
||
1. **`current_block_label_for_phi` is a heuristic** (see codegen). On nested
|
||
`if` terms it will return the wrong block label, because it scans the body
|
||
backwards. Ticking, because no test cases trigger it yet. Must be fixed
|
||
next, before new language features arrive.
|
||
2. **No typed AST.** Codegen reads the source AST directly and relies on
|
||
the typechecker having run before. Fine for the MVP; once ADTs or
|
||
closures arrive, I will need a separate typed IR stage (TIR).
|
||
3. **The `hash` field is not in the AST.** Right now we hash the def object
|
||
directly. Once I serialise hashes as fields (caching), the hash will
|
||
need to exclude that field before computation.
|
||
|
||
**Plan iteration 2 (now):**
|
||
|
||
1. Clean up block-label tracking, with a nested-if test.
|
||
2. Strings as a literal + `io/print_str`.
|
||
3. Hello-world example as a second E2E test.
|
||
4. CLI: `--json` output for machine consumers wherever it fits.
|
||
|
||
**Plan iteration 3:**
|
||
|
||
ADTs + pattern matching. That is the next big jump. Requires a typed IR
|
||
stage (TIR), because pattern matching lowers into decision trees, which
|
||
have a different shape from the AST.
|
||
|
||
## 2026-05-07 — iteration 2 done
|
||
|
||
- Block-label tracking is now robust (nested `if`s work). Test
|
||
`max3_picks_largest` protects it.
|
||
- Strings as `Lit::Str { value }`, type `Str` -> LLVM `ptr`, with
|
||
`io/print_str` effect op. `examples/hello.ail.json` prints a string.
|
||
- CLI: `manifest --json`, `builtins --json` for tool consumers.
|
||
- `ail deps [--of NAME] [--json]` lists call edges. Effect ops are tagged
|
||
`effect:NAME` so a consumer can filter them.
|
||
|
||
**Architecture check:** no structural deviations. Codegen still reads the
|
||
source AST directly (a TIR stage will become necessary with ADTs in
|
||
iteration 3).
|
||
|
||
## 2026-05-07 — iteration 3 done: ADTs
|
||
|
||
- TypeDef in the AST with ctors. A ctor has `name` and `fields: [Type...]`.
|
||
- Term::Ctor (construction) and Term::Match (pattern matching).
|
||
- Patterns: `Wild`, `Var`, `Lit`, `Ctor { ctor, fields }`. In the MVP,
|
||
nested ctor patterns are NOT allowed — sub-patterns must be `Var` or
|
||
`Wild`.
|
||
- Typechecker with a type registry and `ctor_index` (ctor name → ADT). In
|
||
Match, exhaustiveness is checked against the full constructor set. A
|
||
negative test protects this.
|
||
- Codegen: boxed heap layout. Per ctor application, `malloc(8 + 8*n)`
|
||
bytes; tag at offset 0, fields from offset 8 (8-byte slots, native typed
|
||
load/store). Match: load tag + switch + arm blocks + phi at the join.
|
||
- `examples/list.ail.json` (Cons/Nil list, sum_list via match) returns 42.
|
||
|
||
**Surprisingly painless.** The architecture decisions from day 0 paid off:
|
||
opaque ptr in LLVM 22 makes the boxed layout almost glue-free; effect
|
||
tracking was untouched by ADTs; the JSON AST takes new node types
|
||
cleanly.
|
||
|
||
**Debt accrued:**
|
||
|
||
1. **Codegen still reads the source AST directly.** The temptation to push
|
||
on without TIR was strong — and worked, because my Match restrictions
|
||
are flat (no nested patterns). Once nested patterns arrive, decision-
|
||
tree lowering will not stay clean without TIR. Debt acknowledged; not
|
||
due now.
|
||
2. **No GC.** The heap leaks. Acceptable for demo programs; must be
|
||
addressed before any longer-running program. Options for Phase 4:
|
||
refcount, Boehm-GC linkage, region inference.
|
||
3. **No runtime pretty-printer for ADT values.** `io/print_int` is enough
|
||
for demos, but a generic `show :: a -> Str` for ADTs would be valuable.
|
||
Requires dispatch over the tag — feasible, but not now.
|
||
|
||
**Plan iteration 4:**
|
||
|
||
The next steps are less obvious. Three candidates in priority order:
|
||
|
||
1. **Module system (imports).** Right now everything is in a single module.
|
||
With multiple modules + cross-module hashing the language only becomes
|
||
practical for several defs.
|
||
2. **Structured error output (`ail check --json`).** So tools can react to
|
||
type errors without parsing text.
|
||
3. **Closures / higher-order functions.** Requires closure conversion and
|
||
is a bigger step.
|
||
|
||
Iteration 4 will be (1) + (2) — both strengthen the LLM tooling and have
|
||
moderate risk.
|
||
|
||
## 2026-05-07 — workflow change: orchestrator + agent repo
|
||
|
||
At the user's suggestion, switching to **orchestrator mode**: I delegate
|
||
clearly bounded implementation chunks to sub-agents and keep only
|
||
architecture decisions, reviews, and commit discipline. Four specialised
|
||
agents drafted: implementer, architect, tester, debugger.
|
||
|
||
**Important correction:** the user required the agents not to be hidden in
|
||
`.claude/agents/`, but versioned as a visible part of the project under
|
||
`agents/`. DESIGN.md gained a new section "Project ecosystem", which
|
||
records this: AILang is not just a language, but language core + CLI +
|
||
examples + agents + docs + tests, all of equal weight.
|
||
|
||
Invocation scheme: the system-prompt body from `agents/<name>.md` as a
|
||
prefix before the concrete task + sent to the `general-purpose` agent.
|
||
Functionally identical to subagent loading from `.claude/agents/`, but
|
||
visible in the repo.
|
||
|
||
**Plan iteration 4 (revised):**
|
||
|
||
The module system is more involved than expected (cross-module hashing,
|
||
import resolution). First the smaller tooling wins, then the module system
|
||
as iteration 5:
|
||
|
||
1. **Structured error output** (`ail check --json` with a Diagnostic
|
||
struct, stable codes like `unbound-var`, `type-mismatch`).
|
||
2. **`ail diff <a> <b>`** — semantic module diff via per-def hash
|
||
comparison.
|
||
3. **IR snapshot tests** — regression protection for the codegen pipeline.
|
||
|
||
## 2026-05-07 — iteration 4 done: LLM tooling consolidation
|
||
|
||
Three sub-commits, each produced by an `ailang-implementer` invocation
|
||
and spot-checked by the orchestrator:
|
||
|
||
- `93fe723` Iter 4a: `ail check --json` with a `Diagnostic` struct
|
||
(`severity`, `code`, `message`, `def`, `ctx`). Stable codes:
|
||
`unbound-var`, `type-mismatch`, `arity-mismatch`,
|
||
`non-exhaustive-match`, `unknown-ctor`, `unknown-ctor-in-pattern`,
|
||
`nested-ctor-pattern-not-allowed`, `duplicate-def`,
|
||
`unknown-effect-op`, `unknown-type`, `schema-mismatch`. API:
|
||
`check_module(&Module) -> Vec<Diagnostic>`.
|
||
- `c652b12` Iter 4b: `ail diff <a> <b> [--json]` as a structural top-level
|
||
def diff via BLAKE3 hash. Four categories (added/removed/changed/
|
||
unchanged), sorted alphabetically, exit code 1 on diff.
|
||
- `74a2005` Iter 4c: IR snapshot tests in
|
||
`crates/ail/tests/snapshots/{sum,max3,hello,list}.ll`.
|
||
Normalisation of `target triple`. Update via `UPDATE_SNAPSHOTS=1
|
||
cargo test ir_snapshot_`. Mismatch produces an `.actual` file.
|
||
|
||
Test count: 28 (previously 19). 7 E2E + 4 IR snapshot + 9 ailang-check + 1
|
||
ailang-codegen + 7 ailang-core.
|
||
|
||
**Closed from the debt register:**
|
||
|
||
- Block tracking in codegen has not been a heuristic risk since Iter 2;
|
||
the explicit `current_block: String` track is now additionally
|
||
protected against regression by Iter 4c snapshot tests. Debt closed.
|
||
|
||
**New / sharpened debt:**
|
||
|
||
1. `check_module` is **single-shot** — the first error aborts, no
|
||
multi-diagnostic gathering. The spec was that way, but the format
|
||
suggests Vec semantics. A real multi-diagnostic refactor will be
|
||
cheaper once TIR exists (a central error accumulator via a separate
|
||
stage). Not due now.
|
||
2. `source_filename` in the IR is hard-coded to `"<module>.ail"`. As long
|
||
as there is only one top-level module, that is platform-stable. With
|
||
Iter 5 (module system + imports) the path becomes relevant — keep it
|
||
path-independent at construction time, otherwise the snapshots will
|
||
tip over.
|
||
|
||
**Plan iteration 5:** module system with imports. Cross-module hashing,
|
||
import resolution, multiple `.ail.json` files in one build. The multi-
|
||
diagnostic refactor only after that.
|
||
|
||
Sub-steps:
|
||
|
||
- **5a — workspace loader.** `ailang_core::Workspace { modules:
|
||
BTreeMap<String, Module> }` plus `load_workspace(entry: &Path)`, which
|
||
follows `imports` recursively from the entry module. Convention:
|
||
`import { module: "foo" }` resolves to `<dir>/foo.ail.json` next to the
|
||
entry. Cycle detection. CLI: existing subcommands keep working on a
|
||
single module; a new `ail workspace <entry>` lists all reachable
|
||
modules with hash. Tests: two small example modules with an import
|
||
relation; cycle test.
|
||
- **5b — cross-module typecheck.** The typechecker takes `&Workspace`
|
||
instead of `&Module`. Imports are mounted in the env as a namespace
|
||
(`alias.def` or, with no alias, `module.def`). New diagnostic codes:
|
||
`unknown-module`, `unknown-import`, `import-cycle`, `ambiguous-name`.
|
||
Tests per code.
|
||
- **5c — cross-module codegen.** The emitter produces IR for all modules
|
||
in the workspace, prefix-mangled with `@ail_<module>_<def>`. E2E test:
|
||
a program that uses a function from module B in module A returns the
|
||
correct result in the binary.
|
||
- **5d — tooling adjustments.** `manifest`, `describe`, `deps`, `diff`
|
||
gain a `--workspace` mode (recursive). The single mode stays the
|
||
default for backwards compatibility.
|
||
|
||
During Iter 5, at construction time **keep `source_filename`
|
||
path-independent** (module name only, no directory prefix), otherwise
|
||
the IR snapshots will tip over.
|
||
|
||
## 2026-05-07 — Iter 5b done: cross-module typecheck
|
||
|
||
- `check_workspace(&Workspace) -> Vec<Diagnostic>` as the top-level API.
|
||
`check_module` is preserved and internally lifts the module into a
|
||
trivial workspace.
|
||
- Convention for qualified references (recorded in DESIGN.md):
|
||
`Term::Var { name }` with exactly one dot = `<prefix>.<def>`. Prefix is
|
||
an import alias or module name. No new AST node, no renamed fields ⇒
|
||
hashes stay stable; all `ir_snapshot_*` still green.
|
||
- Three new diagnostic codes: `unknown-module`, `unknown-import`,
|
||
`invalid-def-name` (with `ctx.reason: "contains-dot"`).
|
||
- CLI: `ail check <entry>` now **always** loads via `load_workspace`.
|
||
Workspace load failures become structured diagnostics in JSON mode with
|
||
codes `module-not-found`, `module-cycle`, `module-name-mismatch`,
|
||
`module-hash-mismatch`, `schema-mismatch`. `ail build` and
|
||
`ail emit-ir` stay per single module (cross-module codegen is 5c).
|
||
- Examples: `ws_main.ail.json` now calls `ws_lib.add` (observable). New:
|
||
`ws_broken.ail.json` (`unknown-import`),
|
||
`ws_unknown_module.ail.json` (`unknown-module`).
|
||
- Tests: 37 green (previously 32). 4 new workspace integration tests in
|
||
`crates/ailang-check/tests/workspace.rs`, one new e2e test
|
||
`check_workspace_resolves_import`.
|
||
- Debt: single-shot diagnostics still in place (multi-diagnostic after
|
||
5c). The dot convention covers exactly one dot — nested module paths
|
||
(`a.b.c`) do not exist; that would only be a topic with hierarchical
|
||
modules and currently falls through as `unbound-var`.
|
||
|
||
## 2026-05-07 — Iter 5c done: cross-module codegen
|
||
|
||
- **Mangling break (deliberate).** All AILang functions are now called
|
||
`@ail_<module>_<def>`, even in single-module programs. The old form
|
||
`@ail_<def>` is gone. Strings/const globals analogously
|
||
(`@.str_<module>_<hint>_<idx>`, `@ail_<module>_<const>`). The entry
|
||
point stays `main` as C ABI: a `define i32 @main()` trampoline calls
|
||
`@ail_<entry-module>_main()`. If the entry module has no
|
||
`main : () -> Unit !IO`, the build fails with `MissingEntryMain`.
|
||
- **Workspace lowering.** New top-level API
|
||
`ailang_codegen::lower_workspace(ws: &Workspace) -> Result<String>`
|
||
produces a single `.ll` for the whole workspace. Modules in
|
||
alphabetical order (BTreeMap order); defs in AST order. Cross-module
|
||
calls are resolved in codegen via the import map of the calling module
|
||
— same logic as in the typechecker, locally duplicated with a
|
||
cross-reference (no shared helper module, because the type worlds
|
||
differ: the typechecker handles `Type`, codegen handles `FnSig` from
|
||
llvm types).
|
||
- **CLI.** `ail build` and `ail emit-ir` now always load the workspace
|
||
and check/lower it fully. Single-module programs keep working
|
||
(trivial workspace with one module). `emit_ir(m)` stays in the codegen
|
||
crate as a convenience API and internally wraps into a trivial
|
||
workspace.
|
||
- **Snapshots regenerated.** `sum.ll`, `max3.ll`, `hello.ll`, `list.ll`
|
||
show the new mangling. New `ws_main.ll` snapshot documents the
|
||
cross-module build: `@ail_ws_main_main` calls `@ail_ws_lib_add`.
|
||
- **Tests.** 40 green (previously 37). New: `workspace_build_runs_imported_fn`
|
||
(e2e: prints 5), `ir_snapshot_ws_main`, `missing_entry_main_is_error`
|
||
(codegen unit). Existing behaviour tests
|
||
(`sum_1_to_10_is_55`, `max3_picks_largest`, `hello_world_str_lit`,
|
||
`list_sum_via_match`) stay green — behaviour unchanged, only the
|
||
mangling is new.
|
||
|
||
**Debt closed:**
|
||
|
||
- **#19 (`source_filename` hardening).** In the workspace world,
|
||
`source_filename` is now uniformly `<entry-module>.ail`, once per
|
||
workspace. The previous hard-coded path dot is gone with it.
|
||
|
||
**State:** the module system is closed end to end — loader + typecheck +
|
||
codegen + build see the workspace as a coherent unit. The multi-
|
||
diagnostic refactor and possibly cross-module ADTs remain for later.
|
||
|
||
## 2026-05-07 — Iter 5d done: tooling extended to the workspace
|
||
|
||
- `ail manifest|describe|deps|diff <entry> --workspace` now operate
|
||
across all modules of the workspace. The default without the flag stays
|
||
single-module for backwards compatibility. Manifest sorts by
|
||
`(module, name)`, describe accepts dotted notation `ws_lib.add`, deps
|
||
emits `{from_module, from_def, to_module, to_def}` edges, diff compares
|
||
workspace-wide with added/removed/changed/unchanged_modules and a
|
||
nested sub-diff per changed_module.
|
||
- Refactor: `diff_def_lists` is the single source of the four-category
|
||
logic; single and workspace diff share it.
|
||
- Tests: 44 green (previously 40). New: `manifest_workspace_lists_all_defs`,
|
||
`describe_workspace_resolves_qualified_name`,
|
||
`deps_workspace_includes_cross_module`, `diff_workspace_added_module`.
|
||
|
||
**Observation (debt):** `deps` does not filter builtins/locals/function
|
||
parameters. In workspace mode that becomes more visible than in single
|
||
mode — `ws_lib.add` lists edges to `ws_lib.+` (builtin) and
|
||
`ws_lib.a`/`ws_lib.b` (function parameters). A known pre-existing
|
||
issue from Iter 2; Task #22 in the backlog.
|
||
|
||
## 2026-05-07 — architecture review after Iter 5
|
||
|
||
Architect agent invoked. Findings:
|
||
|
||
1. **Mangling consistency holds.** `@ail_<module>_<def>` is consistent
|
||
across functions, constants, string globals, and cross-module calls.
|
||
The trampoline is correct. ADT constructors are deliberately
|
||
symbol-free (inline malloc).
|
||
2. **Module hashes bit-identical since Iter 4.** The Iter 5c snapshot
|
||
regeneration was a codegen-output change, not a hash break.
|
||
3. **Drift, due now:**
|
||
- DESIGN.md says `define i64 @main()`, codegen emits
|
||
`define i32 @main()` (see `sum.ll:35`).
|
||
- String-schema notation in DESIGN.md was shortened
|
||
(`@.str_<module>_<idx>` instead of `@.str_<module>_<hint>_<idx>`).
|
||
4. **Debt that accrues interest:** the `deps` builtin leak (Task #22) has
|
||
become a falsehood in workspace mode — close it before the next big
|
||
jump.
|
||
|
||
**Plan iteration 6 — clean-up:**
|
||
|
||
1. **Fix DESIGN.md drift.** Update the mangling-scheme block, correct the
|
||
`@main` signature, and note the string globals precisely.
|
||
2. **`deps` hardening (#22).** Build a top-level def table per workspace;
|
||
filter edges whose target is not a top-level symbol, or emit them as
|
||
separate `builtin:`/`local:` categories. Function parameters via
|
||
lexical scope tracking from walk_term.
|
||
3. **Multi-diagnostic refactor (#20).** `check_workspace` accumulates
|
||
`Vec<Diagnostic>` across all defs instead of short-circuiting on the
|
||
first error. Intra-def may still short-circuit — the value is "see
|
||
all broken defs at once", not "see all broken sub-terms of one def".
|
||
|
||
Order: 1 first (doc triviality), then 2 before 3 (deps is a tooling-
|
||
truth fix, multi-diag is a structural extension).
|
||
|
||
## 2026-05-07 — Iter 6 done: deps hardening + multi-diagnose + DESIGN audit
|
||
|
||
Three things landed together. All small, all KISS — no architecture move,
|
||
just paying off recorded debt.
|
||
|
||
**1. `ail deps` filters builtins, params, and let/match bindings (#22).**
|
||
|
||
Before: `sum -> +, -, ==, n, sum` and `ws_lib.add -> ws_lib.+`,
|
||
`ws_lib.a`, `ws_lib.b`. After: `sum -> sum`, `ws_lib.add -> ws_lib.add`
|
||
gone (no real deps; only the cross-module call from `ws_main` remains).
|
||
|
||
Implementation:
|
||
- New helper `ailang_check::builtins::value_names()`: derives the
|
||
Var-level builtin names (`+ - * / % == != < <= > >= not`) from
|
||
`list()`, so the install-list and the deps-filter share one source of
|
||
truth.
|
||
- `walk_term` in `crates/ail/src/main.rs` now threads a `scope` set:
|
||
fn-params seed it; `Let` adds the bound name for the body only;
|
||
`Match` arms add their pattern variables (`bind_pattern` helper, MVP
|
||
rule "ctor sub-patterns are Var/Wild") and roll them back after. Var
|
||
refs that hit `scope` or `builtins` are dropped; qualified names
|
||
(`prefix.def`) are passed through unconditionally — the typechecker
|
||
forbids dots in def names, so no shadowing risk.
|
||
- Tests added: `deps_filters_builtins_params_locals`,
|
||
`deps_workspace_filters_builtins_and_params`. The Iter 5d test
|
||
(`deps_workspace_includes_cross_module`) keeps passing — the only
|
||
edge it asserted is the legitimate one.
|
||
|
||
**2. `check_module` / `check_workspace` are multi-diagnose (#20).**
|
||
|
||
`check_in_workspace` returns `Vec<CheckError>` instead of `Result<()>`.
|
||
Pass-1 (top-level symbol table) stays fail-fast — corrupt globals would
|
||
taint every later diagnostic. Type-def installation is fail-fast within
|
||
a module (env corruption) but the outer module loop continues. The
|
||
body-check loop is the multi-diagnose layer: each def is checked against
|
||
the assembled env, errors accumulate, the next def is attempted.
|
||
|
||
Test: `body_errors_accumulate_across_defs` — one module with two
|
||
independent body errors (arity mismatch + unknown var) yields two
|
||
diagnostics with the right `def` field. The legacy single-error `check`
|
||
keeps working by `.into_iter().next()`-ing the Vec, so internal snapshot
|
||
tests in `crates/ailang-check/src/lib.rs` are unchanged.
|
||
|
||
Out of scope: intra-def collection. A single fn body with three type
|
||
errors still reports one. The "see all broken defs at once" goal is met;
|
||
intra-def will require unification deferral and isn't due now.
|
||
|
||
**3. DESIGN.md `What the MVP is NOT` audit (#24).**
|
||
|
||
The section was lying: it claimed "No ADTs / pattern matching" (delivered
|
||
Iter 3) and "Only ints + bools + unit" (strings landed Iter 2). Renamed
|
||
to `What is not (yet) supported`, restructured into "not yet" + "what is
|
||
supported (smoke-tested)". New invariant: this section is meant to be
|
||
the truth at the **end of the latest iteration**, not a 2026-05-07-day-0
|
||
scope statement.
|
||
|
||
**Architecture check (the user-asked self-questioning):**
|
||
|
||
- *Would I use this language now?* For non-recursive arithmetic + ADT
|
||
programs over int/bool/str: yes, comfortably. For anything that needs
|
||
mapping, folding, generic data structures: no, closures are the
|
||
blocker. That's the next big sprint, not Iter 7.
|
||
- *Consistency:* DESIGN.md, JOURNAL.md, code, and CLI output now agree
|
||
on what the language can do. The "What is not (yet) supported" block
|
||
is the canonical truth surface.
|
||
- *Visualisation:* `ail deps --workspace --json` is now a clean
|
||
cross-module call graph (no builtin noise). Good enough for an
|
||
external graph renderer to consume; a built-in DOT/ASCII renderer is
|
||
*possible future tooling*, not "we need it now". KISS.
|
||
- *Documentation:* the agents/ directory is the sub-prompt layer, the
|
||
JOURNAL is the iteration log, DESIGN.md is the contract. No new doc
|
||
axes needed at this scale.
|
||
|
||
**Tests:** 47 green (previously 44). +2 deps tests in `e2e.rs`, +1
|
||
multi-diag test in `crates/ailang-check/tests/workspace.rs`.
|
||
|
||
**Plan iteration 7:**
|
||
|
||
Closures + higher-order functions. This is the big jump that
|
||
DESIGN.md / Day 0 has been pointing at: it requires a typed IR (TIR)
|
||
stage, closure conversion in lowering, and a heap-aware ABI. The
|
||
multi-diag refactor in Iter 6 was scoped intentionally minimal — when
|
||
TIR lands, intra-def diagnostics become structurally cheap and Task #20
|
||
gets revisited.
|
||
|
||
## 2026-05-07 — Iter 7 done: first-class function references (no capture)
|
||
|
||
Iter 6 outlined Iter 7 as "closures + HOFs + TIR". KISS course-correct
|
||
on inspection: that bundle had three independent things in it, and the
|
||
HOF use-cases (passing functions around, calling through fn-typed
|
||
parameters) need none of TIR or capture. Splitting paid off — what
|
||
landed here is ~120 LOC of codegen, no TIR, no heap, no ABI churn.
|
||
Closures with capture stay queued for Iter 8 (where TIR is the
|
||
correct precondition).
|
||
|
||
**What works now:**
|
||
- Top-level fn name (or qualified `prefix.def`) used as a value yields
|
||
an LLVM fn-pointer (`@ail_<m>_<def>`, type `ptr`).
|
||
- Fn-typed parameters can be called as `f(args)` — the body emits an
|
||
indirect `call <ret> (<param-tys>) %f(...)`.
|
||
- Pass through `let`: `let g = inc in g(x)` works (the local just
|
||
aliases the global SSA, the sidetable lookup still hits).
|
||
- Pass to another fn: `apply(inc, 41) == 42` — see
|
||
`examples/hof.ail.json`, exercised end-to-end.
|
||
|
||
**What does not (yet) work — by design:**
|
||
- No anonymous lambdas. The only fn-value source is a top-level def
|
||
reference.
|
||
- No capture. A fn-value is always a constant pointer to a top-level
|
||
def; there is no environment to allocate.
|
||
- Both deferred to Iter 8 where they share the TIR + closure-conversion
|
||
preconditions.
|
||
|
||
**Implementation, in order of where the rubber meets the road:**
|
||
|
||
1. `llvm_type` learned `Type::Fn { .. } -> "ptr"`. The actual signature
|
||
travels separately. New helper `fn_sig_from_type` lifts an AILang
|
||
fn-type into an `FnSig` (LLVM types only).
|
||
2. `Emitter` got a sidetable: `ssa_fn_sigs: BTreeMap<String, FnSig>`,
|
||
keyed by SSA value (or `@global`). It's reset per function body.
|
||
3. At `emit_fn` entry, every fn-typed parameter registers
|
||
`(%arg_<name>, sig)` in the sidetable.
|
||
4. `lower_term(Term::Var)` now falls through to a top-level fn lookup
|
||
(`resolve_top_level_fn`) when the name isn't a local. The returned
|
||
SSA is the global symbol; the sidetable gets the sig.
|
||
5. `lower_term(Term::App)` dispatches:
|
||
- if callee is a `Var` AND not shadowed AND statically known
|
||
(`is_static_callee` covers builtin operators, qualified
|
||
`prefix.def`, current-module fns), keep the existing direct
|
||
`lower_app` path — no extra indirection in the IR;
|
||
- otherwise lower the callee, expect type `ptr`, look up the sig in
|
||
the sidetable, emit `emit_indirect_call`.
|
||
6. `Term::If` propagates the sig to its phi SSA when both branches are
|
||
fn-pointers with matching sigs (cheap two-line copy; no separate
|
||
test, falls out of the `apply`-on-conditional pattern).
|
||
|
||
**Why no typechecker change?** The typechecker already accepted
|
||
fn-typed locals (`Term::Var` against `env.globals`, App via `synth(callee)`
|
||
unifying with `Type::Fn`). The only blocker was `MVP: callee must be a
|
||
variable` in codegen.
|
||
|
||
**Tests:** 48 green (previously 47).
|
||
|
||
- `crates/ail/tests/e2e.rs::higher_order_apply_inc` builds and runs
|
||
`examples/hof.ail.json`, asserts the binary prints `42`.
|
||
- Existing tests unchanged (incl. snapshot tests around the IR
|
||
emission for `sum`, `list`, `max3`).
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Yes for `apply`-style and "pass a
|
||
predicate" patterns. Still no for capturing closures (`let n = 3 in
|
||
map(\x -> x + n, xs)`-equivalent), but the ergonomic gap shrank.
|
||
- *Consistency:* DESIGN.md "What is not (yet) supported" rewritten in
|
||
the same edit; first-class fn-refs now have a positive bullet, the
|
||
closures bullet is precise about what it means (no capture, no
|
||
lambdas).
|
||
- *Visualisation:* `ail describe`/`manifest` already render fn-typed
|
||
params correctly via the existing `pretty::type_to_string`
|
||
(`((Int) -> Int, Int) -> Int`). No tooling change required.
|
||
- *KISS:* every alternative I considered (full `LocalType` enum,
|
||
swapping `(String, String)` returns to a typed wrapper, lifting
|
||
lambdas to defs as syntactic sugar) was strictly more code than the
|
||
sidetable approach, with no expressivity gain.
|
||
|
||
**Plan iteration 8:**
|
||
|
||
Closures with capture, anonymous lambdas, the typed IR (TIR) layer,
|
||
closure conversion in lowering. Now that we have indirect calls
|
||
working, the main delta is: a fn-value also needs an environment
|
||
pointer, the sidetable becomes per-value (heap-allocated), and the
|
||
calling convention shifts to `(env_ptr, args...)`. Touches every
|
||
existing call path — that's why it gets its own iteration.
|
||
|
||
## 2026-05-07 — Iter 8 done: closures with capture (no TIR needed)
|
||
|
||
Iter 7's plan named TIR as the prerequisite for closures. On
|
||
inspection that bundling was wrong — TIR is one possible
|
||
implementation strategy, not a structural requirement. The
|
||
typechecker already attaches enough type information through `synth`
|
||
that the codegen can read capture types out of `self.locals`
|
||
directly. So Iter 8 ships closures **without** introducing TIR. KISS
|
||
won.
|
||
|
||
The work split into two commits:
|
||
|
||
**Iter 8a — closure-pair ABI flip.** Every fn-value is now a `ptr` to
|
||
a heap or static closure pair `{ thunk_ptr, env_ptr }`, regardless of
|
||
whether it came from a lambda or a top-level def reference. To keep
|
||
top-level-fn references cheap, every top-level fn auto-emits:
|
||
|
||
```llvm
|
||
define <ret> @ail_<m>_<f>_adapter(ptr %_env, <params>) {
|
||
%r = call <ret> @ail_<m>_<f>(<args>)
|
||
ret <ret> %r
|
||
}
|
||
@ail_<m>_<f>_clos = constant { ptr, ptr } { @adapter, null }
|
||
```
|
||
|
||
`Term::Var` resolving to a top-level fn returns the address of
|
||
`_clos`, never the bare fn pointer. `emit_indirect_call` was
|
||
rewritten to GEP+load both halves and call `thunk(env, args...)`.
|
||
Direct calls (statically-known callees in `Term::App`) bypass the
|
||
adapter and stay at the original speed.
|
||
|
||
The Iter 7 hof example (`apply(inc, 41)`) continues to print 42
|
||
unchanged — only the IR shape changed, not the source. IR snapshot
|
||
files for sum/list/max3/hello/ws_main were refreshed.
|
||
|
||
**Iter 8b — Term::Lam + capture + lambda lifting.** New AST node:
|
||
|
||
```jsonc
|
||
{ "t": "lam",
|
||
"params": ["x"...],
|
||
"paramTypes": [Type...],
|
||
"retType": Type,
|
||
"effects": ["..."],
|
||
"body": Term }
|
||
```
|
||
|
||
Param/return types are explicit. The typechecker accepts the
|
||
declared `Type::Fn` shape, checks the body's type against `retType`,
|
||
and verifies that body effects are a subset of the declared lambda
|
||
effects (no row polymorphism in the MVP). Constructing a lambda is
|
||
pure; the act of *calling* picks up the declared effects, via the
|
||
existing App branch.
|
||
|
||
Codegen does textbook closure conversion:
|
||
|
||
1. **Free-variable analysis.** `collect_captures` walks the body
|
||
skipping builtins (`+`, `==`, ...), the current module's top-
|
||
level fns, and qualified `prefix.def` names. The remainder are
|
||
captures. Inner lambdas contribute their own free vars upward.
|
||
2. **Lift to thunk.** For each lambda, generate a fresh
|
||
`@ail_<m>_<def>_lam<id>(ptr %env, params...)`. State the body
|
||
into a side buffer (the emitter's `body`/`locals`/`counter` are
|
||
saved and reset, then restored). Captures and lambda params are
|
||
pushed as named locals so the body lowering finds them. The
|
||
thunk text goes into a `deferred_thunks` queue and is appended
|
||
after the parent fn's `}` — LLVM IR doesn't care about fn order.
|
||
3. **Pack at the use site.** In the OUTER body emit:
|
||
|
||
```llvm
|
||
%env = call ptr @malloc(i64 <8 * captures>)
|
||
; for each capture i: store at offset 8*i
|
||
%clos = call ptr @malloc(i64 16)
|
||
; store thunk_ptr at offset 0, env at offset 8
|
||
```
|
||
|
||
`%clos` is the value returned by the Lam term. Its sig is
|
||
registered in the sidetable so subsequent indirect calls work.
|
||
|
||
4. **Capture sigs propagate.** A fn-typed capture (e.g. capturing a
|
||
fn-typed param of an outer scope) keeps its FnSig in the thunk's
|
||
sidetable, so the captured fn can still be indirect-called from
|
||
inside the lambda.
|
||
|
||
Capture layout uses 8-byte slots regardless of LLVM type. Typed
|
||
load/store reads only the bytes it needs — wasted padding for `i1`
|
||
and `i8` is fine at this scale.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Yes for substantially more cases.
|
||
`let n = 3 in apply(\\x. x + n, 39)` is the example I would have
|
||
reached for in Iter 6 and bounced off. It now compiles and runs.
|
||
`map`/`fold`/`filter` over user-supplied predicates are within
|
||
reach — only the absence of polymorphism still forces author-side
|
||
monomorphisation.
|
||
- *Did I think of everything?* Hash stability checked manually:
|
||
`examples/sum.ail.json` produced the same fn hashes
|
||
(`db33f57cb329935e`, `d9a916a0ed10a3d3`) before and after Iter 8.
|
||
Existing modules without `Term::Lam` serialise bit-identically. ✓
|
||
- *Consistency:* DESIGN.md "What is not (yet) supported" rewritten
|
||
in the same edit. The Term schema gained `lam`, `ctor`, `match`
|
||
rows that were already supported but had been omitted from the
|
||
schema fragment. Now the doc is exhaustive for the supported
|
||
language.
|
||
- *Visualisation:* `ail describe` already renders Lam terms (added
|
||
pretty-printer rule), and the codegen IR for `closure.ail.json`
|
||
reads as a textbook closure-conversion lowering.
|
||
- *KISS check:* I considered three alternatives and all were
|
||
strictly worse — fat-pointer ABI (aggregate-passing concerns), full
|
||
TIR layer (large rewrite), uniform heap pair without static-closure
|
||
optimisation (regressed Iter 7 to one malloc per fn-value escape).
|
||
|
||
**Tests:** 49 green (was 48 after Iter 7). One new e2e:
|
||
`closure_captures_let_n` builds and runs `examples/closure.ail.json`
|
||
asserting "42". IR snapshot files refreshed for the per-fn adapter +
|
||
static-closure scaffold — only structural delta.
|
||
|
||
**Plan iteration 9:**
|
||
|
||
Two candidates, both real pain points:
|
||
|
||
1. **Polymorphic inference.** Make `Type::Forall` actually work in
|
||
`synth` — instantiate fresh type variables at each use site, allow
|
||
`let id = \\x. x in (id 1, id true)`. This unblocks generic
|
||
`map`/`fold`/etc. without per-type clones. Probably small (~150
|
||
LOC in the typechecker; codegen already monomorphises by
|
||
instantiation when it lowers the call).
|
||
|
||
2. **GC / region reclamation.** Right now ADT boxes, lambda envs,
|
||
and closure pairs all leak through the program's lifetime. A
|
||
minimal mark-and-sweep over a tagged heap would let us run real
|
||
programs. Bigger lift, ~400-600 LOC plus runtime support.
|
||
|
||
Leaning toward (1) for the next iteration: it's the smaller bite
|
||
*and* the bigger expressivity unlock. (2) becomes acute only when
|
||
someone tries to run an unbounded loop, which the current examples
|
||
don't.
|
||
|
||
## 2026-05-07 — Iter 9 done: dogfood + `ail run`
|
||
|
||
Course-corrected from the Iter-8 plan. Polymorphism is the bigger
|
||
expressivity unlock on paper, but I hadn't actually proved that the
|
||
language was sufficient for "small but real" programs without it. So
|
||
Iter 9 became a dogfood iteration: write a non-trivial program
|
||
that exercises everything Iter 1-8 shipped, and use `ail run` /
|
||
errors / type-checker output as the user would. If something broke,
|
||
fix it. If nothing broke, document the boundary moved.
|
||
|
||
**`examples/list_map.ail.json`**:
|
||
|
||
```jsonc
|
||
type IntList = Nil | Cons Int IntList
|
||
|
||
map_int :: ((Int) -> Int, IntList) -> IntList
|
||
map_int(f, xs) = match xs {
|
||
Nil -> Nil
|
||
Cons(h, t) -> Cons(f(h), map_int(f, t))
|
||
}
|
||
|
||
print_list :: (IntList) -> Unit !IO
|
||
print_list(xs) = match xs {
|
||
Nil -> ()
|
||
Cons(h, t) -> let _ = do io/print_int(h) in print_list(t)
|
||
}
|
||
|
||
main = let xs = Cons 1 (Cons 2 (Cons 3 Nil)) in
|
||
print_list(map_int(\\x. x * 2, xs))
|
||
```
|
||
|
||
**Result:** nothing broke. Output `2\\n4\\n6\\n`, exit 0. The full
|
||
pipeline (`ail run`) covers: ADTs with two ctors of different
|
||
arity; pattern matching with nested `Var` fields; recursion over
|
||
ADT; closures (with no captures here, so env is null but the
|
||
closure-pair plumbing still gets exercised); fn-typed parameters in
|
||
a top-level def; `do io/...` inside a match arm body, with `let _`
|
||
to sequence two effectful operations; effect propagation through
|
||
the call chain. This validates Iter 1-8 as a self-contained
|
||
foundation.
|
||
|
||
**Friction surfaced:** writing the AST by hand is tedious — the
|
||
JSON for this 4-def module is 200+ lines. That's not surprising
|
||
(the format is for LLMs, not humans), but it suggests an Iter 10
|
||
priority: a richer pretty-print form, or an `ail snippet` helper
|
||
for common boilerplate (`mk_list_int`, etc.). Not blocking; noted.
|
||
|
||
**`ail run` (Iter 9b):** Builds into a tempdir + execs the binary,
|
||
exit code passthrough. Saves a `cd && ./bin` step in the dogfood
|
||
loop. Tiny addition — `Cmd::Build`'s body factored into a shared
|
||
`build_to` helper.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For self-contained Int-typed
|
||
programs over recursive ADTs: yes. The list_map example is what
|
||
I would have wanted to write since Iter 6 and bounced off
|
||
repeatedly. It now compiles and runs without me adapting the
|
||
source — the language is what its authors said it was, end to
|
||
end.
|
||
- *Did I think of everything?* Two cracks observed during the
|
||
dogfood:
|
||
- `(Int)` parens around single-param fn-types in pretty-print are
|
||
visual noise. Cosmetic, can wait.
|
||
- `let _ = do <effect> in <body>` is the only way to sequence
|
||
effects today. Working as intended given KISS, but a `;`
|
||
operator (sequencing) would be cheap polish.
|
||
- *Consistency:* DESIGN.md CLI block + smoke-test list updated.
|
||
Iter 8c invariant — "What is not (yet) supported" ≡ truth at end
|
||
of latest iteration — held; no new pending items.
|
||
- *KISS:* Iter 9 added 0 LOC of language semantics. All gain came
|
||
from validating the existing surface and a small CLI helper.
|
||
|
||
**Tests:** 50 green (was 49). New e2e
|
||
`list_map_doubles_then_prints`. No test for `ail run` itself —
|
||
`build_and_run` already exercises the equivalent path.
|
||
|
||
**Plan iteration 10:**
|
||
|
||
The dogfood revealed two real-but-not-blocking pain points and one
|
||
big architectural gap. Candidates, ranked:
|
||
|
||
1. **Polymorphic let-bindings with monomorphisation at codegen.**
|
||
Allows `let id = \\x. x in (id 1, id true)` and ultimately
|
||
`map :: (a -> b) -> List a -> List b`. The ground truth-ier
|
||
answer for the "would I use it for X?" question, but a
|
||
non-trivial pipeline change (typechecker→codegen needs to thread
|
||
instantiation info to the call site).
|
||
|
||
2. **Sequencing operator `;` and richer effect ergonomics.** A
|
||
`Term::Seq { lhs, rhs }` (or compile sugar to `Let { name: "_",
|
||
value: lhs, body: rhs }`) plus a small pretty-print update.
|
||
Cheap, satisfying.
|
||
|
||
3. **GC.** Heap reclamation for ADT boxes, lambda envs, closure
|
||
pairs. Real architecture step. Becomes acute the moment someone
|
||
writes a long-running loop; the current examples don't.
|
||
|
||
Tentative pick: (2) for the next sprint as a satisfying small
|
||
polish, then (1) as Iter 11. (3) bides its time until a real
|
||
program needs it.
|
||
|
||
## 2026-05-07 — Iter 10 done: Term::Seq sequencing
|
||
|
||
Followed the Iter 9 plan and shipped (2). New AST node
|
||
`Term::Seq { lhs, rhs }` with serde tag "seq". Semantics: evaluate
|
||
lhs (which must be Unit), discard the value, return rhs. Effects
|
||
from both sides accumulate.
|
||
|
||
This is sugar for `let _ = lhs in rhs`, but it's a first-class node
|
||
because:
|
||
- The pretty-print renders cleanly (`(seq lhs rhs)` instead of
|
||
borrowing the `let` form with a discard binding).
|
||
- Diagnostics are sharper: a non-Unit lhs gets a "type mismatch"
|
||
error pointing at the seq site, not "binding `_` had type X" at
|
||
a let site.
|
||
- Future tooling (effect inference visualisation, dataflow) can
|
||
treat sequencing as a structural concept instead of a special-
|
||
cased let.
|
||
|
||
Codegen is trivial: lower lhs (drop SSA), lower rhs (return).
|
||
|
||
Refactored `examples/list_map.ail.json`'s `print_list` to use seq
|
||
instead of `let _ = ...`. Output unchanged (`2\\n4\\n6\\n`); the
|
||
JSON shed a few lines and reads more honestly.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Same answer as Iter 9 (yes for
|
||
small but real programs), but the seq node makes IO-heavy
|
||
recursion read better — closer to "call this effect, then this
|
||
one" instead of "bind this effect to nothing, then this one".
|
||
- *Did I break anything?* Hash stability check: existing examples
|
||
without `Term::Seq` serialise identically; their fn hashes are
|
||
unchanged. `list_map.ail.json`'s hashes shifted as expected since
|
||
its body changed.
|
||
- *KISS:* +30 LOC across AST/pretty/check/codegen/walker. One
|
||
unit test for the lhs-must-be-Unit rule. The dogfood example
|
||
proves the e2e path.
|
||
|
||
**Tests:** 51 green (was 50). New `seq_lhs_must_be_unit` unit test
|
||
in ailang-check. Existing list_map e2e still passes after the
|
||
refactor.
|
||
|
||
**Plan iteration 11:**
|
||
|
||
Polymorphism, as queued in the Iter 9 plan. Concretely: HM-style
|
||
unification + let-generalisation in the typechecker, monomorph-
|
||
isation at codegen time. Touches the typechecker→codegen pipeline.
|
||
Bigger commit than the recent stretch, will probably need to be
|
||
phased (typechecker substitution machinery, then codegen
|
||
specialisation, then docs).
|
||
|
||
## 2026-05-07 — Iter 11 done: deeper dogfood (insertion sort)
|
||
|
||
Pulled back from polymorphism for one more validation cycle before
|
||
the architectural step. Polymorphism is a substantial pipeline
|
||
change (typechecker substitution + codegen monomorphisation) and I
|
||
wanted one more "small but real" program to confirm the existing
|
||
foundation holds before disturbing it.
|
||
|
||
`examples/sort.ail.json` — insertion sort over `IntList`:
|
||
|
||
```jsonc
|
||
insert :: Int -> IntList -> IntList
|
||
insert(y, xs) = match xs {
|
||
Nil -> [y]
|
||
Cons(h, t) -> if y <= h then Cons(y, Cons(h, t))
|
||
else Cons(h, insert(y, t))
|
||
}
|
||
|
||
sort :: IntList -> IntList
|
||
sort(xs) = match xs {
|
||
Nil -> Nil
|
||
Cons(h, t) -> insert(h, sort(t))
|
||
}
|
||
|
||
print_list :: IntList -> Unit !IO // uses Iter 10 seq
|
||
|
||
main = print_list(sort([3,1,4,1,5,9,2,6,5,3,5]))
|
||
```
|
||
|
||
**Result:** typechecks first try, runs first try, prints
|
||
`1 1 2 3 3 4 5 5 5 6 9` (each on its own line). 11-element input,
|
||
correct sorted output. The combination of recursive ADT pattern
|
||
match + comparison ops + branching + leaf recursion + IO
|
||
sequencing all worked end to end without the language tripping me
|
||
up. Iter 10's seq made `print_list` notably cleaner than the
|
||
`let _ = ...` form would have been.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For "small but real"
|
||
monomorphic programs over Int, Bool, Unit, Str, and ADTs of
|
||
those: confidently yes. Insertion sort writes out as the
|
||
textbook recursion, no bookkeeping that the language couldn't
|
||
do for me.
|
||
- *Did I think of everything?* The remaining wall is still
|
||
polymorphism. Sort over `IntList` needs hand-monomorphisation;
|
||
a generic `sort :: (a -> a -> Bool) -> List a -> List a` is
|
||
what the language eventually wants. No new architectural cracks
|
||
surfaced from this dogfood.
|
||
- *Visualisation:* `ail describe sort.ail.json sort` reads the
|
||
way I'd expect a sort definition to read, with `IntList` types
|
||
inline and the recursive call rendered cleanly.
|
||
- *KISS:* Iter 11 added 0 LOC of language semantics and 1 e2e
|
||
test. The 250-line JSON for the example is verbose but
|
||
mechanical — no friction once you accept that the JSON is the
|
||
surface for LLM authors.
|
||
|
||
**Tests:** 52 green (was 51). New e2e
|
||
`insertion_sort_orders_list`. Pure addition; existing tests
|
||
untouched.
|
||
|
||
**Plan iteration 12:**
|
||
|
||
Now polymorphism. Two more dogfood programs would just keep
|
||
producing the "the language is fine for monomorphic programs"
|
||
result, which is already established. The real expressivity
|
||
unlock — and the answer to "would I use it for X?" for X that
|
||
actually needs generic data — is HM inference + let-generalisation
|
||
+ monomorphisation. Phased plan:
|
||
|
||
12a. Typechecker: introduce a `Subst` (type variable substitution)
|
||
and unification. Thread through `synth`. At `let`, generalise
|
||
syntactic values (lambdas) — no value-restriction subtlety
|
||
needed yet, the MVP has no mutable refs.
|
||
12b. Codegen: at each polymorphic call site, the typechecker
|
||
records the instantiation. Codegen walks the AST a second
|
||
time per (def, instantiation) pair and emits a specialised
|
||
version with the type variables substituted by concrete
|
||
types in fn signatures.
|
||
12c. Docs + a polymorphic `id` test + a generic `map :: (a -> b)
|
||
-> List a -> List b` rewrite of `list_map.ail.json`.
|
||
|
||
## 2026-05-07 — Iter 12a/b done: polymorphism reaches the binary
|
||
|
||
Skipped 12c's "polymorphic map" — without parameterised ADTs (which
|
||
the MVP doesn't have), the rewrite would still be over a concrete
|
||
`IntList`, defeating the purpose. So 12c becomes lighter: docs +
|
||
two new examples (`poly_id`, `poly_apply`) that prove polymorphism
|
||
end-to-end on primitive types and on fn-typed parameters. The big
|
||
test is whether *I* would use the language now for a poly-flavoured
|
||
program; the answer below.
|
||
|
||
**12a — typechecker:**
|
||
|
||
`Type::Forall { vars, body }` is now legal at top-level fn types.
|
||
Implementation is the textbook ML rule: peel the Forall when
|
||
checking the body (rigid vars go into `Env.rigid_vars` so
|
||
`check_type_well_formed` accepts them), instantiate fresh metavars
|
||
at every var-resolution site, unify on every formerly-`expect_eq`
|
||
edge.
|
||
|
||
The metavar encoding sidesteps an AST schema change: a metavar is
|
||
just `Type::Var { name: "$m<id>" }`. The `$` prefix can't collide
|
||
with source identifiers, the JSON layout doesn't shift, and module
|
||
hashes stay bit-identical (verified: `sum.ail.json` keeps
|
||
`db33f57cb329935e` / `d9a916a0ed10a3d3`). I considered adding a
|
||
new `Type::Meta` variant under `#[serde(skip)]` but that would
|
||
have pulled hashing concerns into serde; the naming convention
|
||
keeps the AST untouched.
|
||
|
||
`Subst` is a flat `BTreeMap<u32, Type>`; `unify` is the standard
|
||
occurs-check version with effects compared as a set. Constants
|
||
still reject Forall outright; ADT fields still reject vars. No
|
||
let-generalisation: lambdas inside fn bodies are checked
|
||
monomorphically against their declared types — keeps the
|
||
implementation small and matches DESIGN.md's "top-level types
|
||
must always be explicitly annotated".
|
||
|
||
**12b — codegen:**
|
||
|
||
Direct calls to a polymorphic def get monomorphised on demand.
|
||
Each unique (def, instantiation) pair emits a specialised LLVM fn
|
||
with mangling `@ail_<m>_<def>__<descriptor>`. Descriptor scheme:
|
||
`Int → I`, `Bool → B`, `Unit → U`, `Str → S`, ADT `Foo → FFoo`,
|
||
`Fn(a)→b → Fn_<a>__r_<b>`. So `id(42)` and `id(true)` produce
|
||
`@ail_poly_id_id__I` and `@ail_poly_id_id__B` side by side.
|
||
|
||
Pass 1 of `lower_workspace` now splits fn-typed defs into mono
|
||
(`module_user_fns`, LLVM-typed FnSig as before) and poly
|
||
(`module_polymorphic_fns`, full FnDef). A unified
|
||
`module_def_ail_types` carries AILang types for both, used by
|
||
the codegen-side type tracker.
|
||
|
||
The hard part was getting AILang types at call sites. The
|
||
typechecker has them but doesn't hand its annotations down (no
|
||
TIR yet). I considered three paths:
|
||
1. Typechecker sidetable keyed by AST node ids — would need
|
||
to assign ids deterministically, brittle.
|
||
2. Uniform representation (everything passes as ptr/i64) —
|
||
contradicts CLAUDE.md's "performance is extremely important".
|
||
3. Codegen replays the type derivation locally.
|
||
Picked (3). The trade-off is duplication (`synth_arg_type`
|
||
mirrors what the typechecker already did), but it's contained
|
||
to a small recursive walk and uses the same `locals`/`extras`
|
||
shadowing pattern. Worth it for the MVP — once a TIR stage
|
||
materialises (it's still on the debt list), the duplication
|
||
collapses into a single pass.
|
||
|
||
`locals` grew from 3-tuple to 4-tuple `(name, ssa, llvm_type,
|
||
ail_type)`. Six push sites updated mechanically. Lambda capture
|
||
metadata grew the same way. `CtorRef` got `ail_fields` so match
|
||
arm bindings inherit the AILang type.
|
||
|
||
The drain phase iterates until `mono_queue` is empty —
|
||
specialised bodies can themselves invoke polymorphic defs and
|
||
queue further entries. `apply_subst_to_term` substitutes rigid
|
||
vars in `Term::Lam` annotations (the only Term arm carrying
|
||
types).
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For monomorphic programs:
|
||
yes (already established). For polymorphism over primitives
|
||
and fn-typed parameters: yes — `id` and `apply` write out the
|
||
way the textbook says they should, with no language-level
|
||
bookkeeping leaking into the source. The `poly_apply` example
|
||
was particularly revealing: the closure-pair ABI (Iter 8a)
|
||
composes cleanly with monomorphisation. Specialised body of
|
||
`apply__I_I` keeps `f` as a fn-typed local; the existing
|
||
indirect-call path already handles the lower from there.
|
||
- *Did I think of everything?* No, two known gaps:
|
||
1. **Polymorphic fn passed as a value** (`let f = id in f(42)`)
|
||
fails in codegen — `resolve_top_level_fn` looks in
|
||
`module_user_fns` only. Adding this means emitting one
|
||
closure-pair global per instantiation, possibly via the
|
||
same drain pass. Defer.
|
||
2. **Higher-rank polymorphism** (`apply(id, 42)`) trips
|
||
`unify_for_subst` which doesn't handle Forall on the param
|
||
side. Real higher-rank polymorphism is a substantial step
|
||
and not on the near horizon — deferred to a later iter.
|
||
- *Visualisation:* `ail manifest poly_id.ail.json` now shows
|
||
`forall a. (a) -> a` correctly. The pretty-printer carried
|
||
`Type::Forall` rendering since Iter 1; nothing to do.
|
||
- *KISS:* +1 typechecker file edit (~430 LOC inserted, mostly
|
||
Subst+unify+four tests), +1 codegen extension (~600 LOC
|
||
inserted, mostly the drain path + helpers + locals widening).
|
||
Two new examples, two new e2e tests. Could be smaller if I
|
||
bit the bullet on TIR; not yet worth the upfront cost.
|
||
|
||
**Tests:** 58/58 (was 56/56). Added 4 typechecker unit tests in
|
||
12a, 2 e2e tests in 12b. Hash invariant holds.
|
||
|
||
**Plan iteration 13 (queued, not started):**
|
||
|
||
The natural next step depends on what I want to use the language
|
||
for. Two candidates, in order of expected payoff:
|
||
|
||
13a. **Parameterised ADTs** — `List a`, `Maybe a`, etc. Without
|
||
these, polymorphism is half-useful: a generic `map` still
|
||
can't transform an `IntList` into a `BoolList`. ADT defs
|
||
would gain a `vars: Vec<String>` field; ctor field types
|
||
could mention them; codegen monomorphises ADT instances
|
||
just like fns. This is the bigger expressivity unlock.
|
||
13b. **GC or arena** — every ADT box, lambda env, and closure
|
||
pair currently leaks. For sort over an 11-element list,
|
||
fine. For anything longer-running, required. The current
|
||
lifetime model is "leak"; the right MVP is probably
|
||
bumpalloc per top-level fn invocation. Could be done
|
||
before parameterised ADTs but doesn't unlock new examples.
|
||
|
||
Leaning 13a — it's the more interesting architectural step and
|
||
makes the "polymorphic map" rewrite from the original 12c plan
|
||
finally meaningful.
|
||
|
||
## 2026-05-07 — Iter 13 done: parameterised ADTs reach the binary
|
||
|
||
**Why now.** End of Iter 12 left polymorphism half-useful: `id`
|
||
and `apply` worked, but every container was monomorphic
|
||
(`IntList`, `Maybe_Int`). A generic `map :: forall a b. ((a) -> b,
|
||
List a) -> List b` was unwritable. 13 lifts that.
|
||
|
||
**Three commits:**
|
||
|
||
- `0782622` 13a — schema (`TypeDef.vars`, `Type::Con.args`) +
|
||
checker (substitution at ctor + match + arity validation in
|
||
`check_fn`).
|
||
- `1631f60` 13b — codegen: per-use-site substitution of LLVM
|
||
field types in `lower_ctor` and `lower_match`. No mono-queue
|
||
for types — ctor code was already inlined at every use site,
|
||
so 13b only had to thread substitution through, not invent a
|
||
symbol scheme. `synth_arg_type` for `Term::Ctor` now returns
|
||
concrete type-args, and `llvm_type(Type::Var)` is a hard
|
||
error instead of a silent `ptr` fallback (the latter was
|
||
flagged by the architect review and is the most defensive
|
||
single change in 13).
|
||
- `<this>` 13c — DESIGN.md flipped (parameterised ADTs out of
|
||
the gap list, into the supported list); two new example lines.
|
||
|
||
**Hash invariant.** Both new fields are
|
||
`#[serde(default, skip_serializing_if = "Vec::is_empty")]`. A new
|
||
regression test in `crates/ailang-core/src/hash.rs` deserialises
|
||
the actual `examples/sum.ail.json` and `examples/list.ail.json`
|
||
from disk and asserts `db33f57cb329935e` and `b082192bd0c99202` —
|
||
the recorded pre-13a hashes. It's deliberately phrased against
|
||
the on-disk JSON rather than reconstructed code, so the test
|
||
fails if anyone resaves the examples in a way that drifts the
|
||
canonical bytes.
|
||
|
||
**Architect-flagged debt I deliberately did NOT touch in 13:**
|
||
|
||
- `is_static_callee` returns true for poly fns but
|
||
`resolve_top_level_fn` only consults `module_user_fns`. A
|
||
poly fn used as a value (`let f = id in f(42)`) passes the
|
||
static check then surfaces as `UnknownVar`. Would need one
|
||
closure-pair global per instantiation. Out of 13 scope; same
|
||
hole that was queued at the end of Iter 12.
|
||
- Triple source of truth for builtins (`builtins::install`,
|
||
`builtins::list`, `codegen::builtin_ail_type` /
|
||
`builtin_effect_op_ret`). Every new operator costs three
|
||
edits. Low interest today, escalates with every effect op.
|
||
Worth a future tidy iter — not blocking expressivity.
|
||
- `synth_arg_type` for `Term::If` returns `synth(then)` only;
|
||
for `Term::Match`, the first arm. Masked today by the
|
||
typechecker having already unified, but it's the kind of
|
||
duplication that decays. Same fundamental cost as the absence
|
||
of a TIR.
|
||
|
||
**Architecture self-check.**
|
||
|
||
- *Would I use this language now?* For polymorphism over
|
||
primitives, fn-typed values, AND parameterised containers —
|
||
yes. The `box.ail.json` and `maybe_int.ail.json` examples
|
||
read like the textbook says they should. No type-arg
|
||
bookkeeping leaks into the source.
|
||
- *KISS.* 13b was much smaller than I feared at the start of
|
||
the design phase: ~150 LOC in codegen, no new structures, no
|
||
mono-queue. The reason: ADT ctor code is already inlined.
|
||
The architect's recommendation to *not* mutate `ctor_index`
|
||
but derive `CtorRef` per use site was the right call —
|
||
preserved the static template, made the substitution local.
|
||
- *Did I think of everything?* Two known gaps remain. **(1)**
|
||
Polymorphic ADTs as the type-arg of a polymorphic fn —
|
||
works today because `unify_for_subst` recurses through
|
||
`Type::Con.args` (added in 13a). **(2)** A polymorphic fn
|
||
taking a polymorphic ADT and returning a different
|
||
parameterised ADT (`map : forall a b. ((a)->b, List a) -> List
|
||
b`) — should also work, but I haven't dogfooded it yet
|
||
because `List a`-as-a-rewrite-of-`list_map` would need the
|
||
schema bumps elsewhere (paramaterised list builder). Queued
|
||
for Iter 14.
|
||
- *Visualisation.* `ail manifest examples/box.ail.json` shows
|
||
`type Box :: forall a. MkBox(a)` and
|
||
`fn unbox :: forall a. (Box<a>) -> a`. The pretty-printer
|
||
picked up `args` and `vars` cleanly (Iter 13a).
|
||
|
||
**Tests:** 64/64 (was 58/58). Added: 1 hash-stability regression
|
||
(13a), 3 checker unit tests for parameterised ADTs (13a),
|
||
2 e2e tests over `box.ail.json` and `maybe_int.ail.json` (13b).
|
||
|
||
**Process note (orchestration).** First iter where I worked
|
||
strictly through the agents in `/agents/`: `ailang-architect`
|
||
ran a drift review on HEAD before 13b started; `ailang-implementer`
|
||
got a fixed brief that incorporated the architect's three
|
||
recommendations (don't mutate `ctor_index`, fix `synth_arg_type`
|
||
for `Term::Ctor`, harden `llvm_type`); 13c (this) is the
|
||
orchestrator's own work. The role split landed in `3df943d` after
|
||
I caught myself doing implementer work on 13a directly. The
|
||
agents pay off in proportion to iter size — for 13b they were
|
||
clearly worth the round-trip; for 13a's checker work, marginal.
|
||
|
||
**Plan iteration 14 (queued, not started):**
|
||
|
||
Two candidates, in order of expected payoff:
|
||
|
||
14a. **Polymorphic `List a` rewrite of `list_map`.** Replaces
|
||
`IntList` with `List a`, rewrites `list_map` to return
|
||
`List b`, and lets the polymorphic-map version be the
|
||
dogfood smoke test. Pure exercise — should fall out of
|
||
13b — but worth the dogfood beat. Also: `Maybe a` used
|
||
in a non-trivial fn (e.g. `find : forall a. ((a) -> Bool,
|
||
List a) -> Maybe a`).
|
||
14b. **GC or arena.** Same pitch as before: every ADT box,
|
||
lambda env, closure pair leaks. For `box.ail.json` and
|
||
`maybe_int.ail.json`, fine. For anything that allocates
|
||
in a loop, required. Bumpalloc per top-level fn
|
||
invocation is the natural MVP.
|
||
14c. **Poly fn as value.** Closes the asymmetry the architect
|
||
flagged; gates `let f = id in f(42)`. One closure-pair
|
||
global per instantiation, emitted via the same mono-queue
|
||
drain path. Smaller surface than 14a/b.
|
||
|
||
Leaning 14a — the dogfood payoff for one iter of polish is
|
||
high, and `Maybe`-in-a-real-fn is a missing piece I haven't
|
||
exercised yet. 14b stays second; 14c is a candidate if I want
|
||
a small palate cleanser.
|
||
|
||
---
|
||
|
||
## Iter 13d — rustdoc polish for `ailang-core` + new `ailang-docwriter` agent
|
||
|
||
User triggered: ran `cargo doc --open` for fun and reported
|
||
that the rendered docs were thin — crate headers existed,
|
||
but `pub` items had no `///` strings, there were no `# Examples`
|
||
sections, and intra-doc links were missing. Internal references
|
||
showed up as prose ("see Builtins") rather than as clickable
|
||
links. Two stale `[Builtins]` and `[code]` warnings had been
|
||
bleeding into every `cargo doc` invocation since Iter 6 or so.
|
||
|
||
Recurring task → new agent. Wrote `agents/ailang-docwriter.md`
|
||
with a tight mandate: rustdoc only, no API changes, no edits in
|
||
`docs/` or `agents/`, three verification gates (rustdoc clean,
|
||
build green, tests green incl. doctests). Updated
|
||
`agents/README.md`. Updated DESIGN.md item 6 of "Verification
|
||
and correctness" to make rustdoc cleanliness a project-wide
|
||
invariant rather than an iter-local cleanup.
|
||
|
||
First mission: `ailang-core` only. The foundation crate every
|
||
other crate depends on — biggest reader-leverage per diff. The
|
||
agent rewrote the crate root so a newcomer learns: what `core`
|
||
owns, where it sits in the pipeline (`core` → `check` →
|
||
`codegen` → `ail`), the central invariant (canonical JSON is
|
||
deterministic, hashes content-addressed, schema = `ailang/v0`),
|
||
and the entry points. Module roots in `ast.rs`, `canonical.rs`,
|
||
`hash.rs`, `pretty.rs`, `workspace.rs` got the same treatment.
|
||
Every `pub` struct / enum / variant / fn / const got a `///`
|
||
string. The Iter-13a additions (`TypeDef.vars`, `Type::Con.args`)
|
||
got an explicit backwards-compat note that points back to the
|
||
hash-stability regression test.
|
||
|
||
`# Examples` blocks landed where they shorten understanding
|
||
(`canonical::to_bytes`, `def_hash`, `Workspace`), all marked
|
||
`ignore` so the workspace doctest run stays cheap (3 ignored,
|
||
0 run, 0 failed). The two stale broken-link warnings in
|
||
`ailang-check` got prose-only fixes — out-of-scope for this
|
||
iter conceptually, but a one-line fix per file means rustdoc
|
||
is now globally clean.
|
||
|
||
**Findings reported by the docwriter** (judgement deferred to
|
||
me; none made it into the diff):
|
||
|
||
- `Term::Lam.param_tys` (JSON: `paramTypes`) is positionally
|
||
paired with `Term::Lam.params: Vec<String>`. Naming hints
|
||
at the convention but a cold reader has to deduce it. **Not
|
||
fixing**: renaming would touch the schema, breaks every hash.
|
||
The `///` string makes the convention explicit, which is
|
||
enough.
|
||
- `Type::Var` is overloaded: source-level rigid vars and
|
||
checker metavars (`$m<id>`) share the same variant. A reader
|
||
of `core` alone sees no hint of the metavar half — it's
|
||
documented in `ailang-check`'s lib doc instead. **Not fixing**:
|
||
splitting the variant would balloon the schema and
|
||
invalidate every hash. Acceptable as long as `check`'s lib
|
||
doc explains it (it does, post-13d).
|
||
- `Def::Type(TypeDef)` versus the type-expression enum `Type`
|
||
in the same module: name collision is real but unavoidable
|
||
without renaming `Type` (which would touch every crate). The
|
||
`///` strings now disambiguate at point of contact.
|
||
|
||
**Process note (orchestration).** Second iter where I worked
|
||
strictly through agents (after 13b). The docwriter brief was
|
||
written from the diagnostic in this conversation, not from a
|
||
DESIGN-doc design pass — there was no architecture decision to
|
||
make, just a discipline gap to close. That's the right shape
|
||
for a docwriter: low-judgement, repeatable, runs after every
|
||
iter that touches public surface. Cost ≈ 25 tool uses for ~390
|
||
LOC of doc additions across 9 files; smaller-grain than
|
||
`ailang-implementer` runs typically are.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(new). `cargo doc --no-deps`: 0 warnings (was 2). `cargo build
|
||
--workspace` green. `cargo test --workspace` green.
|
||
|
||
**Plan iteration 13e/13f (queued, not started).** Two natural
|
||
follow-ups for the docwriter:
|
||
|
||
13e. **`ailang-check` rustdoc**: type-checker is the next
|
||
biggest crate by `pub`-surface and the most algorithmically
|
||
dense. Rigid/metavar split, `Forall` instantiation, the
|
||
match-arm exhaustiveness logic, the Iter-13a substitution
|
||
machinery — all of it benefits more from prose explanation
|
||
than `core` did.
|
||
13f. **`ailang-codegen` + `ail` CLI rustdoc**: codegen is dense
|
||
but mechanical (mangling, ABI, block tracking); the CLI
|
||
is mostly clap derive macros. Lower payoff per LOC of doc
|
||
than 13e but rounds out the warning-free invariant across
|
||
the whole workspace.
|
||
|
||
After 13d there's no urgency on 13e/f — `cargo doc` is already
|
||
warning-free. They're "polish iterations to be slotted in
|
||
between feature iters when context budget is short". 14a
|
||
(`List a` rewrite) remains the next-feature default.
|
||
|
||
## Iter 13e — rustdoc polish for `ailang-check`
|
||
|
||
Second docwriter mission. Same mandate as 13d, applied to the
|
||
typechecker crate. `lib.rs` (1922 LOC) had a strong crate root
|
||
already — covers HM-with-effects, top-level forall, the
|
||
rigid-vs-metavar split, the `$m<id>` encoding — but its `pub`
|
||
surface (the `Env` struct, `CheckError` + 24 variants,
|
||
`CheckedModule`, `CtorRef`, the `check_module` entry point) was
|
||
mostly undocumented. `builtins.rs` had a one-line module
|
||
header and zero `///` strings on `EffectOpSig` or `install`.
|
||
`diagnostic.rs` had a strong module header (lists every stable
|
||
diagnostic code) but `Diagnostic`, `Severity`, and the
|
||
construction helpers were undocumented.
|
||
|
||
Agent added 188 LOC of pure rustdoc across the three files; no
|
||
non-doc lines changed (verified by filtering the diff). Each
|
||
`CheckError` variant now carries the AST term/type that
|
||
triggers it and the stable kebab-case `code` it maps to. `Env`
|
||
fields (`globals`, `effect_ops`, `types`, `module_globals`,
|
||
`current_module`) got individual `///` strings that name the
|
||
invariants — most importantly that `module_globals` includes
|
||
the current module, which the agent flagged as undocumented at
|
||
field level (now fixed). Crate-root prose got an upgraded
|
||
intra-doc link to `[`check_module`]`; `super::` reference in
|
||
`diagnostic.rs` rewritten to `crate::` (cosmetic, but the
|
||
canonical form).
|
||
|
||
**Findings reported** (judgement deferred to me):
|
||
|
||
- `Env` is `pub` with all-`pub` fields but `Env::new` is
|
||
private and there's no public builder — external callers can
|
||
only construct via field-by-field literal, which is fragile
|
||
if a field is added later. **Not fixing**: changing this is
|
||
an API decision, not a doc one. Worth raising next time we
|
||
touch the crate's public surface deliberately.
|
||
- `CheckError::CtorArity` and `CheckError::ArityMismatch` both
|
||
serialize to the public diagnostic code `arity-mismatch`. The
|
||
`///` strings now flag the collision per-variant; tooling
|
||
that consumes the diagnostic JSON sees only the merged code
|
||
and that's intentional from the iter-5b vintage. **Not
|
||
fixing.**
|
||
- `Diagnostic` and `Severity` are reachable both via the crate
|
||
re-export and via `crate::diagnostic::*` because the
|
||
`diagnostic` module is itself `pub`. Rustdoc renders both
|
||
pages; harmless but slightly noisy. **Not fixing**: the
|
||
re-export is the documented entry point and we don't want to
|
||
hide the module.
|
||
|
||
**Process note.** Same shape as 13d: I wrote a brief naming
|
||
the deficiencies (numbers of pub items, which files had thin
|
||
roots), the agent did the doc additions inside its mandate,
|
||
the orchestrator-side work was authoring DESIGN/JOURNAL and
|
||
verifying the diff. Cost ≈ 32 tool uses for 188 LOC of doc
|
||
across 3 files — denser than 13d (more sentences per pub item
|
||
because the typechecker invariants need explicit
|
||
articulation), but the per-LOC payoff for a future reader is
|
||
also higher.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(unchanged). `cargo doc --no-deps`: 0 warnings (was 0; the
|
||
agent introduced 4 transient broken-link warnings during the
|
||
work and resolved all of them before reporting done). `cargo
|
||
build --workspace` green. `cargo test --workspace` green.
|
||
|
||
**Plan 13f / 14a unchanged.** 13f (`ailang-codegen` + `ail`
|
||
CLI) is the natural next polish iter; 14a (`List a` rewrite of
|
||
`list_map`) remains the next-feature default. Auto-mode is on,
|
||
so I'll continue into 13f directly unless context budget
|
||
pressures a switch.
|
||
|
||
## Iter 13f — rustdoc polish for `ailang-codegen` + `ail` CLI
|
||
|
||
Combined docwriter mission. Codegen has a small public
|
||
surface (only 3 top-level `pub` items: `CodegenError` enum +
|
||
7 variants, `emit_ir`, `lower_workspace`); the CLI is a
|
||
binary with zero `pub` items, so the only useful rustdoc is
|
||
the module header. One agent run, both files.
|
||
|
||
What landed (148 LOC of doc additions, no non-doc lines
|
||
changed, verified by filtering the diff):
|
||
|
||
- `ailang-codegen` crate root got intra-doc-link upgrades
|
||
(`[`emit_ir`]`, `[`lower_workspace`]`,
|
||
`[`CodegenError::MissingEntryMain`]`) and a precondition
|
||
sentence — both entry points assume their input has already
|
||
passed the typechecker; codegen does not call the checker
|
||
itself.
|
||
- Every `CodegenError` variant got a `///` string naming the
|
||
AST term/condition that triggers it. Same shape as
|
||
`CheckError` post-13e, so the two error enums now read
|
||
similarly and a reader can grep across them. `Internal` is
|
||
flagged in its doc as a catch-all that covers ~30
|
||
invariant-violation sites.
|
||
- `emit_ir` and `lower_workspace` now make the
|
||
single-vs-multi-module split explicit and cross-link to
|
||
each other.
|
||
- `ail/main.rs` module header expanded from a 5-line stub to
|
||
a full subcommand list with one-liners (`manifest`,
|
||
`render`, `describe`, `deps`, `check`, `emit-ir`, `build`,
|
||
`run`, `builtins`, `diff`, `workspace`), the `clang`-on-PATH
|
||
prerequisite for `build`/`run`, the design-intent paragraph
|
||
about each subcommand being narrowly scoped for LLM
|
||
consumption (already partly there), and an explicit "no
|
||
`pub` items, `--help` text comes from clap" note for anyone
|
||
who lands here from rustdoc.
|
||
|
||
**Findings reported** (judgement deferred to me):
|
||
|
||
- `CodegenError::Internal(String)` is a single opaque catch-all
|
||
for ~30 distinct invariant-violation sites (mono-queue
|
||
desync, ctor-index miss, lambda-env shape, ...). Tests can
|
||
only substring-match on it. **Not fixing**: splitting is a
|
||
test-ergonomics decision, not a doc one. Worth raising the
|
||
next time codegen tests get a serious rewrite.
|
||
- `emit_ir` synthesises an internal `Workspace` with
|
||
`root_dir = "."`. No codegen path reads `root_dir` today, so
|
||
this is harmless; if a future feature reaches `root_dir`
|
||
from codegen, the assumption surfaces. **Not fixing**: the
|
||
agent flagged it correctly as "would change behaviour, out
|
||
of scope".
|
||
|
||
**Process note: brief drift caught by the agent.** I told the
|
||
docwriter the CLI had nine subcommands; it found eleven
|
||
(`Deps` and `Diff` were missing from my brief, which was
|
||
written off a `head -40` of the source). Agent silently
|
||
corrected and flagged the drift in its findings. Useful
|
||
counter-pressure to the orchestrator pattern: my survey was
|
||
sloppy and the agent did not propagate the sloppiness into
|
||
the doc. This is one of the things sub-agents are good at and
|
||
why I keep delegating even on small jobs.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(unchanged). `cargo doc --no-deps`: 0 warnings. Also verified
|
||
under `RUSTDOCFLAGS='-D rustdoc::broken_intra_doc_links'` per
|
||
agent report. `cargo build --workspace` green. `cargo test
|
||
--workspace` green.
|
||
|
||
**Workspace-wide rustdoc invariant achieved.** All four
|
||
crates (`ailang-core`, `ailang-check`, `ailang-codegen`,
|
||
`ail`) now have:
|
||
- crate-root `//!` that names ownership, position in
|
||
pipeline, and entry points;
|
||
- module-root `//!` on every file with non-trivial content;
|
||
- `///` on every public item (struct, enum, variant, fn,
|
||
field, const) that names the contract, not just the type;
|
||
- intra-doc links wherever prose previously referred to
|
||
another item by name.
|
||
|
||
DESIGN.md item 6 ("rustdoc cleanliness") is now load-bearing
|
||
across the whole workspace, not just `core`. The
|
||
`ailang-docwriter` agent's job from here is **maintenance**:
|
||
run after any iter that adds public surface, not full sweeps.
|
||
|
||
**Next.** 14a is now unblocked: write a polymorphic `list_map`
|
||
that uses `List a` (Iter-13a parameterised ADTs) and `Maybe a`,
|
||
then extend an existing demo program (`hello_print` or one of
|
||
the dogfood sources) to call it end-to-end. That exercises
|
||
the parameterised-ADT pipeline through type-check, codegen,
|
||
and runtime — the missing piece in 13a/b/c was that the
|
||
feature shipped but no real program used it. If context
|
||
budget at the start of 14a is tight, alternative is 14b
|
||
(GC/arena scaffolding) or 14c (poly fn as value); both have
|
||
real design questions that need an orchestrator design pass
|
||
first, so 14a stays the default.
|
||
|
||
## Iter 14a — polymorphic `List a` end-to-end + monomorphisation bug fixed
|
||
|
||
The dogfood payoff for parameterised ADTs (13a/b/c). Prior
|
||
to this iter, the only programs exercising the feature were
|
||
`box.ail.json` (single `MkBox(42)` round-trip) and
|
||
`maybe_int.ail.json` (single `or_else` call). That is a thin
|
||
slice — neither program builds a recursive parameterised ADT
|
||
nor calls a polymorphic higher-order fn. 14a closes that gap
|
||
with `data List a` + `map : forall a b. ((a) -> b, List<a>) ->
|
||
List<b>` recursive, then prints the result.
|
||
|
||
Three-agent run (tester → debugger → no implementer needed):
|
||
|
||
1. **Tester** wrote `examples/list_map_poly.ail.json` (5 defs:
|
||
`List`, `inc`, `map`, `print_list`, `main`) and a new e2e
|
||
test `list_map_poly_inc_then_prints` that asserts stdout
|
||
`["2", "3", "4"]`. Typecheck passed, build crashed:
|
||
`internal: monomorphisation: var \`a\` bound to two
|
||
distinct types`. Tester correctly stopped — fixture
|
||
encodes the contract; bug is in the compiler — and
|
||
reported with a hypothesis (recursion / shared
|
||
substitution slot in `map`).
|
||
|
||
2. **Debugger** refuted the hypothesis with a non-recursive
|
||
repro (`Cons(7, Nil)` triggers the same crash without
|
||
`map` in the picture at all). Real cause was much smaller:
|
||
in `synth_arg_type` (codegen, ~line 2087) for `Term::Ctor`,
|
||
any type var of the parent ADT that the ctor's args
|
||
couldn't pin was filled with `Type::unit()` as a
|
||
placeholder. For nullary ctors of a parameterised ADT
|
||
(`Nil : List<a>`, `None : Maybe<a>`) that placeholder
|
||
leaked upward. Inside a parent like `Cons(Int, Nil) :
|
||
List<a>`, `unify_for_subst` would walk
|
||
`cref.ail_fields = [Var{a}, Con{List,[Var{a}]}]` against
|
||
`[Int, Con{List,[Unit]}]`, bind `a = Int` from the head,
|
||
then collide with `a = Unit` from the tail. The recursive
|
||
`map` fixture surfaced it because it is the first program
|
||
to nest a nullary ctor of a parameterised ADT inside a
|
||
parent ctor — the existing 13b regressions never did. The
|
||
tester's hypothesis was reasonable from the symptom but
|
||
wrong on mechanism; refuting it via a smaller repro is
|
||
exactly the discipline the debugger role is for.
|
||
|
||
3. **Fix** (`crates/ailang-codegen/src/lib.rs`, +30/-9 LOC,
|
||
no new variant, no API change):
|
||
- Replace `Type::unit()` placeholder with a synth-only
|
||
wildcard `Type::Var { name: "$u" }`. The `$u` prefix is
|
||
a reserved-namespace convention that mirrors the
|
||
checker's `$m<id>` for instantiation metavars — same
|
||
trick (source-level identifiers can't start with `$`),
|
||
same goal (extra semantics without schema change).
|
||
- `unify_for_subst` short-circuits on a `$u`-prefixed
|
||
**arg-side** var: accept without binding, let a sibling
|
||
arg pin the type var instead. Param-side semantics
|
||
untouched. `derive_substitution`'s "var not pinned"
|
||
check still fires for genuinely under-determined calls
|
||
(those would have failed typechecking, so codegen never
|
||
sees them, but the safety net stands).
|
||
|
||
**Tests:** 25/25 e2e (was 24, +1 new poly test). All five
|
||
named regressions stayed green: `list_map_doubles_then_prints`,
|
||
`parameterised_box_round_trip`, `parameterised_maybe_match`,
|
||
`polymorphic_id_at_int_and_bool`, `polymorphic_apply_with_fn_param`.
|
||
Insertion sort still green. `cargo doc --no-deps`: 0 warnings
|
||
(workspace invariant from 13d/e/f preserved). `cargo build
|
||
--workspace` green.
|
||
|
||
**Process note: tester→debugger→done in one iter.** No
|
||
implementer dispatch needed — the debugger's mandate covers
|
||
"propose **and apply** a minimally invasive fix" once the
|
||
diagnosis is solid. 30 LOC across two sites in one file is
|
||
inside the role's scope (the role doc says "stop and report"
|
||
only on >50 LOC across multiple files). The orchestrator-side
|
||
work was the design (the source program), the scoped briefs,
|
||
and verification. This is the cleanest agent-flow shape so
|
||
far: each agent did exactly its job, the tester's wrong
|
||
hypothesis didn't propagate because the debugger tested it,
|
||
and the orchestrator never wrote compiler code.
|
||
|
||
**Findings flagged** (judgement deferred, not fixed):
|
||
|
||
- `CodegenError::Internal` (the catch-all string variant the
|
||
docwriter flagged in 13f) is now used by one more invariant
|
||
— the `$u` short-circuit could in principle be reached by a
|
||
malformed input; it's currently silent because typecheck
|
||
rejects under-determined calls upstream. Worth splitting
|
||
into typed variants the next time codegen tests get a real
|
||
rewrite. (Same finding as 13f, now reinforced by another
|
||
use site.)
|
||
- The `$u` / `$m` reserved-prefix convention (`$u` for
|
||
codegen synth wildcards, `$m<id>` for checker metavars)
|
||
is undocumented as a project-wide naming rule. Two
|
||
prefixes is fine; if a third appears, this should be
|
||
promoted to a DESIGN.md note.
|
||
|
||
**Next.** Parameterised ADTs are now genuinely usable for
|
||
real programs. The standard library starter set (an
|
||
`examples/std_*` series with `List`, `Maybe`, `Either`,
|
||
basic combinators `length`, `filter`, `fold`, `concat`)
|
||
becomes worthwhile in a way it wasn't pre-14a — that's a
|
||
plausible 14b alternative, smaller than GC/arena, and would
|
||
itself surface more dogfood bugs. The original 14b
|
||
(GC/arena) and 14c (poly fn as value) remain on the queue
|
||
but have not had a design pass.
|
||
|
||
## Iter 14b — design pass for the authoring surface
|
||
|
||
User redirected the project at the iter boundary. I had been
|
||
about to write a stdlib in JSON-AST form; user pushed back: do
|
||
I really program *best* in JSON, given the language is supposed
|
||
to be the one I'm most accurate in? Honest answer was no — JSON
|
||
was rationalisation. The constraint added in this iter:
|
||
|
||
> "Die Syntax sollte gut formalisierbar sein, damit man auch
|
||
> einem fremden LLM eine Spec geben kann, die es dann fehlerfrei
|
||
> befolgt."
|
||
|
||
That single sentence ruled out about half the design space:
|
||
anything with operator precedence (precedence is the #1 thing
|
||
an LLM gets wrong when handed a spec, because it requires
|
||
building a parse-tree mental model rather than just following
|
||
production rules), anything with semantic indentation, anything
|
||
with maximal-munch lexing or context-sensitive reductions.
|
||
|
||
What remains is roughly: S-expressions, or dialects close to
|
||
S-expressions. Decision 6 in DESIGN.md captures the constraints
|
||
in priority order, sketches three candidate forms (A: tagged
|
||
S-expression, B: indented record-style, C: pretty-printer-as-
|
||
source), and picks (A) as the first attempt with explicit
|
||
rollback to (C) if (A) hurts authoring more than it helps.
|
||
|
||
**Key shape of (A):** every AST node has a unique head keyword.
|
||
No case-disambiguation rule (no "capitalised head means ctor").
|
||
Bare atoms in positional slots get their sort from the parent
|
||
slot (inside `(con NAME args...)` second-and-later positions
|
||
are types; inside `(app HEAD args...)` first position is a term;
|
||
etc.). To construct a value with a ctor, write
|
||
`(term-ctor TypeName CtorName args...)` explicitly. To match,
|
||
`(pat-ctor CtorName fields...)`. The capitalisation/case of an
|
||
identifier carries no semantic weight to the parser.
|
||
|
||
This rules out the silent-error class "I forgot to capitalise
|
||
`Cons` and it parsed as a function call" — which I had been
|
||
relying on case-conventions to prevent in the JSON form too,
|
||
but only by hand-discipline. With explicit tags the discipline
|
||
becomes a parse rule.
|
||
|
||
**Empirical test (this iter).** Hand-encoded three examples in
|
||
form (A) as `examples/*.ailx`:
|
||
|
||
```
|
||
hello.ailx — 5 LOC (was JSON: 36 pretty / 21 canonical)
|
||
box.ailx — 25 LOC (was JSON: 160 pretty / 88 canonical)
|
||
list_map_poly.ailx — 50 LOC (was JSON: 394 pretty / 230 canonical)
|
||
```
|
||
|
||
Roughly 4–8× line reduction, ~4× character reduction. Bigger
|
||
gains on bigger programs (overhead is proportional to AST
|
||
depth, not to program size, so the form scales well). All three
|
||
mapped unambiguously to AST nodes by inspection — no
|
||
ambiguities surfaced during writing that the spec didn't
|
||
already cover.
|
||
|
||
Two small lex/grammar issues I discovered while writing and
|
||
folded back into DESIGN.md before committing:
|
||
|
||
1. **Operator idents** like `+`, `==`, `<=`. Initial spec had a
|
||
word-shaped `[A-Za-z_]...` regex; that excluded operators.
|
||
Fix: lexer recognises only `(`/`)`/whitespace as delimiters.
|
||
Every other maximal token is classified by first character
|
||
(digit ⇒ integer, `"` ⇒ string, else ⇒ ident). `+`, `42`,
|
||
`io/print_int`, `==` all become single ident or integer
|
||
tokens with no special rule.
|
||
2. **Bool literals.** Bare `true`/`false` are reserved in term
|
||
context; outside term context they would be ill-formed
|
||
anyway. Unit is explicit: `(lit-unit)`.
|
||
|
||
**Files touched:**
|
||
- `docs/DESIGN.md` — Decision 6 added (~140 lines), with
|
||
constraints, three candidates, first-choice rationale, and
|
||
an implementation outline for Iter 14c.
|
||
- `examples/hello.ailx`, `examples/box.ailx`,
|
||
`examples/list_map_poly.ailx` — three design exhibits. Not
|
||
parseable yet (header comment says so).
|
||
- `docs/JOURNAL.md` — this entry.
|
||
|
||
**Tests:** None new. Existing 25/25 e2e + 3 ignored doctests
|
||
unchanged (this iter is paper, not code). `cargo doc --no-deps`
|
||
0 warnings (workspace invariant from 13d/e/f preserved).
|
||
|
||
**Process note: orchestration with explicit licence to be
|
||
wrong.** User said "du darfst auch ausprobieren und dich irren
|
||
(und es dann rückgängig machen). Wir wollen das beste Design
|
||
für den propagierten Zweck." That changes the cost model for
|
||
design iteration: I should optimise for *information per iter*,
|
||
not for *correctness on first commit*. So I committed form (A)
|
||
as a working hypothesis with a documented rollback path to
|
||
form (C), rather than design-by-committee until I was sure.
|
||
|
||
**Plan 14c (next).**
|
||
1. New crate `ailang-surface` with a small PEG parser → existing
|
||
`ailang-core::ast` types. No new AST nodes.
|
||
2. Round-trip test gate: every existing `examples/*.ail.json`
|
||
gets a sibling `*.ailx` written by hand or by an
|
||
AST→surface emitter; the test parses the surface, canonises
|
||
to JSON, and asserts hash-equivalence to the original. If a
|
||
single fixture loses its hash, the form does not ship.
|
||
3. CLI: `ail parse <file.ailx> -o <file.ail.json>`. Symmetric
|
||
to existing `ail render`.
|
||
4. If round-trip works for all current fixtures, mark form (A)
|
||
confirmed and start the stdlib in `.ailx` directly. If it
|
||
fails, document why in JOURNAL and try (C).
|
||
|
||
**Plan 14d (after 14c).** First stdlib module: `std_list.ailx`
|
||
with `length`, `filter`, `fold`, `concat`, `reverse`, `head`,
|
||
`tail`. Each combinator is a fresh test vector for codegen and
|
||
for the still-young parameterised-ADT pipeline. Written in the
|
||
new surface from day one — no JSON authoring of stdlib.
|
||
|
||
## Iter 14c — `ailang-surface` parser + pretty-printer ships
|
||
|
||
Implements Decision 6's form (A). New crate `ailang-surface`
|
||
(~1843 LOC across `lex.rs`, `parse.rs`, `print.rs`, lib root,
|
||
plus tests) ships as a strictly additive producer of
|
||
`ailang-core::ast::Module` values. `ailang-check` and
|
||
`ailang-codegen` were not modified — projection-agnostic, as
|
||
the architectural pin requires.
|
||
|
||
**Implementer dispatch went clean.** Brief gave the EBNF, the
|
||
fixtures to round-trip, the lexer rule, and the architectural
|
||
constraints. Implementer hand-wrote a recursive-descent parser
|
||
(one Rust fn per EBNF production), a deterministic pretty-
|
||
printer, an integration test that runs the round-trip gate on
|
||
every fixture, and the `ail parse` CLI subcommand. No
|
||
parser-combinator dependency. No AST-shape changes.
|
||
|
||
**Two AST-driven form refinements** that were not in the 14b
|
||
sketch (both folded into DESIGN.md Decision 6 by the
|
||
implementer before commit):
|
||
|
||
1. `lam-term` had to carry `param_tys`, `ret_ty`, and
|
||
`effects` because the AST's `Term::Lam` stores parallel
|
||
typed-param data. Production became `(lam (params (typed
|
||
x Int) ...) (ret T) (effects ...) (body ...))`. Same
|
||
shape-style as the rest of the form; no new lex rules.
|
||
2. `import-clause` had to admit `Option<String>` aliases.
|
||
Production became `(import name (as alias)?)` with `as`
|
||
as a bare ident token. No new lex rules.
|
||
|
||
The 14b 30-production budget held: implementer reports ~28
|
||
named productions in the parser. Constraint 1 (formalisable
|
||
for foreign LLM) intact.
|
||
|
||
**Verification gates, all green:**
|
||
|
||
- `cargo build --workspace`: 0 warnings, finished.
|
||
- `cargo test --workspace`: 76 tests pass (was 64; +9 unit
|
||
tests in `ailang-surface`, +2 integration tests in
|
||
`tests/round_trip.rs`, +1 e2e regression preserved). All
|
||
17 `examples/*.ail.json` fixtures round-trip
|
||
byte-identical through `print → parse → canonical JSON`;
|
||
3 hand-written `.ailx` exhibits parse to canonical JSON
|
||
byte-identical to their `.ail.json` siblings.
|
||
- `cargo doc --no-deps`: 0 warnings (workspace invariant
|
||
from 13d/e/f preserved; new crate's rustdoc landed
|
||
correctly with crate-root `//!` plus all `pub` items
|
||
documented).
|
||
|
||
**Manual smoke test (orchestrator-side after agent reported
|
||
done):** `ail parse <file.ailx> -o <file.ail.json>` followed
|
||
by `ail run <file.ail.json>` for all three exhibits:
|
||
|
||
```
|
||
hello.ailx → "Hello, AILang."
|
||
box.ailx → "42"
|
||
list_map_poly.ailx → "2\n3\n4"
|
||
```
|
||
|
||
End-to-end pipeline form (A) → AST → typecheck → codegen →
|
||
clang → binary works on all three.
|
||
|
||
**Important non-issue.** `examples/*.ail.json` fixtures on
|
||
disk are hand-formatted JSON with non-canonical key order;
|
||
the parser produces canonical (lex-sorted) JSON. Diff at
|
||
the file-byte level is not zero. Diff at the canonical-byte
|
||
level is zero — which is the only thing that matters per
|
||
Decision 1, since hashing uses canonical form. The
|
||
round-trip test gates on canonical bytes, not file bytes.
|
||
This is correct behaviour; flagged here so a future reader
|
||
who runs `diff` doesn't think the surface is broken.
|
||
|
||
**Files created:**
|
||
|
||
- `crates/ailang-surface/Cargo.toml`
|
||
- `crates/ailang-surface/src/lib.rs` (~40 LOC, rustdoc heavy)
|
||
- `crates/ailang-surface/src/lex.rs` (~264 LOC)
|
||
- `crates/ailang-surface/src/parse.rs` (~1041 LOC, one fn
|
||
per production)
|
||
- `crates/ailang-surface/src/print.rs` (~371 LOC)
|
||
- `crates/ailang-surface/tests/round_trip.rs` (~128 LOC)
|
||
|
||
**Files modified:**
|
||
|
||
- `Cargo.toml` (workspace) — `ailang-surface` member +
|
||
workspace dep.
|
||
- `Cargo.lock` — refresh.
|
||
- `crates/ail/Cargo.toml` — `ailang-surface` dep.
|
||
- `crates/ail/src/main.rs` — new `Parse { path, output }`
|
||
subcommand (~36 LOC).
|
||
- `docs/DESIGN.md` — form refinements appendix to
|
||
Decision 6.
|
||
- `examples/list_map_poly.ailx` — implementer added doc
|
||
strings to match the JSON original (was a 14b design
|
||
exhibit, not byte-aligned).
|
||
|
||
**Known debt:** none reported. `ailang-check` /
|
||
`ailang-codegen` untouched per the architectural pin.
|
||
|
||
**Plan 14d (next, queued).** With form (A) now ergonomic
|
||
and round-trip-verified, the std-lib path opens up. First
|
||
target: `examples/std/std_list.ailx` with `length`,
|
||
`filter`, `fold` (left and right), `concat`, `reverse`,
|
||
`head`, `tail`. Each combinator a fresh test vector for
|
||
the parameterised-ADT pipeline (which 14a opened end-to-
|
||
end). Authored in form (A); the resulting `.ail.json` is
|
||
what tests consume. If 14d surfaces more codegen bugs in
|
||
the parameterised-ADT path (it likely will — 14a found
|
||
one already), debugger handles them inline.
|
||
|
||
The 14b/c form-A hypothesis has held under the empirical
|
||
test of round-tripping every existing fixture. The
|
||
documented rollback to form (C) is now off the table for
|
||
this iter cycle, though the architectural pin keeps it
|
||
open for future replacement of `ailang-surface` should
|
||
the form prove inadequate at stdlib scale.
|
||
|
||
## Language-completion sequence (14d → 14f, then stdlib in 15a)
|
||
|
||
User redirected at the 14d boundary: write the language to
|
||
"finished" before starting on a stdlib. Reasoning:
|
||
authoring a stdlib in an unfinished language wastes work —
|
||
each gap discovered later forces a rewrite of code already
|
||
written. The user also confirmed: **no schema version bump
|
||
needed**. AILang has exactly one consumer (me), so version
|
||
ceremony for compatibility management is pure overhead.
|
||
Edit AST and fixtures in place; pin new hashes where the
|
||
hash regression test demands it.
|
||
|
||
Updated planning sequence:
|
||
|
||
- **14d** — remove `Term::If` redundancy. Pure subtraction.
|
||
- **14e** — explicit tail-call annotation (`tail` flag on
|
||
`Term::App`/`Term::Do`, `musttail` in codegen, tail-position
|
||
verifier in checker).
|
||
- **14f** — memory management. Currently every ADT allocation
|
||
leaks. Likely Boehm conservative GC (`GC_malloc` + `-lgc`)
|
||
for minimum surface change; design pass first.
|
||
- **15a** — first stdlib module (`std_list`).
|
||
|
||
Deferred (not stdlib-blocking; can land later without
|
||
rewriting code that already exists): records/tuples (use
|
||
ADT pairs), nested patterns (use pyramid `match`), local
|
||
recursive `let` (hoist to top level).
|
||
|
||
## Iter 14d — `Term::If` removed
|
||
|
||
Subtraction iter. `Term::If { cond, then, else_ }` was
|
||
semantically a subset of `Term::Match` on `Bool`. CLAUDE.md
|
||
forbids redundancies; two AST nodes for the same operation
|
||
was an authoring decision with no semantic content and a
|
||
duplicate codegen path.
|
||
|
||
The migration shape was the canonical one named in the
|
||
brief:
|
||
|
||
```
|
||
(if c a b) → (match c (case (lit-bool true) a) (case _ b))
|
||
```
|
||
|
||
Wildcard arm satisfies the existing `primitive-needs-wildcard`
|
||
rule. A future iter may upgrade exhaustiveness to recognise
|
||
`true`+`false` as covering Bool without a wildcard, but the
|
||
wildcard form works with the current checker and that was
|
||
enough for 14d.
|
||
|
||
**Implementer dispatch went clean with one documented
|
||
deviation.** Removing the `Term::If` codegen path was not
|
||
sufficient on its own: the existing match codegen rejects
|
||
`i1` (Bool) scrutinees and `Pattern::Lit` patterns — both
|
||
of which the migration shape requires. The implementer
|
||
added a tightly-scoped `lower_bool_match` helper (~95 LOC)
|
||
that handles **only** the two-arm Bool migration shape
|
||
(`(lit-bool true) -> A | _ -> B` or its mirror), errors on
|
||
anything else, and emits the same `br i1`/phi IR the old
|
||
`Term::If` path emitted. No generalisation of the
|
||
ADT-match codegen.
|
||
|
||
The deviation was the right call. **Lesson for future
|
||
subtraction iters**: when removing a specialised AST node,
|
||
the codegen for the migration target may need a small
|
||
extension. Pre-emptively scope this in the brief next time.
|
||
|
||
**Diff size**: 13 files, +286/-221 LOC. Net +65 LOC across
|
||
the workspace, but the AST got smaller (one variant gone),
|
||
the form-(A) grammar got smaller (one production gone),
|
||
and the typechecker got smaller (one branch gone). Codegen
|
||
got slightly larger because of the bool-match helper, but
|
||
the alternative was reusing the existing match path and
|
||
generalising it — which would have been a bigger and
|
||
riskier change.
|
||
|
||
**Hash deltas** (intentional, per Decision 7):
|
||
|
||
| def | before | after |
|
||
|---|---|---|
|
||
| `sum.sum` | `db33f57cb329935e` | `7f5fe7f72c63a9fd` |
|
||
| `sort.insert` | `697fcb9f30f8633a` | `07ff6ee7db17565d` |
|
||
| `max3.max` | `65c45d6a45dd0a72` | `2aa1576f3fbf5b3d` |
|
||
| `max3.max3` | `624b14429bf302f5` | `c452ec2e36c0af27` |
|
||
|
||
Untouched defs (e.g. `sum.main`, all `sort.*` except
|
||
`insert`, `sort.IntList`, `sort.print_list`, `max3.main`)
|
||
keep bit-identical hashes. That's the canary that the
|
||
canonical-JSON byte format was not perturbed — only the
|
||
migrated bodies changed identity.
|
||
|
||
**Verification**: 76/76 tests green (unchanged count; no
|
||
new tests in this iter, by design — subtraction). Manual
|
||
smoke: `sum` → `55`, `max3` → `17`, `sort` → ordered list.
|
||
Identical to pre-migration stdout for all three. `cargo
|
||
doc --no-deps` 0 warnings.
|
||
|
||
**Tail-call survey from the implementer (gold finding,
|
||
informs 14e).** While reading the migrated fixtures the
|
||
implementer surveyed tail positions. Result:
|
||
|
||
- `print_list` (in both `sort.ail.json` and
|
||
`list_map_poly.ail.json`): the recursive call is the
|
||
rhs of a `seq` which is the body of a match arm —
|
||
**already in tail position**. TCO would convert these
|
||
to actual loops.
|
||
- `main` chains: the outer call is in tail position;
|
||
inner calls are not.
|
||
- `insert`, `sort`, `map`: the recursive calls are
|
||
**arguments to a `Cons` ctor construction** (e.g.
|
||
`Cons (f h) (map f t)`). NOT in tail position.
|
||
Constructor-blocking is the standard ML/Haskell case
|
||
where TCO does not apply without a CPS transform or an
|
||
accumulator-form rewrite.
|
||
|
||
**Implications for 14e.** Adding a `tail` flag to
|
||
`Term::App`/`Do` will work for `print_list`-style
|
||
recursions and for terminal call chains, but **will not**
|
||
help the `map`/`sort`/`insert`-style ctor-blocked
|
||
recursions. Those need either an accumulator-form rewrite
|
||
in the source program (the standard ML/Haskell move) or a
|
||
CPS transform (much more intrusive). 14e ships only the
|
||
annotation + verification; accumulator forms become an
|
||
authoring pattern in the stdlib, not a compiler feature.
|
||
|
||
This sharpens what 14e can promise: tail-call **wins**
|
||
will be visible in `print_list`-style terminal-recursion
|
||
patterns; `map`/`sort` style stays stack-bounded by depth,
|
||
which makes 14f (GC) the more important iter for handling
|
||
long lists than 14e by itself.
|
||
|
||
## Iter 14e — explicit, verified tail calls
|
||
|
||
Decision 8 ships. `Term::App` and `Term::Do` carry a `tail: bool`
|
||
flag (serde-default false, skip-when-false in serialisation).
|
||
A new `verify_tail_positions` typecheck pass walks each fn body
|
||
and Lam body with an `is_tail_context: bool` threaded down per
|
||
the standard Scheme rules, rejecting any `tail: true` call that
|
||
sits outside tail position. Codegen emits `musttail call` for
|
||
marked App calls.
|
||
|
||
**Hash deltas (intentional, only the migrated calls):**
|
||
|
||
| def | hash before | hash after |
|
||
|---|---|---|
|
||
| `list_map_poly.print_list` | unchanged 14c value | new |
|
||
| `sort.print_list` | unchanged 14d value | new |
|
||
|
||
All other defs across all 18 fixtures kept bit-identical hashes.
|
||
This is the canary for the `skip_serializing_if = "is_false"`
|
||
serde rule: an unmarked `App`/`Do` serialises identically to its
|
||
pre-14e form, so untouched defs cannot drift.
|
||
|
||
**Tests: 79/79 (was 76, +3).** New tests:
|
||
- `tail_call_in_non_tail_position_is_rejected` (check unit) —
|
||
asserts the diagnostic fires on a deliberately-misplaced
|
||
`tail: true` call (e.g. as a `Cons` arg).
|
||
- `tail_call_in_tail_position_is_accepted` (check unit) —
|
||
asserts the verifier accepts the canonical `print_list` shape.
|
||
- `iter14e_print_list_recursion_emits_musttail` (e2e IR-grep) —
|
||
builds `list_map_poly`, dumps IR, asserts the recursive call
|
||
site uses `musttail call`. This is the only direct evidence
|
||
that the AST flag actually reaches LLVM.
|
||
|
||
**IR-snapshot evidence** at the recursive site of
|
||
`print_list` after migration:
|
||
|
||
```
|
||
%v7 = musttail call i8 @ail_list_map_poly_print_list(ptr %v6)
|
||
ret i8 %v7
|
||
```
|
||
|
||
`musttail` followed immediately by `ret` of the same SSA value —
|
||
LLVM's terminator rule satisfied. Smoke: `list_map_poly` →
|
||
`2/3/4`, `insertion_sort_orders_list` → identical sorted list.
|
||
Behavior unchanged; only the calling shape did.
|
||
|
||
**Two implementer deviations (called out, both reasonable):**
|
||
|
||
1. **`tail-do` falls back to `tail call`, not `musttail`.** LLVM
|
||
`musttail` requires identical caller/callee return types. AILang
|
||
IO ops dispatch through runtime helpers (`printf`/`puts`)
|
||
returning `i32`, while AILang's `Unit` lowers to `i8`. Cross-type
|
||
`musttail` would be rejected by the verifier. So `Term::Do` with
|
||
`tail: true` lowers to `tail call` (the LLVM optimisation hint,
|
||
not the guarantee), then `ret i8 0`. No fixture currently uses
|
||
`tail-do`, so the path is implemented but not exercised
|
||
end-to-end. Proper fix: change runtime helper signatures to
|
||
return `i8`. Punted; not blocking.
|
||
|
||
2. **`block_terminated` plumbing in codegen.** A `tail-app` /
|
||
`tail-do` emits `musttail call ... ret ...` directly and sets
|
||
`self.block_terminated = true`. Surrounding code (match-arm phi
|
||
construction, fn-body trailing-ret, lambda-thunk trailing-ret)
|
||
checks the flag and skips the fall-through emit. When every
|
||
match arm is a tail call, the join block is omitted entirely.
|
||
This was unavoidable to keep the IR well-formed — adding a
|
||
second `ret` after a `musttail call`+`ret` would be a verifier
|
||
error. The flag is a small piece of state but it's the right
|
||
shape for "the current basic block has been definitively
|
||
terminated by a sub-emission".
|
||
|
||
**Form (A) at constraint ceiling.** Two new productions
|
||
(`tail-app-term`, `tail-do-term`) bring the count to ~30, which
|
||
is exactly the constraint-1 budget. Future productions need to
|
||
either retire something or accept an explicit budget rebalancing
|
||
in DESIGN.md.
|
||
|
||
**GC notes from the implementer (informs 14f).**
|
||
|
||
- **Allocations cluster in `lower_ctor`** (~line 850 of codegen).
|
||
Every `term-ctor` does `malloc(8 + 8 * n)`. In `print_list` we
|
||
allocate nothing per recursion (just match + read fields +
|
||
recurse); allocations come from `map`, from `main`'s
|
||
list-building Cons chain, and from any other user code that
|
||
builds ADTs.
|
||
- **Lambda envs and closure pairs allocate too** (`lower_lambda`).
|
||
Closure pair: `malloc(16)`. Env block: `malloc(env_size)`.
|
||
Direct-application closures (the common case for HOF args)
|
||
could be arena'd cleanly because the closure dies after the
|
||
call returns. Stored or returned closures escape.
|
||
- **Tail recursion does NOT reduce allocation pressure**, only
|
||
stack depth. For `print_list`-style recursions there's no
|
||
allocation to begin with, so the win is purely stack-bounded.
|
||
For `map`-style ctor-blocked recursions, each step allocates
|
||
one new `Cons` box — that's where allocation-side work pays
|
||
off.
|
||
- **The "obviously safe" arena boundary** is a fn whose return
|
||
type contains no boxed ADT (i.e. returns `Int`/`Bool`/`Unit`/
|
||
`Str` only). All ADT boxes allocated inside such a fn cannot
|
||
escape; an arena freed at fn return is sound by construction.
|
||
Most current fixtures violate this — `map`, `sort` return ADTs
|
||
— so a per-fn-arena scheme alone won't carry. Need a heap
|
||
with GC for escaping allocations.
|
||
|
||
This narrows 14f's design space: probably **Boehm conservative
|
||
GC across the board** as a first cut (`GC_malloc` substituted
|
||
for `malloc`, `-lgc` linked, no language change). Add a
|
||
per-fn-arena optimisation later for non-escaping cases if the
|
||
profile justifies it. Boehm is a one-iter shot; arenas would be
|
||
a multi-iter design pass with escape analysis.
|
||
|
||
**Plan 14f.** Boehm-GC integration. Concretely:
|
||
- `GC_malloc` instead of `malloc` in lowered IR.
|
||
- `-lgc` added to the clang link command in `ailang-codegen`'s
|
||
build path (probably in the CLI, since the codegen crate
|
||
emits IR text and clang is invoked downstream).
|
||
- Conservative scan handles AILang's stack and globals out of
|
||
the box.
|
||
- Verify on a stress test: build a list of 100k Cons cells in
|
||
`map`, run, observe RSS doesn't blow up. Boehm collects
|
||
unreachable boxes during allocation pressure.
|
||
- No AST or schema change. No language-level change.
|
||
|
||
After 14f, the language is "done enough" for stdlib (15a).
|
||
Anything else (records as a primitive, nested patterns, local
|
||
recursive let, type classes) can layer on later without forcing
|
||
stdlib rewrites.
|
||
|
||
## Iter 14f — Boehm conservative GC
|
||
|
||
Decision 9 ships. Through Iter 14e every ADT box, lambda env,
|
||
and closure pair was leaked. This iter substitutes
|
||
`GC_malloc` for `malloc` in all four IR allocation sites and
|
||
links `-lgc`. No language change, no AST change, no schema
|
||
change.
|
||
|
||
**Diff: 5 files, ~30 LOC net.**
|
||
|
||
- `crates/ailang-codegen/src/lib.rs`: 4× `@malloc` → `@GC_malloc`
|
||
(declare line + 3 call sites: `lower_ctor`, `lower_lambda`'s
|
||
env, `lower_lambda`'s closure pair).
|
||
- `crates/ail/src/main.rs`: `.arg("-lgc")` added to the clang
|
||
invocation in `build_to`.
|
||
- `crates/ail/tests/snapshots/{hello,sum,list,max3,ws_main}.ll`:
|
||
mechanical s/@malloc/@GC_malloc/, 9 occurrences across 5
|
||
files. The IR is bit-identical to pre-14f modulo this
|
||
substitution — exactly Decision 9's promise.
|
||
- `crates/ail/tests/e2e.rs`: new test
|
||
`gc_handles_recursive_list_construction` (+19 LOC).
|
||
- `examples/gc_stress.{ailx,ail.json}`: new fixture.
|
||
|
||
**Hash invariance verified.** Every existing fixture's def
|
||
hashes are unchanged. The codegen and link line are downstream
|
||
of canonical bytes; the AST schema didn't move; nothing on
|
||
disk in `examples/*.ail.json` was touched. The new
|
||
`gc_stress` module adds 4 new hashes, all unrelated.
|
||
|
||
**Tests: 80/80 (was 79).** Existing 79 produce byte-identical
|
||
stdout — only the allocator changed, semantics unchanged. New:
|
||
`gc_handles_recursive_list_construction` builds a List<Int> of
|
||
length 50 via recursive `Cons`, sums it (`1275`). Manual smoke:
|
||
|
||
- `gc_stress.ail.json` → `1275`.
|
||
- `list_map_poly` → `2 3 4` (unchanged).
|
||
- `sort` → sorted list (unchanged).
|
||
|
||
`cargo doc --no-deps`: 0 warnings (DESIGN.md item 6 invariant
|
||
preserved through nine iters of feature work).
|
||
|
||
**Pattern shape used in `gc_stress`** (caught a small typechecker
|
||
constraint). The first proposed shape `(case (lit-int 0) Nil)`
|
||
doesn't parse — `pat-lit` takes the bare literal token, not a
|
||
keyword-prefixed form, and `case` requires a pattern. Worked
|
||
shape: comparison-and-bool-match, mirroring `sort.ail.json`'s
|
||
`<=` arm:
|
||
|
||
```
|
||
(match (app == n 0)
|
||
(case (pat-lit true) (term-ctor List Nil))
|
||
(case (pat-wild) (term-ctor List Cons n (app build (app - n 1)))))
|
||
```
|
||
|
||
This is the canonical "if-then-else" pattern post-14d. Worth
|
||
flagging for the stdlib brief: predicates that need to branch
|
||
go through the `==` / `<` / `<=` builtin returning Bool, then
|
||
match on that Bool with a wildcard fallback. Three lines for
|
||
what `if` used to do in one — but uniform with the rest of the
|
||
language, no special case.
|
||
|
||
**GC integration notes.**
|
||
- `GC_INIT()` is **not needed** on this build host (Arch
|
||
with `libgc 1.5.6`). libgc auto-inits via
|
||
`__attribute__((constructor))`.
|
||
- No conservative-scan over-retention symptom observed: every
|
||
existing test's stdout byte-identical; behaviour preserved.
|
||
- `-lgc` alone is sufficient for the link; pthread/dl come in
|
||
transitively from libgc.so's NEEDED entries.
|
||
|
||
**Language is feature-complete enough for stdlib.** Iters 14d
|
||
(redundancy removal), 14e (explicit tail calls), 14f (GC) are
|
||
the three blockers identified at the 14b boundary. They are
|
||
all done. Anything else (records as primitive, nested patterns,
|
||
local rec let, type classes) layers on later without forcing
|
||
stdlib rewrite.
|
||
|
||
**Plan 15a.** First stdlib module: `examples/std/std_list.ailx`.
|
||
Combinators: `length`, `append`, `reverse`, `map`, `filter`,
|
||
`fold_left`, `fold_right`, `head`, `tail`, `is_empty`. Each
|
||
combinator a fresh test vector for the parameterised-ADT +
|
||
GC + tail-call combination. Authored in form (A) from day one;
|
||
`.ail.json` produced via `ail parse`. Each combinator gets a
|
||
dedicated e2e test.
|
||
|
||
If 15a surfaces compiler bugs (likely — every prior dogfood
|
||
iter has, see 14a's monomorphisation bug), debugger handles
|
||
them inline. If a compiler limitation surfaces that genuinely
|
||
blocks the stdlib (e.g. nested patterns turn out to be needed),
|
||
that becomes its own iter before 15a continues.
|
||
|
||
The architectural pin from Decision 6 governs: stdlib lives
|
||
under `examples/std/` as `.ailx` source; tests load the
|
||
generated `.ail.json`. `ailang-check` and `ailang-codegen`
|
||
remain projection-agnostic.
|
||
|
||
## Iter 14g — `Term::If` restored (revert of 14d)
|
||
|
||
Reconsidered 14d's removal of `Term::If`. The decision was wrong;
|
||
restored.
|
||
|
||
**Why 14d was wrong.** "No redundancies" from CLAUDE.md is a real
|
||
rule but it requires judgment to apply. `Term::If` reduces to
|
||
`Term::Match` on Bool, but reducibility is not redundancy in a
|
||
strong sense — `1 + 1` reduces to `2`, you don't remove `+` from
|
||
the language because of it. `Term::If` is a primitive control-
|
||
flow shape that every programming language has for good reason:
|
||
bool branching is the second most common control flow shape after
|
||
sequencing.
|
||
|
||
**Quantitative.** `(if c a b)` is 4 tokens. The post-14d
|
||
replacement `(match c (case (pat-lit true) a) (case (pat-wild) b))`
|
||
is 12. That's a 3× token-economy hit on every Bool branch — in
|
||
exactly the language whose authoring constraint was supposed to
|
||
be token-efficient. The match-on-Bool form is also asymmetric
|
||
(false case via `pat-wild` because the typechecker rejects
|
||
`pat-lit false` as exhaustive) and structurally lopsided.
|
||
|
||
**Meta-pattern that produced the wrong call.** I had been treating
|
||
user observations as directives. The user said "if is a subset of
|
||
match" — which is a factual observation; I jumped to remove it,
|
||
citing CLAUDE.md as cover. There was no independent conviction
|
||
behind the change, only doctrinal hooking-up of a user remark.
|
||
The leak showed up in 14f's JOURNAL prose ("three lines for what
|
||
`if` used to do in one"), which the user correctly read as me
|
||
regretting the decision.
|
||
|
||
Two feedback memories saved to head this off in future iters
|
||
(`/home/brummel/.claude/projects/-home-brummel-dev-ailang/memory/`):
|
||
- `feedback_user_suggestions_not_directives.md` — observations
|
||
are input, not output. Form an opinion before acting.
|
||
- `feedback_no_nostalgia_for_removed_features.md` — describe
|
||
canonical form on its own merits, not as compensation for
|
||
what was deleted.
|
||
|
||
**Implementation (revert).** Mechanically reverse-applied 14d's
|
||
diff at every site (`ast.rs`, `check/lib.rs` (incl. the new
|
||
14e `verify_tail_positions` arm), `codegen/lib.rs` (4 sites),
|
||
`surface/{parse,print}.rs`, `core/pretty.rs`, `ail/main.rs`,
|
||
`e2e.rs` test mutation). Removed the `lower_bool_match` helper
|
||
that 14d had introduced — it existed only because the 14d
|
||
migration shape needed codegen for non-`ptr` match scrutinees;
|
||
with `Term::If` back, match-on-Bool returns to its pre-14d
|
||
unsupported state and the helper is dead weight. Three fixtures
|
||
(`sum`, `sort`, `max3`) restored to their pre-14d shape.
|
||
|
||
**One additional fixture migration.** `gc_stress` was authored
|
||
in 14f using the 14d match-on-Bool migration shape (because
|
||
14f sat between 14d and this revert). After removing
|
||
`lower_bool_match` it would have failed to compile. Migrated
|
||
`gc_stress.{ail.json,ailx}` to use `(if ...)` directly. Output
|
||
unchanged: `1275`.
|
||
|
||
**14e and 14f are intact.** Verified by spot-emit of
|
||
`list_map_poly`'s IR: `musttail call i8 @ail_list_map_poly_print_list`
|
||
and `call ptr @GC_malloc(i64 8)` both present. The revert is
|
||
strictly local to `Term::If`-related code paths.
|
||
|
||
**Hash check.** All four pre-14d hashes returned:
|
||
|
||
| def | restored hash |
|
||
|---|---|
|
||
| `sum.sum` | `db33f57cb329935e` |
|
||
| `sort.insert` | `697fcb9f30f8633a` |
|
||
| `max3.max` | `65c45d6a45dd0a72` |
|
||
| `max3.max3` | `624b14429bf302f5` |
|
||
|
||
Untouched defs across all 18 fixtures (incl. the 14e print_list
|
||
hash deltas) keep their post-14f hashes. The revert's hash
|
||
movement is exactly the four 14d-migrated defs reverting plus
|
||
the one accidental 14f-victim (`gc_stress` defs, never previously
|
||
shipped under any other hash).
|
||
|
||
**DESIGN.md.** Decision 7 is preserved with a `Status: REVERTED`
|
||
header. Audit trail matters; future reads should see the
|
||
decision and its reversal both. Form-(A) productions in
|
||
Decision 6's appendix have `if-term` restored.
|
||
|
||
**Tests: 80/80 green.** Identical stdout for every existing
|
||
fixture. `cargo doc --no-deps` 0 warnings.
|
||
|
||
**LOC delta.** +265/-295 net −30. Net cleanup: the `lower_bool_match`
|
||
helper was bigger than the restored `Term::If` codegen.
|
||
|
||
**Plan.** Back to 15a — first stdlib module `std_maybe`. The
|
||
brief I had drafted included an "authoring note: post-14d
|
||
if-then-else" section that's now obsolete. Re-issue without
|
||
that, using `if` naturally where appropriate.
|
||
|
||
|
||
|
||
|