Brummel
706f90bacd
Strictly additive new crate per Decision 6 architectural pin. JSON-AST stays the source of truth; ailang-surface is one producer/consumer of ailang_core::ast::Module values. ailang-check and ailang-codegen unchanged. Crate contents: - src/lex.rs (~264 LOC): 3-rule lexical core. Whitespace and parens delimit tokens; semicolon to EOL is comment; first-character classifier (digit -> int, " -> string, else -> ident). - src/parse.rs (~1041 LOC): hand-written recursive descent. One Rust fn per EBNF production. No parser-combinator dep. - src/print.rs (~371 LOC): deterministic pretty-printer. Round-trip contract with parse() is the surface's correctness gate. - tests/round_trip.rs (~128 LOC): integration test runs every examples/*.ail.json fixture through print -> parse -> canonical JSON, asserts canonical-byte equality with the original. Two AST-driven form widenings beyond the 14b sketch (both folded into DESIGN.md Decision 6): - lam-term carries (typed name type) params, ret type, and optional effects (Term::Lam has parallel param_tys/ret_ty/ effects fields). - import-clause admits (import name (as alias)?) (Import.alias is Option<String>). Production count ~28, under 30-rule budget. Constraint 1 (formalisable for foreign LLM) intact. Verification: - cargo build --workspace green. - cargo test --workspace: 76 tests green (was 64; +9 surface unit, +2 round-trip integration). All 17 fixtures round-trip byte-identical at canonical level. 3 hand-written .ailx exhibits parse to canonical JSON identical to their .ail.json siblings. - cargo doc --no-deps zero warnings (DESIGN.md item 6 invariant). Manual smoke test (ail parse → ail run): hello, box, list_map_poly all produce expected output through the form-(A) authoring lane end-to-end. CLI: ail parse <file.ailx> [-o <file.ail.json>]. .ail.json remains a first-class input to every existing subcommand. Plan 14d: stdlib (std_list.ailx with length/filter/fold/concat/ reverse/head/tail), authored in form (A) from day one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1749 lines
80 KiB
Markdown
1749 lines
80 KiB
Markdown
# JOURNAL
|
||
|
||
Chronological notes for myself. Not every change; only decisions, obstacles,
|
||
and observations that future iterations will need.
|
||
|
||
## 2026-05-07 — Day 0
|
||
|
||
- Repo initialised. Assignment in `CLAUDE.md`: LLM-native language, LLVM backend.
|
||
- Design decisions captured in `docs/DESIGN.md`.
|
||
- Toolchain: `rustc 1.94`, `llvm-config 22.1.3`, `clang` available.
|
||
- Decided against `inkwell` in favour of LLVM IR text emit. Rationale in DESIGN.md.
|
||
- Workspace layout:
|
||
- `crates/ailang-core` — AST, type, hash, JSON schema
|
||
- `crates/ailang-check` — typechecker (comes later)
|
||
- `crates/ailang-codegen` — lowering + LLVM IR emit
|
||
- `crates/ail` — CLI
|
||
- MVP goal: `examples/sum.ail.json` → binary that prints 55. **Achieved.**
|
||
|
||
## 2026-05-07 — architecture review after the MVP
|
||
|
||
Still on track? Broadly yes. Concrete observations:
|
||
|
||
**What holds:**
|
||
|
||
- JSON AST + canonical form + content hash are all lego bricks that later
|
||
tools can build on without a refactor (`ail deps`, `ail diff`).
|
||
- The LLVM IR text pipeline works as planned. No libllvm version pain.
|
||
- The effect set is wired into the type system from the start. Extensible to
|
||
row-poly without touching the core.
|
||
|
||
**Debt that accrues interest:**
|
||
|
||
1. **`current_block_label_for_phi` is a heuristic** (see codegen). On nested
|
||
`if` terms it will return the wrong block label, because it scans the body
|
||
backwards. Ticking, because no test cases trigger it yet. Must be fixed
|
||
next, before new language features arrive.
|
||
2. **No typed AST.** Codegen reads the source AST directly and relies on
|
||
the typechecker having run before. Fine for the MVP; once ADTs or
|
||
closures arrive, I will need a separate typed IR stage (TIR).
|
||
3. **The `hash` field is not in the AST.** Right now we hash the def object
|
||
directly. Once I serialise hashes as fields (caching), the hash will
|
||
need to exclude that field before computation.
|
||
|
||
**Plan iteration 2 (now):**
|
||
|
||
1. Clean up block-label tracking, with a nested-if test.
|
||
2. Strings as a literal + `io/print_str`.
|
||
3. Hello-world example as a second E2E test.
|
||
4. CLI: `--json` output for machine consumers wherever it fits.
|
||
|
||
**Plan iteration 3:**
|
||
|
||
ADTs + pattern matching. That is the next big jump. Requires a typed IR
|
||
stage (TIR), because pattern matching lowers into decision trees, which
|
||
have a different shape from the AST.
|
||
|
||
## 2026-05-07 — iteration 2 done
|
||
|
||
- Block-label tracking is now robust (nested `if`s work). Test
|
||
`max3_picks_largest` protects it.
|
||
- Strings as `Lit::Str { value }`, type `Str` -> LLVM `ptr`, with
|
||
`io/print_str` effect op. `examples/hello.ail.json` prints a string.
|
||
- CLI: `manifest --json`, `builtins --json` for tool consumers.
|
||
- `ail deps [--of NAME] [--json]` lists call edges. Effect ops are tagged
|
||
`effect:NAME` so a consumer can filter them.
|
||
|
||
**Architecture check:** no structural deviations. Codegen still reads the
|
||
source AST directly (a TIR stage will become necessary with ADTs in
|
||
iteration 3).
|
||
|
||
## 2026-05-07 — iteration 3 done: ADTs
|
||
|
||
- TypeDef in the AST with ctors. A ctor has `name` and `fields: [Type...]`.
|
||
- Term::Ctor (construction) and Term::Match (pattern matching).
|
||
- Patterns: `Wild`, `Var`, `Lit`, `Ctor { ctor, fields }`. In the MVP,
|
||
nested ctor patterns are NOT allowed — sub-patterns must be `Var` or
|
||
`Wild`.
|
||
- Typechecker with a type registry and `ctor_index` (ctor name → ADT). In
|
||
Match, exhaustiveness is checked against the full constructor set. A
|
||
negative test protects this.
|
||
- Codegen: boxed heap layout. Per ctor application, `malloc(8 + 8*n)`
|
||
bytes; tag at offset 0, fields from offset 8 (8-byte slots, native typed
|
||
load/store). Match: load tag + switch + arm blocks + phi at the join.
|
||
- `examples/list.ail.json` (Cons/Nil list, sum_list via match) returns 42.
|
||
|
||
**Surprisingly painless.** The architecture decisions from day 0 paid off:
|
||
opaque ptr in LLVM 22 makes the boxed layout almost glue-free; effect
|
||
tracking was untouched by ADTs; the JSON AST takes new node types
|
||
cleanly.
|
||
|
||
**Debt accrued:**
|
||
|
||
1. **Codegen still reads the source AST directly.** The temptation to push
|
||
on without TIR was strong — and worked, because my Match restrictions
|
||
are flat (no nested patterns). Once nested patterns arrive, decision-
|
||
tree lowering will not stay clean without TIR. Debt acknowledged; not
|
||
due now.
|
||
2. **No GC.** The heap leaks. Acceptable for demo programs; must be
|
||
addressed before any longer-running program. Options for Phase 4:
|
||
refcount, Boehm-GC linkage, region inference.
|
||
3. **No runtime pretty-printer for ADT values.** `io/print_int` is enough
|
||
for demos, but a generic `show :: a -> Str` for ADTs would be valuable.
|
||
Requires dispatch over the tag — feasible, but not now.
|
||
|
||
**Plan iteration 4:**
|
||
|
||
The next steps are less obvious. Three candidates in priority order:
|
||
|
||
1. **Module system (imports).** Right now everything is in a single module.
|
||
With multiple modules + cross-module hashing the language only becomes
|
||
practical for several defs.
|
||
2. **Structured error output (`ail check --json`).** So tools can react to
|
||
type errors without parsing text.
|
||
3. **Closures / higher-order functions.** Requires closure conversion and
|
||
is a bigger step.
|
||
|
||
Iteration 4 will be (1) + (2) — both strengthen the LLM tooling and have
|
||
moderate risk.
|
||
|
||
## 2026-05-07 — workflow change: orchestrator + agent repo
|
||
|
||
At the user's suggestion, switching to **orchestrator mode**: I delegate
|
||
clearly bounded implementation chunks to sub-agents and keep only
|
||
architecture decisions, reviews, and commit discipline. Four specialised
|
||
agents drafted: implementer, architect, tester, debugger.
|
||
|
||
**Important correction:** the user required the agents not to be hidden in
|
||
`.claude/agents/`, but versioned as a visible part of the project under
|
||
`agents/`. DESIGN.md gained a new section "Project ecosystem", which
|
||
records this: AILang is not just a language, but language core + CLI +
|
||
examples + agents + docs + tests, all of equal weight.
|
||
|
||
Invocation scheme: the system-prompt body from `agents/<name>.md` as a
|
||
prefix before the concrete task + sent to the `general-purpose` agent.
|
||
Functionally identical to subagent loading from `.claude/agents/`, but
|
||
visible in the repo.
|
||
|
||
**Plan iteration 4 (revised):**
|
||
|
||
The module system is more involved than expected (cross-module hashing,
|
||
import resolution). First the smaller tooling wins, then the module system
|
||
as iteration 5:
|
||
|
||
1. **Structured error output** (`ail check --json` with a Diagnostic
|
||
struct, stable codes like `unbound-var`, `type-mismatch`).
|
||
2. **`ail diff <a> <b>`** — semantic module diff via per-def hash
|
||
comparison.
|
||
3. **IR snapshot tests** — regression protection for the codegen pipeline.
|
||
|
||
## 2026-05-07 — iteration 4 done: LLM tooling consolidation
|
||
|
||
Three sub-commits, each produced by an `ailang-implementer` invocation
|
||
and spot-checked by the orchestrator:
|
||
|
||
- `93fe723` Iter 4a: `ail check --json` with a `Diagnostic` struct
|
||
(`severity`, `code`, `message`, `def`, `ctx`). Stable codes:
|
||
`unbound-var`, `type-mismatch`, `arity-mismatch`,
|
||
`non-exhaustive-match`, `unknown-ctor`, `unknown-ctor-in-pattern`,
|
||
`nested-ctor-pattern-not-allowed`, `duplicate-def`,
|
||
`unknown-effect-op`, `unknown-type`, `schema-mismatch`. API:
|
||
`check_module(&Module) -> Vec<Diagnostic>`.
|
||
- `c652b12` Iter 4b: `ail diff <a> <b> [--json]` as a structural top-level
|
||
def diff via BLAKE3 hash. Four categories (added/removed/changed/
|
||
unchanged), sorted alphabetically, exit code 1 on diff.
|
||
- `74a2005` Iter 4c: IR snapshot tests in
|
||
`crates/ail/tests/snapshots/{sum,max3,hello,list}.ll`.
|
||
Normalisation of `target triple`. Update via `UPDATE_SNAPSHOTS=1
|
||
cargo test ir_snapshot_`. Mismatch produces an `.actual` file.
|
||
|
||
Test count: 28 (previously 19). 7 E2E + 4 IR snapshot + 9 ailang-check + 1
|
||
ailang-codegen + 7 ailang-core.
|
||
|
||
**Closed from the debt register:**
|
||
|
||
- Block tracking in codegen has not been a heuristic risk since Iter 2;
|
||
the explicit `current_block: String` track is now additionally
|
||
protected against regression by Iter 4c snapshot tests. Debt closed.
|
||
|
||
**New / sharpened debt:**
|
||
|
||
1. `check_module` is **single-shot** — the first error aborts, no
|
||
multi-diagnostic gathering. The spec was that way, but the format
|
||
suggests Vec semantics. A real multi-diagnostic refactor will be
|
||
cheaper once TIR exists (a central error accumulator via a separate
|
||
stage). Not due now.
|
||
2. `source_filename` in the IR is hard-coded to `"<module>.ail"`. As long
|
||
as there is only one top-level module, that is platform-stable. With
|
||
Iter 5 (module system + imports) the path becomes relevant — keep it
|
||
path-independent at construction time, otherwise the snapshots will
|
||
tip over.
|
||
|
||
**Plan iteration 5:** module system with imports. Cross-module hashing,
|
||
import resolution, multiple `.ail.json` files in one build. The multi-
|
||
diagnostic refactor only after that.
|
||
|
||
Sub-steps:
|
||
|
||
- **5a — workspace loader.** `ailang_core::Workspace { modules:
|
||
BTreeMap<String, Module> }` plus `load_workspace(entry: &Path)`, which
|
||
follows `imports` recursively from the entry module. Convention:
|
||
`import { module: "foo" }` resolves to `<dir>/foo.ail.json` next to the
|
||
entry. Cycle detection. CLI: existing subcommands keep working on a
|
||
single module; a new `ail workspace <entry>` lists all reachable
|
||
modules with hash. Tests: two small example modules with an import
|
||
relation; cycle test.
|
||
- **5b — cross-module typecheck.** The typechecker takes `&Workspace`
|
||
instead of `&Module`. Imports are mounted in the env as a namespace
|
||
(`alias.def` or, with no alias, `module.def`). New diagnostic codes:
|
||
`unknown-module`, `unknown-import`, `import-cycle`, `ambiguous-name`.
|
||
Tests per code.
|
||
- **5c — cross-module codegen.** The emitter produces IR for all modules
|
||
in the workspace, prefix-mangled with `@ail_<module>_<def>`. E2E test:
|
||
a program that uses a function from module B in module A returns the
|
||
correct result in the binary.
|
||
- **5d — tooling adjustments.** `manifest`, `describe`, `deps`, `diff`
|
||
gain a `--workspace` mode (recursive). The single mode stays the
|
||
default for backwards compatibility.
|
||
|
||
During Iter 5, at construction time **keep `source_filename`
|
||
path-independent** (module name only, no directory prefix), otherwise
|
||
the IR snapshots will tip over.
|
||
|
||
## 2026-05-07 — Iter 5b done: cross-module typecheck
|
||
|
||
- `check_workspace(&Workspace) -> Vec<Diagnostic>` as the top-level API.
|
||
`check_module` is preserved and internally lifts the module into a
|
||
trivial workspace.
|
||
- Convention for qualified references (recorded in DESIGN.md):
|
||
`Term::Var { name }` with exactly one dot = `<prefix>.<def>`. Prefix is
|
||
an import alias or module name. No new AST node, no renamed fields ⇒
|
||
hashes stay stable; all `ir_snapshot_*` still green.
|
||
- Three new diagnostic codes: `unknown-module`, `unknown-import`,
|
||
`invalid-def-name` (with `ctx.reason: "contains-dot"`).
|
||
- CLI: `ail check <entry>` now **always** loads via `load_workspace`.
|
||
Workspace load failures become structured diagnostics in JSON mode with
|
||
codes `module-not-found`, `module-cycle`, `module-name-mismatch`,
|
||
`module-hash-mismatch`, `schema-mismatch`. `ail build` and
|
||
`ail emit-ir` stay per single module (cross-module codegen is 5c).
|
||
- Examples: `ws_main.ail.json` now calls `ws_lib.add` (observable). New:
|
||
`ws_broken.ail.json` (`unknown-import`),
|
||
`ws_unknown_module.ail.json` (`unknown-module`).
|
||
- Tests: 37 green (previously 32). 4 new workspace integration tests in
|
||
`crates/ailang-check/tests/workspace.rs`, one new e2e test
|
||
`check_workspace_resolves_import`.
|
||
- Debt: single-shot diagnostics still in place (multi-diagnostic after
|
||
5c). The dot convention covers exactly one dot — nested module paths
|
||
(`a.b.c`) do not exist; that would only be a topic with hierarchical
|
||
modules and currently falls through as `unbound-var`.
|
||
|
||
## 2026-05-07 — Iter 5c done: cross-module codegen
|
||
|
||
- **Mangling break (deliberate).** All AILang functions are now called
|
||
`@ail_<module>_<def>`, even in single-module programs. The old form
|
||
`@ail_<def>` is gone. Strings/const globals analogously
|
||
(`@.str_<module>_<hint>_<idx>`, `@ail_<module>_<const>`). The entry
|
||
point stays `main` as C ABI: a `define i32 @main()` trampoline calls
|
||
`@ail_<entry-module>_main()`. If the entry module has no
|
||
`main : () -> Unit !IO`, the build fails with `MissingEntryMain`.
|
||
- **Workspace lowering.** New top-level API
|
||
`ailang_codegen::lower_workspace(ws: &Workspace) -> Result<String>`
|
||
produces a single `.ll` for the whole workspace. Modules in
|
||
alphabetical order (BTreeMap order); defs in AST order. Cross-module
|
||
calls are resolved in codegen via the import map of the calling module
|
||
— same logic as in the typechecker, locally duplicated with a
|
||
cross-reference (no shared helper module, because the type worlds
|
||
differ: the typechecker handles `Type`, codegen handles `FnSig` from
|
||
llvm types).
|
||
- **CLI.** `ail build` and `ail emit-ir` now always load the workspace
|
||
and check/lower it fully. Single-module programs keep working
|
||
(trivial workspace with one module). `emit_ir(m)` stays in the codegen
|
||
crate as a convenience API and internally wraps into a trivial
|
||
workspace.
|
||
- **Snapshots regenerated.** `sum.ll`, `max3.ll`, `hello.ll`, `list.ll`
|
||
show the new mangling. New `ws_main.ll` snapshot documents the
|
||
cross-module build: `@ail_ws_main_main` calls `@ail_ws_lib_add`.
|
||
- **Tests.** 40 green (previously 37). New: `workspace_build_runs_imported_fn`
|
||
(e2e: prints 5), `ir_snapshot_ws_main`, `missing_entry_main_is_error`
|
||
(codegen unit). Existing behaviour tests
|
||
(`sum_1_to_10_is_55`, `max3_picks_largest`, `hello_world_str_lit`,
|
||
`list_sum_via_match`) stay green — behaviour unchanged, only the
|
||
mangling is new.
|
||
|
||
**Debt closed:**
|
||
|
||
- **#19 (`source_filename` hardening).** In the workspace world,
|
||
`source_filename` is now uniformly `<entry-module>.ail`, once per
|
||
workspace. The previous hard-coded path dot is gone with it.
|
||
|
||
**State:** the module system is closed end to end — loader + typecheck +
|
||
codegen + build see the workspace as a coherent unit. The multi-
|
||
diagnostic refactor and possibly cross-module ADTs remain for later.
|
||
|
||
## 2026-05-07 — Iter 5d done: tooling extended to the workspace
|
||
|
||
- `ail manifest|describe|deps|diff <entry> --workspace` now operate
|
||
across all modules of the workspace. The default without the flag stays
|
||
single-module for backwards compatibility. Manifest sorts by
|
||
`(module, name)`, describe accepts dotted notation `ws_lib.add`, deps
|
||
emits `{from_module, from_def, to_module, to_def}` edges, diff compares
|
||
workspace-wide with added/removed/changed/unchanged_modules and a
|
||
nested sub-diff per changed_module.
|
||
- Refactor: `diff_def_lists` is the single source of the four-category
|
||
logic; single and workspace diff share it.
|
||
- Tests: 44 green (previously 40). New: `manifest_workspace_lists_all_defs`,
|
||
`describe_workspace_resolves_qualified_name`,
|
||
`deps_workspace_includes_cross_module`, `diff_workspace_added_module`.
|
||
|
||
**Observation (debt):** `deps` does not filter builtins/locals/function
|
||
parameters. In workspace mode that becomes more visible than in single
|
||
mode — `ws_lib.add` lists edges to `ws_lib.+` (builtin) and
|
||
`ws_lib.a`/`ws_lib.b` (function parameters). A known pre-existing
|
||
issue from Iter 2; Task #22 in the backlog.
|
||
|
||
## 2026-05-07 — architecture review after Iter 5
|
||
|
||
Architect agent invoked. Findings:
|
||
|
||
1. **Mangling consistency holds.** `@ail_<module>_<def>` is consistent
|
||
across functions, constants, string globals, and cross-module calls.
|
||
The trampoline is correct. ADT constructors are deliberately
|
||
symbol-free (inline malloc).
|
||
2. **Module hashes bit-identical since Iter 4.** The Iter 5c snapshot
|
||
regeneration was a codegen-output change, not a hash break.
|
||
3. **Drift, due now:**
|
||
- DESIGN.md says `define i64 @main()`, codegen emits
|
||
`define i32 @main()` (see `sum.ll:35`).
|
||
- String-schema notation in DESIGN.md was shortened
|
||
(`@.str_<module>_<idx>` instead of `@.str_<module>_<hint>_<idx>`).
|
||
4. **Debt that accrues interest:** the `deps` builtin leak (Task #22) has
|
||
become a falsehood in workspace mode — close it before the next big
|
||
jump.
|
||
|
||
**Plan iteration 6 — clean-up:**
|
||
|
||
1. **Fix DESIGN.md drift.** Update the mangling-scheme block, correct the
|
||
`@main` signature, and note the string globals precisely.
|
||
2. **`deps` hardening (#22).** Build a top-level def table per workspace;
|
||
filter edges whose target is not a top-level symbol, or emit them as
|
||
separate `builtin:`/`local:` categories. Function parameters via
|
||
lexical scope tracking from walk_term.
|
||
3. **Multi-diagnostic refactor (#20).** `check_workspace` accumulates
|
||
`Vec<Diagnostic>` across all defs instead of short-circuiting on the
|
||
first error. Intra-def may still short-circuit — the value is "see
|
||
all broken defs at once", not "see all broken sub-terms of one def".
|
||
|
||
Order: 1 first (doc triviality), then 2 before 3 (deps is a tooling-
|
||
truth fix, multi-diag is a structural extension).
|
||
|
||
## 2026-05-07 — Iter 6 done: deps hardening + multi-diagnose + DESIGN audit
|
||
|
||
Three things landed together. All small, all KISS — no architecture move,
|
||
just paying off recorded debt.
|
||
|
||
**1. `ail deps` filters builtins, params, and let/match bindings (#22).**
|
||
|
||
Before: `sum -> +, -, ==, n, sum` and `ws_lib.add -> ws_lib.+`,
|
||
`ws_lib.a`, `ws_lib.b`. After: `sum -> sum`, `ws_lib.add -> ws_lib.add`
|
||
gone (no real deps; only the cross-module call from `ws_main` remains).
|
||
|
||
Implementation:
|
||
- New helper `ailang_check::builtins::value_names()`: derives the
|
||
Var-level builtin names (`+ - * / % == != < <= > >= not`) from
|
||
`list()`, so the install-list and the deps-filter share one source of
|
||
truth.
|
||
- `walk_term` in `crates/ail/src/main.rs` now threads a `scope` set:
|
||
fn-params seed it; `Let` adds the bound name for the body only;
|
||
`Match` arms add their pattern variables (`bind_pattern` helper, MVP
|
||
rule "ctor sub-patterns are Var/Wild") and roll them back after. Var
|
||
refs that hit `scope` or `builtins` are dropped; qualified names
|
||
(`prefix.def`) are passed through unconditionally — the typechecker
|
||
forbids dots in def names, so no shadowing risk.
|
||
- Tests added: `deps_filters_builtins_params_locals`,
|
||
`deps_workspace_filters_builtins_and_params`. The Iter 5d test
|
||
(`deps_workspace_includes_cross_module`) keeps passing — the only
|
||
edge it asserted is the legitimate one.
|
||
|
||
**2. `check_module` / `check_workspace` are multi-diagnose (#20).**
|
||
|
||
`check_in_workspace` returns `Vec<CheckError>` instead of `Result<()>`.
|
||
Pass-1 (top-level symbol table) stays fail-fast — corrupt globals would
|
||
taint every later diagnostic. Type-def installation is fail-fast within
|
||
a module (env corruption) but the outer module loop continues. The
|
||
body-check loop is the multi-diagnose layer: each def is checked against
|
||
the assembled env, errors accumulate, the next def is attempted.
|
||
|
||
Test: `body_errors_accumulate_across_defs` — one module with two
|
||
independent body errors (arity mismatch + unknown var) yields two
|
||
diagnostics with the right `def` field. The legacy single-error `check`
|
||
keeps working by `.into_iter().next()`-ing the Vec, so internal snapshot
|
||
tests in `crates/ailang-check/src/lib.rs` are unchanged.
|
||
|
||
Out of scope: intra-def collection. A single fn body with three type
|
||
errors still reports one. The "see all broken defs at once" goal is met;
|
||
intra-def will require unification deferral and isn't due now.
|
||
|
||
**3. DESIGN.md `What the MVP is NOT` audit (#24).**
|
||
|
||
The section was lying: it claimed "No ADTs / pattern matching" (delivered
|
||
Iter 3) and "Only ints + bools + unit" (strings landed Iter 2). Renamed
|
||
to `What is not (yet) supported`, restructured into "not yet" + "what is
|
||
supported (smoke-tested)". New invariant: this section is meant to be
|
||
the truth at the **end of the latest iteration**, not a 2026-05-07-day-0
|
||
scope statement.
|
||
|
||
**Architecture check (the user-asked self-questioning):**
|
||
|
||
- *Would I use this language now?* For non-recursive arithmetic + ADT
|
||
programs over int/bool/str: yes, comfortably. For anything that needs
|
||
mapping, folding, generic data structures: no, closures are the
|
||
blocker. That's the next big sprint, not Iter 7.
|
||
- *Consistency:* DESIGN.md, JOURNAL.md, code, and CLI output now agree
|
||
on what the language can do. The "What is not (yet) supported" block
|
||
is the canonical truth surface.
|
||
- *Visualisation:* `ail deps --workspace --json` is now a clean
|
||
cross-module call graph (no builtin noise). Good enough for an
|
||
external graph renderer to consume; a built-in DOT/ASCII renderer is
|
||
*possible future tooling*, not "we need it now". KISS.
|
||
- *Documentation:* the agents/ directory is the sub-prompt layer, the
|
||
JOURNAL is the iteration log, DESIGN.md is the contract. No new doc
|
||
axes needed at this scale.
|
||
|
||
**Tests:** 47 green (previously 44). +2 deps tests in `e2e.rs`, +1
|
||
multi-diag test in `crates/ailang-check/tests/workspace.rs`.
|
||
|
||
**Plan iteration 7:**
|
||
|
||
Closures + higher-order functions. This is the big jump that
|
||
DESIGN.md / Day 0 has been pointing at: it requires a typed IR (TIR)
|
||
stage, closure conversion in lowering, and a heap-aware ABI. The
|
||
multi-diag refactor in Iter 6 was scoped intentionally minimal — when
|
||
TIR lands, intra-def diagnostics become structurally cheap and Task #20
|
||
gets revisited.
|
||
|
||
## 2026-05-07 — Iter 7 done: first-class function references (no capture)
|
||
|
||
Iter 6 outlined Iter 7 as "closures + HOFs + TIR". KISS course-correct
|
||
on inspection: that bundle had three independent things in it, and the
|
||
HOF use-cases (passing functions around, calling through fn-typed
|
||
parameters) need none of TIR or capture. Splitting paid off — what
|
||
landed here is ~120 LOC of codegen, no TIR, no heap, no ABI churn.
|
||
Closures with capture stay queued for Iter 8 (where TIR is the
|
||
correct precondition).
|
||
|
||
**What works now:**
|
||
- Top-level fn name (or qualified `prefix.def`) used as a value yields
|
||
an LLVM fn-pointer (`@ail_<m>_<def>`, type `ptr`).
|
||
- Fn-typed parameters can be called as `f(args)` — the body emits an
|
||
indirect `call <ret> (<param-tys>) %f(...)`.
|
||
- Pass through `let`: `let g = inc in g(x)` works (the local just
|
||
aliases the global SSA, the sidetable lookup still hits).
|
||
- Pass to another fn: `apply(inc, 41) == 42` — see
|
||
`examples/hof.ail.json`, exercised end-to-end.
|
||
|
||
**What does not (yet) work — by design:**
|
||
- No anonymous lambdas. The only fn-value source is a top-level def
|
||
reference.
|
||
- No capture. A fn-value is always a constant pointer to a top-level
|
||
def; there is no environment to allocate.
|
||
- Both deferred to Iter 8 where they share the TIR + closure-conversion
|
||
preconditions.
|
||
|
||
**Implementation, in order of where the rubber meets the road:**
|
||
|
||
1. `llvm_type` learned `Type::Fn { .. } -> "ptr"`. The actual signature
|
||
travels separately. New helper `fn_sig_from_type` lifts an AILang
|
||
fn-type into an `FnSig` (LLVM types only).
|
||
2. `Emitter` got a sidetable: `ssa_fn_sigs: BTreeMap<String, FnSig>`,
|
||
keyed by SSA value (or `@global`). It's reset per function body.
|
||
3. At `emit_fn` entry, every fn-typed parameter registers
|
||
`(%arg_<name>, sig)` in the sidetable.
|
||
4. `lower_term(Term::Var)` now falls through to a top-level fn lookup
|
||
(`resolve_top_level_fn`) when the name isn't a local. The returned
|
||
SSA is the global symbol; the sidetable gets the sig.
|
||
5. `lower_term(Term::App)` dispatches:
|
||
- if callee is a `Var` AND not shadowed AND statically known
|
||
(`is_static_callee` covers builtin operators, qualified
|
||
`prefix.def`, current-module fns), keep the existing direct
|
||
`lower_app` path — no extra indirection in the IR;
|
||
- otherwise lower the callee, expect type `ptr`, look up the sig in
|
||
the sidetable, emit `emit_indirect_call`.
|
||
6. `Term::If` propagates the sig to its phi SSA when both branches are
|
||
fn-pointers with matching sigs (cheap two-line copy; no separate
|
||
test, falls out of the `apply`-on-conditional pattern).
|
||
|
||
**Why no typechecker change?** The typechecker already accepted
|
||
fn-typed locals (`Term::Var` against `env.globals`, App via `synth(callee)`
|
||
unifying with `Type::Fn`). The only blocker was `MVP: callee must be a
|
||
variable` in codegen.
|
||
|
||
**Tests:** 48 green (previously 47).
|
||
|
||
- `crates/ail/tests/e2e.rs::higher_order_apply_inc` builds and runs
|
||
`examples/hof.ail.json`, asserts the binary prints `42`.
|
||
- Existing tests unchanged (incl. snapshot tests around the IR
|
||
emission for `sum`, `list`, `max3`).
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Yes for `apply`-style and "pass a
|
||
predicate" patterns. Still no for capturing closures (`let n = 3 in
|
||
map(\x -> x + n, xs)`-equivalent), but the ergonomic gap shrank.
|
||
- *Consistency:* DESIGN.md "What is not (yet) supported" rewritten in
|
||
the same edit; first-class fn-refs now have a positive bullet, the
|
||
closures bullet is precise about what it means (no capture, no
|
||
lambdas).
|
||
- *Visualisation:* `ail describe`/`manifest` already render fn-typed
|
||
params correctly via the existing `pretty::type_to_string`
|
||
(`((Int) -> Int, Int) -> Int`). No tooling change required.
|
||
- *KISS:* every alternative I considered (full `LocalType` enum,
|
||
swapping `(String, String)` returns to a typed wrapper, lifting
|
||
lambdas to defs as syntactic sugar) was strictly more code than the
|
||
sidetable approach, with no expressivity gain.
|
||
|
||
**Plan iteration 8:**
|
||
|
||
Closures with capture, anonymous lambdas, the typed IR (TIR) layer,
|
||
closure conversion in lowering. Now that we have indirect calls
|
||
working, the main delta is: a fn-value also needs an environment
|
||
pointer, the sidetable becomes per-value (heap-allocated), and the
|
||
calling convention shifts to `(env_ptr, args...)`. Touches every
|
||
existing call path — that's why it gets its own iteration.
|
||
|
||
## 2026-05-07 — Iter 8 done: closures with capture (no TIR needed)
|
||
|
||
Iter 7's plan named TIR as the prerequisite for closures. On
|
||
inspection that bundling was wrong — TIR is one possible
|
||
implementation strategy, not a structural requirement. The
|
||
typechecker already attaches enough type information through `synth`
|
||
that the codegen can read capture types out of `self.locals`
|
||
directly. So Iter 8 ships closures **without** introducing TIR. KISS
|
||
won.
|
||
|
||
The work split into two commits:
|
||
|
||
**Iter 8a — closure-pair ABI flip.** Every fn-value is now a `ptr` to
|
||
a heap or static closure pair `{ thunk_ptr, env_ptr }`, regardless of
|
||
whether it came from a lambda or a top-level def reference. To keep
|
||
top-level-fn references cheap, every top-level fn auto-emits:
|
||
|
||
```llvm
|
||
define <ret> @ail_<m>_<f>_adapter(ptr %_env, <params>) {
|
||
%r = call <ret> @ail_<m>_<f>(<args>)
|
||
ret <ret> %r
|
||
}
|
||
@ail_<m>_<f>_clos = constant { ptr, ptr } { @adapter, null }
|
||
```
|
||
|
||
`Term::Var` resolving to a top-level fn returns the address of
|
||
`_clos`, never the bare fn pointer. `emit_indirect_call` was
|
||
rewritten to GEP+load both halves and call `thunk(env, args...)`.
|
||
Direct calls (statically-known callees in `Term::App`) bypass the
|
||
adapter and stay at the original speed.
|
||
|
||
The Iter 7 hof example (`apply(inc, 41)`) continues to print 42
|
||
unchanged — only the IR shape changed, not the source. IR snapshot
|
||
files for sum/list/max3/hello/ws_main were refreshed.
|
||
|
||
**Iter 8b — Term::Lam + capture + lambda lifting.** New AST node:
|
||
|
||
```jsonc
|
||
{ "t": "lam",
|
||
"params": ["x"...],
|
||
"paramTypes": [Type...],
|
||
"retType": Type,
|
||
"effects": ["..."],
|
||
"body": Term }
|
||
```
|
||
|
||
Param/return types are explicit. The typechecker accepts the
|
||
declared `Type::Fn` shape, checks the body's type against `retType`,
|
||
and verifies that body effects are a subset of the declared lambda
|
||
effects (no row polymorphism in the MVP). Constructing a lambda is
|
||
pure; the act of *calling* picks up the declared effects, via the
|
||
existing App branch.
|
||
|
||
Codegen does textbook closure conversion:
|
||
|
||
1. **Free-variable analysis.** `collect_captures` walks the body
|
||
skipping builtins (`+`, `==`, ...), the current module's top-
|
||
level fns, and qualified `prefix.def` names. The remainder are
|
||
captures. Inner lambdas contribute their own free vars upward.
|
||
2. **Lift to thunk.** For each lambda, generate a fresh
|
||
`@ail_<m>_<def>_lam<id>(ptr %env, params...)`. State the body
|
||
into a side buffer (the emitter's `body`/`locals`/`counter` are
|
||
saved and reset, then restored). Captures and lambda params are
|
||
pushed as named locals so the body lowering finds them. The
|
||
thunk text goes into a `deferred_thunks` queue and is appended
|
||
after the parent fn's `}` — LLVM IR doesn't care about fn order.
|
||
3. **Pack at the use site.** In the OUTER body emit:
|
||
|
||
```llvm
|
||
%env = call ptr @malloc(i64 <8 * captures>)
|
||
; for each capture i: store at offset 8*i
|
||
%clos = call ptr @malloc(i64 16)
|
||
; store thunk_ptr at offset 0, env at offset 8
|
||
```
|
||
|
||
`%clos` is the value returned by the Lam term. Its sig is
|
||
registered in the sidetable so subsequent indirect calls work.
|
||
|
||
4. **Capture sigs propagate.** A fn-typed capture (e.g. capturing a
|
||
fn-typed param of an outer scope) keeps its FnSig in the thunk's
|
||
sidetable, so the captured fn can still be indirect-called from
|
||
inside the lambda.
|
||
|
||
Capture layout uses 8-byte slots regardless of LLVM type. Typed
|
||
load/store reads only the bytes it needs — wasted padding for `i1`
|
||
and `i8` is fine at this scale.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Yes for substantially more cases.
|
||
`let n = 3 in apply(\\x. x + n, 39)` is the example I would have
|
||
reached for in Iter 6 and bounced off. It now compiles and runs.
|
||
`map`/`fold`/`filter` over user-supplied predicates are within
|
||
reach — only the absence of polymorphism still forces author-side
|
||
monomorphisation.
|
||
- *Did I think of everything?* Hash stability checked manually:
|
||
`examples/sum.ail.json` produced the same fn hashes
|
||
(`db33f57cb329935e`, `d9a916a0ed10a3d3`) before and after Iter 8.
|
||
Existing modules without `Term::Lam` serialise bit-identically. ✓
|
||
- *Consistency:* DESIGN.md "What is not (yet) supported" rewritten
|
||
in the same edit. The Term schema gained `lam`, `ctor`, `match`
|
||
rows that were already supported but had been omitted from the
|
||
schema fragment. Now the doc is exhaustive for the supported
|
||
language.
|
||
- *Visualisation:* `ail describe` already renders Lam terms (added
|
||
pretty-printer rule), and the codegen IR for `closure.ail.json`
|
||
reads as a textbook closure-conversion lowering.
|
||
- *KISS check:* I considered three alternatives and all were
|
||
strictly worse — fat-pointer ABI (aggregate-passing concerns), full
|
||
TIR layer (large rewrite), uniform heap pair without static-closure
|
||
optimisation (regressed Iter 7 to one malloc per fn-value escape).
|
||
|
||
**Tests:** 49 green (was 48 after Iter 7). One new e2e:
|
||
`closure_captures_let_n` builds and runs `examples/closure.ail.json`
|
||
asserting "42". IR snapshot files refreshed for the per-fn adapter +
|
||
static-closure scaffold — only structural delta.
|
||
|
||
**Plan iteration 9:**
|
||
|
||
Two candidates, both real pain points:
|
||
|
||
1. **Polymorphic inference.** Make `Type::Forall` actually work in
|
||
`synth` — instantiate fresh type variables at each use site, allow
|
||
`let id = \\x. x in (id 1, id true)`. This unblocks generic
|
||
`map`/`fold`/etc. without per-type clones. Probably small (~150
|
||
LOC in the typechecker; codegen already monomorphises by
|
||
instantiation when it lowers the call).
|
||
|
||
2. **GC / region reclamation.** Right now ADT boxes, lambda envs,
|
||
and closure pairs all leak through the program's lifetime. A
|
||
minimal mark-and-sweep over a tagged heap would let us run real
|
||
programs. Bigger lift, ~400-600 LOC plus runtime support.
|
||
|
||
Leaning toward (1) for the next iteration: it's the smaller bite
|
||
*and* the bigger expressivity unlock. (2) becomes acute only when
|
||
someone tries to run an unbounded loop, which the current examples
|
||
don't.
|
||
|
||
## 2026-05-07 — Iter 9 done: dogfood + `ail run`
|
||
|
||
Course-corrected from the Iter-8 plan. Polymorphism is the bigger
|
||
expressivity unlock on paper, but I hadn't actually proved that the
|
||
language was sufficient for "small but real" programs without it. So
|
||
Iter 9 became a dogfood iteration: write a non-trivial program
|
||
that exercises everything Iter 1-8 shipped, and use `ail run` /
|
||
errors / type-checker output as the user would. If something broke,
|
||
fix it. If nothing broke, document the boundary moved.
|
||
|
||
**`examples/list_map.ail.json`**:
|
||
|
||
```jsonc
|
||
type IntList = Nil | Cons Int IntList
|
||
|
||
map_int :: ((Int) -> Int, IntList) -> IntList
|
||
map_int(f, xs) = match xs {
|
||
Nil -> Nil
|
||
Cons(h, t) -> Cons(f(h), map_int(f, t))
|
||
}
|
||
|
||
print_list :: (IntList) -> Unit !IO
|
||
print_list(xs) = match xs {
|
||
Nil -> ()
|
||
Cons(h, t) -> let _ = do io/print_int(h) in print_list(t)
|
||
}
|
||
|
||
main = let xs = Cons 1 (Cons 2 (Cons 3 Nil)) in
|
||
print_list(map_int(\\x. x * 2, xs))
|
||
```
|
||
|
||
**Result:** nothing broke. Output `2\\n4\\n6\\n`, exit 0. The full
|
||
pipeline (`ail run`) covers: ADTs with two ctors of different
|
||
arity; pattern matching with nested `Var` fields; recursion over
|
||
ADT; closures (with no captures here, so env is null but the
|
||
closure-pair plumbing still gets exercised); fn-typed parameters in
|
||
a top-level def; `do io/...` inside a match arm body, with `let _`
|
||
to sequence two effectful operations; effect propagation through
|
||
the call chain. This validates Iter 1-8 as a self-contained
|
||
foundation.
|
||
|
||
**Friction surfaced:** writing the AST by hand is tedious — the
|
||
JSON for this 4-def module is 200+ lines. That's not surprising
|
||
(the format is for LLMs, not humans), but it suggests an Iter 10
|
||
priority: a richer pretty-print form, or an `ail snippet` helper
|
||
for common boilerplate (`mk_list_int`, etc.). Not blocking; noted.
|
||
|
||
**`ail run` (Iter 9b):** Builds into a tempdir + execs the binary,
|
||
exit code passthrough. Saves a `cd && ./bin` step in the dogfood
|
||
loop. Tiny addition — `Cmd::Build`'s body factored into a shared
|
||
`build_to` helper.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For self-contained Int-typed
|
||
programs over recursive ADTs: yes. The list_map example is what
|
||
I would have wanted to write since Iter 6 and bounced off
|
||
repeatedly. It now compiles and runs without me adapting the
|
||
source — the language is what its authors said it was, end to
|
||
end.
|
||
- *Did I think of everything?* Two cracks observed during the
|
||
dogfood:
|
||
- `(Int)` parens around single-param fn-types in pretty-print are
|
||
visual noise. Cosmetic, can wait.
|
||
- `let _ = do <effect> in <body>` is the only way to sequence
|
||
effects today. Working as intended given KISS, but a `;`
|
||
operator (sequencing) would be cheap polish.
|
||
- *Consistency:* DESIGN.md CLI block + smoke-test list updated.
|
||
Iter 8c invariant — "What is not (yet) supported" ≡ truth at end
|
||
of latest iteration — held; no new pending items.
|
||
- *KISS:* Iter 9 added 0 LOC of language semantics. All gain came
|
||
from validating the existing surface and a small CLI helper.
|
||
|
||
**Tests:** 50 green (was 49). New e2e
|
||
`list_map_doubles_then_prints`. No test for `ail run` itself —
|
||
`build_and_run` already exercises the equivalent path.
|
||
|
||
**Plan iteration 10:**
|
||
|
||
The dogfood revealed two real-but-not-blocking pain points and one
|
||
big architectural gap. Candidates, ranked:
|
||
|
||
1. **Polymorphic let-bindings with monomorphisation at codegen.**
|
||
Allows `let id = \\x. x in (id 1, id true)` and ultimately
|
||
`map :: (a -> b) -> List a -> List b`. The ground truth-ier
|
||
answer for the "would I use it for X?" question, but a
|
||
non-trivial pipeline change (typechecker→codegen needs to thread
|
||
instantiation info to the call site).
|
||
|
||
2. **Sequencing operator `;` and richer effect ergonomics.** A
|
||
`Term::Seq { lhs, rhs }` (or compile sugar to `Let { name: "_",
|
||
value: lhs, body: rhs }`) plus a small pretty-print update.
|
||
Cheap, satisfying.
|
||
|
||
3. **GC.** Heap reclamation for ADT boxes, lambda envs, closure
|
||
pairs. Real architecture step. Becomes acute the moment someone
|
||
writes a long-running loop; the current examples don't.
|
||
|
||
Tentative pick: (2) for the next sprint as a satisfying small
|
||
polish, then (1) as Iter 11. (3) bides its time until a real
|
||
program needs it.
|
||
|
||
## 2026-05-07 — Iter 10 done: Term::Seq sequencing
|
||
|
||
Followed the Iter 9 plan and shipped (2). New AST node
|
||
`Term::Seq { lhs, rhs }` with serde tag "seq". Semantics: evaluate
|
||
lhs (which must be Unit), discard the value, return rhs. Effects
|
||
from both sides accumulate.
|
||
|
||
This is sugar for `let _ = lhs in rhs`, but it's a first-class node
|
||
because:
|
||
- The pretty-print renders cleanly (`(seq lhs rhs)` instead of
|
||
borrowing the `let` form with a discard binding).
|
||
- Diagnostics are sharper: a non-Unit lhs gets a "type mismatch"
|
||
error pointing at the seq site, not "binding `_` had type X" at
|
||
a let site.
|
||
- Future tooling (effect inference visualisation, dataflow) can
|
||
treat sequencing as a structural concept instead of a special-
|
||
cased let.
|
||
|
||
Codegen is trivial: lower lhs (drop SSA), lower rhs (return).
|
||
|
||
Refactored `examples/list_map.ail.json`'s `print_list` to use seq
|
||
instead of `let _ = ...`. Output unchanged (`2\\n4\\n6\\n`); the
|
||
JSON shed a few lines and reads more honestly.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* Same answer as Iter 9 (yes for
|
||
small but real programs), but the seq node makes IO-heavy
|
||
recursion read better — closer to "call this effect, then this
|
||
one" instead of "bind this effect to nothing, then this one".
|
||
- *Did I break anything?* Hash stability check: existing examples
|
||
without `Term::Seq` serialise identically; their fn hashes are
|
||
unchanged. `list_map.ail.json`'s hashes shifted as expected since
|
||
its body changed.
|
||
- *KISS:* +30 LOC across AST/pretty/check/codegen/walker. One
|
||
unit test for the lhs-must-be-Unit rule. The dogfood example
|
||
proves the e2e path.
|
||
|
||
**Tests:** 51 green (was 50). New `seq_lhs_must_be_unit` unit test
|
||
in ailang-check. Existing list_map e2e still passes after the
|
||
refactor.
|
||
|
||
**Plan iteration 11:**
|
||
|
||
Polymorphism, as queued in the Iter 9 plan. Concretely: HM-style
|
||
unification + let-generalisation in the typechecker, monomorph-
|
||
isation at codegen time. Touches the typechecker→codegen pipeline.
|
||
Bigger commit than the recent stretch, will probably need to be
|
||
phased (typechecker substitution machinery, then codegen
|
||
specialisation, then docs).
|
||
|
||
## 2026-05-07 — Iter 11 done: deeper dogfood (insertion sort)
|
||
|
||
Pulled back from polymorphism for one more validation cycle before
|
||
the architectural step. Polymorphism is a substantial pipeline
|
||
change (typechecker substitution + codegen monomorphisation) and I
|
||
wanted one more "small but real" program to confirm the existing
|
||
foundation holds before disturbing it.
|
||
|
||
`examples/sort.ail.json` — insertion sort over `IntList`:
|
||
|
||
```jsonc
|
||
insert :: Int -> IntList -> IntList
|
||
insert(y, xs) = match xs {
|
||
Nil -> [y]
|
||
Cons(h, t) -> if y <= h then Cons(y, Cons(h, t))
|
||
else Cons(h, insert(y, t))
|
||
}
|
||
|
||
sort :: IntList -> IntList
|
||
sort(xs) = match xs {
|
||
Nil -> Nil
|
||
Cons(h, t) -> insert(h, sort(t))
|
||
}
|
||
|
||
print_list :: IntList -> Unit !IO // uses Iter 10 seq
|
||
|
||
main = print_list(sort([3,1,4,1,5,9,2,6,5,3,5]))
|
||
```
|
||
|
||
**Result:** typechecks first try, runs first try, prints
|
||
`1 1 2 3 3 4 5 5 5 6 9` (each on its own line). 11-element input,
|
||
correct sorted output. The combination of recursive ADT pattern
|
||
match + comparison ops + branching + leaf recursion + IO
|
||
sequencing all worked end to end without the language tripping me
|
||
up. Iter 10's seq made `print_list` notably cleaner than the
|
||
`let _ = ...` form would have been.
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For "small but real"
|
||
monomorphic programs over Int, Bool, Unit, Str, and ADTs of
|
||
those: confidently yes. Insertion sort writes out as the
|
||
textbook recursion, no bookkeeping that the language couldn't
|
||
do for me.
|
||
- *Did I think of everything?* The remaining wall is still
|
||
polymorphism. Sort over `IntList` needs hand-monomorphisation;
|
||
a generic `sort :: (a -> a -> Bool) -> List a -> List a` is
|
||
what the language eventually wants. No new architectural cracks
|
||
surfaced from this dogfood.
|
||
- *Visualisation:* `ail describe sort.ail.json sort` reads the
|
||
way I'd expect a sort definition to read, with `IntList` types
|
||
inline and the recursive call rendered cleanly.
|
||
- *KISS:* Iter 11 added 0 LOC of language semantics and 1 e2e
|
||
test. The 250-line JSON for the example is verbose but
|
||
mechanical — no friction once you accept that the JSON is the
|
||
surface for LLM authors.
|
||
|
||
**Tests:** 52 green (was 51). New e2e
|
||
`insertion_sort_orders_list`. Pure addition; existing tests
|
||
untouched.
|
||
|
||
**Plan iteration 12:**
|
||
|
||
Now polymorphism. Two more dogfood programs would just keep
|
||
producing the "the language is fine for monomorphic programs"
|
||
result, which is already established. The real expressivity
|
||
unlock — and the answer to "would I use it for X?" for X that
|
||
actually needs generic data — is HM inference + let-generalisation
|
||
+ monomorphisation. Phased plan:
|
||
|
||
12a. Typechecker: introduce a `Subst` (type variable substitution)
|
||
and unification. Thread through `synth`. At `let`, generalise
|
||
syntactic values (lambdas) — no value-restriction subtlety
|
||
needed yet, the MVP has no mutable refs.
|
||
12b. Codegen: at each polymorphic call site, the typechecker
|
||
records the instantiation. Codegen walks the AST a second
|
||
time per (def, instantiation) pair and emits a specialised
|
||
version with the type variables substituted by concrete
|
||
types in fn signatures.
|
||
12c. Docs + a polymorphic `id` test + a generic `map :: (a -> b)
|
||
-> List a -> List b` rewrite of `list_map.ail.json`.
|
||
|
||
## 2026-05-07 — Iter 12a/b done: polymorphism reaches the binary
|
||
|
||
Skipped 12c's "polymorphic map" — without parameterised ADTs (which
|
||
the MVP doesn't have), the rewrite would still be over a concrete
|
||
`IntList`, defeating the purpose. So 12c becomes lighter: docs +
|
||
two new examples (`poly_id`, `poly_apply`) that prove polymorphism
|
||
end-to-end on primitive types and on fn-typed parameters. The big
|
||
test is whether *I* would use the language now for a poly-flavoured
|
||
program; the answer below.
|
||
|
||
**12a — typechecker:**
|
||
|
||
`Type::Forall { vars, body }` is now legal at top-level fn types.
|
||
Implementation is the textbook ML rule: peel the Forall when
|
||
checking the body (rigid vars go into `Env.rigid_vars` so
|
||
`check_type_well_formed` accepts them), instantiate fresh metavars
|
||
at every var-resolution site, unify on every formerly-`expect_eq`
|
||
edge.
|
||
|
||
The metavar encoding sidesteps an AST schema change: a metavar is
|
||
just `Type::Var { name: "$m<id>" }`. The `$` prefix can't collide
|
||
with source identifiers, the JSON layout doesn't shift, and module
|
||
hashes stay bit-identical (verified: `sum.ail.json` keeps
|
||
`db33f57cb329935e` / `d9a916a0ed10a3d3`). I considered adding a
|
||
new `Type::Meta` variant under `#[serde(skip)]` but that would
|
||
have pulled hashing concerns into serde; the naming convention
|
||
keeps the AST untouched.
|
||
|
||
`Subst` is a flat `BTreeMap<u32, Type>`; `unify` is the standard
|
||
occurs-check version with effects compared as a set. Constants
|
||
still reject Forall outright; ADT fields still reject vars. No
|
||
let-generalisation: lambdas inside fn bodies are checked
|
||
monomorphically against their declared types — keeps the
|
||
implementation small and matches DESIGN.md's "top-level types
|
||
must always be explicitly annotated".
|
||
|
||
**12b — codegen:**
|
||
|
||
Direct calls to a polymorphic def get monomorphised on demand.
|
||
Each unique (def, instantiation) pair emits a specialised LLVM fn
|
||
with mangling `@ail_<m>_<def>__<descriptor>`. Descriptor scheme:
|
||
`Int → I`, `Bool → B`, `Unit → U`, `Str → S`, ADT `Foo → FFoo`,
|
||
`Fn(a)→b → Fn_<a>__r_<b>`. So `id(42)` and `id(true)` produce
|
||
`@ail_poly_id_id__I` and `@ail_poly_id_id__B` side by side.
|
||
|
||
Pass 1 of `lower_workspace` now splits fn-typed defs into mono
|
||
(`module_user_fns`, LLVM-typed FnSig as before) and poly
|
||
(`module_polymorphic_fns`, full FnDef). A unified
|
||
`module_def_ail_types` carries AILang types for both, used by
|
||
the codegen-side type tracker.
|
||
|
||
The hard part was getting AILang types at call sites. The
|
||
typechecker has them but doesn't hand its annotations down (no
|
||
TIR yet). I considered three paths:
|
||
1. Typechecker sidetable keyed by AST node ids — would need
|
||
to assign ids deterministically, brittle.
|
||
2. Uniform representation (everything passes as ptr/i64) —
|
||
contradicts CLAUDE.md's "performance is extremely important".
|
||
3. Codegen replays the type derivation locally.
|
||
Picked (3). The trade-off is duplication (`synth_arg_type`
|
||
mirrors what the typechecker already did), but it's contained
|
||
to a small recursive walk and uses the same `locals`/`extras`
|
||
shadowing pattern. Worth it for the MVP — once a TIR stage
|
||
materialises (it's still on the debt list), the duplication
|
||
collapses into a single pass.
|
||
|
||
`locals` grew from 3-tuple to 4-tuple `(name, ssa, llvm_type,
|
||
ail_type)`. Six push sites updated mechanically. Lambda capture
|
||
metadata grew the same way. `CtorRef` got `ail_fields` so match
|
||
arm bindings inherit the AILang type.
|
||
|
||
The drain phase iterates until `mono_queue` is empty —
|
||
specialised bodies can themselves invoke polymorphic defs and
|
||
queue further entries. `apply_subst_to_term` substitutes rigid
|
||
vars in `Term::Lam` annotations (the only Term arm carrying
|
||
types).
|
||
|
||
**Architecture self-check:**
|
||
|
||
- *Would I use this language now?* For monomorphic programs:
|
||
yes (already established). For polymorphism over primitives
|
||
and fn-typed parameters: yes — `id` and `apply` write out the
|
||
way the textbook says they should, with no language-level
|
||
bookkeeping leaking into the source. The `poly_apply` example
|
||
was particularly revealing: the closure-pair ABI (Iter 8a)
|
||
composes cleanly with monomorphisation. Specialised body of
|
||
`apply__I_I` keeps `f` as a fn-typed local; the existing
|
||
indirect-call path already handles the lower from there.
|
||
- *Did I think of everything?* No, two known gaps:
|
||
1. **Polymorphic fn passed as a value** (`let f = id in f(42)`)
|
||
fails in codegen — `resolve_top_level_fn` looks in
|
||
`module_user_fns` only. Adding this means emitting one
|
||
closure-pair global per instantiation, possibly via the
|
||
same drain pass. Defer.
|
||
2. **Higher-rank polymorphism** (`apply(id, 42)`) trips
|
||
`unify_for_subst` which doesn't handle Forall on the param
|
||
side. Real higher-rank polymorphism is a substantial step
|
||
and not on the near horizon — deferred to a later iter.
|
||
- *Visualisation:* `ail manifest poly_id.ail.json` now shows
|
||
`forall a. (a) -> a` correctly. The pretty-printer carried
|
||
`Type::Forall` rendering since Iter 1; nothing to do.
|
||
- *KISS:* +1 typechecker file edit (~430 LOC inserted, mostly
|
||
Subst+unify+four tests), +1 codegen extension (~600 LOC
|
||
inserted, mostly the drain path + helpers + locals widening).
|
||
Two new examples, two new e2e tests. Could be smaller if I
|
||
bit the bullet on TIR; not yet worth the upfront cost.
|
||
|
||
**Tests:** 58/58 (was 56/56). Added 4 typechecker unit tests in
|
||
12a, 2 e2e tests in 12b. Hash invariant holds.
|
||
|
||
**Plan iteration 13 (queued, not started):**
|
||
|
||
The natural next step depends on what I want to use the language
|
||
for. Two candidates, in order of expected payoff:
|
||
|
||
13a. **Parameterised ADTs** — `List a`, `Maybe a`, etc. Without
|
||
these, polymorphism is half-useful: a generic `map` still
|
||
can't transform an `IntList` into a `BoolList`. ADT defs
|
||
would gain a `vars: Vec<String>` field; ctor field types
|
||
could mention them; codegen monomorphises ADT instances
|
||
just like fns. This is the bigger expressivity unlock.
|
||
13b. **GC or arena** — every ADT box, lambda env, and closure
|
||
pair currently leaks. For sort over an 11-element list,
|
||
fine. For anything longer-running, required. The current
|
||
lifetime model is "leak"; the right MVP is probably
|
||
bumpalloc per top-level fn invocation. Could be done
|
||
before parameterised ADTs but doesn't unlock new examples.
|
||
|
||
Leaning 13a — it's the more interesting architectural step and
|
||
makes the "polymorphic map" rewrite from the original 12c plan
|
||
finally meaningful.
|
||
|
||
## 2026-05-07 — Iter 13 done: parameterised ADTs reach the binary
|
||
|
||
**Why now.** End of Iter 12 left polymorphism half-useful: `id`
|
||
and `apply` worked, but every container was monomorphic
|
||
(`IntList`, `Maybe_Int`). A generic `map :: forall a b. ((a) -> b,
|
||
List a) -> List b` was unwritable. 13 lifts that.
|
||
|
||
**Three commits:**
|
||
|
||
- `0782622` 13a — schema (`TypeDef.vars`, `Type::Con.args`) +
|
||
checker (substitution at ctor + match + arity validation in
|
||
`check_fn`).
|
||
- `1631f60` 13b — codegen: per-use-site substitution of LLVM
|
||
field types in `lower_ctor` and `lower_match`. No mono-queue
|
||
for types — ctor code was already inlined at every use site,
|
||
so 13b only had to thread substitution through, not invent a
|
||
symbol scheme. `synth_arg_type` for `Term::Ctor` now returns
|
||
concrete type-args, and `llvm_type(Type::Var)` is a hard
|
||
error instead of a silent `ptr` fallback (the latter was
|
||
flagged by the architect review and is the most defensive
|
||
single change in 13).
|
||
- `<this>` 13c — DESIGN.md flipped (parameterised ADTs out of
|
||
the gap list, into the supported list); two new example lines.
|
||
|
||
**Hash invariant.** Both new fields are
|
||
`#[serde(default, skip_serializing_if = "Vec::is_empty")]`. A new
|
||
regression test in `crates/ailang-core/src/hash.rs` deserialises
|
||
the actual `examples/sum.ail.json` and `examples/list.ail.json`
|
||
from disk and asserts `db33f57cb329935e` and `b082192bd0c99202` —
|
||
the recorded pre-13a hashes. It's deliberately phrased against
|
||
the on-disk JSON rather than reconstructed code, so the test
|
||
fails if anyone resaves the examples in a way that drifts the
|
||
canonical bytes.
|
||
|
||
**Architect-flagged debt I deliberately did NOT touch in 13:**
|
||
|
||
- `is_static_callee` returns true for poly fns but
|
||
`resolve_top_level_fn` only consults `module_user_fns`. A
|
||
poly fn used as a value (`let f = id in f(42)`) passes the
|
||
static check then surfaces as `UnknownVar`. Would need one
|
||
closure-pair global per instantiation. Out of 13 scope; same
|
||
hole that was queued at the end of Iter 12.
|
||
- Triple source of truth for builtins (`builtins::install`,
|
||
`builtins::list`, `codegen::builtin_ail_type` /
|
||
`builtin_effect_op_ret`). Every new operator costs three
|
||
edits. Low interest today, escalates with every effect op.
|
||
Worth a future tidy iter — not blocking expressivity.
|
||
- `synth_arg_type` for `Term::If` returns `synth(then)` only;
|
||
for `Term::Match`, the first arm. Masked today by the
|
||
typechecker having already unified, but it's the kind of
|
||
duplication that decays. Same fundamental cost as the absence
|
||
of a TIR.
|
||
|
||
**Architecture self-check.**
|
||
|
||
- *Would I use this language now?* For polymorphism over
|
||
primitives, fn-typed values, AND parameterised containers —
|
||
yes. The `box.ail.json` and `maybe_int.ail.json` examples
|
||
read like the textbook says they should. No type-arg
|
||
bookkeeping leaks into the source.
|
||
- *KISS.* 13b was much smaller than I feared at the start of
|
||
the design phase: ~150 LOC in codegen, no new structures, no
|
||
mono-queue. The reason: ADT ctor code is already inlined.
|
||
The architect's recommendation to *not* mutate `ctor_index`
|
||
but derive `CtorRef` per use site was the right call —
|
||
preserved the static template, made the substitution local.
|
||
- *Did I think of everything?* Two known gaps remain. **(1)**
|
||
Polymorphic ADTs as the type-arg of a polymorphic fn —
|
||
works today because `unify_for_subst` recurses through
|
||
`Type::Con.args` (added in 13a). **(2)** A polymorphic fn
|
||
taking a polymorphic ADT and returning a different
|
||
parameterised ADT (`map : forall a b. ((a)->b, List a) -> List
|
||
b`) — should also work, but I haven't dogfooded it yet
|
||
because `List a`-as-a-rewrite-of-`list_map` would need the
|
||
schema bumps elsewhere (paramaterised list builder). Queued
|
||
for Iter 14.
|
||
- *Visualisation.* `ail manifest examples/box.ail.json` shows
|
||
`type Box :: forall a. MkBox(a)` and
|
||
`fn unbox :: forall a. (Box<a>) -> a`. The pretty-printer
|
||
picked up `args` and `vars` cleanly (Iter 13a).
|
||
|
||
**Tests:** 64/64 (was 58/58). Added: 1 hash-stability regression
|
||
(13a), 3 checker unit tests for parameterised ADTs (13a),
|
||
2 e2e tests over `box.ail.json` and `maybe_int.ail.json` (13b).
|
||
|
||
**Process note (orchestration).** First iter where I worked
|
||
strictly through the agents in `/agents/`: `ailang-architect`
|
||
ran a drift review on HEAD before 13b started; `ailang-implementer`
|
||
got a fixed brief that incorporated the architect's three
|
||
recommendations (don't mutate `ctor_index`, fix `synth_arg_type`
|
||
for `Term::Ctor`, harden `llvm_type`); 13c (this) is the
|
||
orchestrator's own work. The role split landed in `3df943d` after
|
||
I caught myself doing implementer work on 13a directly. The
|
||
agents pay off in proportion to iter size — for 13b they were
|
||
clearly worth the round-trip; for 13a's checker work, marginal.
|
||
|
||
**Plan iteration 14 (queued, not started):**
|
||
|
||
Two candidates, in order of expected payoff:
|
||
|
||
14a. **Polymorphic `List a` rewrite of `list_map`.** Replaces
|
||
`IntList` with `List a`, rewrites `list_map` to return
|
||
`List b`, and lets the polymorphic-map version be the
|
||
dogfood smoke test. Pure exercise — should fall out of
|
||
13b — but worth the dogfood beat. Also: `Maybe a` used
|
||
in a non-trivial fn (e.g. `find : forall a. ((a) -> Bool,
|
||
List a) -> Maybe a`).
|
||
14b. **GC or arena.** Same pitch as before: every ADT box,
|
||
lambda env, closure pair leaks. For `box.ail.json` and
|
||
`maybe_int.ail.json`, fine. For anything that allocates
|
||
in a loop, required. Bumpalloc per top-level fn
|
||
invocation is the natural MVP.
|
||
14c. **Poly fn as value.** Closes the asymmetry the architect
|
||
flagged; gates `let f = id in f(42)`. One closure-pair
|
||
global per instantiation, emitted via the same mono-queue
|
||
drain path. Smaller surface than 14a/b.
|
||
|
||
Leaning 14a — the dogfood payoff for one iter of polish is
|
||
high, and `Maybe`-in-a-real-fn is a missing piece I haven't
|
||
exercised yet. 14b stays second; 14c is a candidate if I want
|
||
a small palate cleanser.
|
||
|
||
---
|
||
|
||
## Iter 13d — rustdoc polish for `ailang-core` + new `ailang-docwriter` agent
|
||
|
||
User triggered: ran `cargo doc --open` for fun and reported
|
||
that the rendered docs were thin — crate headers existed,
|
||
but `pub` items had no `///` strings, there were no `# Examples`
|
||
sections, and intra-doc links were missing. Internal references
|
||
showed up as prose ("see Builtins") rather than as clickable
|
||
links. Two stale `[Builtins]` and `[code]` warnings had been
|
||
bleeding into every `cargo doc` invocation since Iter 6 or so.
|
||
|
||
Recurring task → new agent. Wrote `agents/ailang-docwriter.md`
|
||
with a tight mandate: rustdoc only, no API changes, no edits in
|
||
`docs/` or `agents/`, three verification gates (rustdoc clean,
|
||
build green, tests green incl. doctests). Updated
|
||
`agents/README.md`. Updated DESIGN.md item 6 of "Verification
|
||
and correctness" to make rustdoc cleanliness a project-wide
|
||
invariant rather than an iter-local cleanup.
|
||
|
||
First mission: `ailang-core` only. The foundation crate every
|
||
other crate depends on — biggest reader-leverage per diff. The
|
||
agent rewrote the crate root so a newcomer learns: what `core`
|
||
owns, where it sits in the pipeline (`core` → `check` →
|
||
`codegen` → `ail`), the central invariant (canonical JSON is
|
||
deterministic, hashes content-addressed, schema = `ailang/v0`),
|
||
and the entry points. Module roots in `ast.rs`, `canonical.rs`,
|
||
`hash.rs`, `pretty.rs`, `workspace.rs` got the same treatment.
|
||
Every `pub` struct / enum / variant / fn / const got a `///`
|
||
string. The Iter-13a additions (`TypeDef.vars`, `Type::Con.args`)
|
||
got an explicit backwards-compat note that points back to the
|
||
hash-stability regression test.
|
||
|
||
`# Examples` blocks landed where they shorten understanding
|
||
(`canonical::to_bytes`, `def_hash`, `Workspace`), all marked
|
||
`ignore` so the workspace doctest run stays cheap (3 ignored,
|
||
0 run, 0 failed). The two stale broken-link warnings in
|
||
`ailang-check` got prose-only fixes — out-of-scope for this
|
||
iter conceptually, but a one-line fix per file means rustdoc
|
||
is now globally clean.
|
||
|
||
**Findings reported by the docwriter** (judgement deferred to
|
||
me; none made it into the diff):
|
||
|
||
- `Term::Lam.param_tys` (JSON: `paramTypes`) is positionally
|
||
paired with `Term::Lam.params: Vec<String>`. Naming hints
|
||
at the convention but a cold reader has to deduce it. **Not
|
||
fixing**: renaming would touch the schema, breaks every hash.
|
||
The `///` string makes the convention explicit, which is
|
||
enough.
|
||
- `Type::Var` is overloaded: source-level rigid vars and
|
||
checker metavars (`$m<id>`) share the same variant. A reader
|
||
of `core` alone sees no hint of the metavar half — it's
|
||
documented in `ailang-check`'s lib doc instead. **Not fixing**:
|
||
splitting the variant would balloon the schema and
|
||
invalidate every hash. Acceptable as long as `check`'s lib
|
||
doc explains it (it does, post-13d).
|
||
- `Def::Type(TypeDef)` versus the type-expression enum `Type`
|
||
in the same module: name collision is real but unavoidable
|
||
without renaming `Type` (which would touch every crate). The
|
||
`///` strings now disambiguate at point of contact.
|
||
|
||
**Process note (orchestration).** Second iter where I worked
|
||
strictly through agents (after 13b). The docwriter brief was
|
||
written from the diagnostic in this conversation, not from a
|
||
DESIGN-doc design pass — there was no architecture decision to
|
||
make, just a discipline gap to close. That's the right shape
|
||
for a docwriter: low-judgement, repeatable, runs after every
|
||
iter that touches public surface. Cost ≈ 25 tool uses for ~390
|
||
LOC of doc additions across 9 files; smaller-grain than
|
||
`ailang-implementer` runs typically are.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(new). `cargo doc --no-deps`: 0 warnings (was 2). `cargo build
|
||
--workspace` green. `cargo test --workspace` green.
|
||
|
||
**Plan iteration 13e/13f (queued, not started).** Two natural
|
||
follow-ups for the docwriter:
|
||
|
||
13e. **`ailang-check` rustdoc**: type-checker is the next
|
||
biggest crate by `pub`-surface and the most algorithmically
|
||
dense. Rigid/metavar split, `Forall` instantiation, the
|
||
match-arm exhaustiveness logic, the Iter-13a substitution
|
||
machinery — all of it benefits more from prose explanation
|
||
than `core` did.
|
||
13f. **`ailang-codegen` + `ail` CLI rustdoc**: codegen is dense
|
||
but mechanical (mangling, ABI, block tracking); the CLI
|
||
is mostly clap derive macros. Lower payoff per LOC of doc
|
||
than 13e but rounds out the warning-free invariant across
|
||
the whole workspace.
|
||
|
||
After 13d there's no urgency on 13e/f — `cargo doc` is already
|
||
warning-free. They're "polish iterations to be slotted in
|
||
between feature iters when context budget is short". 14a
|
||
(`List a` rewrite) remains the next-feature default.
|
||
|
||
## Iter 13e — rustdoc polish for `ailang-check`
|
||
|
||
Second docwriter mission. Same mandate as 13d, applied to the
|
||
typechecker crate. `lib.rs` (1922 LOC) had a strong crate root
|
||
already — covers HM-with-effects, top-level forall, the
|
||
rigid-vs-metavar split, the `$m<id>` encoding — but its `pub`
|
||
surface (the `Env` struct, `CheckError` + 24 variants,
|
||
`CheckedModule`, `CtorRef`, the `check_module` entry point) was
|
||
mostly undocumented. `builtins.rs` had a one-line module
|
||
header and zero `///` strings on `EffectOpSig` or `install`.
|
||
`diagnostic.rs` had a strong module header (lists every stable
|
||
diagnostic code) but `Diagnostic`, `Severity`, and the
|
||
construction helpers were undocumented.
|
||
|
||
Agent added 188 LOC of pure rustdoc across the three files; no
|
||
non-doc lines changed (verified by filtering the diff). Each
|
||
`CheckError` variant now carries the AST term/type that
|
||
triggers it and the stable kebab-case `code` it maps to. `Env`
|
||
fields (`globals`, `effect_ops`, `types`, `module_globals`,
|
||
`current_module`) got individual `///` strings that name the
|
||
invariants — most importantly that `module_globals` includes
|
||
the current module, which the agent flagged as undocumented at
|
||
field level (now fixed). Crate-root prose got an upgraded
|
||
intra-doc link to `[`check_module`]`; `super::` reference in
|
||
`diagnostic.rs` rewritten to `crate::` (cosmetic, but the
|
||
canonical form).
|
||
|
||
**Findings reported** (judgement deferred to me):
|
||
|
||
- `Env` is `pub` with all-`pub` fields but `Env::new` is
|
||
private and there's no public builder — external callers can
|
||
only construct via field-by-field literal, which is fragile
|
||
if a field is added later. **Not fixing**: changing this is
|
||
an API decision, not a doc one. Worth raising next time we
|
||
touch the crate's public surface deliberately.
|
||
- `CheckError::CtorArity` and `CheckError::ArityMismatch` both
|
||
serialize to the public diagnostic code `arity-mismatch`. The
|
||
`///` strings now flag the collision per-variant; tooling
|
||
that consumes the diagnostic JSON sees only the merged code
|
||
and that's intentional from the iter-5b vintage. **Not
|
||
fixing.**
|
||
- `Diagnostic` and `Severity` are reachable both via the crate
|
||
re-export and via `crate::diagnostic::*` because the
|
||
`diagnostic` module is itself `pub`. Rustdoc renders both
|
||
pages; harmless but slightly noisy. **Not fixing**: the
|
||
re-export is the documented entry point and we don't want to
|
||
hide the module.
|
||
|
||
**Process note.** Same shape as 13d: I wrote a brief naming
|
||
the deficiencies (numbers of pub items, which files had thin
|
||
roots), the agent did the doc additions inside its mandate,
|
||
the orchestrator-side work was authoring DESIGN/JOURNAL and
|
||
verifying the diff. Cost ≈ 32 tool uses for 188 LOC of doc
|
||
across 3 files — denser than 13d (more sentences per pub item
|
||
because the typechecker invariants need explicit
|
||
articulation), but the per-LOC payoff for a future reader is
|
||
also higher.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(unchanged). `cargo doc --no-deps`: 0 warnings (was 0; the
|
||
agent introduced 4 transient broken-link warnings during the
|
||
work and resolved all of them before reporting done). `cargo
|
||
build --workspace` green. `cargo test --workspace` green.
|
||
|
||
**Plan 13f / 14a unchanged.** 13f (`ailang-codegen` + `ail`
|
||
CLI) is the natural next polish iter; 14a (`List a` rewrite of
|
||
`list_map`) remains the next-feature default. Auto-mode is on,
|
||
so I'll continue into 13f directly unless context budget
|
||
pressures a switch.
|
||
|
||
## Iter 13f — rustdoc polish for `ailang-codegen` + `ail` CLI
|
||
|
||
Combined docwriter mission. Codegen has a small public
|
||
surface (only 3 top-level `pub` items: `CodegenError` enum +
|
||
7 variants, `emit_ir`, `lower_workspace`); the CLI is a
|
||
binary with zero `pub` items, so the only useful rustdoc is
|
||
the module header. One agent run, both files.
|
||
|
||
What landed (148 LOC of doc additions, no non-doc lines
|
||
changed, verified by filtering the diff):
|
||
|
||
- `ailang-codegen` crate root got intra-doc-link upgrades
|
||
(`[`emit_ir`]`, `[`lower_workspace`]`,
|
||
`[`CodegenError::MissingEntryMain`]`) and a precondition
|
||
sentence — both entry points assume their input has already
|
||
passed the typechecker; codegen does not call the checker
|
||
itself.
|
||
- Every `CodegenError` variant got a `///` string naming the
|
||
AST term/condition that triggers it. Same shape as
|
||
`CheckError` post-13e, so the two error enums now read
|
||
similarly and a reader can grep across them. `Internal` is
|
||
flagged in its doc as a catch-all that covers ~30
|
||
invariant-violation sites.
|
||
- `emit_ir` and `lower_workspace` now make the
|
||
single-vs-multi-module split explicit and cross-link to
|
||
each other.
|
||
- `ail/main.rs` module header expanded from a 5-line stub to
|
||
a full subcommand list with one-liners (`manifest`,
|
||
`render`, `describe`, `deps`, `check`, `emit-ir`, `build`,
|
||
`run`, `builtins`, `diff`, `workspace`), the `clang`-on-PATH
|
||
prerequisite for `build`/`run`, the design-intent paragraph
|
||
about each subcommand being narrowly scoped for LLM
|
||
consumption (already partly there), and an explicit "no
|
||
`pub` items, `--help` text comes from clap" note for anyone
|
||
who lands here from rustdoc.
|
||
|
||
**Findings reported** (judgement deferred to me):
|
||
|
||
- `CodegenError::Internal(String)` is a single opaque catch-all
|
||
for ~30 distinct invariant-violation sites (mono-queue
|
||
desync, ctor-index miss, lambda-env shape, ...). Tests can
|
||
only substring-match on it. **Not fixing**: splitting is a
|
||
test-ergonomics decision, not a doc one. Worth raising the
|
||
next time codegen tests get a serious rewrite.
|
||
- `emit_ir` synthesises an internal `Workspace` with
|
||
`root_dir = "."`. No codegen path reads `root_dir` today, so
|
||
this is harmless; if a future feature reaches `root_dir`
|
||
from codegen, the assumption surfaces. **Not fixing**: the
|
||
agent flagged it correctly as "would change behaviour, out
|
||
of scope".
|
||
|
||
**Process note: brief drift caught by the agent.** I told the
|
||
docwriter the CLI had nine subcommands; it found eleven
|
||
(`Deps` and `Diff` were missing from my brief, which was
|
||
written off a `head -40` of the source). Agent silently
|
||
corrected and flagged the drift in its findings. Useful
|
||
counter-pressure to the orchestrator pattern: my survey was
|
||
sloppy and the agent did not propagate the sloppiness into
|
||
the doc. This is one of the things sub-agents are good at and
|
||
why I keep delegating even on small jobs.
|
||
|
||
**Tests:** 64/64 unit + e2e (unchanged), 3 ignored doctests
|
||
(unchanged). `cargo doc --no-deps`: 0 warnings. Also verified
|
||
under `RUSTDOCFLAGS='-D rustdoc::broken_intra_doc_links'` per
|
||
agent report. `cargo build --workspace` green. `cargo test
|
||
--workspace` green.
|
||
|
||
**Workspace-wide rustdoc invariant achieved.** All four
|
||
crates (`ailang-core`, `ailang-check`, `ailang-codegen`,
|
||
`ail`) now have:
|
||
- crate-root `//!` that names ownership, position in
|
||
pipeline, and entry points;
|
||
- module-root `//!` on every file with non-trivial content;
|
||
- `///` on every public item (struct, enum, variant, fn,
|
||
field, const) that names the contract, not just the type;
|
||
- intra-doc links wherever prose previously referred to
|
||
another item by name.
|
||
|
||
DESIGN.md item 6 ("rustdoc cleanliness") is now load-bearing
|
||
across the whole workspace, not just `core`. The
|
||
`ailang-docwriter` agent's job from here is **maintenance**:
|
||
run after any iter that adds public surface, not full sweeps.
|
||
|
||
**Next.** 14a is now unblocked: write a polymorphic `list_map`
|
||
that uses `List a` (Iter-13a parameterised ADTs) and `Maybe a`,
|
||
then extend an existing demo program (`hello_print` or one of
|
||
the dogfood sources) to call it end-to-end. That exercises
|
||
the parameterised-ADT pipeline through type-check, codegen,
|
||
and runtime — the missing piece in 13a/b/c was that the
|
||
feature shipped but no real program used it. If context
|
||
budget at the start of 14a is tight, alternative is 14b
|
||
(GC/arena scaffolding) or 14c (poly fn as value); both have
|
||
real design questions that need an orchestrator design pass
|
||
first, so 14a stays the default.
|
||
|
||
## Iter 14a — polymorphic `List a` end-to-end + monomorphisation bug fixed
|
||
|
||
The dogfood payoff for parameterised ADTs (13a/b/c). Prior
|
||
to this iter, the only programs exercising the feature were
|
||
`box.ail.json` (single `MkBox(42)` round-trip) and
|
||
`maybe_int.ail.json` (single `or_else` call). That is a thin
|
||
slice — neither program builds a recursive parameterised ADT
|
||
nor calls a polymorphic higher-order fn. 14a closes that gap
|
||
with `data List a` + `map : forall a b. ((a) -> b, List<a>) ->
|
||
List<b>` recursive, then prints the result.
|
||
|
||
Three-agent run (tester → debugger → no implementer needed):
|
||
|
||
1. **Tester** wrote `examples/list_map_poly.ail.json` (5 defs:
|
||
`List`, `inc`, `map`, `print_list`, `main`) and a new e2e
|
||
test `list_map_poly_inc_then_prints` that asserts stdout
|
||
`["2", "3", "4"]`. Typecheck passed, build crashed:
|
||
`internal: monomorphisation: var \`a\` bound to two
|
||
distinct types`. Tester correctly stopped — fixture
|
||
encodes the contract; bug is in the compiler — and
|
||
reported with a hypothesis (recursion / shared
|
||
substitution slot in `map`).
|
||
|
||
2. **Debugger** refuted the hypothesis with a non-recursive
|
||
repro (`Cons(7, Nil)` triggers the same crash without
|
||
`map` in the picture at all). Real cause was much smaller:
|
||
in `synth_arg_type` (codegen, ~line 2087) for `Term::Ctor`,
|
||
any type var of the parent ADT that the ctor's args
|
||
couldn't pin was filled with `Type::unit()` as a
|
||
placeholder. For nullary ctors of a parameterised ADT
|
||
(`Nil : List<a>`, `None : Maybe<a>`) that placeholder
|
||
leaked upward. Inside a parent like `Cons(Int, Nil) :
|
||
List<a>`, `unify_for_subst` would walk
|
||
`cref.ail_fields = [Var{a}, Con{List,[Var{a}]}]` against
|
||
`[Int, Con{List,[Unit]}]`, bind `a = Int` from the head,
|
||
then collide with `a = Unit` from the tail. The recursive
|
||
`map` fixture surfaced it because it is the first program
|
||
to nest a nullary ctor of a parameterised ADT inside a
|
||
parent ctor — the existing 13b regressions never did. The
|
||
tester's hypothesis was reasonable from the symptom but
|
||
wrong on mechanism; refuting it via a smaller repro is
|
||
exactly the discipline the debugger role is for.
|
||
|
||
3. **Fix** (`crates/ailang-codegen/src/lib.rs`, +30/-9 LOC,
|
||
no new variant, no API change):
|
||
- Replace `Type::unit()` placeholder with a synth-only
|
||
wildcard `Type::Var { name: "$u" }`. The `$u` prefix is
|
||
a reserved-namespace convention that mirrors the
|
||
checker's `$m<id>` for instantiation metavars — same
|
||
trick (source-level identifiers can't start with `$`),
|
||
same goal (extra semantics without schema change).
|
||
- `unify_for_subst` short-circuits on a `$u`-prefixed
|
||
**arg-side** var: accept without binding, let a sibling
|
||
arg pin the type var instead. Param-side semantics
|
||
untouched. `derive_substitution`'s "var not pinned"
|
||
check still fires for genuinely under-determined calls
|
||
(those would have failed typechecking, so codegen never
|
||
sees them, but the safety net stands).
|
||
|
||
**Tests:** 25/25 e2e (was 24, +1 new poly test). All five
|
||
named regressions stayed green: `list_map_doubles_then_prints`,
|
||
`parameterised_box_round_trip`, `parameterised_maybe_match`,
|
||
`polymorphic_id_at_int_and_bool`, `polymorphic_apply_with_fn_param`.
|
||
Insertion sort still green. `cargo doc --no-deps`: 0 warnings
|
||
(workspace invariant from 13d/e/f preserved). `cargo build
|
||
--workspace` green.
|
||
|
||
**Process note: tester→debugger→done in one iter.** No
|
||
implementer dispatch needed — the debugger's mandate covers
|
||
"propose **and apply** a minimally invasive fix" once the
|
||
diagnosis is solid. 30 LOC across two sites in one file is
|
||
inside the role's scope (the role doc says "stop and report"
|
||
only on >50 LOC across multiple files). The orchestrator-side
|
||
work was the design (the source program), the scoped briefs,
|
||
and verification. This is the cleanest agent-flow shape so
|
||
far: each agent did exactly its job, the tester's wrong
|
||
hypothesis didn't propagate because the debugger tested it,
|
||
and the orchestrator never wrote compiler code.
|
||
|
||
**Findings flagged** (judgement deferred, not fixed):
|
||
|
||
- `CodegenError::Internal` (the catch-all string variant the
|
||
docwriter flagged in 13f) is now used by one more invariant
|
||
— the `$u` short-circuit could in principle be reached by a
|
||
malformed input; it's currently silent because typecheck
|
||
rejects under-determined calls upstream. Worth splitting
|
||
into typed variants the next time codegen tests get a real
|
||
rewrite. (Same finding as 13f, now reinforced by another
|
||
use site.)
|
||
- The `$u` / `$m` reserved-prefix convention (`$u` for
|
||
codegen synth wildcards, `$m<id>` for checker metavars)
|
||
is undocumented as a project-wide naming rule. Two
|
||
prefixes is fine; if a third appears, this should be
|
||
promoted to a DESIGN.md note.
|
||
|
||
**Next.** Parameterised ADTs are now genuinely usable for
|
||
real programs. The standard library starter set (an
|
||
`examples/std_*` series with `List`, `Maybe`, `Either`,
|
||
basic combinators `length`, `filter`, `fold`, `concat`)
|
||
becomes worthwhile in a way it wasn't pre-14a — that's a
|
||
plausible 14b alternative, smaller than GC/arena, and would
|
||
itself surface more dogfood bugs. The original 14b
|
||
(GC/arena) and 14c (poly fn as value) remain on the queue
|
||
but have not had a design pass.
|
||
|
||
## Iter 14b — design pass for the authoring surface
|
||
|
||
User redirected the project at the iter boundary. I had been
|
||
about to write a stdlib in JSON-AST form; user pushed back: do
|
||
I really program *best* in JSON, given the language is supposed
|
||
to be the one I'm most accurate in? Honest answer was no — JSON
|
||
was rationalisation. The constraint added in this iter:
|
||
|
||
> "Die Syntax sollte gut formalisierbar sein, damit man auch
|
||
> einem fremden LLM eine Spec geben kann, die es dann fehlerfrei
|
||
> befolgt."
|
||
|
||
That single sentence ruled out about half the design space:
|
||
anything with operator precedence (precedence is the #1 thing
|
||
an LLM gets wrong when handed a spec, because it requires
|
||
building a parse-tree mental model rather than just following
|
||
production rules), anything with semantic indentation, anything
|
||
with maximal-munch lexing or context-sensitive reductions.
|
||
|
||
What remains is roughly: S-expressions, or dialects close to
|
||
S-expressions. Decision 6 in DESIGN.md captures the constraints
|
||
in priority order, sketches three candidate forms (A: tagged
|
||
S-expression, B: indented record-style, C: pretty-printer-as-
|
||
source), and picks (A) as the first attempt with explicit
|
||
rollback to (C) if (A) hurts authoring more than it helps.
|
||
|
||
**Key shape of (A):** every AST node has a unique head keyword.
|
||
No case-disambiguation rule (no "capitalised head means ctor").
|
||
Bare atoms in positional slots get their sort from the parent
|
||
slot (inside `(con NAME args...)` second-and-later positions
|
||
are types; inside `(app HEAD args...)` first position is a term;
|
||
etc.). To construct a value with a ctor, write
|
||
`(term-ctor TypeName CtorName args...)` explicitly. To match,
|
||
`(pat-ctor CtorName fields...)`. The capitalisation/case of an
|
||
identifier carries no semantic weight to the parser.
|
||
|
||
This rules out the silent-error class "I forgot to capitalise
|
||
`Cons` and it parsed as a function call" — which I had been
|
||
relying on case-conventions to prevent in the JSON form too,
|
||
but only by hand-discipline. With explicit tags the discipline
|
||
becomes a parse rule.
|
||
|
||
**Empirical test (this iter).** Hand-encoded three examples in
|
||
form (A) as `examples/*.ailx`:
|
||
|
||
```
|
||
hello.ailx — 5 LOC (was JSON: 36 pretty / 21 canonical)
|
||
box.ailx — 25 LOC (was JSON: 160 pretty / 88 canonical)
|
||
list_map_poly.ailx — 50 LOC (was JSON: 394 pretty / 230 canonical)
|
||
```
|
||
|
||
Roughly 4–8× line reduction, ~4× character reduction. Bigger
|
||
gains on bigger programs (overhead is proportional to AST
|
||
depth, not to program size, so the form scales well). All three
|
||
mapped unambiguously to AST nodes by inspection — no
|
||
ambiguities surfaced during writing that the spec didn't
|
||
already cover.
|
||
|
||
Two small lex/grammar issues I discovered while writing and
|
||
folded back into DESIGN.md before committing:
|
||
|
||
1. **Operator idents** like `+`, `==`, `<=`. Initial spec had a
|
||
word-shaped `[A-Za-z_]...` regex; that excluded operators.
|
||
Fix: lexer recognises only `(`/`)`/whitespace as delimiters.
|
||
Every other maximal token is classified by first character
|
||
(digit ⇒ integer, `"` ⇒ string, else ⇒ ident). `+`, `42`,
|
||
`io/print_int`, `==` all become single ident or integer
|
||
tokens with no special rule.
|
||
2. **Bool literals.** Bare `true`/`false` are reserved in term
|
||
context; outside term context they would be ill-formed
|
||
anyway. Unit is explicit: `(lit-unit)`.
|
||
|
||
**Files touched:**
|
||
- `docs/DESIGN.md` — Decision 6 added (~140 lines), with
|
||
constraints, three candidates, first-choice rationale, and
|
||
an implementation outline for Iter 14c.
|
||
- `examples/hello.ailx`, `examples/box.ailx`,
|
||
`examples/list_map_poly.ailx` — three design exhibits. Not
|
||
parseable yet (header comment says so).
|
||
- `docs/JOURNAL.md` — this entry.
|
||
|
||
**Tests:** None new. Existing 25/25 e2e + 3 ignored doctests
|
||
unchanged (this iter is paper, not code). `cargo doc --no-deps`
|
||
0 warnings (workspace invariant from 13d/e/f preserved).
|
||
|
||
**Process note: orchestration with explicit licence to be
|
||
wrong.** User said "du darfst auch ausprobieren und dich irren
|
||
(und es dann rückgängig machen). Wir wollen das beste Design
|
||
für den propagierten Zweck." That changes the cost model for
|
||
design iteration: I should optimise for *information per iter*,
|
||
not for *correctness on first commit*. So I committed form (A)
|
||
as a working hypothesis with a documented rollback path to
|
||
form (C), rather than design-by-committee until I was sure.
|
||
|
||
**Plan 14c (next).**
|
||
1. New crate `ailang-surface` with a small PEG parser → existing
|
||
`ailang-core::ast` types. No new AST nodes.
|
||
2. Round-trip test gate: every existing `examples/*.ail.json`
|
||
gets a sibling `*.ailx` written by hand or by an
|
||
AST→surface emitter; the test parses the surface, canonises
|
||
to JSON, and asserts hash-equivalence to the original. If a
|
||
single fixture loses its hash, the form does not ship.
|
||
3. CLI: `ail parse <file.ailx> -o <file.ail.json>`. Symmetric
|
||
to existing `ail render`.
|
||
4. If round-trip works for all current fixtures, mark form (A)
|
||
confirmed and start the stdlib in `.ailx` directly. If it
|
||
fails, document why in JOURNAL and try (C).
|
||
|
||
**Plan 14d (after 14c).** First stdlib module: `std_list.ailx`
|
||
with `length`, `filter`, `fold`, `concat`, `reverse`, `head`,
|
||
`tail`. Each combinator is a fresh test vector for codegen and
|
||
for the still-young parameterised-ADT pipeline. Written in the
|
||
new surface from day one — no JSON authoring of stdlib.
|
||
|
||
## Iter 14c — `ailang-surface` parser + pretty-printer ships
|
||
|
||
Implements Decision 6's form (A). New crate `ailang-surface`
|
||
(~1843 LOC across `lex.rs`, `parse.rs`, `print.rs`, lib root,
|
||
plus tests) ships as a strictly additive producer of
|
||
`ailang-core::ast::Module` values. `ailang-check` and
|
||
`ailang-codegen` were not modified — projection-agnostic, as
|
||
the architectural pin requires.
|
||
|
||
**Implementer dispatch went clean.** Brief gave the EBNF, the
|
||
fixtures to round-trip, the lexer rule, and the architectural
|
||
constraints. Implementer hand-wrote a recursive-descent parser
|
||
(one Rust fn per EBNF production), a deterministic pretty-
|
||
printer, an integration test that runs the round-trip gate on
|
||
every fixture, and the `ail parse` CLI subcommand. No
|
||
parser-combinator dependency. No AST-shape changes.
|
||
|
||
**Two AST-driven form refinements** that were not in the 14b
|
||
sketch (both folded into DESIGN.md Decision 6 by the
|
||
implementer before commit):
|
||
|
||
1. `lam-term` had to carry `param_tys`, `ret_ty`, and
|
||
`effects` because the AST's `Term::Lam` stores parallel
|
||
typed-param data. Production became `(lam (params (typed
|
||
x Int) ...) (ret T) (effects ...) (body ...))`. Same
|
||
shape-style as the rest of the form; no new lex rules.
|
||
2. `import-clause` had to admit `Option<String>` aliases.
|
||
Production became `(import name (as alias)?)` with `as`
|
||
as a bare ident token. No new lex rules.
|
||
|
||
The 14b 30-production budget held: implementer reports ~28
|
||
named productions in the parser. Constraint 1 (formalisable
|
||
for foreign LLM) intact.
|
||
|
||
**Verification gates, all green:**
|
||
|
||
- `cargo build --workspace`: 0 warnings, finished.
|
||
- `cargo test --workspace`: 76 tests pass (was 64; +9 unit
|
||
tests in `ailang-surface`, +2 integration tests in
|
||
`tests/round_trip.rs`, +1 e2e regression preserved). All
|
||
17 `examples/*.ail.json` fixtures round-trip
|
||
byte-identical through `print → parse → canonical JSON`;
|
||
3 hand-written `.ailx` exhibits parse to canonical JSON
|
||
byte-identical to their `.ail.json` siblings.
|
||
- `cargo doc --no-deps`: 0 warnings (workspace invariant
|
||
from 13d/e/f preserved; new crate's rustdoc landed
|
||
correctly with crate-root `//!` plus all `pub` items
|
||
documented).
|
||
|
||
**Manual smoke test (orchestrator-side after agent reported
|
||
done):** `ail parse <file.ailx> -o <file.ail.json>` followed
|
||
by `ail run <file.ail.json>` for all three exhibits:
|
||
|
||
```
|
||
hello.ailx → "Hello, AILang."
|
||
box.ailx → "42"
|
||
list_map_poly.ailx → "2\n3\n4"
|
||
```
|
||
|
||
End-to-end pipeline form (A) → AST → typecheck → codegen →
|
||
clang → binary works on all three.
|
||
|
||
**Important non-issue.** `examples/*.ail.json` fixtures on
|
||
disk are hand-formatted JSON with non-canonical key order;
|
||
the parser produces canonical (lex-sorted) JSON. Diff at
|
||
the file-byte level is not zero. Diff at the canonical-byte
|
||
level is zero — which is the only thing that matters per
|
||
Decision 1, since hashing uses canonical form. The
|
||
round-trip test gates on canonical bytes, not file bytes.
|
||
This is correct behaviour; flagged here so a future reader
|
||
who runs `diff` doesn't think the surface is broken.
|
||
|
||
**Files created:**
|
||
|
||
- `crates/ailang-surface/Cargo.toml`
|
||
- `crates/ailang-surface/src/lib.rs` (~40 LOC, rustdoc heavy)
|
||
- `crates/ailang-surface/src/lex.rs` (~264 LOC)
|
||
- `crates/ailang-surface/src/parse.rs` (~1041 LOC, one fn
|
||
per production)
|
||
- `crates/ailang-surface/src/print.rs` (~371 LOC)
|
||
- `crates/ailang-surface/tests/round_trip.rs` (~128 LOC)
|
||
|
||
**Files modified:**
|
||
|
||
- `Cargo.toml` (workspace) — `ailang-surface` member +
|
||
workspace dep.
|
||
- `Cargo.lock` — refresh.
|
||
- `crates/ail/Cargo.toml` — `ailang-surface` dep.
|
||
- `crates/ail/src/main.rs` — new `Parse { path, output }`
|
||
subcommand (~36 LOC).
|
||
- `docs/DESIGN.md` — form refinements appendix to
|
||
Decision 6.
|
||
- `examples/list_map_poly.ailx` — implementer added doc
|
||
strings to match the JSON original (was a 14b design
|
||
exhibit, not byte-aligned).
|
||
|
||
**Known debt:** none reported. `ailang-check` /
|
||
`ailang-codegen` untouched per the architectural pin.
|
||
|
||
**Plan 14d (next, queued).** With form (A) now ergonomic
|
||
and round-trip-verified, the std-lib path opens up. First
|
||
target: `examples/std/std_list.ailx` with `length`,
|
||
`filter`, `fold` (left and right), `concat`, `reverse`,
|
||
`head`, `tail`. Each combinator a fresh test vector for
|
||
the parameterised-ADT pipeline (which 14a opened end-to-
|
||
end). Authored in form (A); the resulting `.ail.json` is
|
||
what tests consume. If 14d surfaces more codegen bugs in
|
||
the parameterised-ADT path (it likely will — 14a found
|
||
one already), debugger handles them inline.
|
||
|
||
The 14b/c form-A hypothesis has held under the empirical
|
||
test of round-tripping every existing fixture. The
|
||
documented rollback to form (C) is now off the table for
|
||
this iter cycle, though the architectural pin keeps it
|
||
open for future replacement of `ailang-surface` should
|
||
the form prove inadequate at stdlib scale.
|
||
|
||
|
||
|