41d406bcbb
Reconsidered 14d's removal of Term::If. The decision was wrong.
"No redundancies" requires judgment; reducibility (if -> match)
is not redundancy in the strong sense. Term::If is a primitive
control-flow shape; bool branching is the second most common
shape after sequencing, and removing it cost 3x tokens on every
branch site (`(if c a b)` 4 tokens vs the match-on-Bool form
12 tokens).
Meta-pattern fixed: I had been treating user observations as
directives. User said "if is a subset of match"; I jumped to
remove it citing CLAUDE.md, with no independent conviction.
The leak appeared in 14f's JOURNAL prose ("three lines for what
if used to do in one"), which read as regret. Two feedback
memories saved (memory/feedback_user_suggestions_not_directives,
memory/feedback_no_nostalgia_for_removed_features) to head this
off.
Implementation: mechanical reverse-application of 14d's diff at
every site (AST, check including the 14e tail-position arm,
codegen 4 sites, surface parser/printer, pretty, CLI walker,
e2e test mutation). Removed lower_bool_match helper — it existed
only because 14d's migration shape needed codegen for non-ptr
match scrutinees; with Term::If back, match-on-Bool returns to
its pre-14d unsupported state. Three fixtures (sum, sort, max3)
restored to pre-14d shape. gc_stress (added in 14f) also
migrated back to (if ...) since it was authored under the wrong
constraint.
14e (musttail) and 14f (GC_malloc) verified intact in IR.
Hashes restored to pre-14d values:
- sum.sum: db33f57cb329935e
- sort.insert: 697fcb9f30f8633a
- max3.max: 65c45d6a45dd0a72
- max3.max3: 624b14429bf302f5
All other defs across all 18 fixtures keep their post-14f
hashes. Tests 80/80 green; cargo doc 0 warnings. LOC delta
+265/-295 net -30.
DESIGN.md Decision 7 preserved with a "Status: REVERTED" header
for audit trail. Form-(A) `if-term` production restored.
Plan: back to 15a (std_maybe stdlib module).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
805 lines
34 KiB
Markdown
805 lines
34 KiB
Markdown
# AILang — design decisions
|
||
|
||
This document records the core decisions for AILang. It is my contract with
|
||
myself across future iterations. Prefer cuts over growth.
|
||
|
||
## Goal
|
||
|
||
AILang is a programming language for LLM authors. It compiles to LLVM IR.
|
||
Performance: native, no GC for the MVP.
|
||
|
||
Optimised for:
|
||
|
||
- **Machine readability** over human ergonomics. The source is structured.
|
||
- **Local reasoning.** Every definition carries its full type and effects.
|
||
- **Provability.** Pure core language, explicit effects, optional refinements.
|
||
- **Robustness against hallucinations.** Symbols are hashable; tools can verify
|
||
existence without spending context window.
|
||
|
||
## Project ecosystem
|
||
|
||
AILang is not just a language but an ecosystem. The language on its own is
|
||
only valuable when its surroundings make it usable, checkable, and
|
||
extensible for its target user (LLM authors). The repo therefore contains
|
||
several equally important components — none of them optional, all of them
|
||
evolving in lockstep with the language:
|
||
|
||
- **Language core** (`crates/ailang-core`, `crates/ailang-check`,
|
||
`crates/ailang-codegen`): AST, type system, codegen.
|
||
- **CLI** (`crates/ail`): toolchain for tooling consumers — `manifest`,
|
||
`describe`, `deps`, `check`, `build`, etc., preferably with `--json` for
|
||
machine consumption.
|
||
- **Examples** (`examples/`): canonical `.ail.json` programs. They are
|
||
specification anchors, not demos — the E2E suite hangs off them.
|
||
- **Agents** (`agents/`): specialised sub-prompts (implementer,
|
||
architect, tester, debugger) that form the project's own LLM tooling.
|
||
They are a versioned part of the repo. See `agents/README.md`.
|
||
- **Docs** (`docs/`): DESIGN.md (what and why), JOURNAL.md (history).
|
||
- **Tests**: unit tests per crate plus E2E in `crates/ail/tests/e2e.rs`. Every
|
||
new compiler path needs a test, otherwise the feature does not count as done.
|
||
|
||
When the language grows, these components grow with it. New tools that
|
||
strengthen the LLM tooling (e.g. `ail diff`, IR snapshot diffs, new agents)
|
||
explicitly belong in the ecosystem inventory of this section and are added
|
||
here as soon as they are established.
|
||
|
||
## Project language: English
|
||
|
||
All in-tree content is written in English: source code (identifiers,
|
||
comments, string literals, CLI help), design documents, the journal, agent
|
||
prompts, READMEs, commit messages, examples, and `CLAUDE.md`. The live
|
||
conversation between user and me stays German for ergonomic reasons;
|
||
everything that lands in git is English. This keeps diffs and tooling output
|
||
uniform and matches the audience for AILang (LLM authors), for whom English
|
||
is the default.
|
||
|
||
## Decision 1: source = data, not text
|
||
|
||
A module is a JSON object with a fixed schema. There is no parser for
|
||
free-form text. Typos in identifiers turn into hash-lookup errors that the
|
||
compiler proposes a fix for directly.
|
||
|
||
A textual form exists (`.ail`, S-expression-like), but only as a
|
||
bidirectional projection of the JSON form. It is intended for human reviews
|
||
and diffs.
|
||
|
||
**Canonical format:** `.ail.json` with deterministic key order.
|
||
|
||
## Decision 2: content-addressed definitions
|
||
|
||
Every top-level definition has a `hash` value (BLAKE3 over canonical JSON
|
||
without the `hash` field itself). References between definitions go primarily
|
||
by name — names are for readability. The hash is the canonical identity.
|
||
|
||
Advantages:
|
||
|
||
- Refactoring by adding new defs, not by in-place change. Old versions stay
|
||
callable until manually removed.
|
||
- Caching of typecheck results and codegen per hash.
|
||
- Diffs show exactly which def has changed.
|
||
|
||
## Decision 3: pure core language + algebraic effects
|
||
|
||
The default is total, pure functions. Effects are declared as a set in the
|
||
function type: `(Int) -> Int ![IO]`. The effect set is row-polymorphic
|
||
(`![IO | r]`). In the MVP only the effects `IO` and `Diverge` (for infinite
|
||
loops) are wired up.
|
||
|
||
This is the most important LLM property: when I read a function, I can trust
|
||
its signature without reading the body.
|
||
|
||
## Decision 4: Hindley-Milner + optional refinements
|
||
|
||
MVP: HM with let-polymorphism. All types are inferable, but at the top level
|
||
they must always be explicitly annotated (for local reasoning).
|
||
|
||
Later: refinement annotations that escalate to SMT. `(i: Int | i >= 0)`. They
|
||
are reserved in the AST from the start, but in the MVP they are simply passed
|
||
through as opaque strings.
|
||
|
||
## Decision 5: emit LLVM IR as text
|
||
|
||
Instead of `inkwell` or `llvm-sys`: AILang produces `.ll` files as strings
|
||
and hands them to `clang` for linking.
|
||
|
||
Rationale:
|
||
|
||
- The LLVM IR text syntax is largely stable across versions.
|
||
- No build dependency on a specific libllvm version.
|
||
- Generated code is trivially inspectable, which makes debugging much easier.
|
||
- An LLM can read the generated IR directly, which is harder with opaque
|
||
library calls.
|
||
|
||
Trade-off: no inline optimisations through the LLVM API. We rely on
|
||
`clang -O2` as the standard pipeline.
|
||
|
||
## Decision 6: authoring surface (Iter 14b — WIP)
|
||
|
||
**Status: design pass in progress.** Reading without skipping the JOURNAL
|
||
will leave this section ahead of the implementation.
|
||
|
||
### Why this is opening up
|
||
|
||
Iters 1 through 14a authored everything as raw `*.ail.json`. That worked
|
||
for 17 fixture files (each ≤ 60 LOC of JSON) but does not scale. Two
|
||
breaking signals:
|
||
|
||
1. The token-economy cost of the JSON-AST is massive: a single integer
|
||
literal `1` is encoded as `{"t":"lit","lit":{"kind":"int","value":1}}`
|
||
— ~38 tokens of structural overhead per bit of semantics. For a stdlib
|
||
in the 200–500 def range this displaces real attention budget.
|
||
2. JSON-AST authoring exposes a class of errors (wrong field names,
|
||
silently-accepted extra fields under `#[serde(default)]`,
|
||
inconsistent ctor casing) that surface only at load time. The
|
||
schema is correct-by-construction in storage but **error-prone in
|
||
authoring**.
|
||
|
||
Decision 1 anticipated this: it says a textual form exists "as a
|
||
bidirectional projection of the JSON form." The pretty-printer already
|
||
emits S-expression-style text (see `crates/ailang-core/src/pretty.rs`).
|
||
What is missing is the inverse direction — text → AST. Decision 6
|
||
adds that inverse as **one** authoring projection alongside the
|
||
existing pretty-printer; the JSON-AST remains the source of truth.
|
||
|
||
### Architectural pin: data structure is the source of truth
|
||
|
||
The textual surface is **not** a replacement for the JSON-AST. It is
|
||
one projection among potentially many. Concretely:
|
||
|
||
- The JSON-AST keeps its role as the canonical, hashable, content-
|
||
addressed representation of a module. All hashing, content-
|
||
addressing, cross-module references, and typecheck/codegen input
|
||
flow through the JSON-AST. **No new hashable form is introduced.**
|
||
- The textual surface (form A, this Decision) is the **AI authoring
|
||
projection**: optimised for me producing programs token-efficiently
|
||
and for foreign LLMs producing programs from a spec. It is not
|
||
optimised for human authors and does not need to be human-pleasant.
|
||
- Future projections are explicitly anticipated: a visual / graphical
|
||
front-end is a plausible second projection for human review and
|
||
inspection (display being the one case where non-AI eyes matter).
|
||
The architecture leaves room: any producer of well-formed
|
||
`ailang-core::ast::Module` values is a valid front-end.
|
||
- **No human is expected to author AILang seriously.** Authoring is
|
||
AI work. Display and verification, by contrast, are concerns where
|
||
human-facing alternatives may be useful — and which can therefore
|
||
layer their own projections on top of the same AST without
|
||
touching the surface or the core.
|
||
|
||
In code terms: `ailang-core` owns the AST. `ailang-surface` (new in
|
||
Iter 14c) is one producer/consumer pair: text-form-A → AST → text-
|
||
form-A. A hypothetical `ailang-visual` would be a different producer
|
||
of the same AST. `ailang-check` and `ailang-codegen` consume only
|
||
the AST and remain projection-agnostic.
|
||
|
||
### Constraints (hard, in priority order)
|
||
|
||
1. **Formalizable for a foreign LLM.** The grammar must fit in an
|
||
EBNF/PEG spec of ≤ 30 productions. A model that has never seen
|
||
AILang must be able to read the spec and produce conforming source
|
||
zero-shot. Rules out: precedence between binary operators,
|
||
semantic indentation, maximal-munch lexing, context-sensitive
|
||
reductions.
|
||
2. **AST-isomorphic.** Every surface form maps to exactly one AST
|
||
shape. Round-trip surface → AST → canonical JSON → AST → surface
|
||
is the identity (modulo formatting). Hashes computed via the
|
||
round-tripped JSON must equal hashes of the same module written
|
||
directly in JSON.
|
||
3. **No external symbols.** ASCII only. No Greek (`∀`), no arrows
|
||
(`→`), no subscripts. Reasoning: I substitute mojibake for
|
||
non-ASCII characters under context pressure; foreign LLMs vary
|
||
in how they tokenize Unicode.
|
||
4. **No precedence.** Either everything is parenthesized, or there
|
||
are no infix operators. Prefer the latter — `add(x, 1)` over
|
||
`x + 1`. Removes a fail mode for both me and foreign LLMs.
|
||
5. **No semantic indentation.** Block structure expressed by paired
|
||
delimiters or terminator tokens. Indentation is informational
|
||
only; the parser ignores it.
|
||
6. **One construct per token-list.** Every AST node corresponds to
|
||
exactly one parenthesized form (or atom). No "sometimes you can
|
||
omit the parens" rules.
|
||
7. **AST surface stays frozen.** The surface adapts to the AST, not
|
||
the other way around. We do not change the JSON schema or
|
||
invalidate hashes to make the surface prettier.
|
||
|
||
### Candidate notations (same `map` encoded in each)
|
||
|
||
The reference target — the polymorphic `map` from `examples/list_map_poly.ail.json`:
|
||
|
||
```
|
||
data List a where Nil | Cons a (List a)
|
||
fn map : forall a b. ((a) -> b, List a) -> List b
|
||
= \f xs. match xs of Nil -> Nil
|
||
| Cons h t -> Cons(f(h), map(f, t))
|
||
```
|
||
|
||
#### (A) S-expression with fully-tagged AST nodes
|
||
|
||
```
|
||
(module list_map_poly
|
||
|
||
(data List (vars a)
|
||
(ctor Nil)
|
||
(ctor Cons a (con List a)))
|
||
|
||
(fn inc
|
||
(type (fn-type (params (con Int)) (ret (con Int))))
|
||
(params x)
|
||
(body (app + x 1)))
|
||
|
||
(fn map
|
||
(type
|
||
(forall (vars a b)
|
||
(fn-type
|
||
(params (fn-type (params a) (ret b)) (con List a))
|
||
(ret (con List b)))))
|
||
(params f xs)
|
||
(body
|
||
(match xs
|
||
(case (pat-ctor Nil) (term-ctor List Nil))
|
||
(case (pat-ctor Cons h t)
|
||
(term-ctor List Cons
|
||
(app f h)
|
||
(app map f t)))))))
|
||
```
|
||
|
||
Grammar core (3-rule lexical layer + ~25 named-form productions):
|
||
```
|
||
sexpr ::= atom | "(" sexpr* ")"
|
||
atom ::= integer | string | ident
|
||
ident ::= any maximal non-whitespace, non-paren run that is not
|
||
a recognised integer or string literal.
|
||
```
|
||
|
||
The lexer recognises one delimiter (`(` / `)`) and whitespace.
|
||
Every other maximal token is classified post-hoc:
|
||
- All-digit run with optional leading `-` → integer atom.
|
||
- `"`-delimited run → string atom.
|
||
- Otherwise → ident.
|
||
|
||
Consequence: operators like `+`, `==`, `<=`, `**`, qualified
|
||
names like `io/print_int`, and cross-module references like
|
||
`std_list.map` are all single ident tokens with no special lex
|
||
rule. The only reserved tokens are `(`, `)`, and whitespace.
|
||
Bool literals (`true`, `false`) and unit (`(lit-unit)`) are
|
||
disambiguated by parser context, not by lex.
|
||
|
||
Every AST node form has a unique head keyword (`module`, `data`,
|
||
`fn`, `forall`, `fn-type`, `con`, `var`, `app`, `lam`, `match`,
|
||
`case`, `pat-ctor`, `term-ctor`, `do`, `seq`, ...). A bare atom in
|
||
a positional slot (e.g. inside `(con List a)` second position) is
|
||
a name reference whose **sort** is determined by the parent slot:
|
||
|
||
- inside `(con NAME args...)` second-and-later positions → type
|
||
expression. Bare atom there ⇒ `Type::Var { name }`.
|
||
- inside `(app HEAD args...)` first position ⇒ `Term::Var`.
|
||
- inside `(pat-ctor CTOR fields...)` field positions ⇒
|
||
`Pattern::Var`.
|
||
- inside `(case PAT BODY)` second position ⇒ term.
|
||
|
||
There is **no lexical case rule**. To construct a value with a
|
||
ctor, write `(term-ctor TypeName CtorName args...)`. To match
|
||
against one, write `(pat-ctor CtorName fields...)`. Capitalised
|
||
identifiers carry no special meaning to the parser. This rules
|
||
out a class of silent errors ("I forgot to capitalise `Cons` and
|
||
it parsed as a function call").
|
||
|
||
**Pros:** smallest formal grammar of any candidate (the lexical
|
||
core is 3 rules; the named-form productions are uniform — every
|
||
node a tagged list). Foreign-LLM bar lowest. Round-trip with the
|
||
existing pretty-printer is a refactor of `pretty.rs` to emit this
|
||
tagged form, plus a new parser.
|
||
|
||
**Cons:** paren density is high. `(forall (vars a b) (fn-type
|
||
...))` has more visual nesting than the current pretty-printer's
|
||
`forall a. (...) -> ...`. Verbosity is ~2× JSON for the same node
|
||
when measured in characters, but ~8× shorter in lines (the
|
||
existing JSON `box.ail.json` of 160 lines becomes ~20 lines in
|
||
this form).
|
||
|
||
#### (B) Indented record-style with explicit terminators
|
||
|
||
```
|
||
module std_list
|
||
|
||
data List(a):
|
||
Nil
|
||
Cons(a, List(a))
|
||
end
|
||
|
||
fn map:
|
||
type: forall a b. fn(fn(a) -> b, List(a)) -> List(b)
|
||
params: f, xs
|
||
body:
|
||
match xs:
|
||
Nil => Nil
|
||
Cons(h, t) => Cons(f(h), map(f, t))
|
||
end
|
||
end
|
||
```
|
||
|
||
Grammar core (~20–30 productions): module-level (def/data/end), type
|
||
sub-grammar (forall, fn, con, var), term sub-grammar (lam, match,
|
||
ctor, app, lit, var, seq), pattern sub-grammar.
|
||
|
||
**Pros:** higher information density per line, closer to mainstream
|
||
ML/Haskell shape. **Cons:** four sub-grammars instead of one.
|
||
`forall a b. fn(...)` keeps a pseudo-precedence (`->` binds tighter
|
||
than the outer `fn(...)` wrapper). Foreign-LLM bar higher.
|
||
|
||
#### (C) Pretty-printer-as-source
|
||
|
||
Use exactly the format `pretty::module` already emits, plus a parser
|
||
that accepts it. The existing pretty-printer's quirks (`::` for
|
||
type-of, `[params]` for fn-params, `<a>` for type-args, `forall a. ...`,
|
||
`!IO`, `()` ambiguous between unit-arg-list and empty-form) become
|
||
the spec.
|
||
|
||
**Pros:** zero churn — the existing pretty-printer is already the
|
||
spec; only the inverse is missing. Round-trip is the identity by
|
||
construction. **Cons:** the existing format mixes four mini-dialects
|
||
(s-expr at term level, ML-shape at type level, square brackets for
|
||
params, `<>` for type args). Formalising it crisply is harder than
|
||
designing a uniform form from scratch.
|
||
|
||
### First choice and rollback plan
|
||
|
||
**Try (A) first.** Reasoning: constraint 1 (formalizable) outweighs
|
||
constraint readability. (A) has a 3-rule core grammar with one
|
||
lexical disambiguation rule. (B) doubles the rule count and
|
||
re-introduces a soft form of precedence (`->` inside `forall`).
|
||
(C) is tempting because it is zero-design but the resulting spec is
|
||
visibly heterogeneous, which is exactly what constraint 1 was meant
|
||
to rule out.
|
||
|
||
If implementing (A) reveals that paren density actively hurts my
|
||
authoring (measurable: I make more wrong-paren errors than the
|
||
JSON-AST shape produced before), roll back and try (C). (B) stays
|
||
on the shelf for a future iter only if both fail.
|
||
|
||
### Implementation outline (Iter 14c onwards, not done in 14b)
|
||
|
||
- `crates/ailang-surface` — new crate. **Strictly additive.** PEG
|
||
parser produces existing `ailang-core::ast` types. No new AST
|
||
nodes, no schema changes, no new hashable form. Pretty-printer
|
||
for form (A) lives here too (the round-trip is the contract).
|
||
- `ailang-check`, `ailang-codegen` are **not modified**. They
|
||
continue to consume `ailang-core::ast::Module` values regardless
|
||
of which projection produced them.
|
||
- Round-trip test: for every `examples/*.ail.json`, parse the
|
||
corresponding hand-written `*.ailx`, canonicalise, and assert
|
||
hash-equivalence to the original. Hash equivalence is the truth
|
||
check; the surface ships only if every fixture round-trips
|
||
identically.
|
||
- CLI: `ail parse <file.ailx> -o <file.ail.json>`. Symmetric to
|
||
existing `ail render`. **`.ail.json` remains a first-class input**
|
||
to every existing subcommand; the parser is a producer, not a
|
||
gatekeeper.
|
||
- Iter 14d: stdlib (`std_list`, `std_maybe`, ...) authored in
|
||
form (A) from day one **because that is the AI authoring
|
||
projection**, not because JSON authoring is forbidden. The
|
||
resulting `.ail.json` is what tests and downstream tools see.
|
||
|
||
### What this Decision deliberately does not do
|
||
|
||
- It does not promote form (A) to the source of truth. Form (A) is
|
||
one projection; the JSON-AST stays canonical.
|
||
- It does not foreclose visual or graphical front-ends. The crate
|
||
layout (`core` owns AST; `surface`/`visual`/... are siblings)
|
||
reserves that lane.
|
||
- It does not remove `.ail.json` as input. Every existing
|
||
CLI subcommand (`check`, `render`, `describe`, `emit-ir`,
|
||
`build`, `run`, `manifest`, `deps`, `diff`, `workspace`,
|
||
`builtins`) keeps its current `.ail.json` interface.
|
||
|
||
### Form refinements during Iter 14c implementation
|
||
|
||
Two productions in the original 14b sketch had to be widened
|
||
during implementation to round-trip the existing AST faithfully.
|
||
Captured here for the spec record:
|
||
|
||
1. **`lam-term` carries types and effects.** The AST's `Term::Lam`
|
||
stores parallel `params`, `param_tys`, `ret_ty`, and `effects`
|
||
fields. The 14b sketch had only names. The implemented form is
|
||
|
||
```
|
||
lam-term ::= "(" "lam" "(" "params" typed-param* ")"
|
||
"(" "ret" type ")"
|
||
effects-clause? body-attr ")"
|
||
typed-param ::= "(" "typed" ident type ")"
|
||
```
|
||
|
||
This keeps the no-precedence / one-construct-per-token-list
|
||
invariants and adds no new lexical rules.
|
||
|
||
2. **`import-clause` admits an optional alias.** The AST's
|
||
`Import.alias` is `Option<String>`; the original sketch only
|
||
supported the `None` case. The implemented form is
|
||
|
||
```
|
||
import-clause ::= "(" "import" ident ("as" ident)? ")"
|
||
```
|
||
|
||
`as` is a bare ident token in this position; no special lexical
|
||
rule is needed.
|
||
|
||
Neither change extends the grammar's rule budget meaningfully:
|
||
the 30-production ceiling of constraint 1 is intact (the parser
|
||
implements ~28 named productions). All 17 `examples/*.ail.json`
|
||
fixtures round-trip identically through `print → parse → canonical
|
||
JSON`; the three hand-written `.ailx` exhibits parse to canonical
|
||
JSON identical to their corresponding `.ail.json` files.
|
||
|
||
3. **Tail-call surface (Iter 14e).** Decision 8 ships two new
|
||
productions, both positional analogues of their non-tail
|
||
counterparts. Their result terms set `Term::App.tail = true` /
|
||
`Term::Do.tail = true`; the typechecker's `verify_tail_positions`
|
||
pass enforces that the marker is only used in tail position.
|
||
|
||
```
|
||
tail-app-term ::= "(" "tail-app" term term+ ")"
|
||
tail-do-term ::= "(" "tail-do" ident term* ")"
|
||
```
|
||
|
||
Production count after Iter 14e: ~30, still inside the 30-rule
|
||
constraint-1 budget. No new lexical rule (`tail-app` / `tail-do`
|
||
are bare ident tokens; no special casing).
|
||
|
||
## Decision 7: redundancy removal — `Term::If` is not a primitive
|
||
|
||
**Status: REVERTED in Iter 14g.** This decision was made on shaky grounds —
|
||
applying CLAUDE.md's "no redundancies" rule to a case that turned out to
|
||
be primitive control flow, not redundancy. The post-removal match-on-Bool
|
||
form (3× the tokens, asymmetric `pat-wild` for the false case) was
|
||
worse for token economy and worse for the natural shape of the language.
|
||
`Term::If` is restored. The text below is preserved for the audit trail.
|
||
|
||
`Term::If { cond, then, else_ }` is semantically a subset of
|
||
`Term::Match` on `Bool`. Per CLAUDE.md the language must contain no
|
||
redundancies; two AST nodes for the same operation produces an
|
||
authoring decision with no semantic content and an extra codegen
|
||
path. Iter 14d removes `Term::If`. Migration shape on the JSON side:
|
||
|
||
{"t":"if","cond":C,"then":A,"else":B}
|
||
→
|
||
{"t":"match","scrutinee":C,
|
||
"arms":[
|
||
{"pat":{"p":"lit","lit":{"kind":"bool","value":true}},"body":A},
|
||
{"pat":{"p":"wild"},"body":B}]}
|
||
|
||
The wildcard arm satisfies the typechecker's
|
||
`primitive-needs-wildcard` rule. A future iter may upgrade the
|
||
exhaustiveness check to recognise the `true`+`false` arm pair as
|
||
covering Bool without a wildcard; until then, wildcard is the
|
||
canonical migration target.
|
||
|
||
No schema version bump (no third-party consumes `ailang/v0`).
|
||
Hash invalidation for the three migrated fixtures (`sum`, `sort`,
|
||
`max3`) is intentional; the new hashes become the new identity.
|
||
|
||
## Decision 8: explicit, verified tail calls
|
||
|
||
For an LLM author, recursion is the natural iteration form
|
||
(`\n. if n == 0 then () else loop(n-1)` is what I reach for, not
|
||
a `for`-loop). Without a tail-call guarantee, every recursive
|
||
program has a silent stack-depth ceiling that no compile-time
|
||
diagnostic warns about. That is exactly the class of correctness
|
||
hazard the language exists to eliminate.
|
||
|
||
Solution: explicit, verified tail calls.
|
||
|
||
- **AST.** `Term::App { fn, args, tail: bool }` and
|
||
`Term::Do { op, args, tail: bool }` gain a `tail` flag,
|
||
serde-defaulting to `false` so existing fixtures load with
|
||
`tail: false` and their hashes stay bit-identical.
|
||
- **Typecheck.** A new pass `verify_tail_positions(fn_body)`
|
||
runs after the main type-check. It walks the body with an
|
||
`is_tail_context: bool` threaded down. The flag is `true` at
|
||
the start, `true` for the body of every `Term::Match` arm,
|
||
`true` for the right operand of `Term::Seq`, `true` for the
|
||
body of `Term::Let`, `true` for the body of `Term::Lam` (each
|
||
Lam opens its own tail scope). The flag is `false` for: args
|
||
of any `App`/`Do`/`Ctor`, scrutinee of `Match`, left of
|
||
`Seq`, condition of `Let`-bound expression. When the walker
|
||
visits an `App { tail: true }` or `Do { tail: true }`, the
|
||
flag must be `true` at that visit; otherwise emit diagnostic
|
||
`tail-call-not-in-tail-position`.
|
||
- **Codegen.** Emit `musttail call` (LLVM IR) for marked calls
|
||
instead of plain `call`. LLVM rejects at IR-verification
|
||
time if the call cannot physically be a tail call (calling
|
||
convention mismatch, signature divergence, etc.). The reject
|
||
surfaces as a hard build error, not a silent runtime
|
||
surprise.
|
||
- **Form (A).** Two new keywords: `tail-app`, `tail-do`.
|
||
Productions are positional analogues of `app`/`do` with
|
||
`tail: true` set on the resulting term. EBNF gains 2 lines;
|
||
total production count goes from ~28 to ~30, still inside
|
||
the constraint-1 budget.
|
||
|
||
**What this does NOT promise.** Per the 14d tail-call survey,
|
||
many existing recursive calls are *not* in tail position
|
||
because they are arguments to constructor calls (e.g.
|
||
`Cons (f h) (map f t)`). 14e adds annotation + verification;
|
||
it does **not** add a CPS transform or accumulator-form
|
||
rewrite. Programs whose recursion is constructor-blocked
|
||
will continue to be stack-bounded by recursion depth. The
|
||
canonical authoring pattern in such cases is to write the
|
||
accumulator-form variant (`map_acc`, `fold_left`, etc.)
|
||
explicitly. The stdlib (15a onward) ships both forms where
|
||
relevant.
|
||
|
||
The 14d migration of existing fixtures will be partial: only
|
||
`print_list`-style terminal recursions get marked. The
|
||
constructor-blocked recursions in `map`, `sort`, `insert`
|
||
remain unmarked — they cannot benefit from `musttail`
|
||
without a source-level rewrite.
|
||
|
||
## Decision 9: memory management — Boehm conservative GC
|
||
|
||
Through Iter 14e, every ADT box, lambda env, and closure pair was
|
||
allocated with bare `malloc` and never freed. That worked for the
|
||
17 test fixtures (all small, all short-lived) but is incompatible
|
||
with any real workload — a stdlib `fold` over a million-element
|
||
list would leak a million boxes. The "Goal" section's "no GC for
|
||
the MVP" framing predates the parameterised-ADT pipeline (Iter 13)
|
||
and the explicit-recursion expectation (Iter 14e); both make a
|
||
collector necessary.
|
||
|
||
**Choice: Boehm-Demers-Weiser conservative GC.** The simplest
|
||
working option:
|
||
|
||
- Replace `malloc(...)` with `GC_malloc(...)` in every IR site
|
||
(currently `lower_ctor`'s ADT box, `lower_lambda`'s env block
|
||
and closure pair).
|
||
- Replace the IR-level `declare ptr @malloc(i64)` with
|
||
`declare ptr @GC_malloc(i64)`.
|
||
- Add `-lgc` to the `clang` link command (in
|
||
`crates/ail/src/main.rs`'s `Build` / `Run` paths).
|
||
- No language-level change. No AST change. No schema change.
|
||
|
||
Rationale:
|
||
|
||
- **Mature.** Boehm has been the default conservative GC for
|
||
decades. Linux distros ship it as `libgc` / `libgc-dev` /
|
||
`gc` (Arch).
|
||
- **No language work.** Conservative scan of the C stack handles
|
||
AILang's stack frames without LLVM stack-map infrastructure
|
||
(which is its own multi-iter design).
|
||
- **Single-iter integration.** Lift-and-shift of the four
|
||
allocation sites; all existing tests must still pass with
|
||
identical output.
|
||
|
||
Trade-offs accepted:
|
||
|
||
- **Conservative over-retention.** A user-supplied `Int` field
|
||
whose value happens to coincide with a heap address will pin
|
||
that allocation. In practice, vanishingly rare for typical
|
||
values; survivable.
|
||
- **Pause time non-deterministic.** Boehm uses stop-the-world
|
||
mark-sweep. For LLM-author-written stdlib code at MVP scale,
|
||
pause times are not the bottleneck.
|
||
- **Build-time dependency.** `libgc` must be installed on the
|
||
build host. Users without it get a link-time error from
|
||
clang, not a silent failure.
|
||
|
||
A future iter may layer a per-fn-arena optimisation on top: when
|
||
a fn's return type contains no boxed ADT, ADT boxes allocated
|
||
inside that fn cannot escape, so an arena freed at fn return is
|
||
sound by construction (per the 14e GC notes). That requires
|
||
escape analysis, the corresponding AST/IR plumbing, and is its
|
||
own design pass. Boehm-everything is the floor; arena is an
|
||
optimisation above it.
|
||
|
||
## Mangling scheme (Iter 5c)
|
||
|
||
All AILang functions are mangled to `@ail_<module>_<def>` — even in the
|
||
single-module case. Constants likewise (`@ail_<module>_<const>`). Global
|
||
string literals carry a short hint for readability:
|
||
`@.str_<module>_<hint>_<idx>` (e.g. `@.str_sum_fmt_int_0`). The entry point
|
||
is a `define i32 @main()` trampoline (C / LLVM ABI) that calls
|
||
`@ail_<entry-module>_main()`. `source_filename` exists exactly once per
|
||
workspace and carries the entry-module name (`<entry-module>.ail`).
|
||
|
||
## Convention: qualified cross-module references (Iter 5b)
|
||
|
||
Cross-module calls use **no** new AST node. Instead, a `Term::Var { name }`
|
||
with exactly one dot in the name is a qualified reference: `<prefix>.<def>`.
|
||
|
||
- `<prefix>` is an import alias (`import { module: "X", as: "<prefix>" }`)
|
||
or, when imported without an alias, the module name itself.
|
||
- `<def>` is the name of a top-level definition in the target module.
|
||
- Def names MUST NOT contain a dot — the typechecker reports
|
||
`invalid-def-name` with `ctx: { "reason": "contains-dot" }`.
|
||
- The workspace loader (Iter 5a) finds all reachable modules; the
|
||
typechecker (Iter 5b, `check_workspace`) resolves dotted names through the
|
||
import map. Diagnostic codes: `unknown-module` (prefix not imported),
|
||
`unknown-import` (module found, def not).
|
||
|
||
Hash stability: no new AST node, no renamed fields — all previous module
|
||
hashes stay bit-identical.
|
||
|
||
## Data model (MVP)
|
||
|
||
### Module
|
||
|
||
```jsonc
|
||
{
|
||
"schema": "ailang/v0",
|
||
"name": "<id>",
|
||
"imports": [{ "module": "<id>", "as": "<id>" }],
|
||
"defs": [Def...]
|
||
}
|
||
```
|
||
|
||
### Def
|
||
|
||
`kind ∈ { "fn", "type", "effect", "const" }`. In the MVP only `fn` and `const`.
|
||
|
||
```jsonc
|
||
{
|
||
"kind": "fn",
|
||
"name": "<id>",
|
||
"type": Type,
|
||
"params": ["<id>"...],
|
||
"body": Term,
|
||
"doc": "<optional string>"
|
||
}
|
||
```
|
||
|
||
### Term (expression)
|
||
|
||
```jsonc
|
||
{ "t": "lit", "lit": { "kind": "int" | "bool" | "unit", "value": ... } }
|
||
{ "t": "var", "name": "<id>" }
|
||
{ "t": "app", "fn": Term, "args": [Term...] }
|
||
{ "t": "let", "name": "<id>", "value": Term, "body": Term }
|
||
{ "t": "if", "cond": Term, "then": Term, "else": Term }
|
||
{ "t": "do", "op": "<eff>/<op>", "args": [Term...] }
|
||
{ "t": "ctor", "type": "<id>", "ctor": "<id>", "args": [Term...] }
|
||
{ "t": "match", "scrutinee": Term, "arms": [Arm...] }
|
||
{ "t": "lam",
|
||
"params": ["<id>"...],
|
||
"paramTypes": [Type...],
|
||
"retType": Type,
|
||
"effects": ["<id>"...],
|
||
"body": Term }
|
||
{ "t": "seq", "lhs": Term, "rhs": Term }
|
||
```
|
||
|
||
In the MVP, `do` is only a direct call to a built-in effect op (no handler).
|
||
A `lam` term constructs an anonymous function value; free variables of
|
||
its body are captured from the enclosing scope (see Iter 8 closure
|
||
conversion in JOURNAL). A `seq` term evaluates `lhs` for its effects
|
||
(its result must be Unit) and yields `rhs`'s value — equivalent to
|
||
`let _ = lhs in rhs`.
|
||
|
||
### Type
|
||
|
||
```jsonc
|
||
{ "k": "con", "name": "Int" }
|
||
{ "k": "con", "name": "Bool" }
|
||
{ "k": "con", "name": "Unit" }
|
||
{ "k": "fn", "params": [Type...], "ret": Type, "effects": ["IO"...] }
|
||
{ "k": "var", "name": "a" }
|
||
{ "k": "forall", "vars": ["a"...], "body": Type }
|
||
```
|
||
|
||
## Pipeline
|
||
|
||
```
|
||
.ail.json ─┐
|
||
├─ load + validate schema
|
||
├─ resolve names + assign hashes
|
||
├─ typecheck (HM, effect rows)
|
||
├─ lower to MIR (SSA-like, named SSA values)
|
||
├─ emit LLVM IR (.ll)
|
||
└─ clang -O2 *.ll -o binary
|
||
```
|
||
|
||
## CLI
|
||
|
||
```
|
||
ail check <module.ail.json> — loads, validates, typechecks
|
||
ail manifest <module.ail.json> — table: name :: type !effects [hash]
|
||
ail describe <module> <name> — detail of a definition
|
||
ail render <module> — JSON → pretty-print
|
||
ail parse <module.ail> — pretty-print → JSON (for bootstrapping)
|
||
ail emit-ir <module> — writes .ll
|
||
ail build <module> — full pipeline → binary
|
||
ail run <module> — build + execute (tempdir), passthrough exit code
|
||
```
|
||
|
||
## Verification and correctness (across cycles)
|
||
|
||
1. **Snapshot tests** for the pretty-printer and IR emit. The diff makes
|
||
regressions visible immediately.
|
||
2. **Property tests** for the JSON ↔ pretty-print roundtrip.
|
||
3. **End-to-end tests** for `examples/` with expected program output.
|
||
4. **Hash stability**: a test ensures the same def always produces the same
|
||
hash.
|
||
5. **CI pin** of the outputs in `tests/expected/`.
|
||
6. **Rustdoc cleanliness**: `cargo doc --no-deps` runs warning-free.
|
||
Maintained by the `ailang-docwriter` agent (Iter 13d onward); fixing a
|
||
rustdoc warning is part of the iter that introduced it, not a follow-up.
|
||
|
||
## What is not (yet) supported
|
||
|
||
Snapshot of the boundary at the end of Iter 13. Items move out of this list
|
||
as iterations land; the JOURNAL records the exact iteration.
|
||
|
||
- No effect handlers — only the built-in IO and Diverge ops.
|
||
- No refinements / SMT escalation.
|
||
- No HM inference inside bodies. Top-level def types are explicit;
|
||
polymorphism is opt-in via `Type::Forall { vars, body }`. Inside
|
||
a body, lambdas check monomorphically against their declared type.
|
||
- Polymorphic fns must be **directly called** at the use site.
|
||
Passing a polymorphic fn as a value (`let f = id in f(42)`) is
|
||
not yet supported — it would need one closure-pair global per
|
||
instantiation, deferred.
|
||
- No higher-rank polymorphism. Passing a polymorphic fn to another
|
||
polymorphic fn (`apply(id, 42)`) is not supported.
|
||
- No cross-module ADTs. ADTs are local to a module; ctor names must be
|
||
unique within their module but may collide across modules.
|
||
- No visibility rules in imports. Every top-level def of an imported module
|
||
is reachable; there is no `pub` / `priv`.
|
||
- No GC. ADT boxes, lambda envs, and closure pairs all leak. Acceptable
|
||
for current example programs; required before any longer-running
|
||
program.
|
||
|
||
What **is** supported (and used as the smoke test for the pipeline):
|
||
|
||
- Int, Bool, Unit, **Str** as primitive types.
|
||
- `if`, `let`, function calls, recursion.
|
||
- Effects on function signatures, with `do op(args)` for direct effect
|
||
ops (`io/print_int`, `io/print_bool`, `io/print_str`).
|
||
- **ADTs + flat pattern matching** (Iter 3). Sub-patterns of a Ctor
|
||
pattern are restricted to `Var` / `Wild`.
|
||
- **Imports + qualified cross-module references** via dotted names
|
||
(Iter 5).
|
||
- **First-class function references** (Iter 7). A top-level fn name (or
|
||
qualified `prefix.def`) used as a `Term::Var` is a fn-value.
|
||
- **Anonymous lambdas with capture** (Iter 8). `Term::Lam` constructs a
|
||
closure that captures any free variables of its body from the
|
||
enclosing scope. All fn-values share a single ABI: a `ptr` to a
|
||
closure pair `{ thunk_ptr, env_ptr }`. Top-level fns get an auto-
|
||
generated adapter and a static closure pair (env = null) so they
|
||
remain passable as values without heap overhead.
|
||
- **Polymorphism via `Type::Forall`** at top-level def types (Iter 12).
|
||
Use sites instantiate fresh metavars; unification pins them against
|
||
the concrete types of the call args. Codegen monomorphises on
|
||
demand: each unique instantiation emits a specialised LLVM fn
|
||
mangled `@ail_<m>_<def>__<descriptor>` (e.g. `id__I` for `id` at
|
||
`Int`, `apply__I_I` for `apply` at `(Int, Int)`).
|
||
- **Parameterised ADTs** (Iter 13). `TypeDef.vars: Vec<String>`
|
||
declares type parameters; `Type::Con.args: Vec<Type>` carries the
|
||
type arguments at use sites. Both fields default to empty and are
|
||
skipped during serialization, so canonical-JSON hashes of every
|
||
pre-13a definition stay bit-identical (regression test in
|
||
`crates/ailang-core/src/hash.rs`). Ctor and match codegen stay
|
||
inline at every use site — there is no specialised ADT symbol —
|
||
but LLVM field types are derived per use site by substituting
|
||
through `cdef.ail_fields`. The substitution is read off the call's
|
||
arg types (ctor) or the scrutinee's `Type::Con.args` (match). An
|
||
unresolved `Type::Var` reaching `llvm_type` is a hard error
|
||
rather than a silent fallback to `ptr`.
|
||
|
||
Pipeline regression smoke tests:
|
||
|
||
- `examples/sum.ail.json` → prints 55 (recursion, arithmetic).
|
||
- `examples/list.ail.json` → prints 42 (ADTs + match).
|
||
- `examples/hof.ail.json` → prints 42 (first-class fn-refs, indirect call).
|
||
- `examples/closure.ail.json` → prints 42 (lambda capturing a let-bound var).
|
||
- `examples/list_map.ail.json` → prints 2/4/6 (ADTs + closure + recursive
|
||
HOF + IO; the dogfood smoke test).
|
||
- `examples/sort.ail.json` → prints sorted [3,1,4,1,5,9,2,6,5,3,5]
|
||
one-per-line (insertion sort over an 11-element list).
|
||
- `examples/poly_id.ail.json` → prints 42 then "true" (polymorphic
|
||
identity at `Int` and `Bool`; two specialised fns emitted).
|
||
- `examples/poly_apply.ail.json` → prints 42 (polymorphic `apply`
|
||
with a fn-typed parameter; `apply(succ, 41)`).
|
||
- `examples/box.ail.json` → prints 42 (parameterised ADT round-
|
||
trip: `MkBox(42)` constructed, then projected by a polymorphic
|
||
`unbox : forall a. (Box<a>) -> a` and printed).
|
||
- `examples/maybe_int.ail.json` → prints 7 then 99 (pattern match
|
||
over `Maybe<Int>`: `or_else(Some(7), 99)` then
|
||
`or_else(None, 99)`).
|