Iter 14b: design pass for the authoring surface

User redirected at iter boundary: writing a stdlib in JSON was the wrong move. The language is supposed to be the one I program *best* in, and JSON-AST authoring is rationalisation, not strength. DESIGN.md Decision 6 captures the constraints that fall out of the "formalisable for a foreign LLM" hard requirement (no precedence, no semantic indentation, ASCII only, every AST node a uniquely-tagged form), sketches three candidate notations with the same `map` encoded in each, and picks form (A) — fully-tagged S-expressions — as the first attempt with explicit rollback path to form (C) if (A) hurts authoring. Form (A) shape: - 3-rule lexical core: sexpr / atom / token-classified-by- first-character. - Every AST node has a unique head keyword. No case-rule (no "capitalised head means ctor"); ctors are explicit via `(term-ctor TypeName CtorName args)` and `(pat-ctor CtorName fields)`. - Bare atoms get their sort from the parent slot (type-var inside `(con NAME args)`, term-var inside `(app HEAD args)`, pat-var inside `(pat-ctor CTOR fields)`, integer literal in term position, etc.). Empirical exhibits, hand-encoded: - examples/hello.ailx 5 LOC (JSON was 36 pretty / 21 canonical) - examples/box.ailx 25 LOC (JSON was 160 / 88) - examples/list_map_poly.ailx 50 LOC (JSON was 394 / 230) 4-8x line reduction, ~4x character reduction. Bigger gains on bigger programs since overhead is proportional to AST depth. None are parseable yet — header comments say "Iter 14b design exhibit, parser lands in 14c". Two small spec issues caught while writing the exhibits and folded back into DESIGN.md before committing: - Operator idents (`+`, `==`) need the token-by-first-char classification rule, not a word-shaped regex. - Bool literals (`true`/`false`) reserved in term context; unit is explicit `(lit-unit)`. Tests unchanged (this iter is paper). 25/25 e2e green. cargo doc --no-deps zero warnings. Plan 14c: new crate `ailang-surface` with PEG parser, round-trip hash-equivalence gate against every existing `examples/*.ail.json`, CLI subcommand `ail parse`. If round-trip holds, stdlib starts in `.ailx` form (Iter 14d). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:57:24 +02:00
parent 747b7cd05c
commit 2bce825b69
5 changed files with 445 additions and 0 deletions
@@ -113,6 +113,233 @@ Rationale:
 Trade-off: no inline optimisations through the LLVM API. We rely on
 `clang -O2` as the standard pipeline.

+## Decision 6: authoring surface (Iter 14b — WIP)
+
+**Status: design pass in progress.** Reading without skipping the JOURNAL
+will leave this section ahead of the implementation.
+
+### Why this is opening up
+
+Iters 1 through 14a authored everything as raw `*.ail.json`. That worked
+for 17 fixture files (each ≤ 60 LOC of JSON) but does not scale. Two
+breaking signals:
+
+1. The token-economy cost of the JSON-AST is massive: a single integer
+   literal `1` is encoded as `{"t":"lit","lit":{"kind":"int","value":1}}`
+   — ~38 tokens of structural overhead per bit of semantics. For a stdlib
+   in the 200–500 def range this displaces real attention budget.
+2. JSON-AST authoring exposes a class of errors (wrong field names,
+   silently-accepted extra fields under `#[serde(default)]`,
+   inconsistent ctor casing) that surface only at load time. The
+   schema is correct-by-construction in storage but **error-prone in
+   authoring**.
+
+Decision 1 anticipated this: it says a textual form exists "as a
+bidirectional projection of the JSON form." The pretty-printer already
+emits S-expression-style text (see `crates/ailang-core/src/pretty.rs`).
+What is missing is the inverse direction — text → AST. Decision 6
+makes that inverse the canonical authoring surface, with JSON-AST
+demoted to "storage and exchange" only.
+
+### Constraints (hard, in priority order)
+
+1. **Formalizable for a foreign LLM.** The grammar must fit in an
+   EBNF/PEG spec of ≤ 30 productions. A model that has never seen
+   AILang must be able to read the spec and produce conforming source
+   zero-shot. Rules out: precedence between binary operators,
+   semantic indentation, maximal-munch lexing, context-sensitive
+   reductions.
+2. **AST-isomorphic.** Every surface form maps to exactly one AST
+   shape. Round-trip surface → AST → canonical JSON → AST → surface
+   is the identity (modulo formatting). Hashes computed via the
+   round-tripped JSON must equal hashes of the same module written
+   directly in JSON.
+3. **No external symbols.** ASCII only. No Greek (`∀`), no arrows
+   (`→`), no subscripts. Reasoning: I substitute mojibake for
+   non-ASCII characters under context pressure; foreign LLMs vary
+   in how they tokenize Unicode.
+4. **No precedence.** Either everything is parenthesized, or there
+   are no infix operators. Prefer the latter — `add(x, 1)` over
+   `x + 1`. Removes a fail mode for both me and foreign LLMs.
+5. **No semantic indentation.** Block structure expressed by paired
+   delimiters or terminator tokens. Indentation is informational
+   only; the parser ignores it.
+6. **One construct per token-list.** Every AST node corresponds to
+   exactly one parenthesized form (or atom). No "sometimes you can
+   omit the parens" rules.
+7. **AST surface stays frozen.** The surface adapts to the AST, not
+   the other way around. We do not change the JSON schema or
+   invalidate hashes to make the surface prettier.
+
+### Candidate notations (same `map` encoded in each)
+
+The reference target — the polymorphic `map` from `examples/list_map_poly.ail.json`:
+
+```
+data List a where Nil | Cons a (List a)
+fn map : forall a b. ((a) -> b, List a) -> List b
+       = \f xs. match xs of Nil -> Nil
+                          | Cons h t -> Cons(f(h), map(f, t))
+```
+
+#### (A) S-expression with fully-tagged AST nodes
+
+```
+(module list_map_poly
+
+  (data List (vars a)
+    (ctor Nil)
+    (ctor Cons a (con List a)))
+
+  (fn inc
+    (type (fn-type (params (con Int)) (ret (con Int))))
+    (params x)
+    (body (app + x 1)))
+
+  (fn map
+    (type
+      (forall (vars a b)
+        (fn-type
+          (params (fn-type (params a) (ret b)) (con List a))
+          (ret (con List b)))))
+    (params f xs)
+    (body
+      (match xs
+        (case (pat-ctor Nil) (term-ctor List Nil))
+        (case (pat-ctor Cons h t)
+          (term-ctor List Cons
+            (app f h)
+            (app map f t)))))))
+```
+
+Grammar core (3-rule lexical layer + ~25 named-form productions):
+```
+sexpr  ::= atom | "(" sexpr* ")"
+atom   ::= integer | string | ident
+ident  ::= any maximal non-whitespace, non-paren run that is not
+           a recognised integer or string literal.
+```
+
+The lexer recognises one delimiter (`(` / `)`) and whitespace.
+Every other maximal token is classified post-hoc:
+- All-digit run with optional leading `-` → integer atom.
+- `"`-delimited run → string atom.
+- Otherwise → ident.
+
+Consequence: operators like `+`, `==`, `<=`, `**`, qualified
+names like `io/print_int`, and cross-module references like
+`std_list.map` are all single ident tokens with no special lex
+rule. The only reserved tokens are `(`, `)`, and whitespace.
+Bool literals (`true`, `false`) and unit (`(lit-unit)`) are
+disambiguated by parser context, not by lex.
+
+Every AST node form has a unique head keyword (`module`, `data`,
+`fn`, `forall`, `fn-type`, `con`, `var`, `app`, `lam`, `match`,
+`case`, `pat-ctor`, `term-ctor`, `do`, `seq`, ...). A bare atom in
+a positional slot (e.g. inside `(con List a)` second position) is
+a name reference whose **sort** is determined by the parent slot:
+
+- inside `(con NAME args...)` second-and-later positions → type
+  expression. Bare atom there ⇒ `Type::Var { name }`.
+- inside `(app HEAD args...)` first position ⇒ `Term::Var`.
+- inside `(pat-ctor CTOR fields...)` field positions ⇒
+  `Pattern::Var`.
+- inside `(case PAT BODY)` second position ⇒ term.
+
+There is **no lexical case rule**. To construct a value with a
+ctor, write `(term-ctor TypeName CtorName args...)`. To match
+against one, write `(pat-ctor CtorName fields...)`. Capitalised
+identifiers carry no special meaning to the parser. This rules
+out a class of silent errors ("I forgot to capitalise `Cons` and
+it parsed as a function call").
+
+**Pros:** smallest formal grammar of any candidate (the lexical
+core is 3 rules; the named-form productions are uniform — every
+node a tagged list). Foreign-LLM bar lowest. Round-trip with the
+existing pretty-printer is a refactor of `pretty.rs` to emit this
+tagged form, plus a new parser.
+
+**Cons:** paren density is high. `(forall (vars a b) (fn-type
+...))` has more visual nesting than the current pretty-printer's
+`forall a. (...) -> ...`. Verbosity is ~2× JSON for the same node
+when measured in characters, but ~8× shorter in lines (the
+existing JSON `box.ail.json` of 160 lines becomes ~20 lines in
+this form).
+
+#### (B) Indented record-style with explicit terminators
+
+```
+module std_list
+
+data List(a):
+  Nil
+  Cons(a, List(a))
+end
+
+fn map:
+  type: forall a b. fn(fn(a) -> b, List(a)) -> List(b)
+  params: f, xs
+  body:
+    match xs:
+      Nil          => Nil
+      Cons(h, t)   => Cons(f(h), map(f, t))
+    end
+end
+```
+
+Grammar core (~20–30 productions): module-level (def/data/end), type
+sub-grammar (forall, fn, con, var), term sub-grammar (lam, match,
+ctor, app, lit, var, seq), pattern sub-grammar.
+
+**Pros:** higher information density per line, closer to mainstream
+ML/Haskell shape. **Cons:** four sub-grammars instead of one.
+`forall a b. fn(...)` keeps a pseudo-precedence (`->` binds tighter
+than the outer `fn(...)` wrapper). Foreign-LLM bar higher.
+
+#### (C) Pretty-printer-as-source
+
+Use exactly the format `pretty::module` already emits, plus a parser
+that accepts it. The existing pretty-printer's quirks (`::` for
+type-of, `[params]` for fn-params, `<a>` for type-args, `forall a. ...`,
+`!IO`, `()` ambiguous between unit-arg-list and empty-form) become
+the spec.
+
+**Pros:** zero churn — the existing pretty-printer is already the
+spec; only the inverse is missing. Round-trip is the identity by
+construction. **Cons:** the existing format mixes four mini-dialects
+(s-expr at term level, ML-shape at type level, square brackets for
+params, `<>` for type args). Formalising it crisply is harder than
+designing a uniform form from scratch.
+
+### First choice and rollback plan
+
+**Try (A) first.** Reasoning: constraint 1 (formalizable) outweighs
+constraint readability. (A) has a 3-rule core grammar with one
+lexical disambiguation rule. (B) doubles the rule count and
+re-introduces a soft form of precedence (`->` inside `forall`).
+(C) is tempting because it is zero-design but the resulting spec is
+visibly heterogeneous, which is exactly what constraint 1 was meant
+to rule out.
+
+If implementing (A) reveals that paren density actively hurts my
+authoring (measurable: I make more wrong-paren errors than the
+JSON-AST shape produced before), roll back and try (C). (B) stays
+on the shelf for a future iter only if both fail.
+
+### Implementation outline (Iter 14c onwards, not done in 14b)
+
+- `crates/ailang-surface` — new crate. PEG parser → existing
+  `ailang-core::ast` types. No new AST nodes.
+- Round-trip test: every `examples/*.ail.json` → emit via
+  `pretty::module` → parse via new crate → re-canonicalise →
+  hash-equivalent to original. Hash equivalence is the truth check;
+  no surface form ships if a single fixture loses its hash.
+- CLI: `ail parse <surface-file> -o <json-file>`. Symmetric to the
+  existing `ail render`.
+- Then: rewrite one existing fixture (probably `box.ail.json`) in
+  the surface, hash-compare, commit both forms. Stdlib starts in
+  the surface from day one.
+
 ## Mangling scheme (Iter 5c)

 All AILang functions are mangled to `@ail_<module>_<def>` — even in the