spec: form-a-default-authoring milestone — roadmap entry + draft spec

Form A (.ail) becomes the sole authoring surface for every program
in the working tree. The seven negative-test JSON-AST fixtures are
the only post-milestone .ail.json files; everything else is rendered
to .ail and the JSON counterpart is deleted (single representation
per program).

Spec decides the four open design questions inline:
- A1: in-process parse via ailang_surface::parse, no build-time
  target/ artefacts (would reintroduce a second representation).
- A2: prelude.ail ships as iter-0 pilot, validated by the existing
  every_ail_fixture_matches_its_json_counterpart gate.
- A3: per-file deletion cadence; iter 0 is the singular dual-form
  window because the JSON counterpart is the witness the CI gate
  uses.
- A4: CLAUDE.md and DESIGN.md reworded — canonical vs authoring
  forms separated, JSON-AST stays canonical/hashable, .ail becomes
  the authoring entry point.

Roundtrip invariant restates from "two forms agree byte-for-byte"
to "parse is deterministic + idempotent under print"; carve-out
inventory test pins the seven JSON-only fixtures against silent
list drift.

Grounding-check PASS over 8 load-bearing assumptions.

Iteration scope: iter 0 = prelude pilot only (render
prelude.ail.json -> prelude.ail; do not delete the JSON yet; do
not touch CLAUDE.md/DESIGN.md/tests yet).
This commit is contained in:
2026-05-13 09:52:04 +02:00
parent 319b73c281
commit e864c85840
2 changed files with 527 additions and 0 deletions
+65
View File
@@ -89,6 +89,71 @@ context. Pick the next milestone from P1.)_
separate so migration runs against a frozen architecture. Same
call milestone 23 made for `==` / `eq`.
- [ ] **\[todo\]** Author `examples/prelude.ail` alongside
`examples/prelude.ail.json`. Today the prelude exists only as
canonical-form JSON-AST; the textual surface form has to be
reconstructed via `ail render` every time someone wants to read
it. Check in the Form-A text as the authoring source and let
`ail parse` regenerate the JSON-AST at build time (or keep both
and gate equivalence via the existing round-trip invariant).
Either way the LLM-author's natural reading surface (Form A
text) becomes the entry point rather than a derived view.
- context: surfaced 2026-05-13 while demonstrating the
Show + print milestone via `ail prose`; the prelude is the
one file most likely to be read by humans and by LLM authors
learning the language, and it's the only fixture without a
`.ail` sibling.
- [ ] **\[milestone\]** Form-A as the default authoring surface for
examples and docs. Render every `examples/*.ail.json` to its
`.ail` sibling via `ail render`, **delete the now-redundant
`.ail.json`**, and regenerate the JSON-AST per `ail parse` at
build / test time. Same for inline JSON-AST blocks in `docs/`
markdown (~7 in DESIGN.md plus ~29 other md files) — convert to
Form-A snippets where the snippet's purpose is to show "what the
language looks like", not "what the schema is".
After this milestone the working tree contains exactly one
representation per program: the `.ail` source. The JSON-AST is a
build artefact, not a checked-in twin. Tests and benches that
currently glob `*.ail.json` either parse-then-consume or are
rewired to point at `.ail` directly. The round-trip invariant
flips role: today it gates that the two forms agree; afterwards
it gates that `parse` is deterministic.
Carve-outs that MUST stay JSON-AST (no Form-A counterpart, and
no `.ail` sibling shall be created — these are the only seven
files that remain `.ail.json`-only post-milestone):
- Fixtures that test canonical-form rejection — Form A would
reject them at parse, defeating the test:
`test_ct1_bare_xmod_rejected.ail.json`,
`test_ct1_qualified_class_rejected.ail.json`,
`test_ct1_bad_qualifier.ail.json`, `broken_unbound.ail.json`,
`test_22b2_invalid_superclass_param.ail.json`,
`test_22b2_kind_mismatch.ail.json`,
`test_22b2_unbound_constraint_var.ail.json`.
- DESIGN.md schema documentation blocks where the JSON-AST shape
*is* the point (Decision 11, ParamMode, canonical-form
invariants).
- Any other case where the structured form is the artefact under
discussion, not a vehicle for a program.
Touches CLAUDE.md ("source of truth is structured data") — the
language doctrine doesn't change (JSON-AST is still the canonical
hashable form), but the *authoring* doctrine does: authors write
`.ail`, the build derives JSON-AST. Sentence in CLAUDE.md needs
rewording in the same milestone.
Mechanical for the bulk; per-fixture judgement call only for the
carve-out list. Run as one milestone so the corpus flip-over
happens at a single, audit-gated point.
- context: post-Form-A-as-canonical-authoring corollary of the
`examples/prelude.ail` entry above. Generalises the same idea
to the rest of the corpus and follows through on the cross-
model authoring data (textual form cheaper, more first-try
hits) by treating Form A as the privileged surface in the
working tree, not just in flavour text.
- [ ] **\[feature\]** Operator routing through `Eq` / `Ord``==`,
`<` etc. resolved via the typeclass instead of the built-in
primitive comparators. No commitment; gated on bench re-baselining
@@ -0,0 +1,462 @@
# Form-A as the Default Authoring Surface — Design Spec
**Date:** 2026-05-13
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude
## Goal
Make Form A (`.ail`) the sole authoring surface for every program in
the working tree. After this milestone, the only `.ail.json` files
that remain in the repository are seven specific negative-test
fixtures whose subject matter cannot be expressed in Form A by
construction. Every other `.ail.json` is rendered to its `.ail`
sibling via `ail render`, and the now-redundant `.ail.json` is
**deleted** — not retained as a generated artefact, not stashed in a
build directory, not regenerated at test time. The working tree
contains exactly one representation per program: the `.ail` source.
The language doctrine does not change. The canonical, hashable,
content-addressed form remains the JSON-AST (Decision 6,
architectural pin, §"data structure is the source of truth"). What
changes is the **authoring doctrine**: authors write `.ail`; the
build derives the JSON-AST in-process when a consumer needs it,
gated by the round-trip invariant.
Driving evidence: the cross-model authoring-form study (audit-cma,
audit-ms, 2026-05-12) measured that Form A is cheaper in tokens
(~98k vs ~207k for the same four tasks) and produces strictly
more first-shot green programs than JSON-AST authoring, across two
distinct foreign LLMs (Qwen3-Coder-Next and CodeLlama-13b-Instruct).
The internal authoring practice is now drift from the empirical
"this is the form LLMs reach for unprompted" finding.
## Architecture
Four architectural decisions land in this spec; each is the
substantive answer to one of the open questions in the roadmap
entry.
### A1. In-process parse, no build-time artefacts
Tests and benches consume `.ail` fixtures by calling
`ailang_surface::parse` (library, in-process) — not by spawning
`ail parse` as a build prelude that emits `.ail.json` to a target
directory.
**Why.** A build-prelude approach reintroduces a second
representation: `examples/foo.ail` (source) plus
`target/parsed/foo.ail.json` (derived). Even with the derived form
gitignored, it would be a working-tree fact during tests and a
shadow corpus for downstream tools to drift against. The single-
representation discipline that motivates this milestone is broken
the moment a derived `.ail.json` exists on disk anywhere a tool
might read it.
The in-process path is also already plumbed: since iter ext-cli.1
(2026-05-12), every path-taking CLI subcommand accepts `.ail` and
parses through `ailang_surface::parse` in-line. The Python bench
drivers (`bench/check.py`, `bench/compile_check.py`,
`bench/cross_lang.py`, `bench/mono_dispatch.py`) call `ail check`
etc. via subprocess and inherit the `.ail` acceptance for free —
they need only a path-suffix change. The integration tests under
`crates/ail/tests/*.rs` and the workspace tests under
`crates/ailang-check/tests/*.rs` currently `fs::read_to_string` a
`.ail.json` and pass the bytes to a loader; they will switch to
reading the `.ail` and calling `ailang_surface::parse` to recover
the same `Module` value.
### A2. `prelude.ail` ships as iter-0 pilot
Iteration 0 converts exactly one file: `examples/prelude.ail.json`
`examples/prelude.ail`. Nothing else moves. The roundtrip-CI gate
covers it (every `.ail` with a same-stem `.ail.json` counterpart is
already byte-compared by `every_ail_fixture_matches_its_json_counterpart`),
which proves the conversion path end-to-end before the bulk
migration commits to it.
**Why.** Prelude exercises the widest cross-section of language
features in a single file: typeclasses with superclasses, polymorphic
free functions with constraints, IO-effect signatures, primitive
ctor patterns, and the loader's auto-injection path. If prelude
round-trips identically through `ail render → ail parse → canonical
bytes`, no other fixture in the corpus introduces a feature this
file does not. The pilot validates the migration mechanism on the
hardest single file before the bulk migration commits the test-
infrastructure refactor.
Iter 0 keeps `prelude.ail.json` on disk alongside `prelude.ail`
the deletion of redundant JSON-AST starts in iter 1 once the
test-infrastructure refactor lands. This is the *only* iteration
where a fixture is dual-form; it is justified by the existing
`every_ail_fixture_matches_its_json_counterpart` gate, which is
exactly the byte-comparison this iteration wants to leverage as a
witness.
### A3. Per-file deletion cadence with iteration-end checkpoints
`.ail.json` files are deleted on the same iteration that materialises
their `.ail` counterpart, not at milestone end. The bulk migration is
sequenced across ~3 iterations of ~30-40 files each, each iteration
fully reaping its own `.ail.json` files before close. There is no
intermediate state where the bulk of fixtures are dual-form.
The exception is iter 0 (the prelude pilot, A2 above), which is
the singular dual-form window: the JSON counterpart is the witness
the existing CI gate uses, and the gate is the safety net for the
pilot itself.
**Why.** A milestone-end bulk delete would mean the working tree
spends weeks in a "which form is the source?" ambiguity. Anyone
editing a fixture in the interim — to repair an unrelated bug,
add a regression test, fix a hash drift — has no canonical guide
to which form to edit. The single-representation property must hold
on a per-commit basis, not as a milestone-end snapshot.
The trade-off is iteration sequencing pressure: every iteration that
ships in this milestone must leave the tree in a coherent
single-representation state. This is enforced by the per-iteration
roundtrip-CI rewrite (see Testing strategy below): once the
infrastructure flips in iter 1, fixtures without `.ail.json` are
the new normal, and fixtures with `.ail.json` are visibly in-flight.
### A4. CLAUDE.md and DESIGN.md rewording
Two documents need substantive rewording in this milestone. Both
edits land in the same commit as the iteration that retires the
last non-carve-out `.ail.json`.
**CLAUDE.md** §"AILang — a language for LLM authors" opening:
Currently:
> *Machine readability over human readability. Source of truth is
> structured data (`.ail.json`), not text.*
After this milestone:
> *Machine readability over human readability. The canonical,
> hashable, content-addressed form is structured data (`.ail.json`);
> the authoring projection is Form A (`.ail`). Authors write `.ail`;
> the build derives the JSON-AST in-process via
> `ailang_surface::parse`, gated by the round-trip invariant. The
> two forms are byte-isomorphic — picking either does not change
> the identity of the module.*
**DESIGN.md** §"What this Decision deliberately does not do" (line
465-466), currently:
> *It does not promote form (A) to the source of truth. Form (A)
> is one projection; the JSON-AST stays canonical.*
The first sentence is replaced; the second stays. New text:
> *It does not change which form is canonical: the JSON-AST remains
> the hashable, content-addressed representation, and all hashing,
> content-addressing, and cross-module references flow through it
> unchanged. What did change in 2026-05-13 (this milestone) is
> which form is authored: authors now write Form A; the JSON-AST
> is materialised in-process by callers that need it. The two
> forms are byte-isomorphic by the round-trip invariant.*
The "Form (A) is one projection; the JSON-AST stays canonical"
phrasing is preserved verbatim in the second sentence because it
remains true: Form A and the eventual visual / graphical
projections are still parallel projections of the same AST. What
the milestone changes is which projection is the **entry point**,
not the projection's relation to the AST.
## Components
### C1. Migration script
A one-shot Bash script `scripts/migrate-to-form-a.sh` (new file)
walks `examples/*.ail.json`, skips the seven carve-outs (named
explicitly in the script body), invokes `ail render` to produce
the sibling `.ail`, and `rm`s the JSON counterpart. The script is
idempotent (re-running on a partially migrated tree is a no-op for
already-migrated files). The script is retired at milestone close
— it has no second use; keeping it would invite drift the next
time someone wonders "where did the JSON-AST come from".
### C2. Test infrastructure refactor
48 files under `crates/` and `bench/` reference `.ail.json` paths.
The refactor is mechanical but spans three classes of consumer:
1. **Roundtrip tests** (`crates/ailang-surface/tests/round_trip.rs`,
`crates/ail/tests/roundtrip_cli.rs`). The five-test invariant
restructures (see Testing strategy §T below).
2. **Fixture-loading integration tests** (`crates/ail/tests/*.rs`,
~35 files). Each test currently reads
`examples/<stem>.ail.json` with `fs::read_to_string` and
passes the bytes to `Module::from_json` or the workspace
loader. The refactor reads `examples/<stem>.ail` and parses via
`ailang_surface::parse_module_from_str`. The seven carve-outs
continue to use the `.ail.json` path (they have no `.ail` form
by construction).
3. **Bench drivers** (`bench/*.py`, `bench/run.sh`). The drivers
spawn `ail check` etc. with fixture paths. Since ext-cli.1
the subcommands accept both extensions; the drivers' fixture
walks change from `glob("examples/*.ail.json")` to
`glob("examples/*.ail")` plus an explicit allow-list for the
carve-outs.
### C3. Roundtrip invariant restatement
§"Roundtrip Invariant" at DESIGN.md:2027 currently anchors on a
two-way `.ail.json``.ail` bijection. After this milestone the
JSON-AST counterpart no longer exists for the bulk of the corpus,
so the invariant restates as:
- **Parse-determinism.** For every well-formed `.ail` text `t`,
`parse(t)` produces a unique AST (the loader is a pure function;
no randomness, no time dependence, no environment leak).
- **Idempotency under print.**
`canonical(parse(t)) == canonical(parse(print(parse(t))))`
printing and re-parsing is a no-op on the canonical bytes.
- **CLI-pipeline idempotency.** The public CLI pipeline `ail render
| ail parse` (one round) is byte-identical to direct
`ail parse` of the source `.ail`. Pins the user-facing CLI
against drift internal tests cannot see.
- **Carve-out anchor.** The seven `.ail.json`-only fixtures are
negative-tests (load-time / typecheck-time rejection); they
participate only in the existing rejection-shape tests, not in
the roundtrip gate. Their presence is the *only* surviving
reason `.ail.json` remains a first-class file extension.
The "JSON → text → JSON" direction (Direction 1 in the current
spec) loses its corpus once the JSON files are deleted. The
property is replaced by parse-determinism plus idempotency — the
two together discharge the load-bearing concern (hashing is well-
defined for any `.ail` source).
### C4. The seven carve-outs (immutable list)
These seven files stay `.ail.json` post-milestone and gain no `.ail`
sibling:
```
examples/test_ct1_bare_xmod_rejected.ail.json
examples/test_ct1_qualified_class_rejected.ail.json
examples/test_ct1_bad_qualifier.ail.json
examples/broken_unbound.ail.json
examples/test_22b2_invalid_superclass_param.ail.json
examples/test_22b2_kind_mismatch.ail.json
examples/test_22b2_unbound_constraint_var.ail.json
```
Justification per file:
- `test_ct1_*` (3): assert canonical-form rejection of malformed
type references. The Form-A parser would either accept the
reference (test loses its subject) or reject it at parse time
with a different diagnostic (test loses its meaning). The
fixture's purpose is to test the **loader/checker** rejection
shape, which requires the input to bypass the parser.
- `broken_unbound`: asserts unbound-variable rejection at typecheck.
The `.ail.json` form encodes a structurally well-formed but
semantically broken module that the Form-A parser would also
parse cleanly; the file is retained in JSON-form not because the
surface cannot express it but because the test already pins the
exact JSON bytes, and re-deriving from `.ail` would add no value
while adding a regeneration step.
- `test_22b2_*` (3): assert class-definition rejection shapes
(invalid superclass param, kind mismatch, unbound constraint var)
that the Form-A parser would either reject at parse time
(changing the diagnostic) or refuse to encode at all (changing
the test subject).
The orchestrator agrees these are the only legitimate carve-outs.
Any future addition to this list requires brainstorm-level
justification on the same semantic grounds.
## Data flow
Before this milestone:
```
author writes: examples/foo.ail.json
|
v
[loader / serde_json]
|
v
Module value -> typecheck, codegen, ...
```
After iter 0 (prelude pilot only):
```
author writes: examples/prelude.ail (new)
|
v
[ailang_surface::parse]
|
v
Module value -> typecheck, codegen, ...
[byte-witness]
examples/prelude.ail.json ----+ (retained
| | for iter 0,
v | deleted iter 1)
[loader / serde_json] |
| |
v |
Module value -----> canonical -------+
```
After milestone close (bulk migration done, JSON counterparts
deleted):
```
author writes: examples/<name>.ail (every program)
|
v
[ailang_surface::parse]
|
v
Module value -> typecheck, codegen, ...
except for the seven carve-outs:
examples/<carve-out>.ail.json
|
v
[loader / serde_json] (negative-test only)
```
Cross-module imports continue to use module names, not file paths;
the workspace loader walks `examples/*` and accepts either extension
for any module it loads. Since the bulk of the corpus is single-
extension post-migration, the loader's extension-dispatch is exercised
only by the seven carve-outs (and by any test that explicitly mixes
the two forms for migration testing).
## Error handling
The migration is mechanical and the round-trip invariant
discharges correctness. There are three failure modes the spec
must name explicitly because they could escape the existing CI net:
**E1. A fixture round-trips before migration but the migrated `.ail`
fails to parse.** This would be a bug in `ail render` or
`ail parse`; the round-trip CI gate
(`every_ail_fixture_matches_its_json_counterpart`) already covers
it for every fixture with a hand-written `.ail` sibling. The
migration script extends the same property to every fixture by
construction (the script's output is exactly the witness the gate
demands). If iter 0 (prelude pilot) fails this gate, the bug is
in render/parse; the milestone halts and the bug fix takes
precedence.
**E2. A test that depends on the exact byte shape of `.ail.json`
breaks after migration.** Some tests (e.g. snapshot tests) compare
byte-equal canonical JSON to a checked-in expected blob. These
tests do not break — the canonical bytes are derived from the AST,
and the AST is form-agnostic. The migration touches only the
fixture-loading path, not the canonical-encoding path.
**E3. A bench driver receives a `.ail` path and emits stale
`.ail.json` paths in its report.** The bench drivers' report format
echoes the fixture name; the suffix change from `.ail.json` to
`.ail` propagates into report bytes. The audit-* JSONs under
`bench/orchestrator-stats/` will start carrying `.ail` paths after
the next milestone-close audit; this is intended and not a
regression.
A failure mode the spec explicitly does *not* worry about: parse
non-determinism. The roundtrip-CI gate has been green across 9
consecutive audits (audit-cma onward); the parser is well-
exercised.
## Testing strategy
### T1. Roundtrip-CI test restatement
`crates/ailang-surface/tests/round_trip.rs` currently runs three
tests against the `.ail.json` corpus. Post-migration, the three
restate:
| Pre-migration test | Post-migration test | Behaviour |
|--------------------|---------------------|-----------|
| `print_then_parse_round_trips_every_fixture` (over `.ail.json`) | replaced by `parse_is_deterministic_over_every_ail_fixture` (over `.ail`) | Reads every `.ail`, parses twice, asserts canonical-byte equality between the two parses. Asserts pure-function-of-input. |
| `parse_then_print_then_parse_is_idempotent_on_every_ail_fixture` | unchanged | Reads every `.ail`, asserts `parse == parse ∘ print ∘ parse`. |
| `every_ail_fixture_matches_its_json_counterpart` | retired | Loses its corpus. The property it pinned (no drift between two hand-authored forms) is no longer expressible because only one form is hand-authored. |
`crates/ail/tests/roundtrip_cli.rs::cli_render_then_parse_preserves_canonical_bytes_on_every_fixture`
restates as `cli_parse_then_render_then_parse_is_idempotent`:
reads every `.ail`, runs `ail parse` (to canonical JSON tempfile),
runs `ail render` (back to `.ail` tempfile), runs `ail parse`
again, and asserts byte equality across the two parses.
`crates/ailang-core/tests/schema_coverage.rs::every_ast_variant_is_observed_in_the_fixture_corpus`
flips its corpus walk from `.ail.json` to `.ail` (load via
`ailang_surface::parse_module_from_str`); the assertion is
unchanged. The schema-coverage property is the strictest gate
that survives: a new AST variant cannot land without a fixture
that uses it.
### T2. Per-iteration regression discipline
Every iteration in this milestone must leave `cargo test
--workspace` green and `bench/compile_check.py` exit-0. The
migration is mechanical; non-green at iteration close is a defect
in the iteration's test-infrastructure refactor, not in the
language. The iteration does not close until the refactor is
green.
### T3. Test-naming discipline for the migration
A few existing tests carry `.ail.json` in their test-function names
(e.g. `mono_recursive_fn_loads_from_json_ast`). These names are
renamed in the same iteration that migrates the corresponding
fixture: the new name should not lie about what file it reads.
This is mechanical, but it has to land in the migration commit, not
as a follow-up rename.
### T4. Carve-out coverage check
A new test `crates/ailang-core/tests/carve_out_inventory.rs`
asserts that exactly the seven named carve-out files exist under
`examples/*.ail.json` after milestone close. The test fails if a
new `.ail.json` is added or a carve-out goes missing. The list
is hardcoded in the test body — this is intentional: the carve-out
list is part of the language's negative-test surface and any change
to it is a deliberate, brainstorm-level decision.
## Acceptance criteria
The milestone closes when all of the following are true:
1. `ls examples/*.ail.json | wc -l` returns `7`, and the seven
files match the carve-out list verbatim.
2. Every program in `examples/` other than the seven carve-outs
exists as a `.ail` file with no `.ail.json` sibling.
3. The roundtrip-CI restatement (T1) is green: parse is
deterministic, idempotent under print, CLI-pipeline idempotent,
and the carve-out inventory test (T4) passes.
4. `cargo test --workspace` is green at milestone close (no
regressions introduced by the test-infrastructure refactor).
5. `bench/compile_check.py` and `bench/cross_lang.py` exit 0
against the migrated corpus.
6. `bench/check.py` is either exit 0 or exit 1 with audit-
ratifiable noise only (consistent with the 9-audit lineage
since audit-cma); a new regression class attributable to the
migration triggers brainstorm-level escalation.
7. CLAUDE.md and DESIGN.md carry the reworded paragraphs (§A4
above) verbatim.
8. The migration script `scripts/migrate-to-form-a.sh` is deleted
from the working tree (one-shot, retired).
9. `docs/WhatsNew.md` carries a milestone-close entry framed for
the user-as-reader: what changed in the authoring experience,
not what changed in the internal representation.
10. The `[milestone] Form-A as the default authoring surface`
entry in `docs/roadmap.md` is checked off and removed.
## Iteration scope
The first iteration covers **§A2 (prelude pilot)** only:
- Render `examples/prelude.ail.json` → `examples/prelude.ail` via
`ail render`.
- Do not delete `prelude.ail.json` yet (iter 1 deletes it
alongside the bulk migration of test infrastructure).
- Verify the existing
`every_ail_fixture_matches_its_json_counterpart` test is green
on the new `.ail`.
- No test-infrastructure changes; no CLAUDE.md / DESIGN.md edits
yet.
Iter 1 onward is out of scope for the first plan; the planner
re-derives iter scope from this spec each iteration.