fieldtest-form-a: 4 fixtures + spec report; agent doctrine update
Boss-dispatched fieldtest at milestone close. Agent authored 4 .ail fixtures (factorial smoke + Show user-ADT + user-class Describe with two instances + two-module workspace) from DESIGN.md + form_a.md only (no compiler-source reads, no spec-of-milestone reads). All four fixtures `ail check` ok and `ail run` produces expected stdout. Findings (1 bug, 1 friction, 2 spec_gaps, 3 working): - [bug] instance-method-body unbound-var bypasses `ail check` — forma_3 first attempt called `str_concat` inside the instance method body; `ail check` returned ok, `ail build` died with the monomorphise_workspace "unknown identifier" diagnostic. Same shape at fn-body level correctly fires `[unbound-var]` at check time. Next: debug skill, RED-first against `ail check`. - [friction] no `str_concat` primitive. Every realistic Show MyType body wants string concatenation; the absence forced the agent to shape examples around bare int_to_str / ctor-arity-1 patterns. Next: small planner tidy iter wiring `str_concat` symmetric to `str_clone` and `int_to_str`. - [spec_gap] form_a.md has zero references to class, instance, constraint, or method productions — pre-22 surface only. The form-a-default-authoring milestone made .ail the authoring form, but the form_a spec doc remains pre-typeclass. - [spec_gap] form_a.md leaves the class-qualifier ambiguity in `(instance (class X))` unspecified — bare-class-name vs canonical- form rule from mq.1 is not documented at the surface level. The agent picked the bare-name reading from the corpus; the canonical- form reading is equally plausible from the schema. Boss-side cleanup at commit time: - Deleted 8 stale .ail.json sidecars in examples/fieldtest/ (4 forma_* derived by the fieldtester per old dual-form workflow + 4 pre-existing floats_* from the prior fieldtest milestone that iter form-a.1 T8 missed because the bash deletion loop globbed only examples/*.ail.json direct children). - Updated skills/fieldtest/agents/ailang-fieldtester.md to remove the now-obsolete "generate canonical JSON sidecar" step (Phase 3 rewritten + Iron Law + Reading list + Common Rationalisations + Red Flags all aligned to the post-form-a single-form doctrine). cargo test --workspace: 557 passed, 0 failed, 3 ignored (unchanged from audit-form-a — the fieldtest fixtures are standalone, not referenced by any test). Findings deferred to follow-up iters; the milestone close itself is still clean.
This commit is contained in:
@@ -0,0 +1,284 @@
|
||||
# Fieldtest — form-a-default-authoring — 2026-05-13
|
||||
|
||||
**Status:** Draft — awaiting orchestrator triage
|
||||
**Author:** ailang-fieldtester (dispatched by skills/fieldtest)
|
||||
|
||||
## Scope
|
||||
|
||||
Form A (`.ail`) became the sole authoring surface across the working
|
||||
tree. 156 non-carve-out `.ail.json` files were rendered to `.ail`
|
||||
siblings and deleted (8 carve-outs remain). The Roundtrip Invariant
|
||||
was restated as parse-determinism + idempotency-under-print + CLI-
|
||||
pipeline-idempotency. CLI subcommands `ail check / build / run /
|
||||
parse / render` accept `.ail` directly via extension dispatch (since
|
||||
iter ext-cli.1). This fieldtest probes whether a downstream LLM-author
|
||||
working only from DESIGN.md + `crates/ailang-core/specs/form_a.md` +
|
||||
the public corpus can hand-author programs that exercise the new
|
||||
authoring surface, the milestone-23/24 typeclass machinery (`Eq`,
|
||||
`Ord`, `Show`, `print`), and multi-module workspaces.
|
||||
|
||||
## Examples
|
||||
|
||||
### `examples/fieldtest/forma_1_factorial.ail` — tail-recursive factorial
|
||||
- What it does: two top-level fns (`fact_acc/2`, `fact/1`) plus `main`
|
||||
prints `fact 5` and `fact 10`. Exercises `tail-app`, `if`, primitive
|
||||
arithmetic, no class machinery. The LLM-natural shape for "is the
|
||||
basic authoring path frictionless?".
|
||||
- Why it fits the milestone: pure-arithmetic smoke for the
|
||||
authoring-ergonomics axis. No class or workspace surface; just
|
||||
exercises parser → checker → codegen on a spontaneous LLM-author
|
||||
shape derived from the form_a.md few-shot corpus (sum_demo /
|
||||
hello).
|
||||
- Outcome: `ail check` → `ok (23 symbols across 2 modules)`; `ail build`
|
||||
→ `built`; stdout `120\n3628800\n` (expected). First-shot clean.
|
||||
|
||||
### `examples/fieldtest/forma_2_show_color.ail` — `instance Show Color` + polymorphic `print`
|
||||
- What it does: declares `data Color { Red | Green | Blue }`, an
|
||||
`instance prelude.Show Color` whose body pattern-matches and returns
|
||||
a `Str` literal, and calls `print` three times. Exercises milestone
|
||||
24's polymorphic `print` end-to-end on a user ADT with a manually
|
||||
written Show instance.
|
||||
- Why it fits the milestone: the Show+print axis. Tests that the
|
||||
iter-24.3 cross-module-references-in-synthesised-bodies invariants
|
||||
fire correctly on a user-defined enum-like ADT.
|
||||
- Outcome: `ail check` → ok; `ail build` → built; stdout
|
||||
`Red\nGreen\nBlue\n` (expected). First-shot clean.
|
||||
|
||||
### `examples/fieldtest/forma_3_user_class_describe.ail` — user-defined `class Describe` with two instances
|
||||
- What it does: declares `class Describe a where describe : (borrow a)
|
||||
-> Str`, ships `instance Describe Int` and `instance Describe Shape`
|
||||
(a 2-variant ADT), then a polymorphic free fn `announce : forall a.
|
||||
Describe a => (borrow a) -> Unit !IO` that calls `describe` then
|
||||
`io/print_str`. `main` calls `announce` at three concrete types.
|
||||
- Why it fits the milestone: stresses the user-typeclass authoring
|
||||
surface, `(forall (vars a) (constraints (constraint Describe a))
|
||||
...)` shape, monomorphisation of a polymorphic free fn through a
|
||||
user constraint, and dispatch to two distinct user instances.
|
||||
- Outcome: **first attempt failed** because I reached for `str_concat`
|
||||
(a builtin I assumed existed for `"Circle r=" ++ int_to_str r`-style
|
||||
bodies); `ail check` returned `ok` but `ail build` died with
|
||||
`monomorphise_workspace: unknown identifier: str_concat`. Repaired
|
||||
by dropping the prefix-string concatenation. **Repaired version
|
||||
runs clean**: stdout `42\n3\n5\n` (expected). See findings below.
|
||||
|
||||
### `examples/fieldtest/forma_4_intmath_lib.ail` + `forma_4_intmath_main.ail` — two-module workspace
|
||||
- What it does: `forma_4_intmath_lib` exports three pure-`Int` fns
|
||||
(`square`, `cube`, `abs_int`); `forma_4_intmath_main` imports it
|
||||
and calls each via qualified references (`forma_4_intmath_lib.square
|
||||
7`, etc.) inside `main`.
|
||||
- Why it fits the milestone: stresses extension-dispatch on a
|
||||
workspace entry plus canonical-form qualified cross-module refs
|
||||
(ct.1 / mq.1 mature features) on hand-authored `.ail`.
|
||||
- Outcome: `ail check examples/fieldtest/forma_4_intmath_main.ail` →
|
||||
`ok (24 symbols across 3 modules)` (lib + main + prelude); `ail
|
||||
build` → built; stdout `49\n27\n42\n` (expected). First-shot clean.
|
||||
|
||||
## Findings
|
||||
|
||||
### [bug] `ail check` does NOT catch unbound identifiers inside `(instance ... (method (body (lam ... (body APP)))))`
|
||||
- Example: forma_3 first attempt; isolated repro in spec body below.
|
||||
- What happened (verbatim, isolated repro):
|
||||
```
|
||||
(module ft_concat_in_instance
|
||||
(class Describe (param a)
|
||||
(method describe (type (fn-type (params (borrow a)) (ret (con Str))))))
|
||||
(instance (class Describe) (type (con Int))
|
||||
(method describe (body (lam (params (typed n (con Int))) (ret (con Str))
|
||||
(body (app str_concat "n=" (app int_to_str n))))))))
|
||||
(fn main (type (fn-type (params) (ret (con Unit)) (effects IO))) (params)
|
||||
(body (do io/print_str (app describe 5)))))
|
||||
```
|
||||
`ail check`: `ok (23 symbols across 2 modules)`.
|
||||
`ail build`: `Error: monomorphise_workspace: unknown identifier: str_concat`.
|
||||
|
||||
The same shape at fn-body level (a plain `(fn ... (body (app str_concat ...)))`)
|
||||
correctly fires `error: [unbound-var] greet: unknown identifier:
|
||||
str_concat` at `ail check`. So the typechecker has the diagnostic
|
||||
machinery; it does not walk the lambda body inside an instance
|
||||
method's `(method (body (lam ... (body ...))))` when looking up
|
||||
identifiers (or it walks but does not produce an error at the
|
||||
unbound-var). The error eventually fires at the monomorphisation
|
||||
pass, whose diagnostic ("monomorphise_workspace: unknown identifier")
|
||||
is significantly less helpful than the typechecker's
|
||||
`[unbound-var]` — no source location, no symbol kind, no `did you
|
||||
mean ...?`.
|
||||
- Why it is a bug: `ail check` is documented as the typecheck step
|
||||
whose purpose is exactly to catch this class of error before
|
||||
codegen runs. The split-pipeline diagnostic surface is a regression
|
||||
in the user-facing contract.
|
||||
- One-line repro: see the four-line isolated repro above.
|
||||
- Recommended downstream action: `debug` → write a RED test asserting
|
||||
`ail check` fires `[unbound-var]` on the instance-method-body shape,
|
||||
then `implement` mini-mode walks the lambda body during instance
|
||||
validation. (Plausible cause: instance method bodies are validated
|
||||
later than fn bodies, after the registry build, and the unbound-var
|
||||
check is not threaded through that path.)
|
||||
|
||||
### [spec_gap] `crates/ailang-core/specs/form_a.md` does not document the typeclass surface
|
||||
- Example(s): forma_2 (Show user ADT) and forma_3 (user-defined class
|
||||
+ constraint).
|
||||
- What happened: I needed to write `(class ...)`, `(instance ...)`,
|
||||
`(method ...)`, and `(constraints (constraint ClassName tyvar))`
|
||||
inside a `forall`. `form_a.md` — the document explicitly billed as
|
||||
"the **complete LLM-targeted specification**. If you are an LLM and
|
||||
this is in your context, you have everything you need to produce
|
||||
valid Form-A" — has zero references to "class", "instance",
|
||||
"constraint", or "method". The grep is empty. I reverse-engineered
|
||||
the surface entirely from `examples/show_user_adt.ail`,
|
||||
`examples/bench_mono_dispatch.ail`, and
|
||||
`examples/mq1_xmod_constraint_class.ail`.
|
||||
- Reading I picked: the `(class NAME (param TYVAR) (method NAME (type
|
||||
FN-TYPE)))` form for class declarations; `(instance (class CLASS-
|
||||
REF) (type TYPE) (method NAME (body LAM-TERM)))` for instances;
|
||||
`(constraints (constraint CLASS-REF TYVAR))` inside a `(forall ...)`
|
||||
schema. These work but are not in the spec; another reading (e.g.
|
||||
whether `(class CLASS-REF)` must be qualified vs. bare for same-
|
||||
module instances) was not determinable from the spec text and had
|
||||
to be derived by example-mining.
|
||||
- Why spec_gap: a foreign LLM-author with form_a.md in context but
|
||||
without the example corpus cannot author class / instance / poly-
|
||||
constrained-fn programs at all. DESIGN.md §Decision-11 documents
|
||||
the JSON-AST schema for `ClassDef` / `InstanceDef` / `FnDef.type`
|
||||
constraints (lines ~1526-1582) but the Form-A textual projection
|
||||
for any of those keywords is not anchored anywhere. The milestone-
|
||||
24 close ratified `print` + four `Show` instances ship in the
|
||||
prelude — without a documented Form-A surface for user-class
|
||||
authoring, the LLM-utility criterion for typeclasses (Decision 11
|
||||
→ "LLM author reaches for it unprompted") cannot be tested on
|
||||
foreign LLMs by the procedure that cma.3 / ms.2 used for non-class
|
||||
programs.
|
||||
- Recommended downstream action: `tighten DESIGN.md` (or extend
|
||||
`form_a.md`) — `plan` an iter that adds three sections to
|
||||
form_a.md: "Class declarations", "Instance declarations", and
|
||||
"Constraints on polymorphic fns". Each with the EBNF production
|
||||
and one or two few-shot fixtures lifted from the corpus
|
||||
(`show_user_adt.ail`, `mq1_xmod_constraint_class.ail`).
|
||||
|
||||
### [spec_gap] Form-A grammar for `prelude.Show` qualifier in `(instance (class ...))`: when is the prefix required?
|
||||
- Example: forma_2 (`(instance (class prelude.Show) (type (con Color))
|
||||
...)`).
|
||||
- What happened: I copied the qualified `prelude.Show` form from
|
||||
`examples/show_user_adt.ail`. It works. But I also note that
|
||||
`examples/mq3_class_eq_vs_fn_eq_classmod.ail` writes `(instance
|
||||
(class MyEq) ...)` bare (no prefix) — because the class is declared
|
||||
in the SAME module. The rule appears to be the mq.1 canonical-form
|
||||
rule (qualify for cross-module, bare for same-module). DESIGN.md
|
||||
§mq.1 makes this explicit at the JSON-AST `InstanceDef.class` field
|
||||
level, but not at the Form-A surface level.
|
||||
- Reading I picked: a class reference inside `(instance (class ...))`
|
||||
follows the same canonical-form rule as `Type::Con.name` — bare for
|
||||
same-module, `module.Class` for cross-module. Another plausible
|
||||
reading: always-qualified (since `prelude` is auto-imported, bare
|
||||
`Show` could resolve to prelude.Show — but the empirical evidence
|
||||
in `show_user_adt.ail` says no, qualifier is required for cross-
|
||||
module).
|
||||
- Why spec_gap: a downstream LLM-author cannot determine whether
|
||||
`(instance (class Show) (type (con MyType)) ...)` (bare, same module
|
||||
as class? cross-module to prelude?) is well-formed without reading
|
||||
the corpus. form_a.md doesn't carry the rule; DESIGN.md carries
|
||||
it at the JSON-AST level but not for the surface.
|
||||
- Recommended downstream action: same iter as the previous gap —
|
||||
one paragraph in form_a.md's new "Class declarations" / "Instance
|
||||
declarations" sections naming the canonical-form rule for
|
||||
`Constraint.class`, `InstanceDef.class`, `SuperclassRef.class` at
|
||||
the Form-A surface.
|
||||
|
||||
### [friction] No primitive `str_concat` / `str_append` in builtins
|
||||
- Example: forma_3 first attempt.
|
||||
- What happened: while writing a `Describe Shape` instance, I reached
|
||||
for a string-concatenation primitive to format `"Circle r=" ++
|
||||
int_to_str r`. `ail builtins` lists `int_to_str`, `bool_to_str`,
|
||||
`str_clone`, `float_to_str` — no `str_concat`, no `str_append`,
|
||||
no `++`. The natural `Show MyType` body shape for any non-trivial
|
||||
ADT (a record-like print, a `name=value` style, anything with a
|
||||
label) requires string concatenation. Without it, the LLM-author
|
||||
either writes degenerate instance bodies (return just the inner
|
||||
Int as if it were the whole repr — what forma_3's repaired version
|
||||
does) or punts on milestone-24 Show-of-rich-ADTs entirely.
|
||||
- Why friction: milestone 24 shipped polymorphic `print` and four
|
||||
prelude `Show` instances on primitives; the natural LLM-author
|
||||
follow-up is `Show MyRecord = "MyRecord { x = " ++ show x ++ "
|
||||
}"`-shaped instance bodies. With no concatenation primitive, the
|
||||
shipped Show feature is observably under-powered for the very task
|
||||
it was advertised to enable (user-ADT pretty-printing through
|
||||
`print`). The friction is real for the milestone's downstream
|
||||
utility, not just for hypothetical foreign LLMs.
|
||||
- Recommended downstream action: `plan` a small follow-up iter that
|
||||
adds `str_concat : (Str, Str) -> Str` as a runtime + checker +
|
||||
codegen primitive (symmetric to `str_clone` / `int_to_str` plumbing
|
||||
in 24.1 / hs.4). The cost surface is small (single-fn slab-
|
||||
allocating C helper + four-site lockstep registration); the LLM-
|
||||
utility payoff is high because every realistic `Show MyType` body
|
||||
needs it.
|
||||
|
||||
### [working] User-defined typeclass with `forall + Constraint` polymorphic fn dispatches correctly
|
||||
- Example: forma_3 (repaired).
|
||||
- What happened: declared `class Describe a where describe : (borrow
|
||||
a) -> Str`, shipped two instances (`Int` and `Shape`), wrote
|
||||
`announce : forall a. Describe a => (borrow a) -> Unit !IO`, called
|
||||
it three times at two concrete types in `main`. `ail check`
|
||||
reported ok across 2 modules (entry + prelude); `ail build`
|
||||
produced a binary; stdout was correct. The mq.3 type-driven
|
||||
dispatch + the iter-23.4 unified mono pass + the milestone-24
|
||||
`print`-style polymorphic-fn pipeline all visibly cooperate on a
|
||||
workload they were not specifically calibrated against. No
|
||||
diagnostic surfaced spuriously; the surface was discoverable from
|
||||
three corpus examples in ~5 minutes.
|
||||
- Why working: the full Decision-11 trajectory (class declaration +
|
||||
multiple instances + polymorphic free fn over a constraint set +
|
||||
mono-pass synthesis of `describe__Int` and `describe__Shape` +
|
||||
call-site rewrite) ran end-to-end on a hand-authored `.ail`
|
||||
workspace without any compiler crate hint. This is the LLM-utility
|
||||
acceptance test for milestone 22+23+24 cumulative, run on one
|
||||
spontaneous fresh-author shape, and it passes.
|
||||
- Recommended downstream action: `carry-on`. Worth pinning the
|
||||
fixture so a future regression cannot silently break the
|
||||
trajectory. (Already a working-tree artefact;
|
||||
`crates/ailang-core/tests/schema_coverage.rs`'s dynamic walk picks
|
||||
it up automatically per the form-a.1 refactor.)
|
||||
|
||||
### [working] Multi-module workspace via `(import ...)` + qualified refs runs end-to-end on `.ail`
|
||||
- Example: forma_4_intmath_main + forma_4_intmath_lib.
|
||||
- What happened: two `.ail` modules in `examples/fieldtest/`, one
|
||||
importing the other with `(import forma_4_intmath_lib)` and using
|
||||
qualified `forma_4_intmath_lib.square` / `.cube` / `.abs_int`
|
||||
references inside `main`. `ail check
|
||||
examples/fieldtest/forma_4_intmath_main.ail` correctly resolved
|
||||
the import (workspace prefers `.ail` sibling over `.ail.json` per
|
||||
the ext-cli.1 doc), typechecked across 3 modules (main + lib +
|
||||
prelude), built, and ran. No friction from the canonical-form rule
|
||||
— qualified names worked first-shot.
|
||||
- Why working: the ext-cli.1 + form-a.1 stack (extension-dispatching
|
||||
workspace loader + `.ail`-preferred sibling resolution) functions
|
||||
exactly as DESIGN.md promises for a hand-authored two-module
|
||||
workspace.
|
||||
- Recommended downstream action: `carry-on`.
|
||||
|
||||
### [working] Polymorphic `print` on user ADT through `instance prelude.Show MyType`
|
||||
- Example: forma_2.
|
||||
- What happened: declared `data Color { Red | Green | Blue }`, wrote
|
||||
`instance prelude.Show Color` with a 3-arm match body returning a
|
||||
Str literal, called `(app print (term-ctor Color Red))` three times
|
||||
in main. Compiled clean; ran clean; produced `Red\nGreen\nBlue\n`.
|
||||
The whole iter-24.3 cross-module-references-in-synthesised-bodies
|
||||
trajectory (the three invariants in DESIGN.md §Cross-module
|
||||
references in synthesised bodies — canonical-form type_args,
|
||||
import_map-bypass fallback, FreeFnCall residual push) ran on a
|
||||
user-ADT that the implementation tests did not specifically pin.
|
||||
- Why working: the LLM-utility acceptance test for milestone-24's
|
||||
flagship feature (polymorphic `print` reaching user ADTs through a
|
||||
user-written Show instance) passes on a first-shot LLM-author
|
||||
example.
|
||||
- Recommended downstream action: `carry-on`.
|
||||
|
||||
## Recommendation summary
|
||||
|
||||
| Finding | Class | Action |
|
||||
|---|---|---|
|
||||
| `ail check` misses unbound-var inside instance method lambda body | bug | `debug` (RED test on instance-body unbound-var; mini `implement`) |
|
||||
| `form_a.md` lacks class/instance/constraint grammar | spec_gap | `plan` (form_a.md addendum: 3 new sections + corpus fixtures) |
|
||||
| Form-A canonical-form rule for `(instance (class X))` qualifier not anchored | spec_gap | `plan` (paragraph in form_a.md's new class-section; tighten DESIGN.md §Decision 11 surface anchor) |
|
||||
| No `str_concat` builtin | friction | `plan` (small follow-up iter; adds runtime + checker + codegen + lockstep wiring symmetric to `str_clone`) |
|
||||
| User-typeclass + constraint poly fn end-to-end | working | `carry-on` |
|
||||
| Multi-module `.ail` workspace + qualified refs | working | `carry-on` |
|
||||
| `print` on user ADT through user `instance prelude.Show` | working | `carry-on` |
|
||||
Reference in New Issue
Block a user