Brummel
72e54f4fd3
The surface-form file extension changes from .ailx to .ail. AILang's
authoring surface now uses the same .ail stem as its canonical JSON
form (.ail.json), giving the language a single coherent extension
family: .ail is the LLM-authored Form A, .ail.json is the canonical
JSON-AST Form B.
Scope (touched):
- 61 example renames examples/**/*.ailx → .ail (git mv)
- 1 rename experiments/.../rendered/ailx.md → ail.md
- 35 content-edited live-toolchain files (crates/, docs/DESIGN.md,
docs/roadmap.md, docs/PROSE_ROUNDTRIP.md, skills/, bench/reference/*.c,
experiment crates under experiments/.../{render,harness,master})
- Experiment-crate cohort rename Cohort::Ailx → Cohort::Ail,
Form::Ailx → Form::Ail, per_cohort/ailx → per_cohort/ail,
{form-only: ailx} → {form-only: ail}, ```ailx → ```ail
Out of scope (deliberately untouched, to preserve honest history):
- docs/journal-archive.md (content-frozen per CLAUDE.md)
- docs/journals/, docs/specs/, docs/plans/, bench/orchestrator-stats/
- experiments/.../runs/ (frozen LLM-output artefacts; models actually
saw .ailx — renaming would falsify the experimental record)
Verification: cargo build/test --workspace green; experiment crate
cargo test green; bench/check.py + compile_check.py + cross_lang.py
all 0-regressed; negative grep for ailx|Ailx|AILX outside the
out-of-scope paths returns zero matches.
Opens immediate follow-up: roadmap.md P2 todo `ail check`/build/run
accept .ail extension — after this rename, .ail is canonical
authoring surface but the CLI still produces a misleading JSON-parse
error on `ail check foo.ail`. That's the next iter.
6.4 KiB
name, description
| name | description |
|---|---|
| fieldtest | Boss-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world programming tasks within the milestone's scope, implements each in the AIL Surface form (.ail — not raw JSON), runs the resulting binaries, and writes a friction-and-bug spec to docs/specs/<date>-fieldtest-<milestone>.md. The spec feeds the next plan as a reference. Implementer simulates a downstream LLM that has only DESIGN.md plus the public examples — never the language's own implementation. |
fieldtest — LLM-usability field test for a shipped milestone
Violating the letter of these rules is violating the spirit.
Overview
audit measures drift between DESIGN.md and what the codebase claims
to do. fieldtest measures something audit cannot: whether the
language, as it now ships, is usable by an LLM author who has only
DESIGN.md and public examples to work from. The shipping LLM-author
is the user of AILang. If a freshly shipped feature is awkward for
that author to reach for, or if DESIGN.md leaves a corner ambiguous
that real code immediately exercises, the milestone has shipped a
latent debt — even when audit reports clean.
The skill produces a friction-and-bug spec that the next iteration's
planner consumes as a reference. The spec sits next to milestone-design
specs at docs/specs/<date>-fieldtest-<milestone>.md.
The substantive process — read DESIGN.md + JOURNAL + milestone spec,
pick 2-4 real-world programming tasks per milestone axis, implement
each in .ail Surface form, run via ail check/build/run,
classify findings, write the spec — lives in
agents/ailang-fieldtester.md. That file also carries the spec
template, the source-isolation discipline (no reading under
crates/, runtime/, bench/), and the per-finding classification
rules. This skill file only governs trigger, dispatch, and handoff.
When to Use / Skipping
Triggers:
- Boss-dispatched after audit closes cleanly, when the orchestrator
believes the iteration is correct and wants a field test. The
pre-condition is a clean (or
ratify-d) audit; the trigger is the Boss's judgment-call that the iteration is complete. - A shipped feature is suspected of being LLM-hostile in some way the earlier brainstorm could not foresee.
- A user-facing surface change (Decision 6, schema rev, new effect/typeclass mechanic) has just landed.
Findings cluster around two shapes — simple bugs (route via debug →
implement mini) or catastrophic architecture problems (route via next
brainstorm). Friction and spec-gap items are also possible; the
routing table below applies in all cases.
Skipping is permitted only for:
- Bug-fix iterations (use
debug). - Tidy iterations whose scope is purely internal cleanup with zero surface impact.
- Iterations whose entire scope is documentation in
docs/.
Skipping is not permitted for:
- A milestone that introduced or changed surface syntax, schema, effects, types, modes, or any user-visible diagnostic.
- A milestone whose JOURNAL entry claims "LLM author can now write
X".
fieldtestis the gate that empirically substantiates such claims; an unverified claim is drift.
The Iron Law
THE FIELDTESTER WORKS FROM DESIGN.MD AND PUBLIC EXAMPLES — NOT FROM THE COMPILER SOURCE.
EVERY EXAMPLE IS WRITTEN IN .ail (SURFACE) FIRST. RAW .ail.json IS NEVER HAND-AUTHORED.
EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND.
The first clause is load-bearing: the whole point of the field test is to simulate a downstream LLM author who has only the specification and the example corpus. The agent file enforces this with a hard path allowlist; the orchestrator must trust that contract and not feed the agent compiler-internal hints in the carrier.
Dispatch
Dispatch ailang-fieldtester with the carrier from the Handoff
Contract below. The agent picks 2-4 examples (one per axis the
milestone touched), implements them in .ail, runs them through the
public ail CLI, classifies findings, and writes the spec. All
artefacts (fixtures + spec) stay in the working tree as unstaged
changes; the Boss commits them after reviewing the report (suggested
commit subject: fieldtest: <milestone> — <N> examples, <K> findings).
Two to four examples is the right size. One is too few (no signal on variation); five is too many for one report to stay readable.
Handoff Contract
fieldtest consumes (from orchestrator at milestone close):
| Field | Content |
|---|---|
milestone_id |
e.g. milestone-22 |
milestone_scope |
1-3 sentences naming what shipped |
axis_hints |
bullet list, one per milestone axis the test should probe |
commit_range |
<prev-milestone-close>..HEAD |
fieldtest produces:
| Field | Content |
|---|---|
spec_path |
docs/specs/<date>-fieldtest-<milestone>.md |
examples_added |
list of .ail paths committed |
findings |
list, each with class (bug / friction / spec_gap / working) + recommendation |
status |
clean / friction_found / bugs_found / infra_blocked |
The orchestrator drives downstream:
bug→debug(RED-first; GREEN viaimplementmini-mode)friction→ nextbrainstormor a tidy iteration viaplannerspec_gap→ ratify in JOURNAL + DESIGN.md, or tighten DESIGN.mdworking→ carry-on (worth recording, no follow-up)
fieldtest does NOT self-resolve.
Cross-references
- Agent dispatched:
skills/fieldtest/agents/ailang-fieldtester.md— carries the five-phase process, the Iron Law in operational form, the spec template, the per-finding classification rules, the Common Rationalisations table, and the Red Flags list. - Upstream gate:
skills/audit/SKILL.mdruns first; fieldtest comes after a clean (or ratified) audit. - Cadence ordering: fieldtest runs before
skills/docwriter/SKILL.md; docwriter happens at a later, longer stability window. - Downstream targets:
skills/debug/SKILL.md— for eachbugfindingskills/brainstorm/SKILL.mdorskills/planner/SKILL.md— forfrictionandspec_gapfindings worth a follow-up iteration
- Project source:
docs/DESIGN.md"Feature-acceptance criterion" — the field test is the empirical complement to brainstorm's prospective application of that criterion. Brainstorm asks "would an LLM author reach for this?"; fieldtest asks "did one?"