AILang/skills/fieldtest/SKILL.md

name, description

name

description

fieldtest

Use at milestone close after audit, OR when an LLM-usability concern about a shipped feature is suspected. Picks 2-4 real-world programming tasks within the milestone's scope, implements each in the AIL Surface form (.ailx — not raw JSON), runs the resulting binaries, and writes a friction-and-bug spec to docs/specs/<date>-fieldtest-<milestone>.md. The spec feeds the next plan as a reference. Implementer simulates a downstream LLM that has only DESIGN.md plus the public examples — never the language's own implementation.

fieldtest — LLM-usability field test for a shipped milestone

Violating the letter of these rules is violating the spirit.

Overview

audit measures drift between DESIGN.md and what the codebase claims to do. fieldtest measures something audit cannot: whether the language, as it now ships, is usable by an LLM author who has only DESIGN.md and public examples to work from. The shipping LLM-author is the user of AILang. If a freshly shipped feature is awkward for that author to reach for, or if DESIGN.md leaves a corner ambiguous that real code immediately exercises, the milestone has shipped a latent debt — even when audit reports clean.

The skill produces a friction-and-bug spec that the next iteration's plan consumes as a reference. The spec sits next to milestone-design specs at docs/specs/<date>-fieldtest-<milestone>.md.

When to Use / Skipping

Triggers:

• A milestone has just closed audit cleanly (or with ratify-d drift only).
• A shipped feature is suspected of being LLM-hostile in some way the earlier brainstorm could not foresee.
• A user-facing surface change (Decision 6, schema rev, new effect/typeclass mechanic) has just landed.

Skipping is permitted only for:

• Bug-fix iterations (use debug).
• Tidy iterations whose scope is purely internal cleanup with zero surface impact.
• Iterations whose entire scope is documentation in docs/.

Skipping is not permitted for:

• A milestone that introduced or changed surface syntax, schema, effects, types, modes, or any user-visible diagnostic.
• A milestone whose JOURNAL entry claims "LLM author can now write X". fieldtest is the gate that empirically substantiates such claims; an unverified claim is drift.

The Iron Law

THE FIELDTESTER WORKS FROM DESIGN.MD AND PUBLIC EXAMPLES — NOT FROM THE COMPILER SOURCE.
EVERY EXAMPLE IS WRITTEN IN .ailx (SURFACE) FIRST. RAW .ail.json IS NEVER HAND-AUTHORED.
EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND.

The first clause is the load-bearing one. The whole point of the field test is to simulate a downstream LLM author who has only the specification and the example corpus. If the dispatched agent reads crates/ailang-check/, crates/ailang-codegen/, or any other implementation source, the test contaminates: it now reflects what the code happens to support, not what DESIGN.md promises. The carrier contract enforces this; the agent's standing reading list re-states it.

The second clause exists because .ail.json is the canonical storage form, but .ailx is the form an LLM author is meant to reach for. Hand-writing JSON skips exactly the part of the language we want to field-test. If the surface form makes a particular pattern painful, that is a finding — not a reason to drop down to JSON.

The Process — five phases

Phase 1 — Scope the field test

The orchestrator collects:

• the milestone identifier and the JOURNAL entries describing what shipped,
• the milestone's design spec (docs/specs/<milestone>.md) if one exists,
• a 1-line scope hint per axis the milestone touched (e.g. "23 shipped typeclass dictionaries: try a Functor and an Eq instance", "20 shipped prose projection: try one prose-first authoring workflow").

Two to four examples is the right size. One is too few (no signal on variation); five is too many for one report to stay readable.

Phase 2 — Dispatch ailang-fieldtester

Hand the agent the carrier (see Handoff Contract). The agent:

1. Reads DESIGN.md + JOURNAL tail + milestone spec + a representative slice of examples/ (to learn the form of valid AIL — not the content of the compiler).
2. Picks the example tasks. Each task is a real, named programming problem (FizzBuzz, polynomial root via Newton's method, JSON-ish tree depth, BFS over a small graph, run-length encoder, …) chosen so it exercises the milestone's deliveries without being a compiler torture-test. Toy problems find toy bugs.
3. For each task:
   • drafts the program in .ailx Surface form,
   • saves it as examples/fieldtest/<milestone>_<n>_<slug>.ailx,
   • runs ail check and ail build and ail run,
   • records the experience: what the agent reached for, what it had to back out of, what the diagnostic said, what surprised it, where DESIGN.md was silent or ambiguous.

The agent has the ail CLI as a black box. It must NOT open crates/, runtime/, or any internal source. It MAY read other .ailx/.ail.json files in examples/ to learn the form.

Phase 3 — Friction & bug classification

Each finding is classified into exactly one of:

Class	Meaning	Downstream action
bug	Compiler/runtime misbehaves vs. DESIGN.md (panic, wrong stdout, misleading diagnostic, schema break).	Hand off to debug (RED-test first).
friction	Compiles and runs, but the LLM-author had to fight the surface to express the thing. Surface clumsy, redundancy forced, mode/clause dance ugly.	Feeds next milestone's brainstorm or a tidy iteration's plan.
spec_gap	DESIGN.md does not constrain the case; the agent had to guess. The compiler picked a reading; another reading was equally plausible.	Orchestrator decides: tighten DESIGN.md, or ratify the existing reading in DESIGN.md, with a JOURNAL entry.
working	The author reached for the milestone's new surface and got it right on the first try with a clean diagnostic when wrong. Worth recording — these are the wins that protect the feature from drift attempts later.

Phase 4 — Self-review the spec

Before commit, the agent (or the orchestrator on its behalf) checks:

1. Source-isolation check: is any finding phrased as "the compiler does X"? If so, replace with "running ail run produced X". Findings must reference observable behaviour, not internal mechanism.
2. Surface-only check: every shipped example is a .ailx file. No hand-written .ail.json for fieldtest examples.
3. Classification check: every finding has exactly one class. A friction-and-bug item gets two entries.
4. Reproducibility check: every bug carries the one-line repro command and the observed output.

Phase 5 — Commit + handoff

examples/fieldtest/<milestone>_<n>_<slug>.ailx   (Surface form)
examples/fieldtest/<milestone>_<n>_<slug>.ail.json (canonical, generated via `ail render --json` or equivalent)
docs/specs/<date>-fieldtest-<milestone>.md      (the report)

Commit pattern (one commit per fieldtest run):

git add examples/fieldtest/*.ail* docs/specs/*-fieldtest-*.md
git commit -m "fieldtest: <milestone> — <N> examples, <K> findings"

Hand the orchestrator:

• the spec path,
• the per-finding classification,
• a one-line recommendation per finding (debug / plan / ratify / carry-on).

The orchestrator dispatches the follow-ups; fieldtest does NOT self-resolve.

Spec structure

docs/specs/<date>-fieldtest-<milestone>.md:

# Fieldtest — <milestone> — <date>

**Status:** Draft — awaiting orchestrator triage
**Author:** ailang-fieldtester (dispatched by skills/fieldtest)

## Scope
What the milestone shipped. One paragraph.

## Examples
Per example, one subsection:
### `examples/fieldtest/<milestone>_<n>_<slug>.ailx` — <task name>
- What it does
- Why this task fits the milestone's scope
- Outcome: compiles? runs? matches expected stdout?

## Findings
Per finding, one entry:
### [class] <one-line title>
- Example(s) where it surfaced
- What happened (verbatim diagnostic / wrong output / etc.)
- Why it is friction / bug / spec_gap / working
- (for `bug`) one-line repro
- Recommended downstream action

## Recommendation summary
A short table mapping finding → action (`debug` / `plan` / `ratify` /
`carry-on`).

Handoff Contract

fieldtest consumes (from orchestrator at milestone close):

Field	Content
milestone_id	e.g. milestone-22
milestone_scope	1-3 sentences naming what shipped
axis_hints	bullet list, one per milestone axis the test should probe
commit_range	<prev-milestone-close>..HEAD

fieldtest produces:

Field	Content
spec_path	docs/specs/<date>-fieldtest-<milestone>.md
examples_added	list of .ailx paths committed
findings	list, each with class + recommendation
status	clean / friction_found / bugs_found / infra_blocked

The orchestrator drives downstream: bugs → debug; friction or spec_gap → next brainstorm or a tidy iteration via plan.

Common Rationalisations

Excuse	Reality
"I'll just peek at crates/ailang-check/ to understand the new typeclass coherence rule"	Peeking ends the test. The whole skill exists because the downstream LLM author cannot peek. If DESIGN.md does not answer the question, that is a spec_gap finding — the most valuable kind.
"Hand-writing JSON is faster, the surface is just sugar"	The surface is the field-test target. JSON-direct skips it. If you want to skip the surface, you have skipped the field test.
"Diagnostic is unclear but I figured out the fix; the example is fine"	The diagnostic being unclear is the friction. Record it verbatim.
"Two examples is enough, both worked"	Two working examples on the milestone's primary axis is a thin signal. Pick at least one example per axis the milestone touched.
"DESIGN.md does not say what should happen here, I'll just pick the obvious reading"	The "obvious" reading IS the spec gap. Record it. The orchestrator decides whether to ratify or tighten.
"The example would be cleaner if I added one helper, even though the milestone didn't ship it"	The field test runs the language as shipped. Helpers you wish existed go in the friction list, not into the example.
"Bug found, I'll just fix it while I'm here, faster than handing off"	Fix-in-place violates the skill split. Record as bug, hand to debug (RED-first). The fix lands in a separate, tested commit.
"audit said clean, I'm sure the field test will too — skip"	audit and fieldtest answer different questions. audit: does code match DESIGN? fieldtest: can a downstream LLM use what shipped? Both gates exist for a reason.

Red Flags — STOP

• About to open any file under crates/, runtime/, or bench/'s internals
• About to hand-write .ail.json for a fieldtest example
• About to silently swap an example to a smaller form because the intended one didn't compile (the failure-to-compile IS the finding)
• About to merge a bug finding into a friction finding "because it's borderline" — pick one class
• About to skip Phase 4 self-review because "I wrote it carefully"
• About to attempt the bug fix in this dispatch (the skill is RED-by- observation; GREEN goes to debug → implement)
• About to recommend more than one downstream action per finding

Cross-references

• Agent dispatched: skills/fieldtest/agents/ailang-fieldtester.md
• Upstream gate: skills/audit/SKILL.md runs first; fieldtest comes after a clean (or ratified) audit
• Downstream targets:
  • skills/debug/SKILL.md — for each bug finding
  • skills/brainstorm/SKILL.md or skills/plan/SKILL.md — for friction and spec_gap findings worth a follow-up iteration
• Project source: docs/DESIGN.md "Feature-acceptance criterion" — the field test is the empirical complement to brainstorm's prospective application of that criterion. Brainstorm asks "would an LLM author reach for this?"; fieldtest asks "did one?"