Brummel
64b0841c5a
The four-phase debug process and the five-phase fieldtest process each lived twice — once in SKILL.md (orchestrator-facing) and once in the dispatched agent's file (subagent-facing). The orchestrator does not execute these phases; the subagent does. Duplicating them in SKILL.md just bloated the orchestrator's main context with bytes that only the subagent ever needs. SKILL.md now carries only what the orchestrator must consult at dispatch time: trigger gating, Iron Law as headline, the carrier contract, the produced handoff, and cross-references. Iron Law in operational form, full process, Common Rationalisations, Red Flags, and (for fieldtest) the spec template now live solely in the agent file. Net −160 LOC across the skills tree. CLAUDE.md pointer for 'Bug fixes — TDD, always' updated to reflect that the substantive discipline lives in the agent file, not the skill file.
127 lines
5.7 KiB
Markdown
127 lines
5.7 KiB
Markdown
---
|
|
name: fieldtest
|
|
description: Use at milestone close after audit, OR when an LLM-usability concern about a shipped feature is suspected. Picks 2-4 real-world programming tasks within the milestone's scope, implements each in the AIL Surface form (.ailx — not raw JSON), runs the resulting binaries, and writes a friction-and-bug spec to docs/specs/<date>-fieldtest-<milestone>.md. The spec feeds the next plan as a reference. Implementer simulates a downstream LLM that has only DESIGN.md plus the public examples — never the language's own implementation.
|
|
---
|
|
|
|
# fieldtest — LLM-usability field test for a shipped milestone
|
|
|
|
> **Violating the letter of these rules is violating the spirit.**
|
|
|
|
## Overview
|
|
|
|
`audit` measures drift between DESIGN.md and what the codebase claims
|
|
to do. `fieldtest` measures something audit cannot: whether the
|
|
language, *as it now ships*, is usable by an LLM author who has only
|
|
DESIGN.md and public examples to work from. The shipping LLM-author
|
|
*is* the user of AILang. If a freshly shipped feature is awkward for
|
|
that author to reach for, or if DESIGN.md leaves a corner ambiguous
|
|
that real code immediately exercises, the milestone has shipped a
|
|
latent debt — even when audit reports `clean`.
|
|
|
|
The skill produces a friction-and-bug spec that the next iteration's
|
|
`plan` consumes as a reference. The spec sits next to milestone-design
|
|
specs at `docs/specs/<date>-fieldtest-<milestone>.md`.
|
|
|
|
The substantive process — read DESIGN.md + JOURNAL + milestone spec,
|
|
pick 2-4 real-world programming tasks per milestone axis, implement
|
|
each in `.ailx` Surface form, run via `ail check`/`build`/`run`,
|
|
classify findings, write the spec — lives in
|
|
`agents/ailang-fieldtester.md`. That file also carries the spec
|
|
template, the source-isolation discipline (no reading under
|
|
`crates/`, `runtime/`, `bench/`), and the per-finding classification
|
|
rules. This skill file only governs trigger, dispatch, and handoff.
|
|
|
|
## When to Use / Skipping
|
|
|
|
Triggers:
|
|
- A milestone has just closed `audit` cleanly (or with `ratify`-d
|
|
drift only).
|
|
- A shipped feature is suspected of being LLM-hostile in some way the
|
|
earlier brainstorm could not foresee.
|
|
- A user-facing surface change (Decision 6, schema rev, new
|
|
effect/typeclass mechanic) has just landed.
|
|
|
|
**Skipping is permitted only** for:
|
|
- Bug-fix iterations (use `debug`).
|
|
- Tidy iterations whose scope is purely internal cleanup with zero
|
|
surface impact.
|
|
- Iterations whose entire scope is documentation in `docs/`.
|
|
|
|
**Skipping is not permitted** for:
|
|
- A milestone that introduced or changed surface syntax, schema,
|
|
effects, types, modes, or any user-visible diagnostic.
|
|
- A milestone whose JOURNAL entry claims "LLM author can now write
|
|
X". `fieldtest` is the gate that empirically substantiates such
|
|
claims; an unverified claim is drift.
|
|
|
|
## The Iron Law
|
|
|
|
```
|
|
THE FIELDTESTER WORKS FROM DESIGN.MD AND PUBLIC EXAMPLES — NOT FROM THE COMPILER SOURCE.
|
|
EVERY EXAMPLE IS WRITTEN IN .ailx (SURFACE) FIRST. RAW .ail.json IS NEVER HAND-AUTHORED.
|
|
EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND.
|
|
```
|
|
|
|
The first clause is load-bearing: the whole point of the field test is
|
|
to simulate a downstream LLM author who has only the specification and
|
|
the example corpus. The agent file enforces this with a hard path
|
|
allowlist; the orchestrator must trust that contract and not feed the
|
|
agent compiler-internal hints in the carrier.
|
|
|
|
## Dispatch
|
|
|
|
Dispatch `ailang-fieldtester` with the carrier from the Handoff
|
|
Contract below. The agent picks 2-4 examples (one per axis the
|
|
milestone touched), implements them in `.ailx`, runs them through the
|
|
public `ail` CLI, classifies findings, writes the spec, and commits
|
|
both fixtures and spec in one commit
|
|
(`fieldtest: <milestone> — <N> examples, <K> findings`).
|
|
|
|
Two to four examples is the right size. One is too few (no signal on
|
|
variation); five is too many for one report to stay readable.
|
|
|
|
## Handoff Contract
|
|
|
|
`fieldtest` consumes (from orchestrator at milestone close):
|
|
|
|
| Field | Content |
|
|
|-------|---------|
|
|
| `milestone_id` | e.g. `milestone-22` |
|
|
| `milestone_scope` | 1-3 sentences naming what shipped |
|
|
| `axis_hints` | bullet list, one per milestone axis the test should probe |
|
|
| `commit_range` | `<prev-milestone-close>..HEAD` |
|
|
|
|
`fieldtest` produces:
|
|
|
|
| Field | Content |
|
|
|-------|---------|
|
|
| `spec_path` | `docs/specs/<date>-fieldtest-<milestone>.md` |
|
|
| `examples_added` | list of `.ailx` paths committed |
|
|
| `findings` | list, each with class (`bug` / `friction` / `spec_gap` / `working`) + recommendation |
|
|
| `status` | `clean` / `friction_found` / `bugs_found` / `infra_blocked` |
|
|
|
|
The orchestrator drives downstream:
|
|
- `bug` → `debug` (RED-first; GREEN via `implement` mini-mode)
|
|
- `friction` → next `brainstorm` or a tidy iteration via `plan`
|
|
- `spec_gap` → ratify in JOURNAL + DESIGN.md, or tighten DESIGN.md
|
|
- `working` → carry-on (worth recording, no follow-up)
|
|
|
|
`fieldtest` does NOT self-resolve.
|
|
|
|
## Cross-references
|
|
|
|
- **Agent dispatched:** `skills/fieldtest/agents/ailang-fieldtester.md`
|
|
— carries the five-phase process, the Iron Law in operational form,
|
|
the spec template, the per-finding classification rules, the Common
|
|
Rationalisations table, and the Red Flags list.
|
|
- **Upstream gate:** `skills/audit/SKILL.md` runs first; fieldtest
|
|
comes after a clean (or ratified) audit.
|
|
- **Downstream targets:**
|
|
- `skills/debug/SKILL.md` — for each `bug` finding
|
|
- `skills/brainstorm/SKILL.md` or `skills/plan/SKILL.md` — for
|
|
`friction` and `spec_gap` findings worth a follow-up iteration
|
|
- **Project source:** `docs/DESIGN.md` "Feature-acceptance criterion"
|
|
— the field test is the empirical complement to brainstorm's
|
|
prospective application of that criterion. Brainstorm asks "would
|
|
an LLM author reach for this?"; fieldtest asks "did one?"
|