Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
65a4f0aa1629095441d484f8a715083edffa2dc7
AILang/skills/fieldtest/SKILL.md
T
Brummel 51da9fab53 iter disc.1: boss-only commits + main-as-quarantine (no branches in implement) 
Two project-wide rules are now explicit across every skill:

1. Only the Boss commits. No skill agent (implementer,
   brainstormer, planner, debugger, fieldtester, docwriter,
   architect, bencher) runs `git commit`. Agents write their
   artefacts to the working tree as unstaged changes; the Boss
   inspects, decides commit shape, and commits.
2. main HEAD is sacrosanct. No actor runs `git reset` or
   `git revert` on main. Bad work stays in the working tree
   where it is still discardable via `git checkout -- <paths>`.

Implement loses the `iter/<iter_id>` branch mechanic entirely;
Phase 0 of the orchestrator-agent now does a clean-tree check
and refuses to start on a dirty tree. Per-task agent commits
are removed everywhere; reviewers operate against
`git diff HEAD` instead of `pre_task_sha..head_sha`.

Motivation: 2026-05-11 iter 23.4 stranded prep2/prep3 commits on
an iter-branch that never integrated to main, then a corrected
spec falsely claimed those commits had shipped. Branch-per-iter
+ manual-Boss-merge + iter-stacking made the strand structurally
possible. See docs/journals/2026-05-11-iter-disc.1.md for the
full per-task notes and motivation.
2026-05-11 22:44:43 +02:00

6.4 KiB
Raw Blame History

name, description
name description
fieldtest Boss-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world programming tasks within the milestone's scope, implements each in the AIL Surface form (.ailx — not raw JSON), runs the resulting binaries, and writes a friction-and-bug spec to docs/specs/<date>-fieldtest-<milestone>.md. The spec feeds the next plan as a reference. Implementer simulates a downstream LLM that has only DESIGN.md plus the public examples — never the language's own implementation.

fieldtest — LLM-usability field test for a shipped milestone

Violating the letter of these rules is violating the spirit.

Overview

audit measures drift between DESIGN.md and what the codebase claims to do. fieldtest measures something audit cannot: whether the language, as it now ships, is usable by an LLM author who has only DESIGN.md and public examples to work from. The shipping LLM-author is the user of AILang. If a freshly shipped feature is awkward for that author to reach for, or if DESIGN.md leaves a corner ambiguous that real code immediately exercises, the milestone has shipped a latent debt — even when audit reports clean.

The skill produces a friction-and-bug spec that the next iteration's planner consumes as a reference. The spec sits next to milestone-design specs at docs/specs/<date>-fieldtest-<milestone>.md.

The substantive process — read DESIGN.md + JOURNAL + milestone spec, pick 2-4 real-world programming tasks per milestone axis, implement each in .ailx Surface form, run via ail check/build/run, classify findings, write the spec — lives in agents/ailang-fieldtester.md. That file also carries the spec template, the source-isolation discipline (no reading under crates/, runtime/, bench/), and the per-finding classification rules. This skill file only governs trigger, dispatch, and handoff.

When to Use / Skipping

Triggers:

  • Boss-dispatched after audit closes cleanly, when the orchestrator believes the iteration is correct and wants a field test. The pre-condition is a clean (or ratify-d) audit; the trigger is the Boss's judgment-call that the iteration is complete.
  • A shipped feature is suspected of being LLM-hostile in some way the earlier brainstorm could not foresee.
  • A user-facing surface change (Decision 6, schema rev, new effect/typeclass mechanic) has just landed.

Findings cluster around two shapes — simple bugs (route via debug → implement mini) or catastrophic architecture problems (route via next brainstorm). Friction and spec-gap items are also possible; the routing table below applies in all cases.

Skipping is permitted only for:

  • Bug-fix iterations (use debug).
  • Tidy iterations whose scope is purely internal cleanup with zero surface impact.
  • Iterations whose entire scope is documentation in docs/.

Skipping is not permitted for:

  • A milestone that introduced or changed surface syntax, schema, effects, types, modes, or any user-visible diagnostic.
  • A milestone whose JOURNAL entry claims "LLM author can now write X". fieldtest is the gate that empirically substantiates such claims; an unverified claim is drift.

The Iron Law

THE FIELDTESTER WORKS FROM DESIGN.MD AND PUBLIC EXAMPLES — NOT FROM THE COMPILER SOURCE.
EVERY EXAMPLE IS WRITTEN IN .ailx (SURFACE) FIRST. RAW .ail.json IS NEVER HAND-AUTHORED.
EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND.

The first clause is load-bearing: the whole point of the field test is to simulate a downstream LLM author who has only the specification and the example corpus. The agent file enforces this with a hard path allowlist; the orchestrator must trust that contract and not feed the agent compiler-internal hints in the carrier.

Dispatch

Dispatch ailang-fieldtester with the carrier from the Handoff Contract below. The agent picks 2-4 examples (one per axis the milestone touched), implements them in .ailx, runs them through the public ail CLI, classifies findings, and writes the spec. All artefacts (fixtures + spec) stay in the working tree as unstaged changes; the Boss commits them after reviewing the report (suggested commit subject: fieldtest: <milestone> — <N> examples, <K> findings).

Two to four examples is the right size. One is too few (no signal on variation); five is too many for one report to stay readable.

Handoff Contract

fieldtest consumes (from orchestrator at milestone close):

Field Content
milestone_id e.g. milestone-22
milestone_scope 1-3 sentences naming what shipped
axis_hints bullet list, one per milestone axis the test should probe
commit_range <prev-milestone-close>..HEAD

fieldtest produces:

Field Content
spec_path docs/specs/<date>-fieldtest-<milestone>.md
examples_added list of .ailx paths committed
findings list, each with class (bug / friction / spec_gap / working) + recommendation
status clean / friction_found / bugs_found / infra_blocked

The orchestrator drives downstream:

  • bug → debug (RED-first; GREEN via implement mini-mode)
  • friction → next brainstorm or a tidy iteration via planner
  • spec_gap → ratify in JOURNAL + DESIGN.md, or tighten DESIGN.md
  • working → carry-on (worth recording, no follow-up)

fieldtest does NOT self-resolve.

Cross-references

  • Agent dispatched: skills/fieldtest/agents/ailang-fieldtester.md — carries the five-phase process, the Iron Law in operational form, the spec template, the per-finding classification rules, the Common Rationalisations table, and the Red Flags list.
  • Upstream gate: skills/audit/SKILL.md runs first; fieldtest comes after a clean (or ratified) audit.
  • Cadence ordering: fieldtest runs before skills/docwriter/SKILL.md; docwriter happens at a later, longer stability window.
  • Downstream targets:
    • skills/debug/SKILL.md — for each bug finding
    • skills/brainstorm/SKILL.md or skills/planner/SKILL.md — for friction and spec_gap findings worth a follow-up iteration
  • Project source: docs/DESIGN.md "Feature-acceptance criterion" — the field test is the empirical complement to brainstorm's prospective application of that criterion. Brainstorm asks "would an LLM author reach for this?"; fieldtest asks "did one?"
Reference in New Issue View Git Blame Copy Permalink