Skills/fieldtest/SKILL.md

---

name: fieldtest description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope (probe tier, for a single-axis surface delta: 1-2 tasks on that axis), implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.

fieldtest — usability field test for a shipped cycle

Violating the letter of these rules is violating the spirit.

Overview

audit measures drift between the design ledger and what the codebase claims to do. fieldtest measures something audit cannot: whether the project, as it now ships, is usable by a downstream consumer who has only the public interface to work from. If a freshly shipped feature is awkward to reach for, or if the public docs leave a corner ambiguous that real usage immediately exercises, the cycle has shipped a latent debt — even when audit reports clean.

The skill produces a friction-and-bug spec that the next iteration's planner consumes as a reference. The spec sits next to cycle-design specs under docs/specs, with a fieldtest- prefix in the slug.

The substantive process — read the design ledger + cycle spec

• recent iter/audit commit bodies, pick the dispatched tier's examples, implement each as a downstream consumer, run the result, classify findings, write the spec — lives in agents/fieldtester.md. That file also carries the spec template, the source-isolation discipline (no reading under the project's code roots or its benchmark directory, if it has one — its CLAUDE.md project facts), and the per-finding classification rules. This skill file only governs trigger, dispatch, and handoff.

Two scopes: per-cycle and milestone

fieldtest runs at two scopes, both using the same fieldtester agent — the carrier picks the scope:

• Per-cycle fieldtest (the default this file otherwise describes): dispatched after a surface-touching cycle's audit, carrier scoped to that cycle (cycle_id / cycle_scope). Catches cycle-local friction early. Optional / orchestrator judgement, as below.
• Milestone fieldtest: the closing gate for a tracker milestone. Carrier scoped to the milestone's promise (milestone_id / milestone_promise); the fieldtester derives curated end-to-end scenarios top-down from that promise and proves the shipped implementation delivers it. Its green status roll-up is required before a surface-touching milestone may be closed.

Both carriers are defined authoritatively under Carrier contract in agents/fieldtester.md (the milestone variant under its Milestone-scope variant sub-heading). The gate the milestone fieldtest feeds is defined once in ../docs/pipeline.md § Milestone-close gate — not restated here. The milestone fieldtest is skippable on the same terms as the per-cycle one, lifted to milestone level: a milestone whose entire scope is internal (no user-visible surface) is exempt.

Two tiers: probe and full

Orthogonal to scope, the per-cycle fieldtest runs at one of two tiers. The orchestrator picks the tier at dispatch and carries it in the carrier's tier field — the field, its example bounds, and its report cap are defined authoritatively under Carrier contract in agents/fieldtester.md and deliberately not restated here:

• Full (the default; tier absent): one example per axis the cycle touched — the shape the rest of this file describes.
• Probe: for a cycle whose user-visible delta is a single narrow axis — one flag, one diagnostic, one configuration key. The carrier's axis_hints then names exactly that one axis. Field evidence (issue #35): a full fieldtest cost ~89k tokens on a one-flag cycle even at minimal example count, yet that same dispatch found a real self-description bug — the probe tier keeps that signal at a fraction of the cost.

Probe is a tier, not a skip. The skip rules below are unchanged: a surface-touching cycle still runs its fieldtest; probe merely right-sizes the dispatch when the surface delta is one axis. A multi-axis cycle takes the full tier — when in doubt, full.

Probe dispatch mechanics. A probe dispatch passes an explicit model: sonnet override on the agent dispatch — the dispatch-level model override takes precedence over the agent's opus frontmatter; the conditioned exception is documented in ../docs/agent-template.md § model, rule 2. (The direct agent-dispatch surface has no per-dispatch effort override, so the frontmatter effort applies unchanged; the probe's economy comes from the model tier and the one-axis scope.) The judgement surface a probe covers is one axis, and the orchestrator triages every finding anyway. The milestone fieldtest always runs full tier on the frontmatter model — its carrier takes no tier field.

When to Use / Skipping

Triggers:

• Orchestrator-dispatched after audit closes cleanly, when the orchestrator believes the iteration is correct and wants a field test. The pre-condition is a clean (or ratified) audit; the trigger is the orchestrator's judgment-call that the iteration is complete.
• A shipped feature is suspected of being awkward in some way the earlier brainstorm could not foresee.
• A user-facing surface change has just landed.

Findings cluster around two shapes — simple bugs (route via debug → implement mini) or catastrophic architecture problems (route via next brainstorm). Friction and spec-gap items are also possible; the routing table below applies in all cases.

Skipping is permitted only for:

• Bug-fix iterations (use debug).
• Tidy iterations whose scope is purely internal cleanup with zero surface impact.
• Iterations whose entire scope is documentation.

Skipping is not permitted for:

• A cycle that introduced or changed surface syntax, schema, effects, API, configuration format, or any user-visible diagnostic.
• A cycle whose commit body or backlog issue claims "users can now do X". fieldtest is the gate that empirically substantiates such claims; an unverified claim is drift.

The Iron Law

THE FIELDTESTER WORKS FROM THE PUBLIC INTERFACE ONLY — NOT FROM THE IMPLEMENTATION SOURCE.
EVERY EXAMPLE IS WRITTEN IN THE PROJECT'S CANONICAL AUTHORING / CONSUMER FORM.
EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND.

The first clause is load-bearing: the whole point of the field test is to simulate a downstream consumer who has only the public interface. The agent file enforces this with a hard path allowlist (computed from the project's public interface, if it has one — its CLAUDE.md project facts); the orchestrator must trust that contract and not feed the agent implementation-internal hints in the carrier.

Dispatch

Dispatch fieldtester with the carrier from the Handoff Contract below. The agent picks its examples per the dispatched tier (bounds in the tier carrier row; on full, one per axis the cycle touched — fewer loses the signal on variation, more overflows one readable report), implements them as downstream consumers, runs them, classifies findings, and writes the spec. The two artefact kinds part ways at commit: the fixtures are committed as normal code (they are consumer test assets — suggested subject fieldtest: <cycle> — <N> examples, <K> findings), while the fieldtest spec is a git-ignored working file that is never committed.

The fieldtest spec is written after cycle N's audit-close (fieldtest runs only once audit is clean), so N's cycle-close sweep has already run and cannot touch it — it survives naturally as an input to the next cycle's planner. It is shell-rm'd at N+1's audit-close, whose sweep removes every working-tree spec and plan (N+1's own, plus this spent fieldtest spec). The fieldtest- prefix only marks it in the working tree as the next cycle's input; it plays no part in the delete decision (the sweep removes everything present).

Handoff Contract

fieldtest consumes (from orchestrator at cycle close) the carrier fields cycle_id, cycle_scope, axis_hints, commit_range, and tier, defined authoritatively under Carrier contract in agents/fieldtester.md (which also specifies the empty-axis_hints fallback and the absent-tier default).

fieldtest produces, for the orchestrator, the fields spec_path, examples_added, and findings, defined authoritatively under Output format in agents/fieldtester.md — single-sourced in the agent file, deliberately not restated here so the two files cannot drift.

It additionally reports one skill-level status roll-up — one of clean / friction_found / bugs_found / infra_blocked. This field is owned here (it is not part of the agent's per-run status protocol), so it is defined in this file only.

The orchestrator drives downstream:

• bug → debug (RED-first; GREEN via implement mini-mode)
• friction → next brainstorm or a tidy iteration via planner
• spec_gap → ratify by updating the design ledger (and naming the gap in the closing commit body), or tighten the design ledger
• working → carry-on (worth recording, no follow-up)

fieldtest does NOT self-resolve.

Tracker filing — bundle by default. A fieldtest typically yields more findings than a cycle closes issues (3-5 vs 1-2 in the field evidence on issue #36), so filing each finding as its own issue grows the tracker structurally with every fieldtested cycle. When findings are filed onto the tracker rather than routed inline:

• Consolidate before filing: before creating any new issue, check whether an open issue on the same line of work can carry the scope — absorb via a comment there instead.
• Related friction / spec_gap findings of one fieldtest go into one collective issue, not one issue per finding.
• A real bug keeps its own issue — it is a forward-queue work item. A low-severity bug may wait in the collective issue instead.
• The fieldtester's report stays per-finding — classification is the agent's contract and does not change. Bundling is this orchestrator-side filing rule only. The writes themselves follow ../issue/SKILL.md, as every tracker write does.

When settled single-issue items accumulate anyway, the orchestrator drains them as a harvest sweep — the batch shape defined in ../docs/pipeline.md § Harvest sweep.

Cross-references

• Agent dispatched: agents/fieldtester.md — carries the five-phase process, the Iron Law in operational form, the spec template, the per-finding classification rules, the Common Rationalisations table, and the Red Flags list.
• Upstream gate: ../audit/SKILL.md runs first; fieldtest comes after a clean (or ratified) audit.
• Closing gate it feeds: ../docs/pipeline.md § Milestone-close gate — the milestone fieldtest's green roll-up is the functional leg of that gate.
• Batch drain it feeds: ../docs/pipeline.md § Harvest sweep — the batch shape that drains accumulated settled single-issue items (bundled findings included).
• Cadence ordering: fieldtest runs before ../docwriter/SKILL.md; docwriter happens at a later, longer stability window.
• Downstream targets:
  • ../debug/SKILL.md — for each bug finding
  • ../brainstorm/SKILL.md or ../planner/SKILL.md — for friction and spec_gap findings worth a follow-up iteration