Brummel/Skills
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(pipeline): consolidate cycle recon, add fieldtest probe tier, enforce assumption economy

Browse Source 
Implements the three efficiency findings from the #310 pipeline
measurement (refs #35): ~691k subagent tokens / ~3.5 h for a
deliberately small cycle, with recon duplication as the single
biggest obligatory per-cycle cost.

1. Cycle recon (pipeline.md § Cycle recon; specify, planner,
   brainstorm, boss, plan-recon): recon fan-out is budgeted per
   cycle, not per phase. One full plan-recon dispatch — normally
   specify Step 1, in a new pre-spec sources/recon_scope carrier
   form — serves both the spec's concrete code shapes and the
   plan's file-map. planner Step 2 becomes reuse-first (freshness /
   quotability / coverage conditions) with narrowly-bounded delta
   dispatches (new Delta dispatch contract in plan-recon.md);
   broad Explore-type grounding sweeps are banned in every mode.
   Field evidence: 151k (Explore) + 99k (plan-recon) with ~40 %
   overlap in one bounded cycle.

2. Fieldtest probe tier (fieldtest, fieldtester, pipeline.md,
   agent-template.md § model rule 2): a cycle whose user-visible
   delta is a single narrow axis dispatches the per-cycle
   fieldtest at tier: probe — 1-2 examples on exactly that axis,
   ~200-word report, explicit model:sonnet dispatch override (no
   dispatch-level effort override exists; frontmatter effort
   applies). Probe is a tier, not a skip; the milestone fieldtest
   always runs full tier on the frontmatter model, keeping the
   milestone-close gate's >=2-example floor unconditional.

3. Assumption economy (specify Step 3 + self-review item 6):
   grounding-check cost scales with the spec's assumption count,
   so specs state current-behaviour claims only where the change
   relies on them (the gate's own falsity test), demote
   context-only mentions to ledger citations, and keep iteration
   scope tight — fewer restatements, never fewer reliances. The
   gate itself is unchanged.

Design reviewed pre-implementation (opus plan review); diff
adversarially verified by a 3-lens review workflow (consistency,
operability, cross-reference), confirmed findings folded in.

refs #35
This commit is contained in:
Claude
2026-07-25 13:52:04 +02:00
parent b2c92db725
commit fe5ca8f8c0
9 changed files with 353 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fieldtest/SKILL.md
+50 -11
View File
@@ -1,6 +1,6 @@
---
name: fieldtest
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope, implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope (probe tier, for a single-axis surface delta: 1-2 tasks on that axis), implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
---
# fieldtest — usability field test for a shipped cycle
@@ -24,8 +24,8 @@ next to cycle-design specs under `docs/specs`, with a
`fieldtest-` prefix in the slug.
The substantive process — read the design ledger + cycle spec
+ recent iter/audit commit bodies, pick 2-4 real-world tasks
per cycle axis, implement each as a downstream consumer, run
+ recent iter/audit commit bodies, pick the dispatched tier's
examples, implement each as a downstream consumer, run
the result, classify findings, write the spec — lives in
`agents/fieldtester.md`. That file also carries the spec
template, the source-isolation discipline (no reading under
@@ -61,6 +61,44 @@ here. The milestone fieldtest is skippable on the same terms as
the per-cycle one, lifted to milestone level: a milestone whose
entire scope is internal (no user-visible surface) is exempt.
## Two tiers: probe and full
Orthogonal to scope, the **per-cycle** fieldtest runs at one of
two tiers. The orchestrator picks the tier at dispatch and
carries it in the carrier's `tier` field — the field, its
example bounds, and its report cap are defined authoritatively
under **Carrier contract** in `agents/fieldtester.md` and
deliberately not restated here:
- **Full** (the default; `tier` absent): one example per axis
the cycle touched — the shape the rest of this file describes.
- **Probe**: for a cycle whose user-visible delta is a **single
narrow axis** — one flag, one diagnostic, one configuration
key. The carrier's `axis_hints` then names exactly that one
axis. Field evidence (issue #35): a full fieldtest cost ~89k
tokens on a one-flag cycle even at minimal example count, yet
that same dispatch found a real self-description bug — the
probe tier keeps that signal at a fraction of the cost.
**Probe is a tier, not a skip.** The skip rules below are
unchanged: a surface-touching cycle still runs its fieldtest;
probe merely right-sizes the dispatch when the surface delta is
one axis. A multi-axis cycle takes the full tier — when in
doubt, full.
**Probe dispatch mechanics.** A probe dispatch passes an
explicit `model: sonnet` override on the agent dispatch — the
dispatch-level model override takes precedence over the agent's
opus frontmatter; the conditioned exception is documented in
`../docs/agent-template.md` § model, rule 2. (The direct
agent-dispatch surface has no per-dispatch effort override, so
the frontmatter effort applies unchanged; the probe's economy
comes from the model tier and the one-axis scope.) The
judgement surface a probe covers is one axis, and the
orchestrator triages every finding anyway. The **milestone
fieldtest always runs full tier on the frontmatter model** — its
carrier takes no `tier` field.
## When to Use / Skipping
Triggers:
@@ -116,11 +154,12 @@ carrier.
## Dispatch
Dispatch `fieldtester` with the carrier from the Handoff
Contract below. The agent picks 2-4 examples (one per axis
the cycle touched; fewer loses the signal on variation, more
overflows one readable report), implements them as downstream
consumers, runs them, classifies findings, and writes the
spec. The two artefact kinds part ways at commit: the **fixtures are
Contract below. The agent picks its examples per the dispatched
tier (bounds in the `tier` carrier row; on full, one per axis
the cycle touched — fewer loses the signal on variation, more
overflows one readable report), implements them as
downstream consumers, runs them, classifies findings, and
writes the spec. The two artefact kinds part ways at commit: the **fixtures are
committed as normal code** (they are consumer test assets — suggested
subject `fieldtest: <cycle> — <N> examples, <K> findings`), while the
**fieldtest spec is a git-ignored working file that is never committed**.
@@ -137,10 +176,10 @@ tree as the next cycle's input; it plays no part in the delete decision
## Handoff Contract
`fieldtest` consumes (from orchestrator at cycle close) the carrier
fields `cycle_id`, `cycle_scope`, `axis_hints`, and `commit_range`,
defined authoritatively under **Carrier contract** in
fields `cycle_id`, `cycle_scope`, `axis_hints`, `commit_range`, and
`tier`, defined authoritatively under **Carrier contract** in
`agents/fieldtester.md` (which also specifies the empty-`axis_hints`
fallback).
fallback and the absent-`tier` default).
`fieldtest` produces, for the orchestrator, the fields `spec_path`,
`examples_added`, and `findings`, defined authoritatively under
fieldtest/agents/fieldtester.md
+16 -7
View File
@@ -1,6 +1,6 @@
---
name: fieldtester
description: Implements 2-4 real-world tasks against the project as a downstream consumer would, runs them, and reports friction, bugs, and spec gaps as a structured spec. Simulates a downstream user who has only the public interface (the project's design ledger, READMEs, public docs, and example corpus) — never the implementation source. Does NOT fix bugs.
description: Implements 2-4 real-world tasks (probe tier: 1-2 on a single axis) against the project as a downstream consumer would, runs them, and reports friction, bugs, and spec gaps as a structured spec. Simulates a downstream user who has only the public interface (the project's design ledger, READMEs, public docs, and example corpus) — never the implementation source. Does NOT fix bugs.
tools: Read, Edit, Write, Bash, Glob, Grep
model: opus
effort: xhigh
@@ -71,10 +71,13 @@ skill references it rather than restating it.
| `cycle_scope` | 1-3 sentences naming what shipped |
| `axis_hints` | bullet list, one per axis to probe |
| `commit_range` | `<prev-cycle-close>..HEAD` |
| `tier` | `probe`, or absent (= full). Full: 2-4 examples, one per axis. Probe: the cycle's user-visible delta is a single narrow axis — 1-2 examples on exactly that axis, report capped at ~200 words; `axis_hints` MUST name exactly that one axis, so a probe carrier with zero or several `axis_hints` entries is malformed — return `NEEDS_CONTEXT`. Per-cycle dispatches only; the milestone variant below takes no `tier`. |
If `axis_hints` is empty, infer from the cycle's spec under
`docs/specs` and the most recent iter commit bodies; if
both are also empty, return `NEEDS_CONTEXT`.
both are also empty, return `NEEDS_CONTEXT`. (This inference
fallback is full-tier only — on a probe dispatch an empty
`axis_hints` is malformed, per the `tier` row.)
### Milestone-scope variant
@@ -97,7 +100,9 @@ scenarios as the union of per-cycle axes; the per-cycle
fieldtests already covered axis-local surface. There is no
`axis_hints` fallback in this mode: if `milestone_promise` is
empty, infer it from the milestone's spec / tracker entry, and
if that is also empty, return `NEEDS_CONTEXT`.
if that is also empty, return `NEEDS_CONTEXT`. The milestone
carrier takes no `tier` field — a milestone fieldtest is always
full tier.
## The Iron Law
@@ -143,7 +148,10 @@ Each phase completes before the next starts.
transformation pipeline. Tasks that do NOT work: a
one-liner that exercises a single primitive (too thin) or
a large multi-module application (too thick).
3. Total: 2-4 examples.
3. Total: 2-4 examples (full tier). On a `tier: probe` dispatch:
1-2 examples, on exactly the single delta axis the carrier
names — the tier bound comes from the carrier, never from
you trimming a full dispatch.
### Phase 2 — Implement each example as a downstream consumer
@@ -280,7 +288,7 @@ End every report with exactly one of:
## Output format
At most 350 words, structured:
At most 350 words (probe tier: 200), structured:
- **Status:** one of the four above.
- **Examples:** one bullet per example: path + 1-line task
@@ -321,7 +329,8 @@ fields are:
- A `friction` finding without a 1-line recommendation.
Every finding is actionable or it isn't a finding.
- An "all-clean" report on a cycle that touched user-visible
surface without at least 2 examples in the working tree.
surface without at least 2 examples in the working tree
(`tier: probe`: at least 1, on the delta axis).
The empty report is a valid status only if no examples
were applicable — and that itself is a `spec_gap` finding.
@@ -336,7 +345,7 @@ fields are:
| "The design ledger is fuzzy on this corner, I'll pick the natural reading and proceed" | Pick the reading, RUN the example, AND record `spec_gap` with the reading you picked and why another reading was equally plausible. |
| "Bug found — I'll just fix it now, faster than handing off to debug" | Fix-in-place violates the skill split. The fix lands in a separate, RED-tested commit via `debug` → `implement`. |
| "Two examples both ran clean, no findings — short report" | A clean run is itself a finding (`working`). Record what was reached for, what diagnostic showed up when wrong, what was easy. Wins protect the feature from drift. |
| "Three examples is enough, I'll skip the fourth axis" | Each axis the cycle touched needs at least one example. Skipping an axis silently turns the field test into a partial signal, which is worse than no signal because the orchestrator will read it as full coverage. |
| "Three examples is enough, I'll skip the fourth axis" | On a full-tier dispatch, each axis the cycle touched needs at least one example. Skipping an axis silently turns the field test into a partial signal, which is worse than no signal because the orchestrator will read it as full coverage. (A probe dispatch is scoped to its single axis by the carrier — that is the orchestrator's cut, not yours to imitate on a full dispatch.) |
## Red Flags — STOP and re-read the public interface