feat(pipeline): consolidate cycle recon, add fieldtest probe tier, enforce assumption economy
Implements the three efficiency findings from the #310 pipeline measurement (refs #35): ~691k subagent tokens / ~3.5 h for a deliberately small cycle, with recon duplication as the single biggest obligatory per-cycle cost. 1. Cycle recon (pipeline.md § Cycle recon; specify, planner, brainstorm, boss, plan-recon): recon fan-out is budgeted per cycle, not per phase. One full plan-recon dispatch — normally specify Step 1, in a new pre-spec sources/recon_scope carrier form — serves both the spec's concrete code shapes and the plan's file-map. planner Step 2 becomes reuse-first (freshness / quotability / coverage conditions) with narrowly-bounded delta dispatches (new Delta dispatch contract in plan-recon.md); broad Explore-type grounding sweeps are banned in every mode. Field evidence: 151k (Explore) + 99k (plan-recon) with ~40 % overlap in one bounded cycle. 2. Fieldtest probe tier (fieldtest, fieldtester, pipeline.md, agent-template.md § model rule 2): a cycle whose user-visible delta is a single narrow axis dispatches the per-cycle fieldtest at tier: probe — 1-2 examples on exactly that axis, ~200-word report, explicit model:sonnet dispatch override (no dispatch-level effort override exists; frontmatter effort applies). Probe is a tier, not a skip; the milestone fieldtest always runs full tier on the frontmatter model, keeping the milestone-close gate's >=2-example floor unconditional. 3. Assumption economy (specify Step 3 + self-review item 6): grounding-check cost scales with the spec's assumption count, so specs state current-behaviour claims only where the change relies on them (the gate's own falsity test), demote context-only mentions to ledger citations, and keep iteration scope tight — fewer restatements, never fewer reliances. The gate itself is unchanged. Design reviewed pre-implementation (opus plan review); diff adversarially verified by a 3-lens review workflow (consistency, operability, cross-reference), confirmed findings folded in. refs #35
This commit is contained in:
+50
-11
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: fieldtest
|
||||
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope, implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
|
||||
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope (probe tier, for a single-axis surface delta: 1-2 tasks on that axis), implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
|
||||
---
|
||||
|
||||
# fieldtest — usability field test for a shipped cycle
|
||||
@@ -24,8 +24,8 @@ next to cycle-design specs under `docs/specs`, with a
|
||||
`fieldtest-` prefix in the slug.
|
||||
|
||||
The substantive process — read the design ledger + cycle spec
|
||||
+ recent iter/audit commit bodies, pick 2-4 real-world tasks
|
||||
per cycle axis, implement each as a downstream consumer, run
|
||||
+ recent iter/audit commit bodies, pick the dispatched tier's
|
||||
examples, implement each as a downstream consumer, run
|
||||
the result, classify findings, write the spec — lives in
|
||||
`agents/fieldtester.md`. That file also carries the spec
|
||||
template, the source-isolation discipline (no reading under
|
||||
@@ -61,6 +61,44 @@ here. The milestone fieldtest is skippable on the same terms as
|
||||
the per-cycle one, lifted to milestone level: a milestone whose
|
||||
entire scope is internal (no user-visible surface) is exempt.
|
||||
|
||||
## Two tiers: probe and full
|
||||
|
||||
Orthogonal to scope, the **per-cycle** fieldtest runs at one of
|
||||
two tiers. The orchestrator picks the tier at dispatch and
|
||||
carries it in the carrier's `tier` field — the field, its
|
||||
example bounds, and its report cap are defined authoritatively
|
||||
under **Carrier contract** in `agents/fieldtester.md` and
|
||||
deliberately not restated here:
|
||||
|
||||
- **Full** (the default; `tier` absent): one example per axis
|
||||
the cycle touched — the shape the rest of this file describes.
|
||||
- **Probe**: for a cycle whose user-visible delta is a **single
|
||||
narrow axis** — one flag, one diagnostic, one configuration
|
||||
key. The carrier's `axis_hints` then names exactly that one
|
||||
axis. Field evidence (issue #35): a full fieldtest cost ~89k
|
||||
tokens on a one-flag cycle even at minimal example count, yet
|
||||
that same dispatch found a real self-description bug — the
|
||||
probe tier keeps that signal at a fraction of the cost.
|
||||
|
||||
**Probe is a tier, not a skip.** The skip rules below are
|
||||
unchanged: a surface-touching cycle still runs its fieldtest;
|
||||
probe merely right-sizes the dispatch when the surface delta is
|
||||
one axis. A multi-axis cycle takes the full tier — when in
|
||||
doubt, full.
|
||||
|
||||
**Probe dispatch mechanics.** A probe dispatch passes an
|
||||
explicit `model: sonnet` override on the agent dispatch — the
|
||||
dispatch-level model override takes precedence over the agent's
|
||||
opus frontmatter; the conditioned exception is documented in
|
||||
`../docs/agent-template.md` § model, rule 2. (The direct
|
||||
agent-dispatch surface has no per-dispatch effort override, so
|
||||
the frontmatter effort applies unchanged; the probe's economy
|
||||
comes from the model tier and the one-axis scope.) The
|
||||
judgement surface a probe covers is one axis, and the
|
||||
orchestrator triages every finding anyway. The **milestone
|
||||
fieldtest always runs full tier on the frontmatter model** — its
|
||||
carrier takes no `tier` field.
|
||||
|
||||
## When to Use / Skipping
|
||||
|
||||
Triggers:
|
||||
@@ -116,11 +154,12 @@ carrier.
|
||||
## Dispatch
|
||||
|
||||
Dispatch `fieldtester` with the carrier from the Handoff
|
||||
Contract below. The agent picks 2-4 examples (one per axis
|
||||
the cycle touched; fewer loses the signal on variation, more
|
||||
overflows one readable report), implements them as downstream
|
||||
consumers, runs them, classifies findings, and writes the
|
||||
spec. The two artefact kinds part ways at commit: the **fixtures are
|
||||
Contract below. The agent picks its examples per the dispatched
|
||||
tier (bounds in the `tier` carrier row; on full, one per axis
|
||||
the cycle touched — fewer loses the signal on variation, more
|
||||
overflows one readable report), implements them as
|
||||
downstream consumers, runs them, classifies findings, and
|
||||
writes the spec. The two artefact kinds part ways at commit: the **fixtures are
|
||||
committed as normal code** (they are consumer test assets — suggested
|
||||
subject `fieldtest: <cycle> — <N> examples, <K> findings`), while the
|
||||
**fieldtest spec is a git-ignored working file that is never committed**.
|
||||
@@ -137,10 +176,10 @@ tree as the next cycle's input; it plays no part in the delete decision
|
||||
## Handoff Contract
|
||||
|
||||
`fieldtest` consumes (from orchestrator at cycle close) the carrier
|
||||
fields `cycle_id`, `cycle_scope`, `axis_hints`, and `commit_range`,
|
||||
defined authoritatively under **Carrier contract** in
|
||||
fields `cycle_id`, `cycle_scope`, `axis_hints`, `commit_range`, and
|
||||
`tier`, defined authoritatively under **Carrier contract** in
|
||||
`agents/fieldtester.md` (which also specifies the empty-`axis_hints`
|
||||
fallback).
|
||||
fallback and the absent-`tier` default).
|
||||
|
||||
`fieldtest` produces, for the orchestrator, the fields `spec_path`,
|
||||
`examples_added`, and `findings`, defined authoritatively under
|
||||
|
||||
Reference in New Issue
Block a user