feat(implement-loop): standard end-verify gains an independent suite gate

Standard mode was the only execution path without an end-of-iteration
suite gate: tree-check only counted files, so suite-green rode on the
per-task implementers' self-reports while mini-verify and
compiler-driven-edit both re-run the suite as a hard gate
(pipeline-audit finding). The tree-check stage becomes end-verify with
two independently-gating legs: files_touched (untracked-aware no-op
gate, unchanged semantics) and suite_green (full independent suite
run; red -> BLOCKED, route back to debug RED-first). Costs ~+51k real
tokens per standard DONE iteration vs. the old count-only stage; the
deterministic backstop is what makes the quality-review tiering in #30
defensible.

Known trade-off, by design: a flaky or environment-red suite now
blocks a standard iteration — the same exposure mini-verify has.

closes #29
This commit is contained in:
2026-07-17 15:32:24 +02:00
parent 092bca554a
commit f5ba8c3747
2 changed files with 54 additions and 29 deletions
+41 -20
View File
@@ -394,18 +394,26 @@ const MINI_VERIFY_SCHEMA = {
detail: { type: 'string', description: 'on any false: which test is still red, what regressed, or that the tree is clean' },
},
}
// Standard-mode no-op gate (issue #12, facets 1+2): the authoritative ground
// truth that the iteration's per-task work actually landed. Runs after the loop
// but BEFORE E2E, so E2E fixtures cannot inflate the count (stats/BLOCKED.md
// are no longer written in-loop at all — the orchestrator writes them from the
// end-report, after this gate). Uses `git status --porcelain` (counts brand-new
// UNTRACKED files — the implementer leaves edits unstaged, so a new-file deliverable
// is untracked and `git diff HEAD` would miss it). A self-report can never reach
// this verdict; it is git ground truth.
const TREE_SCHEMA = {
// Standard-mode end-verify (issue #12 facets 1+2 + issue #29 / the pipeline
// audit): the authoritative ground truth that the iteration's per-task work
// actually landed AND left the project suite green. Runs after the loop but
// BEFORE E2E, so E2E fixtures cannot inflate the count (stats/BLOCKED.md are
// no longer written in-loop at all — the orchestrator writes them from the
// end-report, after this gate).
// Two legs, both gating:
// • files_touched from `git status --porcelain` (counts brand-new UNTRACKED
// files — the implementer leaves edits unstaged, so a new-file deliverable
// is untracked and `git diff HEAD` would miss it). 0 → no-op BLOCKED.
// • suite_green from a full independent suite run. Standard mode previously
// rode on the per-task implementers' self-reported green while mini and
// compiler-driven both re-ran the suite as a hard gate — this closes that
// asymmetry. red → BLOCKED (route back to debug, RED-first). Same
// flaky-suite exposure mini-verify already has, by design.
// A self-report can never reach this verdict; it is git/suite ground truth.
const STD_VERIFY_SCHEMA = {
type: 'object',
additionalProperties: false,
required: ['files_touched'],
required: ['files_touched', 'suite_green'],
properties: {
files_touched: {
type: 'integer',
@@ -413,6 +421,13 @@ const TREE_SCHEMA = {
'the line count of `git status --porcelain` (every changed/added/untracked path). Run exactly that — do NOT ' +
'use `git diff`, which omits untracked new files. Do NOT edit anything. 0 means the iteration is a no-op.',
},
suite_green: {
type: 'boolean',
description:
"the FULL project suite passes (the test command from the project's CLAUDE.md facts), re-run by you — " +
'never taken from an earlier report',
},
detail: { type: 'string', description: 'on suite_green false: which tests failed' },
},
}
// Cross-task discard guard (issue #23): the per-boundary snapshot verdict.
@@ -999,23 +1014,29 @@ if (mode === 'mini' && outcome === 'DONE') {
} else if (mode === 'standard' && outcome === 'DONE') {
phase('Verify')
treeState = await agent(
`${STANDING}\n\nReport the working-tree footprint of an implement iteration WITHOUT changing anything. Run ` +
'`git status --porcelain` and report files_touched = its line count (every changed, added, or untracked ' +
'path). Use `git status --porcelain`, NOT `git diff`, so brand-new untracked files are counted. Do NOT edit, ' +
'do NOT commit.',
{ model: 'sonnet', effort: 'medium', label: 'tree-check', phase: 'Verify', schema: TREE_SCHEMA },
`${STANDING}\n\nEnd-verify an implement iteration WITHOUT changing any code or test. Two independent checks:\n` +
'1. Run `git status --porcelain` and report files_touched = its line count (every changed, added, or ' +
'untracked path). Use `git status --porcelain`, NOT `git diff`, so brand-new untracked files are counted.\n' +
'2. Run the FULL project suite (the test command from the CLAUDE.md project facts) and report suite_green — ' +
'on red, name the failing tests in detail.\n' +
'Do NOT edit, do NOT commit.',
{ model: 'sonnet', effort: 'medium', label: 'verify', phase: 'Verify', schema: STD_VERIFY_SCHEMA },
)
if (!treeState || treeState.files_touched === 0) {
if (!treeState || treeState.files_touched === 0 || !treeState.suite_green) {
outcome = 'BLOCKED'
synthBlock = {
task: null,
reason: !treeState
? 'infra: tree-check agent did not return — cannot confirm the iteration landed; treat as not-done'
: 'no-op iteration: `git status --porcelain` is empty — every task reported DONE but nothing landed in the working tree',
detail: null,
? 'infra: end-verify agent did not return — cannot confirm the iteration landed; treat as not-done'
: treeState.files_touched === 0
? 'no-op iteration: `git status --porcelain` is empty — every task reported DONE but nothing landed in the working tree'
: 'suite red at end-verify: the iteration regressed the project suite — route back to debug (RED-first)',
detail: treeState && !treeState.suite_green ? treeState.detail || null : null,
}
// a confirmed-empty tree is a clean no-op (no BLOCKED.md); a null agent is infra
// (tree state unknown — possibly dirty), so leave noopClean false.
// (tree state unknown — possibly dirty), so leave noopClean false. A suite-red
// verdict has a dirty tree by definition — BLOCKED.md is written (by the
// orchestrator, from the end-report).
if (treeState && treeState.files_touched === 0) noopClean = true
}
}