feat(implement-loop): standard end-verify gains an independent suite gate
Standard mode was the only execution path without an end-of-iteration suite gate: tree-check only counted files, so suite-green rode on the per-task implementers' self-reports while mini-verify and compiler-driven-edit both re-run the suite as a hard gate (pipeline-audit finding). The tree-check stage becomes end-verify with two independently-gating legs: files_touched (untracked-aware no-op gate, unchanged semantics) and suite_green (full independent suite run; red -> BLOCKED, route back to debug RED-first). Costs ~+51k real tokens per standard DONE iteration vs. the old count-only stage; the deterministic backstop is what makes the quality-review tiering in #30 defensible. Known trade-off, by design: a flaky or environment-red suite now blocks a standard iteration — the same exposure mini-verify has. closes #29
This commit is contained in:
+13
-9
@@ -114,6 +114,7 @@ NEVER PUSH PAST `BLOCKED` BY HAND.
|
|||||||
ON `PARTIAL` OR `BLOCKED`, THE END-REPORT CARRIES THE `BLOCKED.md` CONTENT (`artifacts.blocked_md`); THE ORCHESTRATOR'S FIRST ACTION ON READING SUCH A REPORT IS WRITING IT VERBATIM TO `BLOCKED.md` AT THE REPO ROOT — UNCOMMITTED BY CONVENTION. ON `DONE`, `artifacts.blocked_md` IS NULL. (EXCEPTIONS WHERE IT IS ALSO NULL: A NO-OP ITERATION — ITS TREE IS ALREADY CLEAN — AND ANY PRE-LOOP EARLY EXIT (MALFORMED CARRIER, DIRTY-TREE PREFLIGHT, PLAN-EXTRACT GUARD): NOTHING RAN AND NOTHING WAS WRITTEN, SO THE STATUS + `infra:`-PREFIXED REASON RIDE THE END-REPORT AND THERE IS NOTHING TO CLEAN UP OR EXPLAIN.)
|
ON `PARTIAL` OR `BLOCKED`, THE END-REPORT CARRIES THE `BLOCKED.md` CONTENT (`artifacts.blocked_md`); THE ORCHESTRATOR'S FIRST ACTION ON READING SUCH A REPORT IS WRITING IT VERBATIM TO `BLOCKED.md` AT THE REPO ROOT — UNCOMMITTED BY CONVENTION. ON `DONE`, `artifacts.blocked_md` IS NULL. (EXCEPTIONS WHERE IT IS ALSO NULL: A NO-OP ITERATION — ITS TREE IS ALREADY CLEAN — AND ANY PRE-LOOP EARLY EXIT (MALFORMED CARRIER, DIRTY-TREE PREFLIGHT, PLAN-EXTRACT GUARD): NOTHING RAN AND NOTHING WAS WRITTEN, SO THE STATUS + `infra:`-PREFIXED REASON RIDE THE END-REPORT AND THERE IS NOTHING TO CLEAN UP OR EXPLAIN.)
|
||||||
`DONE` REQUIRES POSITIVE EVIDENCE THAT AN EDIT LANDED — A NO-OP ITERATION (`git status --porcelain` EMPTY, untracked files INCLUDED) IS NEVER `DONE`, IT IS `BLOCKED` (the issue #11 vacuous-green shape, guarded on the main path). THE VERDICT IS THE GIT TREE COUNT, NEVER A SELF-REPORT.
|
`DONE` REQUIRES POSITIVE EVIDENCE THAT AN EDIT LANDED — A NO-OP ITERATION (`git status --porcelain` EMPTY, untracked files INCLUDED) IS NEVER `DONE`, IT IS `BLOCKED` (the issue #11 vacuous-green shape, guarded on the main path). THE VERDICT IS THE GIT TREE COUNT, NEVER A SELF-REPORT.
|
||||||
IN MINI MODE THE RED->GREEN IS OBSERVED, NOT ASSERTED: AN INDEPENDENT VERIFY RE-RUNS THE HANDED-OFF RED TEST AND THE SUITE; A STILL-RED TEST OR A REGRESSION IS `BLOCKED` (route back to debug, RED-first).
|
IN MINI MODE THE RED->GREEN IS OBSERVED, NOT ASSERTED: AN INDEPENDENT VERIFY RE-RUNS THE HANDED-OFF RED TEST AND THE SUITE; A STILL-RED TEST OR A REGRESSION IS `BLOCKED` (route back to debug, RED-first).
|
||||||
|
IN STANDARD MODE THE END-VERIFY RE-RUNS THE FULL SUITE INDEPENDENTLY: SUITE RED IS `BLOCKED` (route back to debug, RED-first) — GREEN IS OBSERVED, NEVER TAKEN FROM PER-TASK SELF-REPORTS.
|
||||||
THE COMPILER-DRIVEN ARM COMMITS ONLY ON CLEAN BUILD + SUITE GREEN UNCHANGED; ELSE IT BOUNCES (specify for a design hole, tdd for discovered new behaviour, debug for a regression).
|
THE COMPILER-DRIVEN ARM COMMITS ONLY ON CLEAN BUILD + SUITE GREEN UNCHANGED; ELSE IT BOUNCES (specify for a design hole, tdd for discovered new behaviour, debug for a regression).
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -141,15 +142,18 @@ positive evidence that an edit landed. The verdict is taken from **git
|
|||||||
ground truth, never a self-report** — a self-report must never be able to
|
ground truth, never a self-report** — a self-report must never be able to
|
||||||
*fail* a run that actually did the work:
|
*fail* a run that actually did the work:
|
||||||
|
|
||||||
- **Iteration gate (standard mode)** — a dedicated `tree-check` agent runs
|
- **Iteration gate (standard mode)** — a dedicated end-`verify` agent runs
|
||||||
`git status --porcelain | wc -l` after the per-task loop but **before**
|
after the per-task loop but **before** E2E, with two independently-gating
|
||||||
E2E and finalize. A `DONE` outcome with a zero count is a no-op iteration
|
legs: `git status --porcelain | wc -l` (a `DONE` outcome with a zero count
|
||||||
→ `BLOCKED`. `git status --porcelain` (not `git diff HEAD`) is used so a
|
is a no-op iteration → `BLOCKED`) and a full **independent suite run**
|
||||||
brand-new **untracked** file — the implementer leaves edits unstaged —
|
(suite red → `BLOCKED`, route back to `debug` RED-first — standard mode
|
||||||
still counts; running before E2E/finalize keeps fixtures and the
|
previously had no end-of-iteration suite gate while mini and
|
||||||
stats/`BLOCKED.md` artefacts from inflating it; and a non-returning agent
|
compiler-driven both re-run the suite; the asymmetry is closed).
|
||||||
is treated as `BLOCKED` (infra), never waved through. A clean-tree no-op
|
`git status --porcelain` (not `git diff HEAD`) is used so a brand-new
|
||||||
writes no `BLOCKED.md` (nothing to clean up — see Step 4).
|
**untracked** file — the implementer leaves edits unstaged — still counts;
|
||||||
|
running before E2E keeps fixtures from inflating it; and a non-returning
|
||||||
|
agent is treated as `BLOCKED` (infra), never waved through. A clean-tree
|
||||||
|
no-op carries no `BLOCKED.md` content (nothing to clean up — see Step 4).
|
||||||
- **Mini mode gate** — an independent `mini-verify` agent re-runs the
|
- **Mini mode gate** — an independent `mini-verify` agent re-runs the
|
||||||
handed-off RED test and the suite and checks the tree is dirty
|
handed-off RED test and the suite and checks the tree is dirty
|
||||||
(`git status --porcelain`); a still-red test, a regression, or a clean
|
(`git status --porcelain`); a still-red test, a regression, or a clean
|
||||||
|
|||||||
@@ -394,18 +394,26 @@ const MINI_VERIFY_SCHEMA = {
|
|||||||
detail: { type: 'string', description: 'on any false: which test is still red, what regressed, or that the tree is clean' },
|
detail: { type: 'string', description: 'on any false: which test is still red, what regressed, or that the tree is clean' },
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
// Standard-mode no-op gate (issue #12, facets 1+2): the authoritative ground
|
// Standard-mode end-verify (issue #12 facets 1+2 + issue #29 / the pipeline
|
||||||
// truth that the iteration's per-task work actually landed. Runs after the loop
|
// audit): the authoritative ground truth that the iteration's per-task work
|
||||||
// but BEFORE E2E, so E2E fixtures cannot inflate the count (stats/BLOCKED.md
|
// actually landed AND left the project suite green. Runs after the loop but
|
||||||
// are no longer written in-loop at all — the orchestrator writes them from the
|
// BEFORE E2E, so E2E fixtures cannot inflate the count (stats/BLOCKED.md are
|
||||||
// end-report, after this gate). Uses `git status --porcelain` (counts brand-new
|
// no longer written in-loop at all — the orchestrator writes them from the
|
||||||
// UNTRACKED files — the implementer leaves edits unstaged, so a new-file deliverable
|
// end-report, after this gate).
|
||||||
// is untracked and `git diff HEAD` would miss it). A self-report can never reach
|
// Two legs, both gating:
|
||||||
// this verdict; it is git ground truth.
|
// • files_touched from `git status --porcelain` (counts brand-new UNTRACKED
|
||||||
const TREE_SCHEMA = {
|
// files — the implementer leaves edits unstaged, so a new-file deliverable
|
||||||
|
// is untracked and `git diff HEAD` would miss it). 0 → no-op BLOCKED.
|
||||||
|
// • suite_green from a full independent suite run. Standard mode previously
|
||||||
|
// rode on the per-task implementers' self-reported green while mini and
|
||||||
|
// compiler-driven both re-ran the suite as a hard gate — this closes that
|
||||||
|
// asymmetry. red → BLOCKED (route back to debug, RED-first). Same
|
||||||
|
// flaky-suite exposure mini-verify already has, by design.
|
||||||
|
// A self-report can never reach this verdict; it is git/suite ground truth.
|
||||||
|
const STD_VERIFY_SCHEMA = {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
additionalProperties: false,
|
additionalProperties: false,
|
||||||
required: ['files_touched'],
|
required: ['files_touched', 'suite_green'],
|
||||||
properties: {
|
properties: {
|
||||||
files_touched: {
|
files_touched: {
|
||||||
type: 'integer',
|
type: 'integer',
|
||||||
@@ -413,6 +421,13 @@ const TREE_SCHEMA = {
|
|||||||
'the line count of `git status --porcelain` (every changed/added/untracked path). Run exactly that — do NOT ' +
|
'the line count of `git status --porcelain` (every changed/added/untracked path). Run exactly that — do NOT ' +
|
||||||
'use `git diff`, which omits untracked new files. Do NOT edit anything. 0 means the iteration is a no-op.',
|
'use `git diff`, which omits untracked new files. Do NOT edit anything. 0 means the iteration is a no-op.',
|
||||||
},
|
},
|
||||||
|
suite_green: {
|
||||||
|
type: 'boolean',
|
||||||
|
description:
|
||||||
|
"the FULL project suite passes (the test command from the project's CLAUDE.md facts), re-run by you — " +
|
||||||
|
'never taken from an earlier report',
|
||||||
|
},
|
||||||
|
detail: { type: 'string', description: 'on suite_green false: which tests failed' },
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
// Cross-task discard guard (issue #23): the per-boundary snapshot verdict.
|
// Cross-task discard guard (issue #23): the per-boundary snapshot verdict.
|
||||||
@@ -999,23 +1014,29 @@ if (mode === 'mini' && outcome === 'DONE') {
|
|||||||
} else if (mode === 'standard' && outcome === 'DONE') {
|
} else if (mode === 'standard' && outcome === 'DONE') {
|
||||||
phase('Verify')
|
phase('Verify')
|
||||||
treeState = await agent(
|
treeState = await agent(
|
||||||
`${STANDING}\n\nReport the working-tree footprint of an implement iteration WITHOUT changing anything. Run ` +
|
`${STANDING}\n\nEnd-verify an implement iteration WITHOUT changing any code or test. Two independent checks:\n` +
|
||||||
'`git status --porcelain` and report files_touched = its line count (every changed, added, or untracked ' +
|
'1. Run `git status --porcelain` and report files_touched = its line count (every changed, added, or ' +
|
||||||
'path). Use `git status --porcelain`, NOT `git diff`, so brand-new untracked files are counted. Do NOT edit, ' +
|
'untracked path). Use `git status --porcelain`, NOT `git diff`, so brand-new untracked files are counted.\n' +
|
||||||
'do NOT commit.',
|
'2. Run the FULL project suite (the test command from the CLAUDE.md project facts) and report suite_green — ' +
|
||||||
{ model: 'sonnet', effort: 'medium', label: 'tree-check', phase: 'Verify', schema: TREE_SCHEMA },
|
'on red, name the failing tests in detail.\n' +
|
||||||
|
'Do NOT edit, do NOT commit.',
|
||||||
|
{ model: 'sonnet', effort: 'medium', label: 'verify', phase: 'Verify', schema: STD_VERIFY_SCHEMA },
|
||||||
)
|
)
|
||||||
if (!treeState || treeState.files_touched === 0) {
|
if (!treeState || treeState.files_touched === 0 || !treeState.suite_green) {
|
||||||
outcome = 'BLOCKED'
|
outcome = 'BLOCKED'
|
||||||
synthBlock = {
|
synthBlock = {
|
||||||
task: null,
|
task: null,
|
||||||
reason: !treeState
|
reason: !treeState
|
||||||
? 'infra: tree-check agent did not return — cannot confirm the iteration landed; treat as not-done'
|
? 'infra: end-verify agent did not return — cannot confirm the iteration landed; treat as not-done'
|
||||||
: 'no-op iteration: `git status --porcelain` is empty — every task reported DONE but nothing landed in the working tree',
|
: treeState.files_touched === 0
|
||||||
detail: null,
|
? 'no-op iteration: `git status --porcelain` is empty — every task reported DONE but nothing landed in the working tree'
|
||||||
|
: 'suite red at end-verify: the iteration regressed the project suite — route back to debug (RED-first)',
|
||||||
|
detail: treeState && !treeState.suite_green ? treeState.detail || null : null,
|
||||||
}
|
}
|
||||||
// a confirmed-empty tree is a clean no-op (no BLOCKED.md); a null agent is infra
|
// a confirmed-empty tree is a clean no-op (no BLOCKED.md); a null agent is infra
|
||||||
// (tree state unknown — possibly dirty), so leave noopClean false.
|
// (tree state unknown — possibly dirty), so leave noopClean false. A suite-red
|
||||||
|
// verdict has a dirty tree by definition — BLOCKED.md is written (by the
|
||||||
|
// orchestrator, from the end-report).
|
||||||
if (treeState && treeState.files_touched === 0) noopClean = true
|
if (treeState && treeState.files_touched === 0) noopClean = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user