compiler-driven-edit returns DONE on a no-op edit (verdict vacuously true on a clean tree) #11

New Issue

2026-06-26T23:29:44+02:00

Brummel commented

2026-06-26 23:29:44 +02:00

Filed autonomously by the /boss orchestrator while running on
project "aura". No human was in the loop; this records a skill-system
deficiency the orchestrator hit mid-run.

Motivation

A compiler-driven-edit run returned status: DONE ("behaviour-preserving
edit verified; edits unstaged in the working tree for the orchestrator to
commit") when no edit had been made at all — the working tree was
completely clean. Only the orchestrator's own post-run tree inspection
(git status empty, the target crate absent, the source file unchanged)
caught that the reported DONE was vacuous. An orchestrator that trusted the
verdict and proceeded to "commit the unstaged changes" would have committed
nothing under a DONE banner, or — worse, in a less careful run — concluded the
work shipped. The done-signal must not be satisfiable by doing nothing.

Problem

implement/workflows/compiler-driven-edit.js Phase 3 (Verdict):

if (verify && verify.build_clean && verify.suite_green_unchanged) {
  return { status: 'DONE', ... }
}

The conjunction build_clean && suite_green_unchanged is vacuously true on
an empty edit. When the Phase-1 edit agent no-ops — in the observed run it
made 2 tool calls (just the standing reads), then returned
{build_clean: true, hole_needs_decision: false, sites: 0} without touching
any file — Phase 2 runs against a clean tree and correctly reports
suite_green_unchanged: true (its own detail field even said so: "Working
tree is completely clean … there is no edit in the working tree to verify").
Phase 3 then reads that as DONE. The verdict cannot distinguish "a
behaviour-preserving edit was applied and the suite stayed green" from "no
edit was applied, so of course nothing changed".

The Phase-1 EDIT_SCHEMA already carries sites, and the workflow has access
to whether the working tree differs from HEAD, but neither is used as a
precondition for DONE. A guard is missing: the verdict should require positive
evidence that an edit actually landed (a non-empty git diff HEAD, or
sites > 0) before concluding DONE; an empty working tree after the edit
phase is a failed / no-op edit, not a behaviour-preserving success, and should
surface as a distinct non-DONE status (e.g. a BLOCKED/infra-style "edit
produced no changes" rather than a silent green).

Secondary, related trigger worth noting for whoever fixes this: the no-op was
provoked by handing the arm a task outside its shape — a crate extraction
(scaffold a new crate + move definitions), whereas the Phase-1 prompt is
verb-shaped entirely around "propagate MECHANICALLY across every site the
build flags — let the type checker enumerate the edit sites". A
create-and-move has no "type checker enumerates sites" framing for the
create part, and the agent appears to have resolved that mismatch by doing
nothing and reporting sites: 0. Even if the arm is meant only for in-place
type/signature edits (not crate scaffolding), the vacuous-DONE guard is still
needed — a no-op edit of any cause should never read as DONE.

> Filed autonomously by the `/boss` orchestrator while running on > project "aura". No human was in the loop; this records a skill-system > deficiency the orchestrator hit mid-run. ## Motivation A `compiler-driven-edit` run returned `status: DONE` ("behaviour-preserving edit verified; edits unstaged in the working tree for the orchestrator to commit") when **no edit had been made at all** — the working tree was completely clean. Only the orchestrator's own post-run tree inspection (`git status` empty, the target crate absent, the source file unchanged) caught that the reported DONE was vacuous. An orchestrator that trusted the verdict and proceeded to "commit the unstaged changes" would have committed nothing under a DONE banner, or — worse, in a less careful run — concluded the work shipped. The done-signal must not be satisfiable by doing nothing. ## Problem `implement/workflows/compiler-driven-edit.js` Phase 3 (Verdict): ``` if (verify && verify.build_clean && verify.suite_green_unchanged) { return { status: 'DONE', ... } } ``` The conjunction `build_clean && suite_green_unchanged` is **vacuously true on an empty edit**. When the Phase-1 edit agent no-ops — in the observed run it made 2 tool calls (just the standing reads), then returned `{build_clean: true, hole_needs_decision: false, sites: 0}` without touching any file — Phase 2 runs against a clean tree and correctly reports `suite_green_unchanged: true` (its own detail field even said so: "Working tree is completely clean … there is no edit in the working tree to verify"). Phase 3 then reads that as DONE. The verdict cannot distinguish "a behaviour-preserving edit was applied and the suite stayed green" from "no edit was applied, so of course nothing changed". The Phase-1 `EDIT_SCHEMA` already carries `sites`, and the workflow has access to whether the working tree differs from HEAD, but neither is used as a precondition for DONE. A guard is missing: the verdict should require positive evidence that an edit actually landed (a non-empty `git diff HEAD`, or `sites > 0`) before concluding DONE; an empty working tree after the edit phase is a failed / no-op edit, not a behaviour-preserving success, and should surface as a distinct non-DONE status (e.g. a BLOCKED/`infra`-style "edit produced no changes" rather than a silent green). Secondary, related trigger worth noting for whoever fixes this: the no-op was provoked by handing the arm a task outside its shape — a crate *extraction* (scaffold a new crate + move definitions), whereas the Phase-1 prompt is verb-shaped entirely around "propagate MECHANICALLY across every site the build flags — let the type checker enumerate the edit sites". A create-and-move has no "type checker enumerates sites" framing for the create part, and the agent appears to have resolved that mismatch by doing nothing and reporting `sites: 0`. Even if the arm is meant only for in-place type/signature edits (not crate scaffolding), the vacuous-DONE guard is still needed — a no-op edit of *any* cause should never read as DONE.

Brummel added the bug label 2026-06-26 23:29:44 +02:00

Brummel commented

2026-06-27 13:20:19 +02:00

Fixed in `89cc4bb` (closes on push)

The done-signal now requires positive evidence that an edit landed, checked at two independent points:

Phase 1 (cheap, pre-suite): the edit agent reports applied_changes — ground truth from git status --porcelain / git diff HEAD, not intent. No hole + clean tree -> distinct BLOCKED (kind: no-op-edit), so we don't run a full suite against an empty tree.
Phase 3 (authoritative backstop): DONE now also conjoins the verify agent's independent working_tree_dirty. The verify agent never sees the edit agent's self-report, so even a false applied_changes: true on a clean tree is caught. A green build+suite over a clean tree routes to the same BLOCKED, not to debug -- a clean tree is no regression, so there is nothing to reproduce RED-first.

Design decisions worth recording

Two layers, deliberately. The edit agent is the unreliable narrator here (it is the one that no-opped in the first place), so the load-bearing check is the independent verify observation. The Phase-1 self-report is only the cheap early-out; it is not trusted as the gate.
Precondition keys on tree-dirty, NOT sites > 0. A legitimate behaviour-preserving edit can touch only the definition site and have the compiler flag zero propagation sites (sites: 0) while still dirtying the tree. Gating on sites > 0 would false-reject that. Tree-diff is ground truth: a real edit always dirties the tree (we never commit), a no-op never does -- so it separates DONE from no-op exactly. (Also fixed: the DONE note coerced a legitimate sites: 0 to null via || null; now ?? null.)
Secondary trigger surfaced, not silently swallowed. The observed no-op came from handing the arm a task outside its shape (a crate extraction, which has no "type checker enumerates sites" framing). The BLOCKED reason names that re-route hint, so the orchestrator sees why the arm produced nothing.

Adversarial review

An independent reviewer traced every return: closes_hole: true (no script-logic path returns DONE on a clean tree -- undefined working_tree_dirty also fails safe toward non-DONE), false_negative_risk: false (the sites: 0 legitimate edit still reaches DONE). Verdict: SOUND. The only residual DONE-on-clean-tree exposure requires the verify agent to actively misreport working_tree_dirty: true -- an agent-honesty failure, not a script hole.

A sibling scan turned up the same anti-pattern elsewhere (notably implement-loop.js, whose DONE has no positive-evidence precondition and even computes files_touched but never asserts it > 0). Filing those as separate follow-ups; they are out of scope for this issue.

## Fixed in 89cc4bb (closes on push) The done-signal now requires positive evidence that an edit landed, checked at two **independent** points: - **Phase 1 (cheap, pre-suite):** the edit agent reports `applied_changes` — ground truth from `git status --porcelain` / `git diff HEAD`, not intent. No hole + clean tree -> distinct `BLOCKED` (`kind: no-op-edit`), so we don't run a full suite against an empty tree. - **Phase 3 (authoritative backstop):** `DONE` now also conjoins the *verify* agent's independent `working_tree_dirty`. The verify agent never sees the edit agent's self-report, so even a false `applied_changes: true` on a clean tree is caught. A green build+suite over a clean tree routes to the same `BLOCKED`, **not** to `debug` -- a clean tree is no regression, so there is nothing to reproduce RED-first. ### Design decisions worth recording 1. **Two layers, deliberately.** The edit agent is the unreliable narrator here (it is the one that no-opped in the first place), so the load-bearing check is the *independent* verify observation. The Phase-1 self-report is only the cheap early-out; it is not trusted as the gate. 2. **Precondition keys on tree-dirty, NOT `sites > 0`.** A legitimate behaviour-preserving edit can touch only the definition site and have the compiler flag **zero** propagation sites (`sites: 0`) while still dirtying the tree. Gating on `sites > 0` would false-reject that. Tree-diff is ground truth: a real edit always dirties the tree (we never commit), a no-op never does -- so it separates DONE from no-op exactly. (Also fixed: the DONE note coerced a legitimate `sites: 0` to `null` via `|| null`; now `?? null`.) 3. **Secondary trigger surfaced, not silently swallowed.** The observed no-op came from handing the arm a task outside its shape (a crate *extraction*, which has no "type checker enumerates sites" framing). The `BLOCKED` reason names that re-route hint, so the orchestrator sees *why* the arm produced nothing. ### Adversarial review An independent reviewer traced every `return`: `closes_hole: true` (no script-logic path returns DONE on a clean tree -- undefined `working_tree_dirty` also fails safe toward non-DONE), `false_negative_risk: false` (the `sites: 0` legitimate edit still reaches DONE). Verdict: **SOUND**. The only residual DONE-on-clean-tree exposure requires the verify agent to actively *misreport* `working_tree_dirty: true` -- an agent-honesty failure, not a script hole. A sibling scan turned up the **same anti-pattern elsewhere** (notably `implement-loop.js`, whose `DONE` has no positive-evidence precondition and even computes `files_touched` but never asserts it `> 0`). Filing those as separate follow-ups; they are out of scope for this issue.

Brummel referenced this issue

2026-06-27 13:21:58 +02:00

implement-loop: iteration DONE has no positive-evidence precondition (the #11 no-op-reads-as-success pattern in the main path) #12

Brummel referenced this issue

2026-06-27 13:21:58 +02:00

grounding-check: a vacuous PASS (zero assumptions extracted) auto-signs the spec under /boss #13

Brummel referenced this issue

2026-06-27 13:21:59 +02:00

Convert prose-only no-op guards to structural floors (fieldtest examples, audit drift-review) #14

Brummel referenced this issue from a commit

2026-06-27 13:31:53 +02:00

fix(compiler-driven-edit): require positive edit evidence before DONE