Convert prose-only no-op guards to structural floors (fieldtest examples, audit drift-review) #14

New Issue

2026-06-27T13:21:59+02:00

Brummel commented

2026-06-27 13:21:59 +02:00

Found by the sibling scan while fixing #11. Two more gates are vacuously
satisfiable by a no-op, but — unlike #11, #(A), #(B) — they are already
defended in agent prose; what's missing is a structural floor at the
consuming gate. Lower severity, hence idea (cuttable): the prose guard
may be judged sufficient.

1 — fieldtest `clean` roll-up vs. zero examples

fieldtest/SKILL.md:143-145: the clean status roll-up is the functional
leg of the milestone-close gate (docs/pipeline.md:90-93 — a milestone
closes only when the milestone fieldtest rolls up clean). A fieldtester
that writes zero examples has zero findings and would roll up clean,
vacuously satisfying that gate.

This is defended in prose: fieldtester.md:320-323 forbids "an all-clean
report … without at least 2 examples in the working tree," and the
rationalisations table turns a clean run into a required working finding.
But the guard is prose-only: there is no structural check that
examples_added >= 2 before clean is honoured, and the consuming
milestone gate trusts the roll-up without re-verifying the example count.

2 — audit `clean` / carry-on vs. an architect that reviewed nothing

audit/agents/architect.md:152-153: "an empty drift list with carry on as
planned is a valid and welcome result" drives the audit carry-on / clean
cycle close (audit/SKILL.md:108-111). For a project with no regression
scripts, Step 2 is "a no-op (architect remains the gate)"
(audit/SKILL.md:66), so the architect is the sole gate — and a
no-op architect (empty drift list returned without actually walking the
contracts/diff) reads as a clean cycle close. The success condition checks
"no drift found," never "a review actually happened." Guarded only by the
architect's own red flag (architect.md:175: "About to return clean
without having read the diff in full").

Direction (not prescribing)

Add a structural floor so these gates do not rest on agent diligence alone:
a fieldtest roll-up clean requires examples_added >= 2 to be honoured
at the gate; an audit clean close on a no-regression-script project
requires some evidence-of-review token (files/contracts actually walked).
Both are mild; the prose guard may be deemed enough — that's the call to
make on this issue.

Refs #11.

Found by the sibling scan while fixing #11. Two more gates are vacuously satisfiable by a no-op, but — unlike #11, #(A), #(B) — they are **already defended in agent prose**; what's missing is a *structural* floor at the consuming gate. Lower severity, hence `idea` (cuttable): the prose guard may be judged sufficient. ## 1 — fieldtest `clean` roll-up vs. zero examples `fieldtest/SKILL.md:143-145`: the `clean` status roll-up is the functional leg of the milestone-close gate (`docs/pipeline.md:90-93` — a milestone closes only when the milestone fieldtest rolls up `clean`). A fieldtester that writes **zero examples** has zero findings and would roll up `clean`, vacuously satisfying that gate. This *is* defended in prose: `fieldtester.md:320-323` forbids "an all-clean report … without at least 2 examples in the working tree," and the rationalisations table turns a clean run into a required `working` finding. But the guard is prose-only: there is no structural check that `examples_added >= 2` before `clean` is honoured, and the consuming milestone gate trusts the roll-up without re-verifying the example count. ## 2 — audit `clean` / carry-on vs. an architect that reviewed nothing `audit/agents/architect.md:152-153`: "an empty drift list with carry on as planned is a valid and welcome result" drives the audit carry-on / clean cycle close (`audit/SKILL.md:108-111`). For a project with **no regression scripts**, Step 2 is "a no-op (architect remains the gate)" (`audit/SKILL.md:66`), so the architect is the **sole** gate — and a no-op architect (empty drift list returned without actually walking the contracts/diff) reads as a clean cycle close. The success condition checks "no drift found," never "a review actually happened." Guarded only by the architect's own red flag (`architect.md:175`: "About to return clean without having read the diff in full"). ## Direction (not prescribing) Add a structural floor so these gates do not rest on agent diligence alone: a fieldtest roll-up `clean` requires `examples_added >= 2` to be honoured at the gate; an audit clean close on a no-regression-script project requires some evidence-of-review token (files/contracts actually walked). Both are mild; the prose guard may be deemed enough — that's the call to make on this issue. Refs #11.

Brummel added the idea label 2026-06-27 13:21:59 +02:00

Brummel commented

2026-06-27 14:28:17 +02:00

Fixed in `5a9a2ae` (closes on push)

Both roll-ups were vacuously satisfiable by a no-op and guarded only in agent prose, never re-verified at the consuming gate:

fieldtest milestone-close (docs/pipeline.md): the functional leg now requires examples_added >= 2 for a clean roll-up to be honoured (the empty-report and internal-milestone escapes preserved); a clean with fewer over user-visible surface is itself a spec_gap.
audit carry-on (audit/SKILL.md): now requires a non-empty What holds (added as an explicit handoff field) as evidence of review; a bare clean with nothing held — worst when no regression scripts run, so the architect is the sole gate — reads as unreviewed and is re-dispatched.

Design note / the call this idea asked for: these are second-layer checks (the consumer verifies the field rather than trusting the agent's discipline) — the same two-layer shape as #11/#12, but prose-on-prose, not code-level floors. Milestone-close and audit carry-on are orchestrator-judgment acts with no script to instrument, so a structural code floor isn't available without converting those skills into workflows (not worth it for this severity). A no-op architect that fabricates a plausible "What holds" still defeats the audit guard; only the empty-report shape — exactly what the issue targeted — is caught.

Adversarially reviewed SOUND (genuine clean cycles / internal milestones / no-applicable-examples runs still pass; no false positive).

## Fixed in 5a9a2ae (closes on push) Both roll-ups were vacuously satisfiable by a no-op and guarded **only in agent prose**, never re-verified at the consuming gate: - **fieldtest milestone-close** (`docs/pipeline.md`): the functional leg now requires `examples_added >= 2` for a `clean` roll-up to be honoured (the empty-report and internal-milestone escapes preserved); a `clean` with fewer over user-visible surface is itself a `spec_gap`. - **audit carry-on** (`audit/SKILL.md`): now requires a non-empty **What holds** (added as an explicit handoff field) as evidence of review; a bare `clean` with nothing held — worst when no regression scripts run, so the architect is the *sole* gate — reads as unreviewed and is re-dispatched. **Design note / the call this `idea` asked for:** these are **second-layer** checks (the consumer verifies the field rather than trusting the agent's discipline) — the same two-layer shape as #11/#12, but **prose-on-prose, not code-level floors**. Milestone-close and audit carry-on are orchestrator-judgment acts with no script to instrument, so a structural code floor isn't available without converting those skills into workflows (not worth it for this severity). A no-op architect that *fabricates* a plausible "What holds" still defeats the audit guard; only the empty-report shape — exactly what the issue targeted — is caught. Adversarially reviewed **SOUND** (genuine clean cycles / internal milestones / no-applicable-examples runs still pass; no false positive).

Brummel referenced this issue from a commit

2026-06-29 11:08:57 +02:00

fix(pipeline,audit): gate-side floors for prose-only no-op guards

Brummel closed this issue

2026-06-29 11:08:57 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Skills#14

Convert prose-only no-op guards to structural floors (fieldtest examples, audit drift-review) #14

1 — fieldtest clean roll-up vs. zero examples

2 — audit clean / carry-on vs. an architect that reviewed nothing

Direction (not prescribing)

Fixed in 5a9a2ae (closes on push)

1 — fieldtest `clean` roll-up vs. zero examples

2 — audit `clean` / carry-on vs. an architect that reviewed nothing

Fixed in `5a9a2ae` (closes on push)