Tier the quality review by independently-measured diff size #30

New Issue

2026-07-17T15:20:05+02:00

claude commented

2026-07-17 15:20:05 +02:00

Motivation

The per-task quality review (qual, opus/xhigh) plus its repair loop is the pipeline's largest cost pool: 51M of 164.5M real tokens (31%) over 206 implement-loop runs (local workflow-run telemetry, 2026-06-17..2026-07-17), and opus weighs ~5× on the quota side. The opus pin is justified in docs/agent-template.md by consequence-of-a-miss — which scales with diff size and drops further once standard mode gains an independent end-of-iteration suite gate (the companion overhead issue).

Mechanism

The spec-compliance reviewer — which runs before the quality review, reads the diff itself, and is independent of the implementer (no producer self-report) — additionally reports diff_magnitude (changed lines) and touches_contract (whether the diff touches design-ledger/contract-referenced paths).
The workflow script picks the quality tier deterministically: diff_magnitude <= 25 AND NOT touches_contract → sonnet/high; otherwise opus/xhigh, unchanged.
Contract-touching diffs are excluded from the downgrade because a five-line change to a contract is exactly where a miss is expensive regardless of size.

Acceptance

Tier choice logged per task in the end-report so the split is auditable.
Sampling phase before the threshold counts as proven: for the first ~10 real small-diff iterations after this lands, each sonnet quality verdict is compared against a one-off opus second review of the same diff; divergences (a finding the sonnet review missed at severity important or higher) are recorded on this issue. The threshold stands only if the sample shows no such divergence; otherwise the threshold drops or the change reverts.
docs/agent-template.md model-tier rule gains the size-conditioned case; quality-reviewer.md notes the tier split.

## Motivation The per-task quality review (`qual`, opus/xhigh) plus its repair loop is the pipeline's largest cost pool: 51M of 164.5M real tokens (31%) over 206 implement-loop runs (local workflow-run telemetry, 2026-06-17..2026-07-17), and opus weighs ~5× on the quota side. The opus pin is justified in `docs/agent-template.md` by consequence-of-a-miss — which scales with diff size and drops further once standard mode gains an independent end-of-iteration suite gate (the companion overhead issue). ## Mechanism - The spec-compliance reviewer — which runs before the quality review, reads the diff itself, and is independent of the implementer (no producer self-report) — additionally reports `diff_magnitude` (changed lines) and `touches_contract` (whether the diff touches design-ledger/contract-referenced paths). - The workflow script picks the quality tier deterministically: `diff_magnitude <= 25` AND NOT `touches_contract` → sonnet/high; otherwise opus/xhigh, unchanged. - Contract-touching diffs are excluded from the downgrade because a five-line change to a contract is exactly where a miss is expensive regardless of size. ## Acceptance - [ ] Tier choice logged per task in the end-report so the split is auditable. - [ ] Sampling phase before the threshold counts as proven: for the first ~10 real small-diff iterations after this lands, each sonnet quality verdict is compared against a one-off opus second review of the same diff; divergences (a finding the sonnet review missed at severity `important` or higher) are recorded on this issue. The threshold stands only if the sample shows no such divergence; otherwise the threshold drops or the change reverts. - [ ] `docs/agent-template.md` model-tier rule gains the size-conditioned case; `quality-reviewer.md` notes the tier split.

claude added the feature label 2026-07-17 15:20:05 +02:00

claude self-assigned this 2026-07-17 15:20:06 +02:00

claude referenced this issue

2026-07-17 15:20:21 +02:00

Remove the postmortem skill #27

Brummel referenced this issue from a commit

2026-07-17 16:01:20 +02:00

feat(implement-loop): standard end-verify gains an independent suite gate

Brummel referenced this issue from a commit

2026-07-17 16:01:20 +02:00

feat(implement-loop): tier the quality review by independently-measured diff size

Brummel closed this issue

2026-07-17 16:01:20 +02:00

claude commented

2026-07-17 16:01:34 +02:00

The tiering mechanism is live as of 44f45d1: the spec-reviewer reports diff_magnitude/touches_contract, the loop picks sonnet/high at <=25 changed lines on non-contract paths, and the chosen tier lands in the end-report (qual_tier). Verified on two sandbox runs: a mini iteration (1-line diff) and a 2-task standard iteration both dispatched quality at sonnet/high; a mocked 50-line diff selected opus/xhigh, and missing measurements fall back to opus (fail-conservative).

The acceptance sampling window is open from the next real consumer small-diff iterations onward: each sonnet quality verdict gets a one-off opus second review of the same diff until ~10 samples exist; any missed important+ finding gets recorded here. Sandbox/trivial iterations do not count toward the sample.

The tiering mechanism is live as of 44f45d1: the spec-reviewer reports `diff_magnitude`/`touches_contract`, the loop picks sonnet/high at <=25 changed lines on non-contract paths, and the chosen tier lands in the end-report (`qual_tier`). Verified on two sandbox runs: a mini iteration (1-line diff) and a 2-task standard iteration both dispatched quality at sonnet/high; a mocked 50-line diff selected opus/xhigh, and missing measurements fall back to opus (fail-conservative). The acceptance sampling window is open from the next real consumer small-diff iterations onward: each sonnet quality verdict gets a one-off opus second review of the same diff until ~10 samples exist; any missed important+ finding gets recorded here. Sandbox/trivial iterations do not count toward the sample.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Skills#30