postmortem: terminal_status text-scan over-reports BLOCKED for shipped implement runs #3

New Issue

2026-06-02T16:16:47+02:00

Brummel commented

2026-06-02 16:16:47 +02:00

The postmortem aggregator (postmortem/scripts/postmortem.py) derives each subagents terminal_statusby scanning the report text for status keywords. Forimplement-orchestratoragents this over-reportsBLOCKED: any report that mentions BLOCKED.md` (the protocol file) or narrates a surmounted blocker in prose is misclassified as a blocked terminal state.

Evidence

In the AILang /boss session 08f4d15c (2026-06-02), six implement-orchestrator runs were flagged BLOCKED by the script. Every one of them demonstrably shipped a commit on main that same session:

poly RawBuf elem-var mono fix → b11a6d9
series kernel module → 35a9bc6
owned-param tail-recursion leak → 68f500c
let-wrapped tail-call leak → e3a9065
type-scoped describe → d565ed8
plan 0126 (#63 leg 3) → d72fe0c

So the field reads 0/6 DONE where git shows 6/6 shipped. The one genuine bounce in that run was a single debugger (leg 3) correctly returning a design fork to brainstorm — which is the success path, not a failure.

Impact

terminal_status is one of the three graded axes (agent effectiveness). A scan that flips every successful implement run to BLOCKED makes the axis actively misleading: a human reading the JSON would conclude the fleet failed when it shipped everything. The postmortem skill`s own Iron Law ("a grade without a cited metric is an opinion") is undermined when the cited metric is wrong.

Fix direction (not prescriptive)

Anchor the scan on the final status line of the end-report (the implement skill emits a fixed end-report structure), not on any keyword occurrence anywhere in the body. The DONE/PARTIAL/BLOCKED end-state is structurally locatable; BLOCKED.md mentions and "we hit a blocker and resolved it" prose are not the terminal status. Until then, the field should be treated as advisory and cross-checked against git, as the 08f4d15c report did manually.

The postmortem aggregator (`postmortem/scripts/postmortem.py`) derives each subagent`s `terminal_status` by scanning the report text for status keywords. For `implement-orchestrator` agents this over-reports `BLOCKED`: any report that mentions `BLOCKED.md` (the protocol file) or narrates a surmounted blocker in prose is misclassified as a blocked terminal state. ## Evidence In the AILang `/boss` session 08f4d15c (2026-06-02), six implement-orchestrator runs were flagged `BLOCKED` by the script. Every one of them demonstrably shipped a commit on `main` that same session: - poly RawBuf elem-var mono fix → `b11a6d9` - series kernel module → `35a9bc6` - owned-param tail-recursion leak → `68f500c` - let-wrapped tail-call leak → `e3a9065` - type-scoped describe → `d565ed8` - plan 0126 (#63 leg 3) → `d72fe0c` So the field reads 0/6 DONE where git shows 6/6 shipped. The one genuine bounce in that run was a single `debugger` (leg 3) correctly returning a design fork to brainstorm — which is the success path, not a failure. ## Impact `terminal_status` is one of the three graded axes (agent effectiveness). A scan that flips every successful implement run to BLOCKED makes the axis actively misleading: a human reading the JSON would conclude the fleet failed when it shipped everything. The postmortem skill`s own Iron Law ("a grade without a cited metric is an opinion") is undermined when the cited metric is wrong. ## Fix direction (not prescriptive) Anchor the scan on the *final* status line of the end-report (the implement skill emits a fixed end-report structure), not on any keyword occurrence anywhere in the body. The DONE/PARTIAL/BLOCKED end-state is structurally locatable; `BLOCKED.md` mentions and "we hit a blocker and resolved it" prose are not the terminal status. Until then, the field should be treated as advisory and cross-checked against git, as the 08f4d15c report did manually.

Brummel referenced this issue from a commit

2026-06-02 16:25:11 +02:00

fix(postmortem): anchor terminal_status on the structured Status line

Brummel closed this issue

2026-06-02 16:25:11 +02:00

claude referenced this issue

2026-07-24 20:11:16 +02:00

Experiment: full pipeline with implement-skills disabled — quality vs token tracking #35

Sign in to join this conversation.