feat(postmortem): aggregate Workflow run logs per stage

Sessions that ran Workflow-substrate pipelines read as near-zero
subagent spend: analyze_subagents() globs one level only, while
Workflow runs log under workflows/wf_*.json (per-agent summary) and
subagents/workflows/<runId>/agent-*.jsonl (raw usage). New
analyze_workflows() joins both — raw transcripts authoritative
(dedup by requestId, as everywhere), summary supplies stage labels
and models, label-prefix grouping (impl:3 -> impl), untracked
transcripts isolated under '?'. Report gains workflows[] and
workflow_totals; totals now include workflow spend.

Live check against a real 17-agent workflow run: raw 1.30M vs
reported 1.22M (the live counter's known ~7-11% under-tracking).

closes #28
This commit is contained in:
2026-07-17 15:23:22 +02:00
parent 22abd37458
commit 7a08259db7
3 changed files with 236 additions and 8 deletions
+19 -5
View File
@@ -1,6 +1,6 @@
---
name: postmortem
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
---
# postmortem — grade the run
@@ -10,9 +10,12 @@ description: Use after a working session (or a `/boss` run) to grade how well th
## Overview
Every session leaves a detailed flight recorder behind — the
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
plus one sidechain log per dispatched subagent under
`<session>/subagents/`. Nobody reads it. So the same expensive
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`,
one sidechain log per dispatched subagent under
`<session>/subagents/`, and for every Workflow run a per-agent
summary in `<session>/workflows/wf_*.json` with raw per-agent
transcripts under `<session>/subagents/workflows/<runId>/`. Nobody
reads it. So the same expensive
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
loop with a 30% error rate, a cache-cold run that re-sends the full
input on every turn. This skill turns that recorder into a graded
@@ -55,7 +58,12 @@ Read before interpreting, so the report never overclaims:
timestamps, every `tool_use` with its input, every `tool_result`
with the authoritative `is_error` flag, skill invocations,
slash commands, and a full separate transcript + `agentType` /
`description` for each subagent.
`description` for each subagent. Workflow runs additionally carry a
per-agent summary (stage label, model, a live token counter) in
`workflows/wf_*.json`; the counter under-tracks vs. the deduped raw
transcripts (~11% in the corpus behind issue #28), so the script
treats the raw transcripts as authoritative and reports the counter
only as `reported_total_tokens` for cross-checking.
- **Absent:** `costUSD` is always `null`; `durationMs` is always
`null`. There is no metered cost in the log, so this skill does
not compute one — it reports **raw token counts** (kept split by
@@ -111,6 +119,12 @@ behind it is an opinion; drop it.
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
fleet that burned real tokens to return `BLOCKED` is the headline
finding. Compare each agent's token spend to what it delivered.
For Workflow runs, grade per stage instead: `workflow_totals` and
each run's `stages` table (label-prefix grouping, e.g. all
`impl:N` under `impl`) show where the run's spend concentrated;
`agents_untracked > 0` flags retried/dropped attempts. Schema-bound
workflow stages carry no `terminal_status` — the workflow script's
own aggregation already judged them.
### Step 3 — Write the report