feat(postmortem): aggregate Workflow run logs per stage
Sessions that ran Workflow-substrate pipelines read as near-zero subagent spend: analyze_subagents() globs one level only, while Workflow runs log under workflows/wf_*.json (per-agent summary) and subagents/workflows/<runId>/agent-*.jsonl (raw usage). New analyze_workflows() joins both — raw transcripts authoritative (dedup by requestId, as everywhere), summary supplies stage labels and models, label-prefix grouping (impl:3 -> impl), untracked transcripts isolated under '?'. Report gains workflows[] and workflow_totals; totals now include workflow spend. Live check against a real 17-agent workflow run: raw 1.30M vs reported 1.22M (the live counter's known ~7-11% under-tracking). closes #28
This commit is contained in:
+19
-5
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: postmortem
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
---
|
||||
|
||||
# postmortem — grade the run
|
||||
@@ -10,9 +10,12 @@ description: Use after a working session (or a `/boss` run) to grade how well th
|
||||
## Overview
|
||||
|
||||
Every session leaves a detailed flight recorder behind — the
|
||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
|
||||
plus one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`. Nobody reads it. So the same expensive
|
||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`,
|
||||
one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`, and for every Workflow run a per-agent
|
||||
summary in `<session>/workflows/wf_*.json` with raw per-agent
|
||||
transcripts under `<session>/subagents/workflows/<runId>/`. Nobody
|
||||
reads it. So the same expensive
|
||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||||
loop with a 30% error rate, a cache-cold run that re-sends the full
|
||||
input on every turn. This skill turns that recorder into a graded
|
||||
@@ -55,7 +58,12 @@ Read before interpreting, so the report never overclaims:
|
||||
timestamps, every `tool_use` with its input, every `tool_result`
|
||||
with the authoritative `is_error` flag, skill invocations,
|
||||
slash commands, and a full separate transcript + `agentType` /
|
||||
`description` for each subagent.
|
||||
`description` for each subagent. Workflow runs additionally carry a
|
||||
per-agent summary (stage label, model, a live token counter) in
|
||||
`workflows/wf_*.json`; the counter under-tracks vs. the deduped raw
|
||||
transcripts (~11% in the corpus behind issue #28), so the script
|
||||
treats the raw transcripts as authoritative and reports the counter
|
||||
only as `reported_total_tokens` for cross-checking.
|
||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||||
`null`. There is no metered cost in the log, so this skill does
|
||||
not compute one — it reports **raw token counts** (kept split by
|
||||
@@ -111,6 +119,12 @@ behind it is an opinion; drop it.
|
||||
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
|
||||
fleet that burned real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare each agent's token spend to what it delivered.
|
||||
For Workflow runs, grade per stage instead: `workflow_totals` and
|
||||
each run's `stages` table (label-prefix grouping, e.g. all
|
||||
`impl:N` under `impl`) show where the run's spend concentrated;
|
||||
`agents_untracked > 0` flags retried/dropped attempts. Schema-bound
|
||||
workflow stages carry no `terminal_status` — the workflow script's
|
||||
own aggregation already judged them.
|
||||
|
||||
### Step 3 — Write the report
|
||||
|
||||
|
||||
Reference in New Issue
Block a user