Brummel
daa7b49f8f
A new utility skill that grades a finished session on three axes — cost & efficiency, toolchain health, agent effectiveness — from the session's own Claude Code flight recorder (the JSONL transcript plus one sidechain log per dispatched subagent). The heavy lifting lives in scripts/postmortem.py, a stdlib-only, read-only aggregator. It handles the two accounting traps the raw log sets: usage is repeated per streamed assistant line under a shared requestId (deduped max-per-field, then summed across requests), and the transcript carries no costUSD/durationMs (both null) — so cost is derived from tokens x an overridable list-price table and labelled an estimate, wall-clock from timestamp deltas. Subagent spend is billed separately from its own usage objects and graded by terminal status. Single-session by design: defaults to the newest transcript in the project's log dir (flagged active if still live), or --session <id> for a finished run. Secret files are out of bounds in both the Iron Law and the Red Flags. Verified end-to-end against a real 15-subagent session.
207 lines
8.6 KiB
Markdown
207 lines
8.6 KiB
Markdown
---
|
||
name: postmortem
|
||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and what the run cost. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/cost/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||
---
|
||
|
||
# postmortem — grade the run
|
||
|
||
> **Violating the letter of these rules is violating the spirit.**
|
||
|
||
## Overview
|
||
|
||
Every session leaves a detailed flight recorder behind — the
|
||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
|
||
plus one sidechain log per dispatched subagent under
|
||
`<session>/subagents/`. Nobody reads it. So the same expensive
|
||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||
loop with a 30% error rate, a cache-cold run that pays full input
|
||
price on every turn. This skill turns that recorder into a graded
|
||
retrospective: **how well did the toolchain work, how effective
|
||
were the agents, and what did it cost.**
|
||
|
||
It is descriptive, not corrective. It produces a report and, where
|
||
warranted, recommends filing tracker issues — it does not change
|
||
code, baselines, or the pipeline.
|
||
|
||
## When to Use / Skipping
|
||
|
||
Invoke:
|
||
|
||
- After a substantial interactive session, when you want to know
|
||
where the time and tokens went.
|
||
- At the close of a `/boss` autonomous run, to grade the
|
||
orchestrator's dispatch choices.
|
||
- When a run *felt* expensive or thrashy and you want the numbers.
|
||
|
||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
||
blocks a cycle.
|
||
|
||
## The Iron Law
|
||
|
||
```
|
||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
||
EVERY DOLLAR FIGURE IS LABELLED "ESTIMATE" — THE TRANSCRIPT CARRIES NO METERED COST
|
||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
||
```
|
||
|
||
## What the data does and does not contain
|
||
|
||
Read before interpreting, so the report never overclaims:
|
||
|
||
- **Present:** per-request `usage` (input / output / cache-creation /
|
||
cache-read / server-tool counts), `model`, `requestId`,
|
||
timestamps, every `tool_use` with its input, every `tool_result`
|
||
with the authoritative `is_error` flag, skill invocations,
|
||
slash commands, and a full separate transcript + `agentType` /
|
||
`description` for each subagent.
|
||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||
`null`. **Cost is derived** from tokens × a list-price table;
|
||
**wall-clock is derived** from timestamp deltas. Both are
|
||
estimates and the report says so.
|
||
|
||
The streamed transcript repeats the same `requestId` across
|
||
several assistant lines with an identical `usage` object. The
|
||
helper script dedups per `requestId`; never sum the raw lines.
|
||
|
||
## The Process
|
||
|
||
### Step 1 — Run the aggregator
|
||
|
||
```
|
||
python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
||
```
|
||
|
||
Defaults to the newest transcript in the current project's log
|
||
dir — i.e. the session you are in (it will be flagged `active`,
|
||
meaning totals are partial). To grade a finished run, pass its
|
||
`--session <uuid>`. Override prices for a non-default plan with
|
||
`--pricing '<json>'` or the `POSTMORTEM_PRICING` env var (same
|
||
shape as `DEFAULT_PRICING` in the script).
|
||
|
||
The script emits one JSON object on stdout. That JSON — not the
|
||
raw log — is your evidence base.
|
||
|
||
### Step 2 — Read the JSON and grade three axes
|
||
|
||
Grade each axis on the cited numbers. A grade with no number
|
||
behind it is an opinion; drop it.
|
||
|
||
1. **Cost & efficiency** — `totals.cost_usd` (estimate),
|
||
main-vs-subagent split, `cache_hit_ratio` (the dominant lever:
|
||
below ~0.7 on a long session means cache-cold turns paying full
|
||
input price), output-token share, `server_tools` (paid
|
||
web_search / web_fetch).
|
||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
||
(the authoritative `is_error` signal), the tool mix (a Read /
|
||
Edit / Bash imbalance hints at thrashing or re-reading),
|
||
`slash_commands` such as `/compact` (context pressure).
|
||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
||
`NEEDS_CONTEXT` / `unknown`), `cost_usd`, and `requests`. A
|
||
fleet that cost real tokens to return `BLOCKED` is the headline
|
||
finding. Compare subagent spend to what they delivered.
|
||
|
||
### Step 3 — Write the report
|
||
|
||
Write a Markdown report to the configured destination (see
|
||
**Output destination**). Structure it as **Output format** below.
|
||
Lead with the scorecard, support every grade with a number from
|
||
the JSON, and keep the prose tight — this is a retrospective, not
|
||
a transcript replay.
|
||
|
||
### Step 4 — Surface, don't fix
|
||
|
||
Translate the sharpest findings into recommendations. Where a
|
||
finding is a concrete, trackable defect (a subagent that reliably
|
||
bounces `BLOCKED`, a tool loop with a runaway error rate),
|
||
recommend filing it via the `issue` skill — name it, but do not
|
||
file it yourself unless the user asks. This skill ends at the
|
||
report.
|
||
|
||
## Output destination
|
||
|
||
Resolve in this order:
|
||
|
||
1. `--out <path>` if the invoker passed one.
|
||
2. The project profile's `paths.postmortem_dir` (if a
|
||
`.claude/dev-cycle-profile.yml` exists and defines it).
|
||
3. Default: `docs/postmortems/<session-id-short>-<yyyy-mm-dd>.md`
|
||
under the project root, creating the directory if needed.
|
||
|
||
Also print a ≤200-word summary to the chat so the headline grade
|
||
and cost are visible without opening the file.
|
||
|
||
## Output format
|
||
|
||
```
|
||
# Post-mortem — <session-id-short> (<date>)
|
||
|
||
**Scorecard**
|
||
| Axis | Grade | Headline metric |
|
||
|------|-------|-----------------|
|
||
| Cost & efficiency | A–F | $<est> total, cache hit <ratio> |
|
||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
||
| Agent effectiveness | A–F | <k>/<n> agents DONE, $<est> on subagents |
|
||
|
||
**Cost** (ESTIMATE — derived from tokens × list price, not metered)
|
||
<main vs subagent split; token breakdown; the one number that
|
||
dominates the bill and why>
|
||
|
||
**Toolchain**
|
||
<tool mix, error ratio + what the errors were, context-pressure
|
||
signals like /compact>
|
||
|
||
**Agents**
|
||
<per-subagent line: type — status — $est — one-clause verdict.
|
||
Omit if no subagents were dispatched.>
|
||
|
||
**Recommendations**
|
||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
||
an `issue`.>
|
||
|
||
_Pricing basis: <pricing_basis from the JSON>. Wall-clock and
|
||
cost are derived, not metered._
|
||
```
|
||
|
||
If the session is flagged `active`, add one line at the top:
|
||
"⚠️ Session still active — figures are partial."
|
||
|
||
## Common Rationalisations
|
||
|
||
| Excuse | Reality |
|
||
|--------|---------|
|
||
| "I'll just skim the transcript and estimate the cost" | The transcript has no cost field and repeats usage per streamed line. Hand-summing is wrong twice over. Run the script. |
|
||
| "$31 is the cost" | It is an *estimate* from list prices. The plan may differ; the basis line and the `ESTIMATE` label are non-optional. |
|
||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent at full price. That is the single biggest lever — grade it. |
|
||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned $2 is a failure that cost money. Read `terminal_status`. |
|
||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
||
|
||
## Red Flags — STOP
|
||
|
||
- Quoting any token or dollar number that did not come out of the
|
||
script's JSON.
|
||
- Presenting a derived dollar figure without the `ESTIMATE` label
|
||
and pricing basis.
|
||
- Reading any file under `~/.claude` other than the session
|
||
transcript and its subagent logs — and never any credential or
|
||
token file.
|
||
- Editing code, baselines, or pipeline config "while you're in
|
||
there". Report only.
|
||
- Grading an axis with adjectives instead of the cited metric.
|
||
|
||
## Cross-references
|
||
|
||
- **Helper script:** `scripts/postmortem.py` — the deterministic
|
||
aggregator. All numbers originate here.
|
||
- **Hand-off target:** the `issue` skill, when a finding is a
|
||
trackable defect worth filing.
|
||
- **Profile slot (optional):** `paths.postmortem_dir` for the
|
||
report destination; `POSTMORTEM_PRICING` to override the price
|
||
table for a non-default plan.
|
||
- **Data-source note:** this skill reads only the current
|
||
session's own transcript and subagent logs. It does not crawl
|
||
other projects or other sessions (single-session by design).
|