feat(postmortem): add session-retrospective skill + aggregator
A new utility skill that grades a finished session on three axes — cost & efficiency, toolchain health, agent effectiveness — from the session's own Claude Code flight recorder (the JSONL transcript plus one sidechain log per dispatched subagent). The heavy lifting lives in scripts/postmortem.py, a stdlib-only, read-only aggregator. It handles the two accounting traps the raw log sets: usage is repeated per streamed assistant line under a shared requestId (deduped max-per-field, then summed across requests), and the transcript carries no costUSD/durationMs (both null) — so cost is derived from tokens x an overridable list-price table and labelled an estimate, wall-clock from timestamp deltas. Subagent spend is billed separately from its own usage objects and graded by terminal status. Single-session by design: defaults to the newest transcript in the project's log dir (flagged active if still live), or --session <id> for a finished run. Secret files are out of bounds in both the Iron Law and the Red Flags. Verified end-to-end against a real 15-subagent session.
This commit is contained in:
@@ -0,0 +1,206 @@
|
||||
---
|
||||
name: postmortem
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and what the run cost. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/cost/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
---
|
||||
|
||||
# postmortem — grade the run
|
||||
|
||||
> **Violating the letter of these rules is violating the spirit.**
|
||||
|
||||
## Overview
|
||||
|
||||
Every session leaves a detailed flight recorder behind — the
|
||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
|
||||
plus one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`. Nobody reads it. So the same expensive
|
||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||||
loop with a 30% error rate, a cache-cold run that pays full input
|
||||
price on every turn. This skill turns that recorder into a graded
|
||||
retrospective: **how well did the toolchain work, how effective
|
||||
were the agents, and what did it cost.**
|
||||
|
||||
It is descriptive, not corrective. It produces a report and, where
|
||||
warranted, recommends filing tracker issues — it does not change
|
||||
code, baselines, or the pipeline.
|
||||
|
||||
## When to Use / Skipping
|
||||
|
||||
Invoke:
|
||||
|
||||
- After a substantial interactive session, when you want to know
|
||||
where the time and tokens went.
|
||||
- At the close of a `/boss` autonomous run, to grade the
|
||||
orchestrator's dispatch choices.
|
||||
- When a run *felt* expensive or thrashy and you want the numbers.
|
||||
|
||||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
||||
blocks a cycle.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
||||
EVERY DOLLAR FIGURE IS LABELLED "ESTIMATE" — THE TRANSCRIPT CARRIES NO METERED COST
|
||||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
||||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
||||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
||||
```
|
||||
|
||||
## What the data does and does not contain
|
||||
|
||||
Read before interpreting, so the report never overclaims:
|
||||
|
||||
- **Present:** per-request `usage` (input / output / cache-creation /
|
||||
cache-read / server-tool counts), `model`, `requestId`,
|
||||
timestamps, every `tool_use` with its input, every `tool_result`
|
||||
with the authoritative `is_error` flag, skill invocations,
|
||||
slash commands, and a full separate transcript + `agentType` /
|
||||
`description` for each subagent.
|
||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||||
`null`. **Cost is derived** from tokens × a list-price table;
|
||||
**wall-clock is derived** from timestamp deltas. Both are
|
||||
estimates and the report says so.
|
||||
|
||||
The streamed transcript repeats the same `requestId` across
|
||||
several assistant lines with an identical `usage` object. The
|
||||
helper script dedups per `requestId`; never sum the raw lines.
|
||||
|
||||
## The Process
|
||||
|
||||
### Step 1 — Run the aggregator
|
||||
|
||||
```
|
||||
python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
||||
```
|
||||
|
||||
Defaults to the newest transcript in the current project's log
|
||||
dir — i.e. the session you are in (it will be flagged `active`,
|
||||
meaning totals are partial). To grade a finished run, pass its
|
||||
`--session <uuid>`. Override prices for a non-default plan with
|
||||
`--pricing '<json>'` or the `POSTMORTEM_PRICING` env var (same
|
||||
shape as `DEFAULT_PRICING` in the script).
|
||||
|
||||
The script emits one JSON object on stdout. That JSON — not the
|
||||
raw log — is your evidence base.
|
||||
|
||||
### Step 2 — Read the JSON and grade three axes
|
||||
|
||||
Grade each axis on the cited numbers. A grade with no number
|
||||
behind it is an opinion; drop it.
|
||||
|
||||
1. **Cost & efficiency** — `totals.cost_usd` (estimate),
|
||||
main-vs-subagent split, `cache_hit_ratio` (the dominant lever:
|
||||
below ~0.7 on a long session means cache-cold turns paying full
|
||||
input price), output-token share, `server_tools` (paid
|
||||
web_search / web_fetch).
|
||||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
||||
(the authoritative `is_error` signal), the tool mix (a Read /
|
||||
Edit / Bash imbalance hints at thrashing or re-reading),
|
||||
`slash_commands` such as `/compact` (context pressure).
|
||||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
||||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
||||
`NEEDS_CONTEXT` / `unknown`), `cost_usd`, and `requests`. A
|
||||
fleet that cost real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare subagent spend to what they delivered.
|
||||
|
||||
### Step 3 — Write the report
|
||||
|
||||
Write a Markdown report to the configured destination (see
|
||||
**Output destination**). Structure it as **Output format** below.
|
||||
Lead with the scorecard, support every grade with a number from
|
||||
the JSON, and keep the prose tight — this is a retrospective, not
|
||||
a transcript replay.
|
||||
|
||||
### Step 4 — Surface, don't fix
|
||||
|
||||
Translate the sharpest findings into recommendations. Where a
|
||||
finding is a concrete, trackable defect (a subagent that reliably
|
||||
bounces `BLOCKED`, a tool loop with a runaway error rate),
|
||||
recommend filing it via the `issue` skill — name it, but do not
|
||||
file it yourself unless the user asks. This skill ends at the
|
||||
report.
|
||||
|
||||
## Output destination
|
||||
|
||||
Resolve in this order:
|
||||
|
||||
1. `--out <path>` if the invoker passed one.
|
||||
2. The project profile's `paths.postmortem_dir` (if a
|
||||
`.claude/dev-cycle-profile.yml` exists and defines it).
|
||||
3. Default: `docs/postmortems/<session-id-short>-<yyyy-mm-dd>.md`
|
||||
under the project root, creating the directory if needed.
|
||||
|
||||
Also print a ≤200-word summary to the chat so the headline grade
|
||||
and cost are visible without opening the file.
|
||||
|
||||
## Output format
|
||||
|
||||
```
|
||||
# Post-mortem — <session-id-short> (<date>)
|
||||
|
||||
**Scorecard**
|
||||
| Axis | Grade | Headline metric |
|
||||
|------|-------|-----------------|
|
||||
| Cost & efficiency | A–F | $<est> total, cache hit <ratio> |
|
||||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, $<est> on subagents |
|
||||
|
||||
**Cost** (ESTIMATE — derived from tokens × list price, not metered)
|
||||
<main vs subagent split; token breakdown; the one number that
|
||||
dominates the bill and why>
|
||||
|
||||
**Toolchain**
|
||||
<tool mix, error ratio + what the errors were, context-pressure
|
||||
signals like /compact>
|
||||
|
||||
**Agents**
|
||||
<per-subagent line: type — status — $est — one-clause verdict.
|
||||
Omit if no subagents were dispatched.>
|
||||
|
||||
**Recommendations**
|
||||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
||||
an `issue`.>
|
||||
|
||||
_Pricing basis: <pricing_basis from the JSON>. Wall-clock and
|
||||
cost are derived, not metered._
|
||||
```
|
||||
|
||||
If the session is flagged `active`, add one line at the top:
|
||||
"⚠️ Session still active — figures are partial."
|
||||
|
||||
## Common Rationalisations
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "I'll just skim the transcript and estimate the cost" | The transcript has no cost field and repeats usage per streamed line. Hand-summing is wrong twice over. Run the script. |
|
||||
| "$31 is the cost" | It is an *estimate* from list prices. The plan may differ; the basis line and the `ESTIMATE` label are non-optional. |
|
||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent at full price. That is the single biggest lever — grade it. |
|
||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned $2 is a failure that cost money. Read `terminal_status`. |
|
||||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
||||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
||||
|
||||
## Red Flags — STOP
|
||||
|
||||
- Quoting any token or dollar number that did not come out of the
|
||||
script's JSON.
|
||||
- Presenting a derived dollar figure without the `ESTIMATE` label
|
||||
and pricing basis.
|
||||
- Reading any file under `~/.claude` other than the session
|
||||
transcript and its subagent logs — and never any credential or
|
||||
token file.
|
||||
- Editing code, baselines, or pipeline config "while you're in
|
||||
there". Report only.
|
||||
- Grading an axis with adjectives instead of the cited metric.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Helper script:** `scripts/postmortem.py` — the deterministic
|
||||
aggregator. All numbers originate here.
|
||||
- **Hand-off target:** the `issue` skill, when a finding is a
|
||||
trackable defect worth filing.
|
||||
- **Profile slot (optional):** `paths.postmortem_dir` for the
|
||||
report destination; `POSTMORTEM_PRICING` to override the price
|
||||
table for a non-default plan.
|
||||
- **Data-source note:** this skill reads only the current
|
||||
session's own transcript and subagent logs. It does not crawl
|
||||
other projects or other sessions (single-session by design).
|
||||
Reference in New Issue
Block a user