Brummel/Skills
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(postmortem): report raw token counts, drop dollar estimates

Browse Source 
Remove the price-table cost derivation entirely. The transcript carries
no metered cost, and list prices vary by plan — so converting tokens to
a dollar figure dressed an estimate up as a number. Report raw token
counts instead, kept split by class (input / output / cache-creation /
cache-read) with a `total_tokens` sum per scope.

Script: drop DEFAULT_PRICING / load_pricing / rate_for / cost_of and the
--pricing flag; aggregate_transcript and analyze_subagents no longer take
a pricing arg; every `cost_usd` field becomes `total_tokens`; the pricing
warning is gone. SKILL.md: the first graded axis is now "Token spend &
efficiency", the scorecard and output format cite token counts, and the
Iron Law / Red Flags now forbid converting tokens to a cost.

cache_hit_ratio and the active-session warning are unchanged.
This commit is contained in:
Brummel
2026-06-02 16:34:53 +02:00
parent 5f1903eef0
commit 97ba871d56
2 changed files with 68 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files
postmortem/SKILL.md
+40 -40
View File
@@ -1,6 +1,6 @@
---
name: postmortem
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and what the run cost. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/cost/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
---
# postmortem — grade the run
@@ -14,10 +14,10 @@ JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
plus one sidechain log per dispatched subagent under
`<session>/subagents/`. Nobody reads it. So the same expensive
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
loop with a 30% error rate, a cache-cold run that pays full input
price on every turn. This skill turns that recorder into a graded
loop with a 30% error rate, a cache-cold run that re-sends the full
input on every turn. This skill turns that recorder into a graded
retrospective: **how well did the toolchain work, how effective
were the agents, and what did it cost.**
were the agents, and how many tokens did it spend.**
It is descriptive, not corrective. It produces a report and, where
warranted, recommends filing tracker issues — it does not change
@@ -31,7 +31,7 @@ Invoke:
where the time and tokens went.
- At the close of a `/boss` autonomous run, to grade the
orchestrator's dispatch choices.
- When a run *felt* expensive or thrashy and you want the numbers.
- When a run *felt* token-heavy or thrashy and you want the numbers.
Not a pipeline phase. Never a hard-gate. It reads logs; it never
blocks a cycle.
@@ -40,7 +40,7 @@ blocks a cycle.
```
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
EVERY DOLLAR FIGURE IS LABELLED "ESTIMATE" — THE TRANSCRIPT CARRIES NO METERED COST
REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
@@ -57,9 +57,11 @@ Read before interpreting, so the report never overclaims:
slash commands, and a full separate transcript + `agentType` /
`description` for each subagent.
- **Absent:** `costUSD` is always `null`; `durationMs` is always
`null`. **Cost is derived** from tokens × a list-price table;
**wall-clock is derived** from timestamp deltas. Both are
estimates and the report says so.
`null`. There is no metered cost in the log, so this skill does
not compute one — it reports **raw token counts** (kept split by
class: input / output / cache-creation / cache-read, plus a
`total_tokens` sum). **Wall-clock is derived** from timestamp
deltas and the report says so.
The streamed transcript repeats the same `requestId` across
several assistant lines with an identical `usage` object. The
@@ -76,9 +78,7 @@ python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
Defaults to the newest transcript in the current project's log
dir — i.e. the session you are in (it will be flagged `active`,
meaning totals are partial). To grade a finished run, pass its
`--session <uuid>`. Override prices for a non-default plan with
`--pricing '<json>'` or the `POSTMORTEM_PRICING` env var (same
shape as `DEFAULT_PRICING` in the script).
`--session <uuid>`.
The script emits one JSON object on stdout. That JSON — not the
raw log — is your evidence base.
@@ -88,20 +88,21 @@ raw log — is your evidence base.
Grade each axis on the cited numbers. A grade with no number
behind it is an opinion; drop it.
1. **Cost & efficiency** — `totals.cost_usd` (estimate),
main-vs-subagent split, `cache_hit_ratio` (the dominant lever:
below ~0.7 on a long session means cache-cold turns paying full
input price), output-token share, `server_tools` (paid
web_search / web_fetch).
1. **Token spend & efficiency** — `totals.total_tokens`, the
main-vs-subagent split, the token breakdown (output share is the
write-heavy signal), `cache_hit_ratio` (the dominant efficiency
lever: below ~0.7 on a long session means cache-cold turns
re-sending the full input), `server_tools` (web_search /
web_fetch counts).
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
(the authoritative `is_error` signal), the tool mix (a Read /
Edit / Bash imbalance hints at thrashing or re-reading),
`slash_commands` such as `/compact` (context pressure).
3. **Agent effectiveness** — per subagent: `terminal_status`
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
`NEEDS_CONTEXT` / `unknown`), `cost_usd`, and `requests`. A
fleet that cost real tokens to return `BLOCKED` is the headline
finding. Compare subagent spend to what they delivered.
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
fleet that burned real tokens to return `BLOCKED` is the headline
finding. Compare each agent's token spend to what it delivered.
### Step 3 — Write the report
@@ -131,7 +132,7 @@ Resolve in this order:
under the project root, creating the directory if needed.
Also print a ≤200-word summary to the chat so the headline grade
and cost are visible without opening the file.
and token spend are visible without opening the file.
## Output format
@@ -141,28 +142,28 @@ and cost are visible without opening the file.
**Scorecard**
| Axis | Grade | Headline metric |
|------|-------|-----------------|
| Cost & efficiency | A–F | $<est> total, cache hit <ratio> |
| Token spend & efficiency | A–F | <total_tokens> total, cache hit <ratio> |
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
| Agent effectiveness | A–F | <k>/<n> agents DONE, $<est> on subagents |
| Agent effectiveness | A–F | <k>/<n> agents DONE, <tokens> on subagents |
**Cost** (ESTIMATE — derived from tokens × list price, not metered)
<main vs subagent split; token breakdown; the one number that
dominates the bill and why>
**Token spend**
<main vs subagent split; token breakdown by class (input / output /
cache-creation / cache-read); the one class that dominates and why>
**Toolchain**
<tool mix, error ratio + what the errors were, context-pressure
signals like /compact>
**Agents**
<per-subagent line: type — status — $est — one-clause verdict.
Omit if no subagents were dispatched.>
<per-subagent line: type — status — <total_tokens> — one-clause
verdict. Omit if no subagents were dispatched.>
**Recommendations**
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
an `issue`.>
_Pricing basis: <pricing_basis from the JSON>. Wall-clock and
cost are derived, not metered._
_Token counts are raw usage. Wall-clock is derived from timestamp
deltas, not metered._
```
If the session is flagged `active`, add one line at the top:
@@ -172,19 +173,19 @@ If the session is flagged `active`, add one line at the top:
| Excuse | Reality |
|--------|---------|
| "I'll just skim the transcript and estimate the cost" | The transcript has no cost field and repeats usage per streamed line. Hand-summing is wrong twice over. Run the script. |
| "$31 is the cost" | It is an *estimate* from list prices. The plan may differ; the basis line and the `ESTIMATE` label are non-optional. |
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent at full price. That is the single biggest lever — grade it. |
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned $2 is a failure that cost money. Read `terminal_status`. |
| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. |
| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. |
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. |
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. |
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
## Red Flags — STOP
- Quoting any token or dollar number that did not come out of the
script's JSON.
- Presenting a derived dollar figure without the `ESTIMATE` label
and pricing basis.
- Quoting any token number that did not come out of the script's
JSON.
- Converting token counts into a cost or dollar figure — report
raw counts only.
- Reading any file under `~/.claude` other than the session
transcript and its subagent logs — and never any credential or
token file.
@@ -199,8 +200,7 @@ If the session is flagged `active`, add one line at the top:
- **Hand-off target:** the `issue` skill, when a finding is a
trackable defect worth filing.
- **Profile slot (optional):** `paths.postmortem_dir` for the
report destination; `POSTMORTEM_PRICING` to override the price
table for a non-default plan.
report destination.
- **Data-source note:** this skill reads only the current
session's own transcript and subagent logs. It does not crawl
other projects or other sessions (single-session by design).
postmortem/scripts/postmortem.py
+28 -87
View File
@@ -13,11 +13,11 @@ traps that only deterministic code gets right every time:
usage object. Counting per-line double-counts tokens. We dedup
per requestId (max per field within a request, then sum across
requests).
2. The transcript carries NO costUSD and NO durationMs (both null).
Cost is derived from tokens x a per-model price table; wall-clock
from timestamp deltas. The price table is a LIST-PRICE ESTIMATE
and is surfaced in the output so the report never presents a
derived dollar figure as if it were metered.
2. The transcript carries NO durationMs (always null), so wall-clock
is derived from timestamp deltas. The numbers reported here are
raw token counts, not a dollar estimate — different token classes
(input / output / cache) are reported separately rather than
collapsed into a single priced figure.
Stdlib only. No network. Read-only against ~/.claude.
"""
@@ -30,21 +30,6 @@ import re
import sys
from datetime import datetime, timezone
# --- Pricing -------------------------------------------------------------
# USD per million tokens. Anthropic public list prices; treated as an
# ESTIMATE, not a metered cost. Override wholesale via the
# POSTMORTEM_PRICING env var (JSON of the same shape) or --pricing.
# Keys are matched as substrings against message.model, longest first.
DEFAULT_PRICING = {
"opus": {"input": 15.0, "output": 75.0,
"cache_write_5m": 18.75, "cache_write_1h": 30.0, "cache_read": 1.5},
"sonnet": {"input": 3.0, "output": 15.0,
"cache_write_5m": 3.75, "cache_write_1h": 6.0, "cache_read": 0.30},
"haiku": {"input": 1.0, "output": 5.0,
"cache_write_5m": 1.25, "cache_write_1h": 2.0, "cache_read": 0.10},
}
PRICING_AS_OF = "2026-01 list prices (estimate, override via POSTMORTEM_PRICING)"
_TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first
# Authoritative signal: the implement end-report leads with a structured
# "Status: DONE" line (optionally wrapped in markdown bold or a list
@@ -63,26 +48,6 @@ STATUS_STANDALONE_RE = re.compile(
re.IGNORECASE | re.MULTILINE)
def load_pricing(override_json):
if override_json:
return json.loads(override_json)
env = os.environ.get("POSTMORTEM_PRICING")
if env:
return json.loads(env)
return DEFAULT_PRICING
def rate_for(model, pricing):
if not model:
return pricing.get("opus"), True
for key in sorted(pricing, key=len, reverse=True):
if key in model:
return pricing[key], False
# Unknown model: fall back to the most expensive known tier so the
# estimate errs high rather than silently undercounting.
return pricing.get("opus", next(iter(pricing.values()))), True
def parse_ts(s):
if not s:
return None
@@ -97,6 +62,14 @@ def empty_tokens():
"cache_creation_5m": 0, "cache_creation_1h": 0}
def token_total(tokens):
"""Raw count of every token the session moved. cache_creation_5m/1h are
a sub-split of cache_creation, so they are excluded to avoid double
counting."""
return (tokens["input"] + tokens["output"]
+ tokens["cache_creation"] + tokens["cache_read"])
def add_tokens(acc, usage):
"""Add one request's deduped usage into an accumulator."""
acc["input"] += usage.get("input_tokens", 0) or 0
@@ -114,14 +87,6 @@ def add_tokens(acc, usage):
acc["cache_creation_1h"] += f1 or 0
def cost_of(tokens, rate):
return (tokens["input"] * rate["input"]
+ tokens["output"] * rate["output"]
+ tokens["cache_creation_5m"] * rate["cache_write_5m"]
+ tokens["cache_creation_1h"] * rate["cache_write_1h"]
+ tokens["cache_read"] * rate["cache_read"]) / 1_000_000.0
def iter_lines(path):
with open(path, "r", encoding="utf-8", errors="replace") as fh:
for line in fh:
@@ -134,9 +99,9 @@ def iter_lines(path):
continue
def aggregate_transcript(path, pricing):
"""Return token totals, cost, model mix, and per-request count for one
JSONL transcript (main or subagent). Dedups usage by requestId."""
def aggregate_transcript(path):
"""Return token totals, model mix, and per-request count for one JSONL
transcript (main or subagent). Dedups usage by requestId."""
# requestId -> {field: max seen}, plus model per request.
req_usage = {}
req_model = {}
@@ -175,18 +140,10 @@ def aggregate_transcript(path, pricing):
tokens = empty_tokens()
server = {"web_search": 0, "web_fetch": 0}
models = {}
cost = 0.0
unknown_model = False
for key, usage in req_usage.items():
add_tokens(tokens, usage)
model = req_model.get(key)
models[model] = models.get(model, 0) + 1
rate, unk = rate_for(model, pricing)
unknown_model = unknown_model or unk
# per-request cost so each request bills at its own model's rate
per = empty_tokens()
add_tokens(per, usage)
cost += cost_of(per, rate)
sv = usage.get("_server", {})
server["web_search"] += sv.get("web_search_requests", 0)
server["web_fetch"] += sv.get("web_fetch_requests", 0)
@@ -194,10 +151,9 @@ def aggregate_transcript(path, pricing):
return {
"requests": len(req_usage),
"tokens": tokens,
"cost_usd": round(cost, 4),
"total_tokens": token_total(tokens),
"models": models,
"server_tools": server,
"unknown_model": unknown_model,
}
@@ -209,7 +165,6 @@ def cache_hit_ratio(tokens):
def analyze_main(path):
"""Tool mix, error rate, skills, slash commands, prompts, wall-clock."""
tool_calls = {}
tool_errors = {}
error_total = 0
interrupted = 0
skills = []
@@ -217,7 +172,7 @@ def analyze_main(path):
tasks = []
user_prompts = 0
first_ts = last_ts = None
cwd = project = version = None
cwd = version = None
for rec in iter_lines(path):
ts = parse_ts(rec.get("timestamp"))
@@ -271,8 +226,6 @@ def analyze_main(path):
if isinstance(tur, dict) and tur.get("interrupted"):
interrupted += 1
# error attribution by tool name needs pairing result->call; we report
# the session-wide error_total and leave per-tool to the weak signal.
return {
"tool_calls": tool_calls,
"total_tool_calls": sum(tool_calls.values()),
@@ -313,7 +266,7 @@ def subagent_terminal_status(path):
return "unknown"
def analyze_subagents(session_dir, pricing):
def analyze_subagents(session_dir):
out = []
sub_dir = os.path.join(session_dir, "subagents")
for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))):
@@ -325,7 +278,7 @@ def analyze_subagents(session_dir, pricing):
meta = json.load(fh)
except (json.JSONDecodeError, OSError):
pass
agg = aggregate_transcript(jl, pricing)
agg = aggregate_transcript(jl)
out.append({
"agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")],
"agent_type": meta.get("agentType"),
@@ -334,7 +287,7 @@ def analyze_subagents(session_dir, pricing):
"models": agg["models"],
"requests": agg["requests"],
"tokens": agg["tokens"],
"cost_usd": agg["cost_usd"],
"total_tokens": agg["total_tokens"],
"terminal_status": subagent_terminal_status(jl),
})
return out
@@ -373,31 +326,26 @@ def main():
ap.add_argument("--file", help="explicit path to a main transcript .jsonl")
ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)")
ap.add_argument("--project-dir", help="explicit ~/.claude/projects/<slug> dir")
ap.add_argument("--pricing", help="JSON price table override")
args = ap.parse_args()
pricing = load_pricing(args.pricing)
main_path, session_dir = resolve_session(args)
session_id = os.path.basename(main_path)[:-len(".jsonl")]
main_agg = aggregate_transcript(main_path, pricing)
main_agg = aggregate_transcript(main_path)
main_meta = analyze_main(main_path)
subagents = []
if session_dir and os.path.isdir(session_dir):
subagents = analyze_subagents(session_dir, pricing)
subagents = analyze_subagents(session_dir)
sub_tokens = empty_tokens()
sub_cost = 0.0
for s in subagents:
for k in sub_tokens:
sub_tokens[k] += s["tokens"][k]
sub_cost += s["cost_usd"]
total_tokens = empty_tokens()
for k in total_tokens:
total_tokens[k] = main_agg["tokens"][k] + sub_tokens[k]
total_cost = round(main_agg["cost_usd"] + sub_cost, 4)
warnings = []
is_active = False
@@ -407,13 +355,8 @@ def main():
is_active = True
warnings.append("Session appears ACTIVE (last event <2 min ago); "
"totals are partial and will grow.")
if main_agg["unknown_model"] or any(s.get("models") and
any(rate_for(m, pricing)[1] for m in s["models"])
for s in subagents):
warnings.append("One or more models were not in the price table; "
"billed at the opus tier (estimate errs high).")
warnings.append("Cost is DERIVED from tokens x list prices, not metered. "
"Price basis: " + PRICING_AS_OF)
warnings.append("Token counts are raw usage; wall-clock is derived from "
"timestamp deltas (the transcript carries no durationMs).")
report = {
"session": {
@@ -430,7 +373,7 @@ def main():
"requests": main_agg["requests"],
"models": main_agg["models"],
"tokens": main_agg["tokens"],
"cost_usd": main_agg["cost_usd"],
"total_tokens": main_agg["total_tokens"],
"cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]),
"server_tools": main_agg["server_tools"],
"user_prompts": main_meta["user_prompts"],
@@ -451,15 +394,13 @@ def main():
"subagent_totals": {
"count": len(subagents),
"tokens": sub_tokens,
"cost_usd": round(sub_cost, 4),
"total_tokens": token_total(sub_tokens),
},
"totals": {
"tokens": total_tokens,
"total_tokens": token_total(total_tokens),
"cache_hit_ratio": cache_hit_ratio(total_tokens),
"cost_usd": total_cost,
},
"pricing_used": pricing,
"pricing_basis": PRICING_AS_OF,
"warnings": warnings,
}
json.dump(report, sys.stdout, indent=2)