From 97ba871d5668ac203400d35e7826ed8dbd019e1a Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 2 Jun 2026 16:34:53 +0200 Subject: [PATCH] refactor(postmortem): report raw token counts, drop dollar estimates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the price-table cost derivation entirely. The transcript carries no metered cost, and list prices vary by plan — so converting tokens to a dollar figure dressed an estimate up as a number. Report raw token counts instead, kept split by class (input / output / cache-creation / cache-read) with a `total_tokens` sum per scope. Script: drop DEFAULT_PRICING / load_pricing / rate_for / cost_of and the --pricing flag; aggregate_transcript and analyze_subagents no longer take a pricing arg; every `cost_usd` field becomes `total_tokens`; the pricing warning is gone. SKILL.md: the first graded axis is now "Token spend & efficiency", the scorecard and output format cite token counts, and the Iron Law / Red Flags now forbid converting tokens to a cost. cache_hit_ratio and the active-session warning are unchanged. --- postmortem/SKILL.md | 80 ++++++++++----------- postmortem/scripts/postmortem.py | 115 ++++++++----------------------- 2 files changed, 68 insertions(+), 127 deletions(-) diff --git a/postmortem/SKILL.md b/postmortem/SKILL.md index 5d63e87..fff8a25 100644 --- a/postmortem/SKILL.md +++ b/postmortem/SKILL.md @@ -1,6 +1,6 @@ --- name: postmortem -description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and what the run cost. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/cost/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate. +description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate. --- # postmortem — grade the run @@ -14,10 +14,10 @@ JSONL transcript under `~/.claude/projects//.jsonl` plus one sidechain log per dispatched subagent under `/subagents/`. Nobody reads it. So the same expensive habits repeat: a subagent fleet that bounces `BLOCKED`, a tool -loop with a 30% error rate, a cache-cold run that pays full input -price on every turn. This skill turns that recorder into a graded +loop with a 30% error rate, a cache-cold run that re-sends the full +input on every turn. This skill turns that recorder into a graded retrospective: **how well did the toolchain work, how effective -were the agents, and what did it cost.** +were the agents, and how many tokens did it spend.** It is descriptive, not corrective. It produces a report and, where warranted, recommends filing tracker issues — it does not change @@ -31,7 +31,7 @@ Invoke: where the time and tokens went. - At the close of a `/boss` autonomous run, to grade the orchestrator's dispatch choices. -- When a run *felt* expensive or thrashy and you want the numbers. +- When a run *felt* token-heavy or thrashy and you want the numbers. Not a pipeline phase. Never a hard-gate. It reads logs; it never blocks a cycle. @@ -40,7 +40,7 @@ blocks a cycle. ``` NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG -EVERY DOLLAR FIGURE IS LABELLED "ESTIMATE" — THE TRANSCRIPT CARRIES NO METERED COST +REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND @@ -57,9 +57,11 @@ Read before interpreting, so the report never overclaims: slash commands, and a full separate transcript + `agentType` / `description` for each subagent. - **Absent:** `costUSD` is always `null`; `durationMs` is always - `null`. **Cost is derived** from tokens × a list-price table; - **wall-clock is derived** from timestamp deltas. Both are - estimates and the report says so. + `null`. There is no metered cost in the log, so this skill does + not compute one — it reports **raw token counts** (kept split by + class: input / output / cache-creation / cache-read, plus a + `total_tokens` sum). **Wall-clock is derived** from timestamp + deltas and the report says so. The streamed transcript repeats the same `requestId` across several assistant lines with an identical `usage` object. The @@ -76,9 +78,7 @@ python3 /scripts/postmortem.py [--session ] [--cwd ] Defaults to the newest transcript in the current project's log dir — i.e. the session you are in (it will be flagged `active`, meaning totals are partial). To grade a finished run, pass its -`--session `. Override prices for a non-default plan with -`--pricing ''` or the `POSTMORTEM_PRICING` env var (same -shape as `DEFAULT_PRICING` in the script). +`--session `. The script emits one JSON object on stdout. That JSON — not the raw log — is your evidence base. @@ -88,20 +88,21 @@ raw log — is your evidence base. Grade each axis on the cited numbers. A grade with no number behind it is an opinion; drop it. -1. **Cost & efficiency** — `totals.cost_usd` (estimate), - main-vs-subagent split, `cache_hit_ratio` (the dominant lever: - below ~0.7 on a long session means cache-cold turns paying full - input price), output-token share, `server_tools` (paid - web_search / web_fetch). +1. **Token spend & efficiency** — `totals.total_tokens`, the + main-vs-subagent split, the token breakdown (output share is the + write-heavy signal), `cache_hit_ratio` (the dominant efficiency + lever: below ~0.7 on a long session means cache-cold turns + re-sending the full input), `server_tools` (web_search / + web_fetch counts). 2. **Toolchain health** — `tools.error_ratio` and `interrupted` (the authoritative `is_error` signal), the tool mix (a Read / Edit / Bash imbalance hints at thrashing or re-reading), `slash_commands` such as `/compact` (context pressure). 3. **Agent effectiveness** — per subagent: `terminal_status` (`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` / - `NEEDS_CONTEXT` / `unknown`), `cost_usd`, and `requests`. A - fleet that cost real tokens to return `BLOCKED` is the headline - finding. Compare subagent spend to what they delivered. + `NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A + fleet that burned real tokens to return `BLOCKED` is the headline + finding. Compare each agent's token spend to what it delivered. ### Step 3 — Write the report @@ -131,7 +132,7 @@ Resolve in this order: under the project root, creating the directory if needed. Also print a ≤200-word summary to the chat so the headline grade -and cost are visible without opening the file. +and token spend are visible without opening the file. ## Output format @@ -141,28 +142,28 @@ and cost are visible without opening the file. **Scorecard** | Axis | Grade | Headline metric | |------|-------|-----------------| -| Cost & efficiency | A–F | $ total, cache hit | +| Token spend & efficiency | A–F | total, cache hit | | Toolchain health | A–F | , interrupts | -| Agent effectiveness | A–F | / agents DONE, $ on subagents | +| Agent effectiveness | A–F | / agents DONE, on subagents | -**Cost** (ESTIMATE — derived from tokens × list price, not metered) -
+**Token spend** +
**Toolchain** **Agents** - + — one-clause +verdict. Omit if no subagents were dispatched.> **Recommendations** <≤5 bullets. Each ties to a cited metric. Mark any that warrant an `issue`.> -_Pricing basis: . Wall-clock and -cost are derived, not metered._ +_Token counts are raw usage. Wall-clock is derived from timestamp +deltas, not metered._ ``` If the session is flagged `active`, add one line at the top: @@ -172,19 +173,19 @@ If the session is flagged `active`, add one line at the top: | Excuse | Reality | |--------|---------| -| "I'll just skim the transcript and estimate the cost" | The transcript has no cost field and repeats usage per streamed line. Hand-summing is wrong twice over. Run the script. | -| "$31 is the cost" | It is an *estimate* from list prices. The plan may differ; the basis line and the `ESTIMATE` label are non-optional. | -| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent at full price. That is the single biggest lever — grade it. | -| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned $2 is a failure that cost money. Read `terminal_status`. | +| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. | +| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. | +| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. | +| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. | | "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. | | "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. | ## Red Flags — STOP -- Quoting any token or dollar number that did not come out of the - script's JSON. -- Presenting a derived dollar figure without the `ESTIMATE` label - and pricing basis. +- Quoting any token number that did not come out of the script's + JSON. +- Converting token counts into a cost or dollar figure — report + raw counts only. - Reading any file under `~/.claude` other than the session transcript and its subagent logs — and never any credential or token file. @@ -199,8 +200,7 @@ If the session is flagged `active`, add one line at the top: - **Hand-off target:** the `issue` skill, when a finding is a trackable defect worth filing. - **Profile slot (optional):** `paths.postmortem_dir` for the - report destination; `POSTMORTEM_PRICING` to override the price - table for a non-default plan. + report destination. - **Data-source note:** this skill reads only the current session's own transcript and subagent logs. It does not crawl other projects or other sessions (single-session by design). diff --git a/postmortem/scripts/postmortem.py b/postmortem/scripts/postmortem.py index 8dc946c..f9538cb 100755 --- a/postmortem/scripts/postmortem.py +++ b/postmortem/scripts/postmortem.py @@ -13,11 +13,11 @@ traps that only deterministic code gets right every time: usage object. Counting per-line double-counts tokens. We dedup per requestId (max per field within a request, then sum across requests). - 2. The transcript carries NO costUSD and NO durationMs (both null). - Cost is derived from tokens x a per-model price table; wall-clock - from timestamp deltas. The price table is a LIST-PRICE ESTIMATE - and is surfaced in the output so the report never presents a - derived dollar figure as if it were metered. + 2. The transcript carries NO durationMs (always null), so wall-clock + is derived from timestamp deltas. The numbers reported here are + raw token counts, not a dollar estimate — different token classes + (input / output / cache) are reported separately rather than + collapsed into a single priced figure. Stdlib only. No network. Read-only against ~/.claude. """ @@ -30,21 +30,6 @@ import re import sys from datetime import datetime, timezone -# --- Pricing ------------------------------------------------------------- -# USD per million tokens. Anthropic public list prices; treated as an -# ESTIMATE, not a metered cost. Override wholesale via the -# POSTMORTEM_PRICING env var (JSON of the same shape) or --pricing. -# Keys are matched as substrings against message.model, longest first. -DEFAULT_PRICING = { - "opus": {"input": 15.0, "output": 75.0, - "cache_write_5m": 18.75, "cache_write_1h": 30.0, "cache_read": 1.5}, - "sonnet": {"input": 3.0, "output": 15.0, - "cache_write_5m": 3.75, "cache_write_1h": 6.0, "cache_read": 0.30}, - "haiku": {"input": 1.0, "output": 5.0, - "cache_write_5m": 1.25, "cache_write_1h": 2.0, "cache_read": 0.10}, -} -PRICING_AS_OF = "2026-01 list prices (estimate, override via POSTMORTEM_PRICING)" - _TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first # Authoritative signal: the implement end-report leads with a structured # "Status: DONE" line (optionally wrapped in markdown bold or a list @@ -63,26 +48,6 @@ STATUS_STANDALONE_RE = re.compile( re.IGNORECASE | re.MULTILINE) -def load_pricing(override_json): - if override_json: - return json.loads(override_json) - env = os.environ.get("POSTMORTEM_PRICING") - if env: - return json.loads(env) - return DEFAULT_PRICING - - -def rate_for(model, pricing): - if not model: - return pricing.get("opus"), True - for key in sorted(pricing, key=len, reverse=True): - if key in model: - return pricing[key], False - # Unknown model: fall back to the most expensive known tier so the - # estimate errs high rather than silently undercounting. - return pricing.get("opus", next(iter(pricing.values()))), True - - def parse_ts(s): if not s: return None @@ -97,6 +62,14 @@ def empty_tokens(): "cache_creation_5m": 0, "cache_creation_1h": 0} +def token_total(tokens): + """Raw count of every token the session moved. cache_creation_5m/1h are + a sub-split of cache_creation, so they are excluded to avoid double + counting.""" + return (tokens["input"] + tokens["output"] + + tokens["cache_creation"] + tokens["cache_read"]) + + def add_tokens(acc, usage): """Add one request's deduped usage into an accumulator.""" acc["input"] += usage.get("input_tokens", 0) or 0 @@ -114,14 +87,6 @@ def add_tokens(acc, usage): acc["cache_creation_1h"] += f1 or 0 -def cost_of(tokens, rate): - return (tokens["input"] * rate["input"] - + tokens["output"] * rate["output"] - + tokens["cache_creation_5m"] * rate["cache_write_5m"] - + tokens["cache_creation_1h"] * rate["cache_write_1h"] - + tokens["cache_read"] * rate["cache_read"]) / 1_000_000.0 - - def iter_lines(path): with open(path, "r", encoding="utf-8", errors="replace") as fh: for line in fh: @@ -134,9 +99,9 @@ def iter_lines(path): continue -def aggregate_transcript(path, pricing): - """Return token totals, cost, model mix, and per-request count for one - JSONL transcript (main or subagent). Dedups usage by requestId.""" +def aggregate_transcript(path): + """Return token totals, model mix, and per-request count for one JSONL + transcript (main or subagent). Dedups usage by requestId.""" # requestId -> {field: max seen}, plus model per request. req_usage = {} req_model = {} @@ -175,18 +140,10 @@ def aggregate_transcript(path, pricing): tokens = empty_tokens() server = {"web_search": 0, "web_fetch": 0} models = {} - cost = 0.0 - unknown_model = False for key, usage in req_usage.items(): add_tokens(tokens, usage) model = req_model.get(key) models[model] = models.get(model, 0) + 1 - rate, unk = rate_for(model, pricing) - unknown_model = unknown_model or unk - # per-request cost so each request bills at its own model's rate - per = empty_tokens() - add_tokens(per, usage) - cost += cost_of(per, rate) sv = usage.get("_server", {}) server["web_search"] += sv.get("web_search_requests", 0) server["web_fetch"] += sv.get("web_fetch_requests", 0) @@ -194,10 +151,9 @@ def aggregate_transcript(path, pricing): return { "requests": len(req_usage), "tokens": tokens, - "cost_usd": round(cost, 4), + "total_tokens": token_total(tokens), "models": models, "server_tools": server, - "unknown_model": unknown_model, } @@ -209,7 +165,6 @@ def cache_hit_ratio(tokens): def analyze_main(path): """Tool mix, error rate, skills, slash commands, prompts, wall-clock.""" tool_calls = {} - tool_errors = {} error_total = 0 interrupted = 0 skills = [] @@ -217,7 +172,7 @@ def analyze_main(path): tasks = [] user_prompts = 0 first_ts = last_ts = None - cwd = project = version = None + cwd = version = None for rec in iter_lines(path): ts = parse_ts(rec.get("timestamp")) @@ -271,8 +226,6 @@ def analyze_main(path): if isinstance(tur, dict) and tur.get("interrupted"): interrupted += 1 - # error attribution by tool name needs pairing result->call; we report - # the session-wide error_total and leave per-tool to the weak signal. return { "tool_calls": tool_calls, "total_tool_calls": sum(tool_calls.values()), @@ -313,7 +266,7 @@ def subagent_terminal_status(path): return "unknown" -def analyze_subagents(session_dir, pricing): +def analyze_subagents(session_dir): out = [] sub_dir = os.path.join(session_dir, "subagents") for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))): @@ -325,7 +278,7 @@ def analyze_subagents(session_dir, pricing): meta = json.load(fh) except (json.JSONDecodeError, OSError): pass - agg = aggregate_transcript(jl, pricing) + agg = aggregate_transcript(jl) out.append({ "agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")], "agent_type": meta.get("agentType"), @@ -334,7 +287,7 @@ def analyze_subagents(session_dir, pricing): "models": agg["models"], "requests": agg["requests"], "tokens": agg["tokens"], - "cost_usd": agg["cost_usd"], + "total_tokens": agg["total_tokens"], "terminal_status": subagent_terminal_status(jl), }) return out @@ -373,31 +326,26 @@ def main(): ap.add_argument("--file", help="explicit path to a main transcript .jsonl") ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)") ap.add_argument("--project-dir", help="explicit ~/.claude/projects/ dir") - ap.add_argument("--pricing", help="JSON price table override") args = ap.parse_args() - pricing = load_pricing(args.pricing) main_path, session_dir = resolve_session(args) session_id = os.path.basename(main_path)[:-len(".jsonl")] - main_agg = aggregate_transcript(main_path, pricing) + main_agg = aggregate_transcript(main_path) main_meta = analyze_main(main_path) subagents = [] if session_dir and os.path.isdir(session_dir): - subagents = analyze_subagents(session_dir, pricing) + subagents = analyze_subagents(session_dir) sub_tokens = empty_tokens() - sub_cost = 0.0 for s in subagents: for k in sub_tokens: sub_tokens[k] += s["tokens"][k] - sub_cost += s["cost_usd"] total_tokens = empty_tokens() for k in total_tokens: total_tokens[k] = main_agg["tokens"][k] + sub_tokens[k] - total_cost = round(main_agg["cost_usd"] + sub_cost, 4) warnings = [] is_active = False @@ -407,13 +355,8 @@ def main(): is_active = True warnings.append("Session appears ACTIVE (last event <2 min ago); " "totals are partial and will grow.") - if main_agg["unknown_model"] or any(s.get("models") and - any(rate_for(m, pricing)[1] for m in s["models"]) - for s in subagents): - warnings.append("One or more models were not in the price table; " - "billed at the opus tier (estimate errs high).") - warnings.append("Cost is DERIVED from tokens x list prices, not metered. " - "Price basis: " + PRICING_AS_OF) + warnings.append("Token counts are raw usage; wall-clock is derived from " + "timestamp deltas (the transcript carries no durationMs).") report = { "session": { @@ -430,7 +373,7 @@ def main(): "requests": main_agg["requests"], "models": main_agg["models"], "tokens": main_agg["tokens"], - "cost_usd": main_agg["cost_usd"], + "total_tokens": main_agg["total_tokens"], "cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]), "server_tools": main_agg["server_tools"], "user_prompts": main_meta["user_prompts"], @@ -451,15 +394,13 @@ def main(): "subagent_totals": { "count": len(subagents), "tokens": sub_tokens, - "cost_usd": round(sub_cost, 4), + "total_tokens": token_total(sub_tokens), }, "totals": { "tokens": total_tokens, + "total_tokens": token_total(total_tokens), "cache_hit_ratio": cache_hit_ratio(total_tokens), - "cost_usd": total_cost, }, - "pricing_used": pricing, - "pricing_basis": PRICING_AS_OF, "warnings": warnings, } json.dump(report, sys.stdout, indent=2)