refactor(postmortem): report raw token counts, drop dollar estimates
Remove the price-table cost derivation entirely. The transcript carries no metered cost, and list prices vary by plan — so converting tokens to a dollar figure dressed an estimate up as a number. Report raw token counts instead, kept split by class (input / output / cache-creation / cache-read) with a `total_tokens` sum per scope. Script: drop DEFAULT_PRICING / load_pricing / rate_for / cost_of and the --pricing flag; aggregate_transcript and analyze_subagents no longer take a pricing arg; every `cost_usd` field becomes `total_tokens`; the pricing warning is gone. SKILL.md: the first graded axis is now "Token spend & efficiency", the scorecard and output format cite token counts, and the Iron Law / Red Flags now forbid converting tokens to a cost. cache_hit_ratio and the active-session warning are unchanged.
This commit is contained in:
+40
-40
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: postmortem
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and what the run cost. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/cost/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
---
|
||||
|
||||
# postmortem — grade the run
|
||||
@@ -14,10 +14,10 @@ JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`
|
||||
plus one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`. Nobody reads it. So the same expensive
|
||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||||
loop with a 30% error rate, a cache-cold run that pays full input
|
||||
price on every turn. This skill turns that recorder into a graded
|
||||
loop with a 30% error rate, a cache-cold run that re-sends the full
|
||||
input on every turn. This skill turns that recorder into a graded
|
||||
retrospective: **how well did the toolchain work, how effective
|
||||
were the agents, and what did it cost.**
|
||||
were the agents, and how many tokens did it spend.**
|
||||
|
||||
It is descriptive, not corrective. It produces a report and, where
|
||||
warranted, recommends filing tracker issues — it does not change
|
||||
@@ -31,7 +31,7 @@ Invoke:
|
||||
where the time and tokens went.
|
||||
- At the close of a `/boss` autonomous run, to grade the
|
||||
orchestrator's dispatch choices.
|
||||
- When a run *felt* expensive or thrashy and you want the numbers.
|
||||
- When a run *felt* token-heavy or thrashy and you want the numbers.
|
||||
|
||||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
||||
blocks a cycle.
|
||||
@@ -40,7 +40,7 @@ blocks a cycle.
|
||||
|
||||
```
|
||||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
||||
EVERY DOLLAR FIGURE IS LABELLED "ESTIMATE" — THE TRANSCRIPT CARRIES NO METERED COST
|
||||
REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE
|
||||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
||||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
||||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
||||
@@ -57,9 +57,11 @@ Read before interpreting, so the report never overclaims:
|
||||
slash commands, and a full separate transcript + `agentType` /
|
||||
`description` for each subagent.
|
||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||||
`null`. **Cost is derived** from tokens × a list-price table;
|
||||
**wall-clock is derived** from timestamp deltas. Both are
|
||||
estimates and the report says so.
|
||||
`null`. There is no metered cost in the log, so this skill does
|
||||
not compute one — it reports **raw token counts** (kept split by
|
||||
class: input / output / cache-creation / cache-read, plus a
|
||||
`total_tokens` sum). **Wall-clock is derived** from timestamp
|
||||
deltas and the report says so.
|
||||
|
||||
The streamed transcript repeats the same `requestId` across
|
||||
several assistant lines with an identical `usage` object. The
|
||||
@@ -76,9 +78,7 @@ python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
||||
Defaults to the newest transcript in the current project's log
|
||||
dir — i.e. the session you are in (it will be flagged `active`,
|
||||
meaning totals are partial). To grade a finished run, pass its
|
||||
`--session <uuid>`. Override prices for a non-default plan with
|
||||
`--pricing '<json>'` or the `POSTMORTEM_PRICING` env var (same
|
||||
shape as `DEFAULT_PRICING` in the script).
|
||||
`--session <uuid>`.
|
||||
|
||||
The script emits one JSON object on stdout. That JSON — not the
|
||||
raw log — is your evidence base.
|
||||
@@ -88,20 +88,21 @@ raw log — is your evidence base.
|
||||
Grade each axis on the cited numbers. A grade with no number
|
||||
behind it is an opinion; drop it.
|
||||
|
||||
1. **Cost & efficiency** — `totals.cost_usd` (estimate),
|
||||
main-vs-subagent split, `cache_hit_ratio` (the dominant lever:
|
||||
below ~0.7 on a long session means cache-cold turns paying full
|
||||
input price), output-token share, `server_tools` (paid
|
||||
web_search / web_fetch).
|
||||
1. **Token spend & efficiency** — `totals.total_tokens`, the
|
||||
main-vs-subagent split, the token breakdown (output share is the
|
||||
write-heavy signal), `cache_hit_ratio` (the dominant efficiency
|
||||
lever: below ~0.7 on a long session means cache-cold turns
|
||||
re-sending the full input), `server_tools` (web_search /
|
||||
web_fetch counts).
|
||||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
||||
(the authoritative `is_error` signal), the tool mix (a Read /
|
||||
Edit / Bash imbalance hints at thrashing or re-reading),
|
||||
`slash_commands` such as `/compact` (context pressure).
|
||||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
||||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
||||
`NEEDS_CONTEXT` / `unknown`), `cost_usd`, and `requests`. A
|
||||
fleet that cost real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare subagent spend to what they delivered.
|
||||
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
|
||||
fleet that burned real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare each agent's token spend to what it delivered.
|
||||
|
||||
### Step 3 — Write the report
|
||||
|
||||
@@ -131,7 +132,7 @@ Resolve in this order:
|
||||
under the project root, creating the directory if needed.
|
||||
|
||||
Also print a ≤200-word summary to the chat so the headline grade
|
||||
and cost are visible without opening the file.
|
||||
and token spend are visible without opening the file.
|
||||
|
||||
## Output format
|
||||
|
||||
@@ -141,28 +142,28 @@ and cost are visible without opening the file.
|
||||
**Scorecard**
|
||||
| Axis | Grade | Headline metric |
|
||||
|------|-------|-----------------|
|
||||
| Cost & efficiency | A–F | $<est> total, cache hit <ratio> |
|
||||
| Token spend & efficiency | A–F | <total_tokens> total, cache hit <ratio> |
|
||||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, $<est> on subagents |
|
||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, <tokens> on subagents |
|
||||
|
||||
**Cost** (ESTIMATE — derived from tokens × list price, not metered)
|
||||
<main vs subagent split; token breakdown; the one number that
|
||||
dominates the bill and why>
|
||||
**Token spend**
|
||||
<main vs subagent split; token breakdown by class (input / output /
|
||||
cache-creation / cache-read); the one class that dominates and why>
|
||||
|
||||
**Toolchain**
|
||||
<tool mix, error ratio + what the errors were, context-pressure
|
||||
signals like /compact>
|
||||
|
||||
**Agents**
|
||||
<per-subagent line: type — status — $est — one-clause verdict.
|
||||
Omit if no subagents were dispatched.>
|
||||
<per-subagent line: type — status — <total_tokens> — one-clause
|
||||
verdict. Omit if no subagents were dispatched.>
|
||||
|
||||
**Recommendations**
|
||||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
||||
an `issue`.>
|
||||
|
||||
_Pricing basis: <pricing_basis from the JSON>. Wall-clock and
|
||||
cost are derived, not metered._
|
||||
_Token counts are raw usage. Wall-clock is derived from timestamp
|
||||
deltas, not metered._
|
||||
```
|
||||
|
||||
If the session is flagged `active`, add one line at the top:
|
||||
@@ -172,19 +173,19 @@ If the session is flagged `active`, add one line at the top:
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "I'll just skim the transcript and estimate the cost" | The transcript has no cost field and repeats usage per streamed line. Hand-summing is wrong twice over. Run the script. |
|
||||
| "$31 is the cost" | It is an *estimate* from list prices. The plan may differ; the basis line and the `ESTIMATE` label are non-optional. |
|
||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent at full price. That is the single biggest lever — grade it. |
|
||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned $2 is a failure that cost money. Read `terminal_status`. |
|
||||
| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. |
|
||||
| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. |
|
||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. |
|
||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. |
|
||||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
||||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
||||
|
||||
## Red Flags — STOP
|
||||
|
||||
- Quoting any token or dollar number that did not come out of the
|
||||
script's JSON.
|
||||
- Presenting a derived dollar figure without the `ESTIMATE` label
|
||||
and pricing basis.
|
||||
- Quoting any token number that did not come out of the script's
|
||||
JSON.
|
||||
- Converting token counts into a cost or dollar figure — report
|
||||
raw counts only.
|
||||
- Reading any file under `~/.claude` other than the session
|
||||
transcript and its subagent logs — and never any credential or
|
||||
token file.
|
||||
@@ -199,8 +200,7 @@ If the session is flagged `active`, add one line at the top:
|
||||
- **Hand-off target:** the `issue` skill, when a finding is a
|
||||
trackable defect worth filing.
|
||||
- **Profile slot (optional):** `paths.postmortem_dir` for the
|
||||
report destination; `POSTMORTEM_PRICING` to override the price
|
||||
table for a non-default plan.
|
||||
report destination.
|
||||
- **Data-source note:** this skill reads only the current
|
||||
session's own transcript and subagent logs. It does not crawl
|
||||
other projects or other sessions (single-session by design).
|
||||
|
||||
Reference in New Issue
Block a user