chore(postmortem): remove the retired postmortem skill
Retirement decided 2026-07-14: no consumers — nothing else in the repo references it, it is not a pipeline phase or dispatch target, and exactly one report was ever produced. Removes postmortem/ (SKILL.md, scripts/, tests/). The global symlink ~/.claude/skills/postmortem and a gitignored __pycache__ leftover were cleaned up on disk; install.sh/uninstall.sh iterate directories generically and need no change. Resolution of the issue's open point (archive): docs/postmortems/ turned out to be untracked all along — gitignored by design as per-run output — so the repo never carried the archive. The single local report stays on disk untouched, and the .gitignore entry stays (with an updated comment) so the local archive does not become git-status noise. closes #27
This commit is contained in:
+3
-1
@@ -5,5 +5,7 @@
|
|||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
|
||||||
# postmortem skill output — reports are per-run artifacts, not tracked
|
# local report archive of the retired postmortem skill (issue #27) —
|
||||||
|
# reports were per-run artifacts, never tracked; the dir may still
|
||||||
|
# exist locally
|
||||||
docs/postmortems/
|
docs/postmortems/
|
||||||
|
|||||||
@@ -1,228 +0,0 @@
|
|||||||
---
|
|
||||||
name: postmortem
|
|
||||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
|
||||||
---
|
|
||||||
|
|
||||||
# postmortem — grade the run
|
|
||||||
|
|
||||||
> **Violating the letter of these rules is violating the spirit.**
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Every session leaves a detailed flight recorder behind — the
|
|
||||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`,
|
|
||||||
one sidechain log per dispatched subagent under
|
|
||||||
`<session>/subagents/`, and for every Workflow run a per-agent
|
|
||||||
summary in `<session>/workflows/wf_*.json` with raw per-agent
|
|
||||||
transcripts under `<session>/subagents/workflows/<runId>/`. Nobody
|
|
||||||
reads it. So the same expensive
|
|
||||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
|
||||||
loop with a 30% error rate, a cache-cold run that re-sends the full
|
|
||||||
input on every turn. This skill turns that recorder into a graded
|
|
||||||
retrospective: **how well did the toolchain work, how effective
|
|
||||||
were the agents, and how many tokens did it spend.**
|
|
||||||
|
|
||||||
It is descriptive, not corrective. It produces a report and, where
|
|
||||||
warranted, recommends filing tracker issues — it does not change
|
|
||||||
code, baselines, or the pipeline.
|
|
||||||
|
|
||||||
## When to Use / Skipping
|
|
||||||
|
|
||||||
Invoke:
|
|
||||||
|
|
||||||
- After a substantial interactive session, when you want to know
|
|
||||||
where the time and tokens went.
|
|
||||||
- At the close of a `/boss` autonomous run, to grade the
|
|
||||||
orchestrator's dispatch choices.
|
|
||||||
- When a run *felt* token-heavy or thrashy and you want the numbers.
|
|
||||||
|
|
||||||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
|
||||||
blocks a cycle.
|
|
||||||
|
|
||||||
## The Iron Law
|
|
||||||
|
|
||||||
```
|
|
||||||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
|
||||||
REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE
|
|
||||||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
|
||||||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
|
||||||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
|
||||||
```
|
|
||||||
|
|
||||||
## What the data does and does not contain
|
|
||||||
|
|
||||||
Read before interpreting, so the report never overclaims:
|
|
||||||
|
|
||||||
- **Present:** per-request `usage` (input / output / cache-creation /
|
|
||||||
cache-read / server-tool counts), `model`, `requestId`,
|
|
||||||
timestamps, every `tool_use` with its input, every `tool_result`
|
|
||||||
with the authoritative `is_error` flag, skill invocations,
|
|
||||||
slash commands, and a full separate transcript + `agentType` /
|
|
||||||
`description` for each subagent. Workflow runs additionally carry a
|
|
||||||
per-agent summary (stage label, model, a live token counter) in
|
|
||||||
`workflows/wf_*.json`; the counter under-tracks vs. the deduped raw
|
|
||||||
transcripts (~11% in the corpus behind issue #28), so the script
|
|
||||||
treats the raw transcripts as authoritative and reports the counter
|
|
||||||
only as `reported_total_tokens` for cross-checking.
|
|
||||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
|
||||||
`null`. There is no metered cost in the log, so this skill does
|
|
||||||
not compute one — it reports **raw token counts** (kept split by
|
|
||||||
class: input / output / cache-creation / cache-read, plus a
|
|
||||||
`total_tokens` sum). **Wall-clock is derived** from timestamp
|
|
||||||
deltas and the report says so.
|
|
||||||
|
|
||||||
The streamed transcript repeats the same `requestId` across
|
|
||||||
several assistant lines with an identical `usage` object. The
|
|
||||||
helper script dedups per `requestId`; never sum the raw lines.
|
|
||||||
|
|
||||||
## The Process
|
|
||||||
|
|
||||||
### Step 1 — Run the aggregator
|
|
||||||
|
|
||||||
```
|
|
||||||
python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
|
||||||
```
|
|
||||||
|
|
||||||
Defaults to the newest transcript in the current project's log
|
|
||||||
dir — i.e. the session you are in (it will be flagged `active`,
|
|
||||||
meaning totals are partial). To grade a finished run, pass its
|
|
||||||
`--session <uuid>`.
|
|
||||||
|
|
||||||
The default log dir is derived from cwd (the path with `/` replaced
|
|
||||||
by `-`), so a session run inside a git worktree resolves to a
|
|
||||||
*different* slug than the primary clone, and after exiting a worktree
|
|
||||||
the default again follows the current cwd. To grade a worktree run
|
|
||||||
from the primary clone, pass `--project-dir
|
|
||||||
~/.claude/projects/<worktree-slug>` (or `--file <path>` to bypass slug
|
|
||||||
derivation entirely).
|
|
||||||
|
|
||||||
The script emits one JSON object on stdout. That JSON — not the
|
|
||||||
raw log — is your evidence base.
|
|
||||||
|
|
||||||
### Step 2 — Read the JSON and grade three axes
|
|
||||||
|
|
||||||
Grade each axis on the cited numbers. A grade with no number
|
|
||||||
behind it is an opinion; drop it.
|
|
||||||
|
|
||||||
1. **Token spend & efficiency** — `totals.total_tokens`, the
|
|
||||||
main-vs-subagent split, the token breakdown (output share is the
|
|
||||||
write-heavy signal), `cache_hit_ratio` (the dominant efficiency
|
|
||||||
lever: below ~0.7 on a long session means cache-cold turns
|
|
||||||
re-sending the full input), `server_tools` (web_search /
|
|
||||||
web_fetch counts).
|
|
||||||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
|
||||||
(the authoritative `is_error` signal), the tool mix (a Read /
|
|
||||||
Edit / Bash imbalance hints at thrashing or re-reading),
|
|
||||||
`slash_commands` such as `/compact` (context pressure).
|
|
||||||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
|
||||||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
|
||||||
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
|
|
||||||
fleet that burned real tokens to return `BLOCKED` is the headline
|
|
||||||
finding. Compare each agent's token spend to what it delivered.
|
|
||||||
For Workflow runs, grade per stage instead: `workflow_totals` and
|
|
||||||
each run's `stages` table (label-prefix grouping, e.g. all
|
|
||||||
`impl:N` under `impl`) show where the run's spend concentrated;
|
|
||||||
`agents_untracked > 0` flags retried/dropped attempts. Schema-bound
|
|
||||||
workflow stages carry no `terminal_status` — the workflow script's
|
|
||||||
own aggregation already judged them.
|
|
||||||
|
|
||||||
### Step 3 — Write the report
|
|
||||||
|
|
||||||
Write a Markdown report to the configured destination (see
|
|
||||||
**Output destination**). Structure it as **Output format** below.
|
|
||||||
Lead with the scorecard, support every grade with a number from
|
|
||||||
the JSON, and keep the prose tight — this is a retrospective, not
|
|
||||||
a transcript replay.
|
|
||||||
|
|
||||||
### Step 4 — Surface, don't fix
|
|
||||||
|
|
||||||
Translate the sharpest findings into recommendations. Where a
|
|
||||||
finding is a concrete, trackable defect (a subagent that reliably
|
|
||||||
bounces `BLOCKED`, a tool loop with a runaway error rate),
|
|
||||||
recommend filing it via the `issue` skill — name it, but do not
|
|
||||||
file it yourself unless the user asks. This skill ends at the
|
|
||||||
report.
|
|
||||||
|
|
||||||
## Output destination
|
|
||||||
|
|
||||||
Resolve in this order:
|
|
||||||
|
|
||||||
1. `--out <path>` if the invoker passed one.
|
|
||||||
2. The project's post-mortem directory, if it has one (its CLAUDE.md
|
|
||||||
project facts).
|
|
||||||
3. Default: `docs/postmortems/<session-id-short>-<yyyy-mm-dd>.md`
|
|
||||||
under the project root, creating the directory if needed.
|
|
||||||
|
|
||||||
Also print a ≤200-word summary to the chat so the headline grade
|
|
||||||
and token spend are visible without opening the file.
|
|
||||||
|
|
||||||
## Output format
|
|
||||||
|
|
||||||
```
|
|
||||||
# Post-mortem — <session-id-short> (<date>)
|
|
||||||
|
|
||||||
**Scorecard**
|
|
||||||
| Axis | Grade | Headline metric |
|
|
||||||
|------|-------|-----------------|
|
|
||||||
| Token spend & efficiency | A–F | <total_tokens> total, cache hit <ratio> |
|
|
||||||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
|
||||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, <tokens> on subagents |
|
|
||||||
|
|
||||||
**Token spend**
|
|
||||||
<main vs subagent split; token breakdown by class (input / output /
|
|
||||||
cache-creation / cache-read); the one class that dominates and why>
|
|
||||||
|
|
||||||
**Toolchain**
|
|
||||||
<tool mix, error ratio + what the errors were, context-pressure
|
|
||||||
signals like /compact>
|
|
||||||
|
|
||||||
**Agents**
|
|
||||||
<per-subagent line: type — status — <total_tokens> — one-clause
|
|
||||||
verdict. Omit if no subagents were dispatched.>
|
|
||||||
|
|
||||||
**Recommendations**
|
|
||||||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
|
||||||
an `issue`.>
|
|
||||||
|
|
||||||
_Token counts are raw usage. Wall-clock is derived from timestamp
|
|
||||||
deltas, not metered._
|
|
||||||
```
|
|
||||||
|
|
||||||
If the session is flagged `active`, add one line at the top:
|
|
||||||
"⚠️ Session still active — figures are partial."
|
|
||||||
|
|
||||||
## Common Rationalisations
|
|
||||||
|
|
||||||
| Excuse | Reality |
|
|
||||||
|--------|---------|
|
|
||||||
| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. |
|
|
||||||
| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. |
|
|
||||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. |
|
|
||||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. |
|
|
||||||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
|
||||||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
|
||||||
|
|
||||||
## Red Flags — STOP
|
|
||||||
|
|
||||||
- Quoting any token number that did not come out of the script's
|
|
||||||
JSON.
|
|
||||||
- Converting token counts into a cost or dollar figure — report
|
|
||||||
raw counts only.
|
|
||||||
- Reading any file under `~/.claude` other than the session
|
|
||||||
transcript and its subagent logs — and never any credential or
|
|
||||||
token file.
|
|
||||||
- Editing code, baselines, or pipeline config "while you're in
|
|
||||||
there". Report only.
|
|
||||||
- Grading an axis with adjectives instead of the cited metric.
|
|
||||||
|
|
||||||
## Cross-references
|
|
||||||
|
|
||||||
- **Helper script:** `scripts/postmortem.py` — the deterministic
|
|
||||||
aggregator. All numbers originate here.
|
|
||||||
- **Hand-off target:** the `issue` skill, when a finding is a
|
|
||||||
trackable defect worth filing.
|
|
||||||
- **Report destination (optional):** the project's post-mortem
|
|
||||||
directory, if its CLAUDE.md project facts name one.
|
|
||||||
- **Data-source note:** this skill reads only the current
|
|
||||||
session's own transcript and subagent logs. It does not crawl
|
|
||||||
other projects or other sessions (single-session by design).
|
|
||||||
@@ -1,501 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Deterministic aggregator for a single Claude Code session.
|
|
||||||
|
|
||||||
Reads the main transcript JSONL for one session plus its subagent
|
|
||||||
sidechain logs and its Workflow-substrate run logs (workflows/wf_*.json
|
|
||||||
+ subagents/workflows/<runId>/agent-*.jsonl), and emits a compact JSON
|
|
||||||
summary on stdout that the post-mortem SKILL interprets into a written
|
|
||||||
report.
|
|
||||||
|
|
||||||
Why a script and not skill prose: transcripts run to >1 MB and a
|
|
||||||
session can spawn dozens of subagent logs. Token accounting has two
|
|
||||||
traps that only deterministic code gets right every time:
|
|
||||||
|
|
||||||
1. Streamed assistant lines repeat the SAME requestId with the SAME
|
|
||||||
usage object. Counting per-line double-counts tokens. We dedup
|
|
||||||
per requestId (max per field within a request, then sum across
|
|
||||||
requests).
|
|
||||||
2. The transcript carries NO durationMs (always null), so wall-clock
|
|
||||||
is derived from timestamp deltas. The numbers reported here are
|
|
||||||
raw token counts, not a dollar estimate — different token classes
|
|
||||||
(input / output / cache) are reported separately rather than
|
|
||||||
collapsed into a single priced figure.
|
|
||||||
|
|
||||||
Stdlib only. No network. Read-only against ~/.claude.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import glob
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
_TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first
|
|
||||||
# Authoritative signal: the implement end-report leads with a structured
|
|
||||||
# "Status: DONE" line (optionally wrapped in markdown bold or a list
|
|
||||||
# marker). Anchoring here is what stops the scan from flipping a shipped
|
|
||||||
# DONE run to BLOCKED on the strength of the report's own template lines
|
|
||||||
# (`BLOCKED file: BLOCKED.md`, `Blocked detail:`) or prose narrating a
|
|
||||||
# surmounted blocker. See issue #3.
|
|
||||||
STATUS_LINE_RE = re.compile(
|
|
||||||
r"^[\s>*#\-]*\**\s*status\s*\**\s*[:\-]\s*\**\s*(" + _TOK_ALT + r")\b",
|
|
||||||
re.IGNORECASE | re.MULTILINE)
|
|
||||||
# Fallback for agents that emit a bare status token as their final line
|
|
||||||
# (no "Status:" prefix). Requires the token to BE the line — excludes
|
|
||||||
# substrings like "BLOCKED.md" or "BLOCKED file:" buried in prose.
|
|
||||||
STATUS_STANDALONE_RE = re.compile(
|
|
||||||
r"^[\s>*#\-]*\**\s*(" + _TOK_ALT + r")\**\s*$",
|
|
||||||
re.IGNORECASE | re.MULTILINE)
|
|
||||||
|
|
||||||
|
|
||||||
def parse_ts(s):
|
|
||||||
if not s:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.fromisoformat(s.replace("Z", "+00:00"))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def empty_tokens():
|
|
||||||
return {"input": 0, "output": 0, "cache_creation": 0, "cache_read": 0,
|
|
||||||
"cache_creation_5m": 0, "cache_creation_1h": 0}
|
|
||||||
|
|
||||||
|
|
||||||
def token_total(tokens):
|
|
||||||
"""Raw count of every token the session moved. cache_creation_5m/1h are
|
|
||||||
a sub-split of cache_creation, so they are excluded to avoid double
|
|
||||||
counting."""
|
|
||||||
return (tokens["input"] + tokens["output"]
|
|
||||||
+ tokens["cache_creation"] + tokens["cache_read"])
|
|
||||||
|
|
||||||
|
|
||||||
def add_tokens(acc, usage):
|
|
||||||
"""Add one request's deduped usage into an accumulator."""
|
|
||||||
acc["input"] += usage.get("input_tokens", 0) or 0
|
|
||||||
acc["output"] += usage.get("output_tokens", 0) or 0
|
|
||||||
cc = usage.get("cache_creation_input_tokens", 0) or 0
|
|
||||||
acc["cache_creation"] += cc
|
|
||||||
acc["cache_read"] += usage.get("cache_read_input_tokens", 0) or 0
|
|
||||||
detail = usage.get("cache_creation") or {}
|
|
||||||
f5 = detail.get("ephemeral_5m_input_tokens")
|
|
||||||
f1 = detail.get("ephemeral_1h_input_tokens")
|
|
||||||
if f5 is None and f1 is None:
|
|
||||||
acc["cache_creation_5m"] += cc # no split available -> assume 5m
|
|
||||||
else:
|
|
||||||
acc["cache_creation_5m"] += f5 or 0
|
|
||||||
acc["cache_creation_1h"] += f1 or 0
|
|
||||||
|
|
||||||
|
|
||||||
def iter_lines(path):
|
|
||||||
with open(path, "r", encoding="utf-8", errors="replace") as fh:
|
|
||||||
for line in fh:
|
|
||||||
line = line.strip()
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
yield json.loads(line)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
|
|
||||||
|
|
||||||
def aggregate_transcript(path):
|
|
||||||
"""Return token totals, model mix, and per-request count for one JSONL
|
|
||||||
transcript (main or subagent). Dedups usage by requestId."""
|
|
||||||
# requestId -> {field: max seen}, plus model per request.
|
|
||||||
req_usage = {}
|
|
||||||
req_model = {}
|
|
||||||
for rec in iter_lines(path):
|
|
||||||
if rec.get("type") != "assistant":
|
|
||||||
continue
|
|
||||||
msg = rec.get("message") or {}
|
|
||||||
usage = msg.get("usage")
|
|
||||||
if not usage:
|
|
||||||
continue
|
|
||||||
key = rec.get("requestId") or rec.get("uuid")
|
|
||||||
model = msg.get("model")
|
|
||||||
if model:
|
|
||||||
req_model[key] = model
|
|
||||||
cur = req_usage.setdefault(key, {})
|
|
||||||
# Max-per-field guards against both identical-repeat and
|
|
||||||
# cumulative-within-request streaming shapes.
|
|
||||||
for f in ("input_tokens", "output_tokens",
|
|
||||||
"cache_creation_input_tokens", "cache_read_input_tokens"):
|
|
||||||
v = usage.get(f, 0) or 0
|
|
||||||
if v > cur.get(f, 0):
|
|
||||||
cur[f] = v
|
|
||||||
det = usage.get("cache_creation") or {}
|
|
||||||
cd = cur.setdefault("cache_creation", {})
|
|
||||||
for f in ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"):
|
|
||||||
v = det.get(f, 0) or 0
|
|
||||||
if v > cd.get(f, 0):
|
|
||||||
cd[f] = v
|
|
||||||
st = usage.get("server_tool_use") or {}
|
|
||||||
sc = cur.setdefault("_server", {})
|
|
||||||
for f in ("web_search_requests", "web_fetch_requests"):
|
|
||||||
v = st.get(f, 0) or 0
|
|
||||||
if v > sc.get(f, 0):
|
|
||||||
sc[f] = v
|
|
||||||
|
|
||||||
tokens = empty_tokens()
|
|
||||||
server = {"web_search": 0, "web_fetch": 0}
|
|
||||||
models = {}
|
|
||||||
for key, usage in req_usage.items():
|
|
||||||
add_tokens(tokens, usage)
|
|
||||||
model = req_model.get(key)
|
|
||||||
models[model] = models.get(model, 0) + 1
|
|
||||||
sv = usage.get("_server", {})
|
|
||||||
server["web_search"] += sv.get("web_search_requests", 0)
|
|
||||||
server["web_fetch"] += sv.get("web_fetch_requests", 0)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"requests": len(req_usage),
|
|
||||||
"tokens": tokens,
|
|
||||||
"total_tokens": token_total(tokens),
|
|
||||||
"models": models,
|
|
||||||
"server_tools": server,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def cache_hit_ratio(tokens):
|
|
||||||
denom = tokens["input"] + tokens["cache_read"] + tokens["cache_creation"]
|
|
||||||
return round(tokens["cache_read"] / denom, 4) if denom else None
|
|
||||||
|
|
||||||
|
|
||||||
def analyze_main(path):
|
|
||||||
"""Tool mix, error rate, skills, slash commands, prompts, wall-clock."""
|
|
||||||
tool_calls = {}
|
|
||||||
error_total = 0
|
|
||||||
interrupted = 0
|
|
||||||
skills = []
|
|
||||||
slash = []
|
|
||||||
tasks = []
|
|
||||||
user_prompts = 0
|
|
||||||
first_ts = last_ts = None
|
|
||||||
cwd = version = None
|
|
||||||
|
|
||||||
for rec in iter_lines(path):
|
|
||||||
ts = parse_ts(rec.get("timestamp"))
|
|
||||||
if ts:
|
|
||||||
if first_ts is None or ts < first_ts:
|
|
||||||
first_ts = ts
|
|
||||||
if last_ts is None or ts > last_ts:
|
|
||||||
last_ts = ts
|
|
||||||
cwd = rec.get("cwd") or cwd
|
|
||||||
version = rec.get("version") or version
|
|
||||||
|
|
||||||
typ = rec.get("type")
|
|
||||||
msg = rec.get("message") or {}
|
|
||||||
content = msg.get("content")
|
|
||||||
|
|
||||||
if typ == "assistant" and isinstance(content, list):
|
|
||||||
for block in content:
|
|
||||||
if block.get("type") != "tool_use":
|
|
||||||
continue
|
|
||||||
name = block.get("name", "?")
|
|
||||||
tool_calls[name] = tool_calls.get(name, 0) + 1
|
|
||||||
if name == "Skill":
|
|
||||||
skills.append({"skill": (block.get("input") or {}).get("skill"),
|
|
||||||
"ts": rec.get("timestamp")})
|
|
||||||
elif name in ("Task", "Agent"):
|
|
||||||
inp = block.get("input") or {}
|
|
||||||
tasks.append({"subagent_type": inp.get("subagent_type"),
|
|
||||||
"description": inp.get("description"),
|
|
||||||
"ts": rec.get("timestamp")})
|
|
||||||
|
|
||||||
if typ == "user":
|
|
||||||
# tool_result blocks carry the authoritative is_error flag
|
|
||||||
if isinstance(content, list):
|
|
||||||
for block in content:
|
|
||||||
if block.get("type") == "tool_result":
|
|
||||||
if block.get("is_error"):
|
|
||||||
error_total += 1
|
|
||||||
elif block.get("type") == "text":
|
|
||||||
for m in re.findall(r"<command-name>(.*?)</command-name>",
|
|
||||||
block.get("text", "")):
|
|
||||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
|
||||||
elif isinstance(content, str):
|
|
||||||
for m in re.findall(r"<command-name>(.*?)</command-name>", content):
|
|
||||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
|
||||||
# genuine user turns (not tool results, not slash plumbing)
|
|
||||||
is_tool_result = (isinstance(content, list)
|
|
||||||
and any(b.get("type") == "tool_result" for b in content))
|
|
||||||
if not is_tool_result and rec.get("toolUseResult") is None:
|
|
||||||
user_prompts += 1
|
|
||||||
tur = rec.get("toolUseResult")
|
|
||||||
if isinstance(tur, dict) and tur.get("interrupted"):
|
|
||||||
interrupted += 1
|
|
||||||
|
|
||||||
return {
|
|
||||||
"tool_calls": tool_calls,
|
|
||||||
"total_tool_calls": sum(tool_calls.values()),
|
|
||||||
"tool_error_total": error_total,
|
|
||||||
"interrupted": interrupted,
|
|
||||||
"skills_invoked": skills,
|
|
||||||
"slash_commands": slash,
|
|
||||||
"tasks_dispatched": tasks,
|
|
||||||
"user_prompts": user_prompts,
|
|
||||||
"wall_clock_sec": round((last_ts - first_ts).total_seconds(), 1)
|
|
||||||
if first_ts and last_ts else None,
|
|
||||||
"started": first_ts.isoformat() if first_ts else None,
|
|
||||||
"ended": last_ts.isoformat() if last_ts else None,
|
|
||||||
"cwd": cwd,
|
|
||||||
"version": version,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def subagent_terminal_status(path):
|
|
||||||
last_text = ""
|
|
||||||
for rec in iter_lines(path):
|
|
||||||
if rec.get("type") != "assistant":
|
|
||||||
continue
|
|
||||||
content = (rec.get("message") or {}).get("content")
|
|
||||||
if isinstance(content, list):
|
|
||||||
for block in content:
|
|
||||||
if block.get("type") == "text" and block.get("text", "").strip():
|
|
||||||
last_text = block["text"]
|
|
||||||
# Primary: the structured "Status:" line. Take the LAST one so a
|
|
||||||
# restated final status wins over any earlier mention.
|
|
||||||
structured = STATUS_LINE_RE.findall(last_text)
|
|
||||||
if structured:
|
|
||||||
return structured[-1].upper()
|
|
||||||
# Fallback: a status token standing alone on its own line.
|
|
||||||
standalone = STATUS_STANDALONE_RE.findall(last_text)
|
|
||||||
if standalone:
|
|
||||||
return standalone[-1].upper()
|
|
||||||
return "unknown"
|
|
||||||
|
|
||||||
|
|
||||||
def analyze_workflows(session_dir):
|
|
||||||
"""Aggregate Workflow-substrate runs: workflows/wf_*.json (per-agent
|
|
||||||
summary: stage label, model, live token counter) joined with the raw
|
|
||||||
per-agent transcripts under subagents/workflows/<runId>/agent-*.jsonl.
|
|
||||||
|
|
||||||
The raw transcripts are authoritative for token counts (deduped by
|
|
||||||
requestId, same as every other transcript here); the wf_*.json
|
|
||||||
'tokens' counters under-track vs. the deduped raw usage (~11% in the
|
|
||||||
206-run corpus that motivated this path, issue #28) and are reported
|
|
||||||
only as a cross-check. Stage = the label up to the first ':'
|
|
||||||
('impl:3' -> 'impl'), so loop stages aggregate across tasks/rounds."""
|
|
||||||
runs = []
|
|
||||||
for wf_path in sorted(glob.glob(os.path.join(session_dir, "workflows",
|
|
||||||
"wf_*.json"))):
|
|
||||||
try:
|
|
||||||
with open(wf_path) as fh:
|
|
||||||
data = json.load(fh)
|
|
||||||
except (json.JSONDecodeError, OSError):
|
|
||||||
continue
|
|
||||||
run_id = data.get("runId") or os.path.basename(wf_path)[:-len(".json")]
|
|
||||||
raw_dir = os.path.join(session_dir, "subagents", "workflows", run_id)
|
|
||||||
meta_by_agent = {}
|
|
||||||
for entry in data.get("workflowProgress") or []:
|
|
||||||
if entry.get("type") != "workflow_agent" or not entry.get("agentId"):
|
|
||||||
continue
|
|
||||||
meta_by_agent[entry["agentId"]] = entry
|
|
||||||
|
|
||||||
run_tokens = empty_tokens()
|
|
||||||
stages = {}
|
|
||||||
agent_count = 0
|
|
||||||
untracked = 0
|
|
||||||
for jl in sorted(glob.glob(os.path.join(raw_dir, "agent-*.jsonl"))):
|
|
||||||
aid = os.path.basename(jl)[len("agent-"):-len(".jsonl")]
|
|
||||||
agg = aggregate_transcript(jl)
|
|
||||||
entry = meta_by_agent.get(aid)
|
|
||||||
if entry is None:
|
|
||||||
# Transcript with no progress entry (retried/dropped
|
|
||||||
# attempt): keep it counted, grouped under '?', never
|
|
||||||
# silently folded into a named stage.
|
|
||||||
untracked += 1
|
|
||||||
label = (entry or {}).get("label") or "?"
|
|
||||||
stage = label.split(":", 1)[0]
|
|
||||||
st = stages.setdefault(stage, {"count": 0, "tokens": empty_tokens(),
|
|
||||||
"models": {}})
|
|
||||||
st["count"] += 1
|
|
||||||
agent_count += 1
|
|
||||||
for k in run_tokens:
|
|
||||||
run_tokens[k] += agg["tokens"][k]
|
|
||||||
st["tokens"][k] += agg["tokens"][k]
|
|
||||||
fallback_model = (entry or {}).get("model")
|
|
||||||
for m, n in agg["models"].items():
|
|
||||||
key = m or fallback_model
|
|
||||||
st["models"][key] = st["models"].get(key, 0) + n
|
|
||||||
|
|
||||||
for st in stages.values():
|
|
||||||
st["total_tokens"] = token_total(st["tokens"])
|
|
||||||
|
|
||||||
runs.append({
|
|
||||||
"run_id": run_id,
|
|
||||||
"name": data.get("workflowName"),
|
|
||||||
"status": data.get("status"),
|
|
||||||
"agents": agent_count,
|
|
||||||
"agents_untracked": untracked,
|
|
||||||
"reported_total_tokens": data.get("totalTokens"),
|
|
||||||
"tokens": run_tokens,
|
|
||||||
"total_tokens": token_total(run_tokens),
|
|
||||||
"stages": stages,
|
|
||||||
})
|
|
||||||
return runs
|
|
||||||
|
|
||||||
|
|
||||||
def analyze_subagents(session_dir):
|
|
||||||
out = []
|
|
||||||
sub_dir = os.path.join(session_dir, "subagents")
|
|
||||||
for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))):
|
|
||||||
meta = {}
|
|
||||||
mp = jl.replace(".jsonl", ".meta.json")
|
|
||||||
if os.path.exists(mp):
|
|
||||||
try:
|
|
||||||
with open(mp) as fh:
|
|
||||||
meta = json.load(fh)
|
|
||||||
except (json.JSONDecodeError, OSError):
|
|
||||||
pass
|
|
||||||
agg = aggregate_transcript(jl)
|
|
||||||
out.append({
|
|
||||||
"agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")],
|
|
||||||
"agent_type": meta.get("agentType"),
|
|
||||||
"description": meta.get("description"),
|
|
||||||
"dispatched_by_tool": meta.get("toolUseId"),
|
|
||||||
"models": agg["models"],
|
|
||||||
"requests": agg["requests"],
|
|
||||||
"tokens": agg["tokens"],
|
|
||||||
"total_tokens": agg["total_tokens"],
|
|
||||||
"terminal_status": subagent_terminal_status(jl),
|
|
||||||
})
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_session(args):
|
|
||||||
"""Return (main_jsonl_path, session_dir_or_None)."""
|
|
||||||
if args.file:
|
|
||||||
path = os.path.abspath(args.file)
|
|
||||||
return path, path[:-len(".jsonl")] if path.endswith(".jsonl") else None
|
|
||||||
|
|
||||||
cwd = args.cwd or os.getcwd()
|
|
||||||
proj_dir = args.project_dir or os.path.join(
|
|
||||||
os.path.expanduser("~/.claude/projects"), cwd.replace("/", "-"))
|
|
||||||
if not os.path.isdir(proj_dir):
|
|
||||||
raise SystemExit(f"no project log dir for cwd {cwd}: {proj_dir} not found")
|
|
||||||
|
|
||||||
if args.session:
|
|
||||||
path = os.path.join(proj_dir, args.session + ".jsonl")
|
|
||||||
if not os.path.exists(path):
|
|
||||||
raise SystemExit(f"session {args.session} not found in {proj_dir}")
|
|
||||||
return path, os.path.join(proj_dir, args.session)
|
|
||||||
|
|
||||||
# default: newest .jsonl in the project dir = current/last session
|
|
||||||
candidates = sorted(glob.glob(os.path.join(proj_dir, "*.jsonl")),
|
|
||||||
key=os.path.getmtime, reverse=True)
|
|
||||||
if not candidates:
|
|
||||||
raise SystemExit(f"no session transcripts in {proj_dir}")
|
|
||||||
path = candidates[0]
|
|
||||||
return path, path[:-len(".jsonl")]
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser(description="Single-session post-mortem aggregator.")
|
|
||||||
ap.add_argument("--session", help="explicit session id (UUID)")
|
|
||||||
ap.add_argument("--file", help="explicit path to a main transcript .jsonl")
|
|
||||||
ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)")
|
|
||||||
ap.add_argument("--project-dir", help="explicit ~/.claude/projects/<slug> dir")
|
|
||||||
args = ap.parse_args()
|
|
||||||
|
|
||||||
main_path, session_dir = resolve_session(args)
|
|
||||||
session_id = os.path.basename(main_path)[:-len(".jsonl")]
|
|
||||||
|
|
||||||
main_agg = aggregate_transcript(main_path)
|
|
||||||
main_meta = analyze_main(main_path)
|
|
||||||
|
|
||||||
subagents = []
|
|
||||||
workflows = []
|
|
||||||
if session_dir and os.path.isdir(session_dir):
|
|
||||||
subagents = analyze_subagents(session_dir)
|
|
||||||
workflows = analyze_workflows(session_dir)
|
|
||||||
|
|
||||||
sub_tokens = empty_tokens()
|
|
||||||
for s in subagents:
|
|
||||||
for k in sub_tokens:
|
|
||||||
sub_tokens[k] += s["tokens"][k]
|
|
||||||
|
|
||||||
wf_tokens = empty_tokens()
|
|
||||||
wf_agents = 0
|
|
||||||
for w in workflows:
|
|
||||||
wf_agents += w["agents"]
|
|
||||||
for k in wf_tokens:
|
|
||||||
wf_tokens[k] += w["tokens"][k]
|
|
||||||
|
|
||||||
total_tokens = empty_tokens()
|
|
||||||
for k in total_tokens:
|
|
||||||
total_tokens[k] = (main_agg["tokens"][k] + sub_tokens[k]
|
|
||||||
+ wf_tokens[k])
|
|
||||||
|
|
||||||
warnings = []
|
|
||||||
is_active = False
|
|
||||||
if main_meta["ended"]:
|
|
||||||
ended = parse_ts(main_meta["ended"])
|
|
||||||
if ended and (datetime.now(timezone.utc) - ended).total_seconds() < 120:
|
|
||||||
is_active = True
|
|
||||||
warnings.append("Session appears ACTIVE (last event <2 min ago); "
|
|
||||||
"totals are partial and will grow.")
|
|
||||||
warnings.append("Token counts are raw usage; wall-clock is derived from "
|
|
||||||
"timestamp deltas (the transcript carries no durationMs).")
|
|
||||||
|
|
||||||
report = {
|
|
||||||
"session": {
|
|
||||||
"id": session_id,
|
|
||||||
"transcript": main_path,
|
|
||||||
"cwd": main_meta["cwd"],
|
|
||||||
"version": main_meta["version"],
|
|
||||||
"started": main_meta["started"],
|
|
||||||
"ended": main_meta["ended"],
|
|
||||||
"wall_clock_sec": main_meta["wall_clock_sec"],
|
|
||||||
"active": is_active,
|
|
||||||
},
|
|
||||||
"main": {
|
|
||||||
"requests": main_agg["requests"],
|
|
||||||
"models": main_agg["models"],
|
|
||||||
"tokens": main_agg["tokens"],
|
|
||||||
"total_tokens": main_agg["total_tokens"],
|
|
||||||
"cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]),
|
|
||||||
"server_tools": main_agg["server_tools"],
|
|
||||||
"user_prompts": main_meta["user_prompts"],
|
|
||||||
},
|
|
||||||
"tools": {
|
|
||||||
"calls": main_meta["tool_calls"],
|
|
||||||
"total": main_meta["total_tool_calls"],
|
|
||||||
"error_total": main_meta["tool_error_total"],
|
|
||||||
"error_ratio": round(main_meta["tool_error_total"]
|
|
||||||
/ main_meta["total_tool_calls"], 4)
|
|
||||||
if main_meta["total_tool_calls"] else None,
|
|
||||||
"interrupted": main_meta["interrupted"],
|
|
||||||
},
|
|
||||||
"skills_invoked": main_meta["skills_invoked"],
|
|
||||||
"slash_commands": main_meta["slash_commands"],
|
|
||||||
"tasks_dispatched": main_meta["tasks_dispatched"],
|
|
||||||
"subagents": subagents,
|
|
||||||
"subagent_totals": {
|
|
||||||
"count": len(subagents),
|
|
||||||
"tokens": sub_tokens,
|
|
||||||
"total_tokens": token_total(sub_tokens),
|
|
||||||
},
|
|
||||||
"workflows": workflows,
|
|
||||||
"workflow_totals": {
|
|
||||||
"runs": len(workflows),
|
|
||||||
"agents": wf_agents,
|
|
||||||
"tokens": wf_tokens,
|
|
||||||
"total_tokens": token_total(wf_tokens),
|
|
||||||
},
|
|
||||||
"totals": {
|
|
||||||
"tokens": total_tokens,
|
|
||||||
"total_tokens": token_total(total_tokens),
|
|
||||||
"cache_hit_ratio": cache_hit_ratio(total_tokens),
|
|
||||||
},
|
|
||||||
"warnings": warnings,
|
|
||||||
}
|
|
||||||
json.dump(report, sys.stdout, indent=2)
|
|
||||||
sys.stdout.write("\n")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,125 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Executable spec for postmortem.subagent_terminal_status.
|
|
||||||
|
|
||||||
Run directly: `python3 postmortem/tests/test_terminal_status.py`.
|
|
||||||
Exits 0 on pass, 1 on failure. No pytest dependency — the plugin has
|
|
||||||
no test harness, and a status-classifier is small enough to pin with
|
|
||||||
plain asserts.
|
|
||||||
|
|
||||||
Pins issue #3: the terminal-status text-scan over-reports BLOCKED for
|
|
||||||
shipped implement runs, because `\\bBLOCKED\\b` fires on the end-report's
|
|
||||||
own template lines (`BLOCKED file: BLOCKED.md`, `Blocked detail:`)
|
|
||||||
and on prose narrating a surmounted blocker. The authoritative signal is
|
|
||||||
the structured `Status:` line of the end-report, not any keyword
|
|
||||||
occurrence in the body.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
|
|
||||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
SCRIPT = os.path.join(HERE, "..", "scripts", "postmortem.py")
|
|
||||||
|
|
||||||
spec = importlib.util.spec_from_file_location("postmortem", SCRIPT)
|
|
||||||
pm = importlib.util.module_from_spec(spec)
|
|
||||||
spec.loader.exec_module(pm)
|
|
||||||
|
|
||||||
|
|
||||||
def write_subagent_log(final_text):
|
|
||||||
"""Write a one-assistant-turn subagent jsonl whose last text block is
|
|
||||||
`final_text`, return its path. Caller cleans up."""
|
|
||||||
fd, path = tempfile.mkstemp(suffix=".jsonl")
|
|
||||||
rec = {"type": "assistant",
|
|
||||||
"message": {"role": "assistant",
|
|
||||||
"content": [{"type": "text", "text": final_text}]}}
|
|
||||||
with os.fdopen(fd, "w") as fh:
|
|
||||||
fh.write(json.dumps(rec) + "\n")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def status_of(final_text):
|
|
||||||
path = write_subagent_log(final_text)
|
|
||||||
try:
|
|
||||||
return pm.subagent_terminal_status(path)
|
|
||||||
finally:
|
|
||||||
os.unlink(path)
|
|
||||||
|
|
||||||
|
|
||||||
# A realistic plain-text iteration end-report that SHIPPED (Status: DONE)
|
|
||||||
# but whose body carries the template's BLOCKED-bearing lines. This is the
|
|
||||||
# exact shape that issue #3 reports as misclassified.
|
|
||||||
SHIPPED_DONE_REPORT = """\
|
|
||||||
Status: DONE
|
|
||||||
Iter: 0126-leg3
|
|
||||||
Started from: a1b2c3d
|
|
||||||
Tasks completed: 3 of 3
|
|
||||||
- poly RawBuf elem-var mono fix
|
|
||||||
- series kernel module
|
|
||||||
- owned-param tail-recursion leak
|
|
||||||
Working tree: dirty (7 files changed)
|
|
||||||
BLOCKED file: BLOCKED.md (uncommitted; only on PARTIAL/BLOCKED)
|
|
||||||
Stats: .claude/stats/0126.json (uncommitted)
|
|
||||||
Files touched: 7
|
|
||||||
Tests: 12 green, 0 red
|
|
||||||
E2E coverage: none (mini mode)
|
|
||||||
Blocked detail: (only if BLOCKED or PARTIAL — also written to BLOCKED.md)
|
|
||||||
|
|
||||||
Note: task 2 hit a blocker early (missing fixture) which was resolved
|
|
||||||
in-phase; no BLOCKED.md was written.
|
|
||||||
"""
|
|
||||||
|
|
||||||
GENUINE_BLOCKED_REPORT = """\
|
|
||||||
Status: BLOCKED
|
|
||||||
Iter: 0127
|
|
||||||
Started from: d4e5f6a
|
|
||||||
Tasks completed: 1 of 3
|
|
||||||
Working tree: dirty (2 files changed)
|
|
||||||
BLOCKED file: BLOCKED.md (uncommitted)
|
|
||||||
Blocked detail: Task: 2
|
|
||||||
Reason: review-loop-exhausted
|
|
||||||
Worker says: spec-compliance never reached approved
|
|
||||||
"""
|
|
||||||
|
|
||||||
PARTIAL_REPORT = """\
|
|
||||||
Status: PARTIAL
|
|
||||||
Iter: 0128
|
|
||||||
Tasks completed: 2 of 3
|
|
||||||
BLOCKED file: BLOCKED.md (uncommitted)
|
|
||||||
"""
|
|
||||||
|
|
||||||
CASES = [
|
|
||||||
# (name, final_text, expected)
|
|
||||||
("shipped DONE with BLOCKED template lines (issue #3)",
|
|
||||||
SHIPPED_DONE_REPORT, "DONE"),
|
|
||||||
("genuine BLOCKED end-report",
|
|
||||||
GENUINE_BLOCKED_REPORT, "BLOCKED"),
|
|
||||||
("PARTIAL end-report mentioning BLOCKED.md",
|
|
||||||
PARTIAL_REPORT, "PARTIAL"),
|
|
||||||
("narrating agent whose final line is a bare status token",
|
|
||||||
"Work complete, tests green.\n\nDONE", "DONE"),
|
|
||||||
("prose mentioning a resolved blocker, no status line",
|
|
||||||
"I was briefly BLOCKED by a missing import but fixed it. All good.",
|
|
||||||
"unknown"),
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
failures = []
|
|
||||||
for name, text, expected in CASES:
|
|
||||||
got = status_of(text)
|
|
||||||
ok = got == expected
|
|
||||||
print(f"[{'PASS' if ok else 'FAIL'}] {name}: expected {expected!r}, got {got!r}")
|
|
||||||
if not ok:
|
|
||||||
failures.append(name)
|
|
||||||
if failures:
|
|
||||||
print(f"\n{len(failures)} failure(s).")
|
|
||||||
return 1
|
|
||||||
print(f"\nAll {len(CASES)} cases passed.")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,124 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Executable spec for postmortem.analyze_workflows.
|
|
||||||
|
|
||||||
Run directly: `python3 postmortem/tests/test_workflow_aggregation.py`.
|
|
||||||
Exits 0 on pass, 1 on failure. No pytest dependency — same plain-assert
|
|
||||||
convention as test_terminal_status.py.
|
|
||||||
|
|
||||||
Pins issue #28: sessions that ran Workflow-substrate pipelines
|
|
||||||
(workflows/wf_*.json + subagents/workflows/<runId>/agent-*.jsonl) must
|
|
||||||
not read as zero subagent spend. The raw transcripts are authoritative
|
|
||||||
(dedup by requestId); the summary supplies stage labels; a transcript
|
|
||||||
missing from workflowProgress groups under '?' instead of being folded
|
|
||||||
into a named stage or dropped.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
|
|
||||||
from postmortem import analyze_workflows # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def _write(path, obj_lines):
|
|
||||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
|
||||||
with open(path, "w") as fh:
|
|
||||||
for obj in obj_lines:
|
|
||||||
fh.write(json.dumps(obj) + "\n")
|
|
||||||
|
|
||||||
|
|
||||||
def _assistant_line(request_id, in_tok, out_tok, model="claude-sonnet-5"):
|
|
||||||
return {
|
|
||||||
"type": "assistant",
|
|
||||||
"requestId": request_id,
|
|
||||||
"message": {
|
|
||||||
"model": model,
|
|
||||||
"usage": {"input_tokens": in_tok, "output_tokens": out_tok,
|
|
||||||
"cache_creation_input_tokens": 0,
|
|
||||||
"cache_read_input_tokens": 0},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
with tempfile.TemporaryDirectory() as session_dir:
|
|
||||||
run_id = "wf_test-1"
|
|
||||||
wf = {
|
|
||||||
"runId": run_id,
|
|
||||||
"workflowName": "spec-fixture",
|
|
||||||
"status": "completed",
|
|
||||||
"totalTokens": 999, # live counter: deliberately wrong
|
|
||||||
"defaultModel": "claude-sonnet-5",
|
|
||||||
"workflowProgress": [
|
|
||||||
{"type": "workflow_phase", "index": 1, "title": "P"},
|
|
||||||
{"type": "workflow_agent", "agentId": "a1",
|
|
||||||
"label": "impl:1", "model": "claude-sonnet-5",
|
|
||||||
"state": "done", "tokens": 1},
|
|
||||||
{"type": "workflow_agent", "agentId": "a2",
|
|
||||||
"label": "impl:2", "model": "claude-sonnet-5",
|
|
||||||
"state": "done", "tokens": 1},
|
|
||||||
{"type": "workflow_agent", "agentId": "a4",
|
|
||||||
"label": "qual:1", "model": "claude-opus-4-8",
|
|
||||||
"state": "error", "tokens": 0}, # no transcript on disk
|
|
||||||
],
|
|
||||||
}
|
|
||||||
os.makedirs(os.path.join(session_dir, "workflows"))
|
|
||||||
with open(os.path.join(session_dir, "workflows",
|
|
||||||
run_id + ".json"), "w") as fh:
|
|
||||||
json.dump(wf, fh)
|
|
||||||
|
|
||||||
raw = os.path.join(session_dir, "subagents", "workflows", run_id)
|
|
||||||
# a1: streamed repeat of the SAME requestId — must dedup to
|
|
||||||
# max-per-field (input 10, output 7), not sum to 20/12.
|
|
||||||
_write(os.path.join(raw, "agent-a1.jsonl"),
|
|
||||||
[_assistant_line("req-1", 10, 5),
|
|
||||||
_assistant_line("req-1", 10, 7)])
|
|
||||||
# a2: two distinct requests — must sum.
|
|
||||||
_write(os.path.join(raw, "agent-a2.jsonl"),
|
|
||||||
[_assistant_line("req-2", 3, 2),
|
|
||||||
_assistant_line("req-3", 4, 1)])
|
|
||||||
# a3: transcript with NO workflowProgress entry — counts under '?'.
|
|
||||||
_write(os.path.join(raw, "agent-a3.jsonl"),
|
|
||||||
[_assistant_line("req-4", 100, 50)])
|
|
||||||
|
|
||||||
runs = analyze_workflows(session_dir)
|
|
||||||
|
|
||||||
assert len(runs) == 1, runs
|
|
||||||
run = runs[0]
|
|
||||||
assert run["run_id"] == run_id
|
|
||||||
assert run["name"] == "spec-fixture"
|
|
||||||
assert run["reported_total_tokens"] == 999
|
|
||||||
|
|
||||||
# Three transcripts on disk, one untracked.
|
|
||||||
assert run["agents"] == 3, run
|
|
||||||
assert run["agents_untracked"] == 1, run
|
|
||||||
|
|
||||||
stages = run["stages"]
|
|
||||||
assert set(stages) == {"impl", "?"}, stages
|
|
||||||
|
|
||||||
impl = stages["impl"]
|
|
||||||
assert impl["count"] == 2, impl
|
|
||||||
# a1 deduped (10 in / 7 out) + a2 summed (7 in / 3 out)
|
|
||||||
assert impl["tokens"]["input"] == 17, impl
|
|
||||||
assert impl["tokens"]["output"] == 10, impl
|
|
||||||
assert impl["total_tokens"] == 27, impl
|
|
||||||
assert impl["models"] == {"claude-sonnet-5": 3}, impl
|
|
||||||
|
|
||||||
unk = stages["?"]
|
|
||||||
assert unk["count"] == 1, unk
|
|
||||||
assert unk["tokens"]["input"] == 100 and unk["tokens"]["output"] == 50, unk
|
|
||||||
|
|
||||||
# Run totals = authoritative raw aggregation, not the live counter.
|
|
||||||
assert run["tokens"]["input"] == 117, run
|
|
||||||
assert run["tokens"]["output"] == 60, run
|
|
||||||
assert run["total_tokens"] == 177, run
|
|
||||||
|
|
||||||
print("ok: analyze_workflows aggregates raw transcripts per stage, "
|
|
||||||
"dedups by requestId, and isolates untracked agents under '?'")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
Reference in New Issue
Block a user