diff --git a/.gitignore b/.gitignore index 4f86d8e..899a4a3 100644 --- a/.gitignore +++ b/.gitignore @@ -5,5 +5,7 @@ __pycache__/ *.pyc -# postmortem skill output — reports are per-run artifacts, not tracked +# local report archive of the retired postmortem skill (issue #27) — +# reports were per-run artifacts, never tracked; the dir may still +# exist locally docs/postmortems/ diff --git a/postmortem/SKILL.md b/postmortem/SKILL.md deleted file mode 100644 index 94a627a..0000000 --- a/postmortem/SKILL.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -name: postmortem -description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate. ---- - -# postmortem — grade the run - -> **Violating the letter of these rules is violating the spirit.** - -## Overview - -Every session leaves a detailed flight recorder behind — the -JSONL transcript under `~/.claude/projects//.jsonl`, -one sidechain log per dispatched subagent under -`/subagents/`, and for every Workflow run a per-agent -summary in `/workflows/wf_*.json` with raw per-agent -transcripts under `/subagents/workflows//`. Nobody -reads it. So the same expensive -habits repeat: a subagent fleet that bounces `BLOCKED`, a tool -loop with a 30% error rate, a cache-cold run that re-sends the full -input on every turn. This skill turns that recorder into a graded -retrospective: **how well did the toolchain work, how effective -were the agents, and how many tokens did it spend.** - -It is descriptive, not corrective. It produces a report and, where -warranted, recommends filing tracker issues — it does not change -code, baselines, or the pipeline. - -## When to Use / Skipping - -Invoke: - -- After a substantial interactive session, when you want to know - where the time and tokens went. -- At the close of a `/boss` autonomous run, to grade the - orchestrator's dispatch choices. -- When a run *felt* token-heavy or thrashy and you want the numbers. - -Not a pipeline phase. Never a hard-gate. It reads logs; it never -blocks a cycle. - -## The Iron Law - -``` -NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG -REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE -NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT -A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT -DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND -``` - -## What the data does and does not contain - -Read before interpreting, so the report never overclaims: - -- **Present:** per-request `usage` (input / output / cache-creation / - cache-read / server-tool counts), `model`, `requestId`, - timestamps, every `tool_use` with its input, every `tool_result` - with the authoritative `is_error` flag, skill invocations, - slash commands, and a full separate transcript + `agentType` / - `description` for each subagent. Workflow runs additionally carry a - per-agent summary (stage label, model, a live token counter) in - `workflows/wf_*.json`; the counter under-tracks vs. the deduped raw - transcripts (~11% in the corpus behind issue #28), so the script - treats the raw transcripts as authoritative and reports the counter - only as `reported_total_tokens` for cross-checking. -- **Absent:** `costUSD` is always `null`; `durationMs` is always - `null`. There is no metered cost in the log, so this skill does - not compute one — it reports **raw token counts** (kept split by - class: input / output / cache-creation / cache-read, plus a - `total_tokens` sum). **Wall-clock is derived** from timestamp - deltas and the report says so. - -The streamed transcript repeats the same `requestId` across -several assistant lines with an identical `usage` object. The -helper script dedups per `requestId`; never sum the raw lines. - -## The Process - -### Step 1 — Run the aggregator - -``` -python3 /scripts/postmortem.py [--session ] [--cwd ] -``` - -Defaults to the newest transcript in the current project's log -dir — i.e. the session you are in (it will be flagged `active`, -meaning totals are partial). To grade a finished run, pass its -`--session `. - -The default log dir is derived from cwd (the path with `/` replaced -by `-`), so a session run inside a git worktree resolves to a -*different* slug than the primary clone, and after exiting a worktree -the default again follows the current cwd. To grade a worktree run -from the primary clone, pass `--project-dir -~/.claude/projects/` (or `--file ` to bypass slug -derivation entirely). - -The script emits one JSON object on stdout. That JSON — not the -raw log — is your evidence base. - -### Step 2 — Read the JSON and grade three axes - -Grade each axis on the cited numbers. A grade with no number -behind it is an opinion; drop it. - -1. **Token spend & efficiency** — `totals.total_tokens`, the - main-vs-subagent split, the token breakdown (output share is the - write-heavy signal), `cache_hit_ratio` (the dominant efficiency - lever: below ~0.7 on a long session means cache-cold turns - re-sending the full input), `server_tools` (web_search / - web_fetch counts). -2. **Toolchain health** — `tools.error_ratio` and `interrupted` - (the authoritative `is_error` signal), the tool mix (a Read / - Edit / Bash imbalance hints at thrashing or re-reading), - `slash_commands` such as `/compact` (context pressure). -3. **Agent effectiveness** — per subagent: `terminal_status` - (`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` / - `NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A - fleet that burned real tokens to return `BLOCKED` is the headline - finding. Compare each agent's token spend to what it delivered. - For Workflow runs, grade per stage instead: `workflow_totals` and - each run's `stages` table (label-prefix grouping, e.g. all - `impl:N` under `impl`) show where the run's spend concentrated; - `agents_untracked > 0` flags retried/dropped attempts. Schema-bound - workflow stages carry no `terminal_status` — the workflow script's - own aggregation already judged them. - -### Step 3 — Write the report - -Write a Markdown report to the configured destination (see -**Output destination**). Structure it as **Output format** below. -Lead with the scorecard, support every grade with a number from -the JSON, and keep the prose tight — this is a retrospective, not -a transcript replay. - -### Step 4 — Surface, don't fix - -Translate the sharpest findings into recommendations. Where a -finding is a concrete, trackable defect (a subagent that reliably -bounces `BLOCKED`, a tool loop with a runaway error rate), -recommend filing it via the `issue` skill — name it, but do not -file it yourself unless the user asks. This skill ends at the -report. - -## Output destination - -Resolve in this order: - -1. `--out ` if the invoker passed one. -2. The project's post-mortem directory, if it has one (its CLAUDE.md - project facts). -3. Default: `docs/postmortems/-.md` - under the project root, creating the directory if needed. - -Also print a ≤200-word summary to the chat so the headline grade -and token spend are visible without opening the file. - -## Output format - -``` -# Post-mortem — () - -**Scorecard** -| Axis | Grade | Headline metric | -|------|-------|-----------------| -| Token spend & efficiency | A–F | total, cache hit | -| Toolchain health | A–F | , interrupts | -| Agent effectiveness | A–F | / agents DONE, on subagents | - -**Token spend** -
- -**Toolchain** - - -**Agents** - — one-clause -verdict. Omit if no subagents were dispatched.> - -**Recommendations** -<≤5 bullets. Each ties to a cited metric. Mark any that warrant -an `issue`.> - -_Token counts are raw usage. Wall-clock is derived from timestamp -deltas, not metered._ -``` - -If the session is flagged `active`, add one line at the top: -"⚠️ Session still active — figures are partial." - -## Common Rationalisations - -| Excuse | Reality | -|--------|---------| -| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. | -| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. | -| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. | -| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. | -| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. | -| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. | - -## Red Flags — STOP - -- Quoting any token number that did not come out of the script's - JSON. -- Converting token counts into a cost or dollar figure — report - raw counts only. -- Reading any file under `~/.claude` other than the session - transcript and its subagent logs — and never any credential or - token file. -- Editing code, baselines, or pipeline config "while you're in - there". Report only. -- Grading an axis with adjectives instead of the cited metric. - -## Cross-references - -- **Helper script:** `scripts/postmortem.py` — the deterministic - aggregator. All numbers originate here. -- **Hand-off target:** the `issue` skill, when a finding is a - trackable defect worth filing. -- **Report destination (optional):** the project's post-mortem - directory, if its CLAUDE.md project facts name one. -- **Data-source note:** this skill reads only the current - session's own transcript and subagent logs. It does not crawl - other projects or other sessions (single-session by design). diff --git a/postmortem/scripts/postmortem.py b/postmortem/scripts/postmortem.py deleted file mode 100755 index 0ec853a..0000000 --- a/postmortem/scripts/postmortem.py +++ /dev/null @@ -1,501 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic aggregator for a single Claude Code session. - -Reads the main transcript JSONL for one session plus its subagent -sidechain logs and its Workflow-substrate run logs (workflows/wf_*.json -+ subagents/workflows//agent-*.jsonl), and emits a compact JSON -summary on stdout that the post-mortem SKILL interprets into a written -report. - -Why a script and not skill prose: transcripts run to >1 MB and a -session can spawn dozens of subagent logs. Token accounting has two -traps that only deterministic code gets right every time: - - 1. Streamed assistant lines repeat the SAME requestId with the SAME - usage object. Counting per-line double-counts tokens. We dedup - per requestId (max per field within a request, then sum across - requests). - 2. The transcript carries NO durationMs (always null), so wall-clock - is derived from timestamp deltas. The numbers reported here are - raw token counts, not a dollar estimate — different token classes - (input / output / cache) are reported separately rather than - collapsed into a single priced figure. - -Stdlib only. No network. Read-only against ~/.claude. -""" - -import argparse -import glob -import json -import os -import re -import sys -from datetime import datetime, timezone - -_TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first -# Authoritative signal: the implement end-report leads with a structured -# "Status: DONE" line (optionally wrapped in markdown bold or a list -# marker). Anchoring here is what stops the scan from flipping a shipped -# DONE run to BLOCKED on the strength of the report's own template lines -# (`BLOCKED file: BLOCKED.md`, `Blocked detail:`) or prose narrating a -# surmounted blocker. See issue #3. -STATUS_LINE_RE = re.compile( - r"^[\s>*#\-]*\**\s*status\s*\**\s*[:\-]\s*\**\s*(" + _TOK_ALT + r")\b", - re.IGNORECASE | re.MULTILINE) -# Fallback for agents that emit a bare status token as their final line -# (no "Status:" prefix). Requires the token to BE the line — excludes -# substrings like "BLOCKED.md" or "BLOCKED file:" buried in prose. -STATUS_STANDALONE_RE = re.compile( - r"^[\s>*#\-]*\**\s*(" + _TOK_ALT + r")\**\s*$", - re.IGNORECASE | re.MULTILINE) - - -def parse_ts(s): - if not s: - return None - try: - return datetime.fromisoformat(s.replace("Z", "+00:00")) - except ValueError: - return None - - -def empty_tokens(): - return {"input": 0, "output": 0, "cache_creation": 0, "cache_read": 0, - "cache_creation_5m": 0, "cache_creation_1h": 0} - - -def token_total(tokens): - """Raw count of every token the session moved. cache_creation_5m/1h are - a sub-split of cache_creation, so they are excluded to avoid double - counting.""" - return (tokens["input"] + tokens["output"] - + tokens["cache_creation"] + tokens["cache_read"]) - - -def add_tokens(acc, usage): - """Add one request's deduped usage into an accumulator.""" - acc["input"] += usage.get("input_tokens", 0) or 0 - acc["output"] += usage.get("output_tokens", 0) or 0 - cc = usage.get("cache_creation_input_tokens", 0) or 0 - acc["cache_creation"] += cc - acc["cache_read"] += usage.get("cache_read_input_tokens", 0) or 0 - detail = usage.get("cache_creation") or {} - f5 = detail.get("ephemeral_5m_input_tokens") - f1 = detail.get("ephemeral_1h_input_tokens") - if f5 is None and f1 is None: - acc["cache_creation_5m"] += cc # no split available -> assume 5m - else: - acc["cache_creation_5m"] += f5 or 0 - acc["cache_creation_1h"] += f1 or 0 - - -def iter_lines(path): - with open(path, "r", encoding="utf-8", errors="replace") as fh: - for line in fh: - line = line.strip() - if not line: - continue - try: - yield json.loads(line) - except json.JSONDecodeError: - continue - - -def aggregate_transcript(path): - """Return token totals, model mix, and per-request count for one JSONL - transcript (main or subagent). Dedups usage by requestId.""" - # requestId -> {field: max seen}, plus model per request. - req_usage = {} - req_model = {} - for rec in iter_lines(path): - if rec.get("type") != "assistant": - continue - msg = rec.get("message") or {} - usage = msg.get("usage") - if not usage: - continue - key = rec.get("requestId") or rec.get("uuid") - model = msg.get("model") - if model: - req_model[key] = model - cur = req_usage.setdefault(key, {}) - # Max-per-field guards against both identical-repeat and - # cumulative-within-request streaming shapes. - for f in ("input_tokens", "output_tokens", - "cache_creation_input_tokens", "cache_read_input_tokens"): - v = usage.get(f, 0) or 0 - if v > cur.get(f, 0): - cur[f] = v - det = usage.get("cache_creation") or {} - cd = cur.setdefault("cache_creation", {}) - for f in ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"): - v = det.get(f, 0) or 0 - if v > cd.get(f, 0): - cd[f] = v - st = usage.get("server_tool_use") or {} - sc = cur.setdefault("_server", {}) - for f in ("web_search_requests", "web_fetch_requests"): - v = st.get(f, 0) or 0 - if v > sc.get(f, 0): - sc[f] = v - - tokens = empty_tokens() - server = {"web_search": 0, "web_fetch": 0} - models = {} - for key, usage in req_usage.items(): - add_tokens(tokens, usage) - model = req_model.get(key) - models[model] = models.get(model, 0) + 1 - sv = usage.get("_server", {}) - server["web_search"] += sv.get("web_search_requests", 0) - server["web_fetch"] += sv.get("web_fetch_requests", 0) - - return { - "requests": len(req_usage), - "tokens": tokens, - "total_tokens": token_total(tokens), - "models": models, - "server_tools": server, - } - - -def cache_hit_ratio(tokens): - denom = tokens["input"] + tokens["cache_read"] + tokens["cache_creation"] - return round(tokens["cache_read"] / denom, 4) if denom else None - - -def analyze_main(path): - """Tool mix, error rate, skills, slash commands, prompts, wall-clock.""" - tool_calls = {} - error_total = 0 - interrupted = 0 - skills = [] - slash = [] - tasks = [] - user_prompts = 0 - first_ts = last_ts = None - cwd = version = None - - for rec in iter_lines(path): - ts = parse_ts(rec.get("timestamp")) - if ts: - if first_ts is None or ts < first_ts: - first_ts = ts - if last_ts is None or ts > last_ts: - last_ts = ts - cwd = rec.get("cwd") or cwd - version = rec.get("version") or version - - typ = rec.get("type") - msg = rec.get("message") or {} - content = msg.get("content") - - if typ == "assistant" and isinstance(content, list): - for block in content: - if block.get("type") != "tool_use": - continue - name = block.get("name", "?") - tool_calls[name] = tool_calls.get(name, 0) + 1 - if name == "Skill": - skills.append({"skill": (block.get("input") or {}).get("skill"), - "ts": rec.get("timestamp")}) - elif name in ("Task", "Agent"): - inp = block.get("input") or {} - tasks.append({"subagent_type": inp.get("subagent_type"), - "description": inp.get("description"), - "ts": rec.get("timestamp")}) - - if typ == "user": - # tool_result blocks carry the authoritative is_error flag - if isinstance(content, list): - for block in content: - if block.get("type") == "tool_result": - if block.get("is_error"): - error_total += 1 - elif block.get("type") == "text": - for m in re.findall(r"(.*?)", - block.get("text", "")): - slash.append({"cmd": m, "ts": rec.get("timestamp")}) - elif isinstance(content, str): - for m in re.findall(r"(.*?)", content): - slash.append({"cmd": m, "ts": rec.get("timestamp")}) - # genuine user turns (not tool results, not slash plumbing) - is_tool_result = (isinstance(content, list) - and any(b.get("type") == "tool_result" for b in content)) - if not is_tool_result and rec.get("toolUseResult") is None: - user_prompts += 1 - tur = rec.get("toolUseResult") - if isinstance(tur, dict) and tur.get("interrupted"): - interrupted += 1 - - return { - "tool_calls": tool_calls, - "total_tool_calls": sum(tool_calls.values()), - "tool_error_total": error_total, - "interrupted": interrupted, - "skills_invoked": skills, - "slash_commands": slash, - "tasks_dispatched": tasks, - "user_prompts": user_prompts, - "wall_clock_sec": round((last_ts - first_ts).total_seconds(), 1) - if first_ts and last_ts else None, - "started": first_ts.isoformat() if first_ts else None, - "ended": last_ts.isoformat() if last_ts else None, - "cwd": cwd, - "version": version, - } - - -def subagent_terminal_status(path): - last_text = "" - for rec in iter_lines(path): - if rec.get("type") != "assistant": - continue - content = (rec.get("message") or {}).get("content") - if isinstance(content, list): - for block in content: - if block.get("type") == "text" and block.get("text", "").strip(): - last_text = block["text"] - # Primary: the structured "Status:" line. Take the LAST one so a - # restated final status wins over any earlier mention. - structured = STATUS_LINE_RE.findall(last_text) - if structured: - return structured[-1].upper() - # Fallback: a status token standing alone on its own line. - standalone = STATUS_STANDALONE_RE.findall(last_text) - if standalone: - return standalone[-1].upper() - return "unknown" - - -def analyze_workflows(session_dir): - """Aggregate Workflow-substrate runs: workflows/wf_*.json (per-agent - summary: stage label, model, live token counter) joined with the raw - per-agent transcripts under subagents/workflows//agent-*.jsonl. - - The raw transcripts are authoritative for token counts (deduped by - requestId, same as every other transcript here); the wf_*.json - 'tokens' counters under-track vs. the deduped raw usage (~11% in the - 206-run corpus that motivated this path, issue #28) and are reported - only as a cross-check. Stage = the label up to the first ':' - ('impl:3' -> 'impl'), so loop stages aggregate across tasks/rounds.""" - runs = [] - for wf_path in sorted(glob.glob(os.path.join(session_dir, "workflows", - "wf_*.json"))): - try: - with open(wf_path) as fh: - data = json.load(fh) - except (json.JSONDecodeError, OSError): - continue - run_id = data.get("runId") or os.path.basename(wf_path)[:-len(".json")] - raw_dir = os.path.join(session_dir, "subagents", "workflows", run_id) - meta_by_agent = {} - for entry in data.get("workflowProgress") or []: - if entry.get("type") != "workflow_agent" or not entry.get("agentId"): - continue - meta_by_agent[entry["agentId"]] = entry - - run_tokens = empty_tokens() - stages = {} - agent_count = 0 - untracked = 0 - for jl in sorted(glob.glob(os.path.join(raw_dir, "agent-*.jsonl"))): - aid = os.path.basename(jl)[len("agent-"):-len(".jsonl")] - agg = aggregate_transcript(jl) - entry = meta_by_agent.get(aid) - if entry is None: - # Transcript with no progress entry (retried/dropped - # attempt): keep it counted, grouped under '?', never - # silently folded into a named stage. - untracked += 1 - label = (entry or {}).get("label") or "?" - stage = label.split(":", 1)[0] - st = stages.setdefault(stage, {"count": 0, "tokens": empty_tokens(), - "models": {}}) - st["count"] += 1 - agent_count += 1 - for k in run_tokens: - run_tokens[k] += agg["tokens"][k] - st["tokens"][k] += agg["tokens"][k] - fallback_model = (entry or {}).get("model") - for m, n in agg["models"].items(): - key = m or fallback_model - st["models"][key] = st["models"].get(key, 0) + n - - for st in stages.values(): - st["total_tokens"] = token_total(st["tokens"]) - - runs.append({ - "run_id": run_id, - "name": data.get("workflowName"), - "status": data.get("status"), - "agents": agent_count, - "agents_untracked": untracked, - "reported_total_tokens": data.get("totalTokens"), - "tokens": run_tokens, - "total_tokens": token_total(run_tokens), - "stages": stages, - }) - return runs - - -def analyze_subagents(session_dir): - out = [] - sub_dir = os.path.join(session_dir, "subagents") - for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))): - meta = {} - mp = jl.replace(".jsonl", ".meta.json") - if os.path.exists(mp): - try: - with open(mp) as fh: - meta = json.load(fh) - except (json.JSONDecodeError, OSError): - pass - agg = aggregate_transcript(jl) - out.append({ - "agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")], - "agent_type": meta.get("agentType"), - "description": meta.get("description"), - "dispatched_by_tool": meta.get("toolUseId"), - "models": agg["models"], - "requests": agg["requests"], - "tokens": agg["tokens"], - "total_tokens": agg["total_tokens"], - "terminal_status": subagent_terminal_status(jl), - }) - return out - - -def resolve_session(args): - """Return (main_jsonl_path, session_dir_or_None).""" - if args.file: - path = os.path.abspath(args.file) - return path, path[:-len(".jsonl")] if path.endswith(".jsonl") else None - - cwd = args.cwd or os.getcwd() - proj_dir = args.project_dir or os.path.join( - os.path.expanduser("~/.claude/projects"), cwd.replace("/", "-")) - if not os.path.isdir(proj_dir): - raise SystemExit(f"no project log dir for cwd {cwd}: {proj_dir} not found") - - if args.session: - path = os.path.join(proj_dir, args.session + ".jsonl") - if not os.path.exists(path): - raise SystemExit(f"session {args.session} not found in {proj_dir}") - return path, os.path.join(proj_dir, args.session) - - # default: newest .jsonl in the project dir = current/last session - candidates = sorted(glob.glob(os.path.join(proj_dir, "*.jsonl")), - key=os.path.getmtime, reverse=True) - if not candidates: - raise SystemExit(f"no session transcripts in {proj_dir}") - path = candidates[0] - return path, path[:-len(".jsonl")] - - -def main(): - ap = argparse.ArgumentParser(description="Single-session post-mortem aggregator.") - ap.add_argument("--session", help="explicit session id (UUID)") - ap.add_argument("--file", help="explicit path to a main transcript .jsonl") - ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)") - ap.add_argument("--project-dir", help="explicit ~/.claude/projects/ dir") - args = ap.parse_args() - - main_path, session_dir = resolve_session(args) - session_id = os.path.basename(main_path)[:-len(".jsonl")] - - main_agg = aggregate_transcript(main_path) - main_meta = analyze_main(main_path) - - subagents = [] - workflows = [] - if session_dir and os.path.isdir(session_dir): - subagents = analyze_subagents(session_dir) - workflows = analyze_workflows(session_dir) - - sub_tokens = empty_tokens() - for s in subagents: - for k in sub_tokens: - sub_tokens[k] += s["tokens"][k] - - wf_tokens = empty_tokens() - wf_agents = 0 - for w in workflows: - wf_agents += w["agents"] - for k in wf_tokens: - wf_tokens[k] += w["tokens"][k] - - total_tokens = empty_tokens() - for k in total_tokens: - total_tokens[k] = (main_agg["tokens"][k] + sub_tokens[k] - + wf_tokens[k]) - - warnings = [] - is_active = False - if main_meta["ended"]: - ended = parse_ts(main_meta["ended"]) - if ended and (datetime.now(timezone.utc) - ended).total_seconds() < 120: - is_active = True - warnings.append("Session appears ACTIVE (last event <2 min ago); " - "totals are partial and will grow.") - warnings.append("Token counts are raw usage; wall-clock is derived from " - "timestamp deltas (the transcript carries no durationMs).") - - report = { - "session": { - "id": session_id, - "transcript": main_path, - "cwd": main_meta["cwd"], - "version": main_meta["version"], - "started": main_meta["started"], - "ended": main_meta["ended"], - "wall_clock_sec": main_meta["wall_clock_sec"], - "active": is_active, - }, - "main": { - "requests": main_agg["requests"], - "models": main_agg["models"], - "tokens": main_agg["tokens"], - "total_tokens": main_agg["total_tokens"], - "cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]), - "server_tools": main_agg["server_tools"], - "user_prompts": main_meta["user_prompts"], - }, - "tools": { - "calls": main_meta["tool_calls"], - "total": main_meta["total_tool_calls"], - "error_total": main_meta["tool_error_total"], - "error_ratio": round(main_meta["tool_error_total"] - / main_meta["total_tool_calls"], 4) - if main_meta["total_tool_calls"] else None, - "interrupted": main_meta["interrupted"], - }, - "skills_invoked": main_meta["skills_invoked"], - "slash_commands": main_meta["slash_commands"], - "tasks_dispatched": main_meta["tasks_dispatched"], - "subagents": subagents, - "subagent_totals": { - "count": len(subagents), - "tokens": sub_tokens, - "total_tokens": token_total(sub_tokens), - }, - "workflows": workflows, - "workflow_totals": { - "runs": len(workflows), - "agents": wf_agents, - "tokens": wf_tokens, - "total_tokens": token_total(wf_tokens), - }, - "totals": { - "tokens": total_tokens, - "total_tokens": token_total(total_tokens), - "cache_hit_ratio": cache_hit_ratio(total_tokens), - }, - "warnings": warnings, - } - json.dump(report, sys.stdout, indent=2) - sys.stdout.write("\n") - - -if __name__ == "__main__": - main() diff --git a/postmortem/tests/test_terminal_status.py b/postmortem/tests/test_terminal_status.py deleted file mode 100644 index f9a6209..0000000 --- a/postmortem/tests/test_terminal_status.py +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env python3 -"""Executable spec for postmortem.subagent_terminal_status. - -Run directly: `python3 postmortem/tests/test_terminal_status.py`. -Exits 0 on pass, 1 on failure. No pytest dependency — the plugin has -no test harness, and a status-classifier is small enough to pin with -plain asserts. - -Pins issue #3: the terminal-status text-scan over-reports BLOCKED for -shipped implement runs, because `\\bBLOCKED\\b` fires on the end-report's -own template lines (`BLOCKED file: BLOCKED.md`, `Blocked detail:`) -and on prose narrating a surmounted blocker. The authoritative signal is -the structured `Status:` line of the end-report, not any keyword -occurrence in the body. -""" - -import importlib.util -import json -import os -import sys -import tempfile - -HERE = os.path.dirname(os.path.abspath(__file__)) -SCRIPT = os.path.join(HERE, "..", "scripts", "postmortem.py") - -spec = importlib.util.spec_from_file_location("postmortem", SCRIPT) -pm = importlib.util.module_from_spec(spec) -spec.loader.exec_module(pm) - - -def write_subagent_log(final_text): - """Write a one-assistant-turn subagent jsonl whose last text block is - `final_text`, return its path. Caller cleans up.""" - fd, path = tempfile.mkstemp(suffix=".jsonl") - rec = {"type": "assistant", - "message": {"role": "assistant", - "content": [{"type": "text", "text": final_text}]}} - with os.fdopen(fd, "w") as fh: - fh.write(json.dumps(rec) + "\n") - return path - - -def status_of(final_text): - path = write_subagent_log(final_text) - try: - return pm.subagent_terminal_status(path) - finally: - os.unlink(path) - - -# A realistic plain-text iteration end-report that SHIPPED (Status: DONE) -# but whose body carries the template's BLOCKED-bearing lines. This is the -# exact shape that issue #3 reports as misclassified. -SHIPPED_DONE_REPORT = """\ -Status: DONE -Iter: 0126-leg3 -Started from: a1b2c3d -Tasks completed: 3 of 3 - - poly RawBuf elem-var mono fix - - series kernel module - - owned-param tail-recursion leak -Working tree: dirty (7 files changed) -BLOCKED file: BLOCKED.md (uncommitted; only on PARTIAL/BLOCKED) -Stats: .claude/stats/0126.json (uncommitted) -Files touched: 7 -Tests: 12 green, 0 red -E2E coverage: none (mini mode) -Blocked detail: (only if BLOCKED or PARTIAL — also written to BLOCKED.md) - -Note: task 2 hit a blocker early (missing fixture) which was resolved -in-phase; no BLOCKED.md was written. -""" - -GENUINE_BLOCKED_REPORT = """\ -Status: BLOCKED -Iter: 0127 -Started from: d4e5f6a -Tasks completed: 1 of 3 -Working tree: dirty (2 files changed) -BLOCKED file: BLOCKED.md (uncommitted) -Blocked detail: Task: 2 - Reason: review-loop-exhausted - Worker says: spec-compliance never reached approved -""" - -PARTIAL_REPORT = """\ -Status: PARTIAL -Iter: 0128 -Tasks completed: 2 of 3 -BLOCKED file: BLOCKED.md (uncommitted) -""" - -CASES = [ - # (name, final_text, expected) - ("shipped DONE with BLOCKED template lines (issue #3)", - SHIPPED_DONE_REPORT, "DONE"), - ("genuine BLOCKED end-report", - GENUINE_BLOCKED_REPORT, "BLOCKED"), - ("PARTIAL end-report mentioning BLOCKED.md", - PARTIAL_REPORT, "PARTIAL"), - ("narrating agent whose final line is a bare status token", - "Work complete, tests green.\n\nDONE", "DONE"), - ("prose mentioning a resolved blocker, no status line", - "I was briefly BLOCKED by a missing import but fixed it. All good.", - "unknown"), -] - - -def main(): - failures = [] - for name, text, expected in CASES: - got = status_of(text) - ok = got == expected - print(f"[{'PASS' if ok else 'FAIL'}] {name}: expected {expected!r}, got {got!r}") - if not ok: - failures.append(name) - if failures: - print(f"\n{len(failures)} failure(s).") - return 1 - print(f"\nAll {len(CASES)} cases passed.") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/postmortem/tests/test_workflow_aggregation.py b/postmortem/tests/test_workflow_aggregation.py deleted file mode 100644 index 8cab991..0000000 --- a/postmortem/tests/test_workflow_aggregation.py +++ /dev/null @@ -1,124 +0,0 @@ -#!/usr/bin/env python3 -"""Executable spec for postmortem.analyze_workflows. - -Run directly: `python3 postmortem/tests/test_workflow_aggregation.py`. -Exits 0 on pass, 1 on failure. No pytest dependency — same plain-assert -convention as test_terminal_status.py. - -Pins issue #28: sessions that ran Workflow-substrate pipelines -(workflows/wf_*.json + subagents/workflows//agent-*.jsonl) must -not read as zero subagent spend. The raw transcripts are authoritative -(dedup by requestId); the summary supplies stage labels; a transcript -missing from workflowProgress groups under '?' instead of being folded -into a named stage or dropped. -""" - -import json -import os -import sys -import tempfile - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts")) -from postmortem import analyze_workflows # noqa: E402 - - -def _write(path, obj_lines): - os.makedirs(os.path.dirname(path), exist_ok=True) - with open(path, "w") as fh: - for obj in obj_lines: - fh.write(json.dumps(obj) + "\n") - - -def _assistant_line(request_id, in_tok, out_tok, model="claude-sonnet-5"): - return { - "type": "assistant", - "requestId": request_id, - "message": { - "model": model, - "usage": {"input_tokens": in_tok, "output_tokens": out_tok, - "cache_creation_input_tokens": 0, - "cache_read_input_tokens": 0}, - }, - } - - -def main(): - with tempfile.TemporaryDirectory() as session_dir: - run_id = "wf_test-1" - wf = { - "runId": run_id, - "workflowName": "spec-fixture", - "status": "completed", - "totalTokens": 999, # live counter: deliberately wrong - "defaultModel": "claude-sonnet-5", - "workflowProgress": [ - {"type": "workflow_phase", "index": 1, "title": "P"}, - {"type": "workflow_agent", "agentId": "a1", - "label": "impl:1", "model": "claude-sonnet-5", - "state": "done", "tokens": 1}, - {"type": "workflow_agent", "agentId": "a2", - "label": "impl:2", "model": "claude-sonnet-5", - "state": "done", "tokens": 1}, - {"type": "workflow_agent", "agentId": "a4", - "label": "qual:1", "model": "claude-opus-4-8", - "state": "error", "tokens": 0}, # no transcript on disk - ], - } - os.makedirs(os.path.join(session_dir, "workflows")) - with open(os.path.join(session_dir, "workflows", - run_id + ".json"), "w") as fh: - json.dump(wf, fh) - - raw = os.path.join(session_dir, "subagents", "workflows", run_id) - # a1: streamed repeat of the SAME requestId — must dedup to - # max-per-field (input 10, output 7), not sum to 20/12. - _write(os.path.join(raw, "agent-a1.jsonl"), - [_assistant_line("req-1", 10, 5), - _assistant_line("req-1", 10, 7)]) - # a2: two distinct requests — must sum. - _write(os.path.join(raw, "agent-a2.jsonl"), - [_assistant_line("req-2", 3, 2), - _assistant_line("req-3", 4, 1)]) - # a3: transcript with NO workflowProgress entry — counts under '?'. - _write(os.path.join(raw, "agent-a3.jsonl"), - [_assistant_line("req-4", 100, 50)]) - - runs = analyze_workflows(session_dir) - - assert len(runs) == 1, runs - run = runs[0] - assert run["run_id"] == run_id - assert run["name"] == "spec-fixture" - assert run["reported_total_tokens"] == 999 - - # Three transcripts on disk, one untracked. - assert run["agents"] == 3, run - assert run["agents_untracked"] == 1, run - - stages = run["stages"] - assert set(stages) == {"impl", "?"}, stages - - impl = stages["impl"] - assert impl["count"] == 2, impl - # a1 deduped (10 in / 7 out) + a2 summed (7 in / 3 out) - assert impl["tokens"]["input"] == 17, impl - assert impl["tokens"]["output"] == 10, impl - assert impl["total_tokens"] == 27, impl - assert impl["models"] == {"claude-sonnet-5": 3}, impl - - unk = stages["?"] - assert unk["count"] == 1, unk - assert unk["tokens"]["input"] == 100 and unk["tokens"]["output"] == 50, unk - - # Run totals = authoritative raw aggregation, not the live counter. - assert run["tokens"]["input"] == 117, run - assert run["tokens"]["output"] == 60, run - assert run["total_tokens"] == 177, run - - print("ok: analyze_workflows aggregates raw transcripts per stage, " - "dedups by requestId, and isolates untracked agents under '?'") - return 0 - - -if __name__ == "__main__": - sys.exit(main())