Compare commits
2 Commits
4996f2d919
...
0951d1f14c
| Author | SHA1 | Date | |
|---|---|---|---|
 claude
|
0951d1f14c | ||
|
claude
|
5d1cffab76 |
+3
-1
@@ -5,5 +5,7 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# postmortem skill output — reports are per-run artifacts, not tracked
|
||||
# local report archive of the retired postmortem skill (issue #27) —
|
||||
# reports were per-run artifacts, never tracked; the dir may still
|
||||
# exist locally
|
||||
docs/postmortems/
|
||||
|
||||
+27
-2
@@ -200,7 +200,13 @@ verdict means *adjudicate against the snapshot*, not proven malice.
|
||||
Renames do not trip it (the boundary diff runs `--no-renames`, so a
|
||||
renamed file's old path stays listed as a deletion). Untracked files
|
||||
sit outside the threat model (`git checkout -- <path>` cannot discard
|
||||
them). Cost: one extra sonnet/medium snapshot call per task of a
|
||||
them). Boundaries are HEAD-aware (issue #32): each snapshot records
|
||||
the HEAD its diff ran against, and a boundary pair straddling a HEAD
|
||||
move — the sanctioned commit-the-subset-then-resume repair — resets
|
||||
the guard with a concern instead of comparing, so a replayed
|
||||
pre-commit boundary never counts the committed subset as lost paths;
|
||||
in a single live run the same concern flags an agent that moved HEAD
|
||||
mid-run. Cost: one extra sonnet/medium snapshot call per task of a
|
||||
multi-task run; single-task and mini runs skip the guard. The
|
||||
detection details live in `workflows/implement-loop.js`, the single
|
||||
source.
|
||||
@@ -220,6 +226,7 @@ Workflow({ name: "implement-loop", args: {
|
||||
mode: "standard",
|
||||
iter_id: "<iter_id>",
|
||||
plan_path: "<path under docs/plans>",
|
||||
plan_sha256: "<sha256sum of the plan file>", // recompute at EVERY (re-)invocation (issue #32)
|
||||
task_range: [3, 8], // optional
|
||||
repo_root: "<abs path of the working tree>" // strongly recommended; effectively required in worktree sessions (issue #25)
|
||||
}})
|
||||
@@ -238,6 +245,14 @@ Workflow({ name: "implement-loop", args: {
|
||||
}})
|
||||
```
|
||||
|
||||
`plan_sha256` content-addresses the plan (issue #32): a resume replays
|
||||
agents cached per (prompt, opts), and with only the plan *path* in the
|
||||
extraction prompts an edit-and-resume would replay the stale pre-edit
|
||||
task text — the repair silently no-ops. Compute the hash fresh
|
||||
(`sha256sum <plan_path>`) at every invocation, including a
|
||||
`resumeFromRunId` re-invocation after a plan repair; never reuse the
|
||||
old value.
|
||||
|
||||
`repo_root` is stamped into every stage prompt as a `git -C` anchor:
|
||||
in worktree sessions a dispatched agent's cwd can resolve to the
|
||||
primary checkout and review the wrong tree (false `infra_blocked`
|
||||
@@ -332,7 +347,17 @@ orchestrator decides what to do with the dirty working tree:
|
||||
before re-running the workflow — its preflight clean-tree check
|
||||
counts the file as dirt. Either stash everything and re-run on a
|
||||
clean tree, or commit the known-good subset, then `rm BLOCKED.md`,
|
||||
then re-run.
|
||||
then re-run. The re-run is either a fresh invocation with
|
||||
`task_range` covering the remaining tasks, or a resume
|
||||
(`resumeFromRunId` on the same script) — for a resume, recompute
|
||||
`plan_sha256` from the edited plan so the extraction prompts
|
||||
change: the repaired task runs live, and a completed task replays
|
||||
from cache when its re-extracted text is byte-identical (the
|
||||
usual case; worst case it re-runs redundantly, never stale —
|
||||
issue #32). The discard guard
|
||||
tolerates the interim subset commit: boundaries are HEAD-aware,
|
||||
so the moved baseline resets the guard with a concern instead of
|
||||
a false lost-paths `BLOCKED`.
|
||||
- **Discard:** `git checkout -- .` to drop unstaged file changes;
|
||||
`git clean -fd` / `rm BLOCKED.md` plus anything else new. main HEAD
|
||||
does NOT move.
|
||||
|
||||
@@ -73,7 +73,18 @@
|
||||
// `stash create` (a bare `git stash` would move the tree; the following
|
||||
// boundary would catch that wipe, but only after the fact). Untracked
|
||||
// files sit outside the threat model (`git checkout -- <path>` cannot
|
||||
// discard them).
|
||||
// discard them). Boundaries are HEAD-aware (issue #32): each snapshot
|
||||
// records the HEAD its diff ran against, and a boundary pair straddling
|
||||
// a HEAD move — the sanctioned commit-the-subset-then-resume repair — is
|
||||
// reset-and-surfaced as a concern instead of compared, so a replayed
|
||||
// pre-commit boundary can never count the committed subset as lost paths.
|
||||
// • Resume content-addressing (issue #32): the carrier's plan_sha256 rides
|
||||
// every plan-reading prompt, so the documented PARTIAL repair (edit the
|
||||
// plan, commit the good subset, resume) re-runs the extraction instead
|
||||
// of replaying the stale pre-edit plan from cache. Downstream, a task
|
||||
// replays iff its re-extracted text is byte-identical — typically every
|
||||
// unchanged task; the failure mode is a redundant live re-run, never
|
||||
// stale text.
|
||||
// • The script has no filesystem or shell access of its own — every
|
||||
// working-tree mutation (code, tests) happens inside an agent() call;
|
||||
// the stats.json / BLOCKED.md contents are built in-script and ride
|
||||
@@ -99,6 +110,20 @@
|
||||
// mode: 'standard' | 'mini'
|
||||
// iter_id: e.g. 'ct.2.3' (standard) or 'bugfix-<symptom>' / 'feat-<behaviour>' (mini)
|
||||
// plan_path: (standard) path under docs/plans
|
||||
// plan_sha256: (standard) sha256 hex of the plan file's CURRENT bytes
|
||||
// (`sha256sum <plan_path>`). Stamped into every plan-reading
|
||||
// prompt as a content-address (issue #32): a resume replays
|
||||
// agents cached per (prompt, opts), and a prompt that
|
||||
// carries only the plan PATH replays a stale extraction
|
||||
// after a plan edit — the documented edit-and-resume repair
|
||||
// silently no-ops. With the bytes hashed into the prompt, a
|
||||
// plan edit re-runs the extraction; downstream, a task
|
||||
// replays from cache iff its re-extracted text is
|
||||
// byte-identical — typically every unchanged task, worst
|
||||
// case a redundant live re-run, never stale text. An
|
||||
// unchanged plan replays fully from cache. Recompute it at
|
||||
// EVERY (re-)invocation — a stale or constant value
|
||||
// re-opens the no-op resume.
|
||||
// task_range: (standard, optional) [from, to] inclusive, 1-based
|
||||
// red_test_path: (mini) absolute path to the RED test from debug / tdd
|
||||
// cause_summary: (mini) 1–2 sentences (debugger cause, or tdd spec_summary)
|
||||
@@ -160,6 +185,7 @@ const {
|
||||
mode = 'standard',
|
||||
iter_id,
|
||||
plan_path,
|
||||
plan_sha256,
|
||||
task_range = null,
|
||||
red_test_path,
|
||||
cause_summary,
|
||||
@@ -180,7 +206,7 @@ if (mode !== 'standard' && mode !== 'mini') {
|
||||
// The required carrier fields per mode (the header comment above is the
|
||||
// contract). Missing or blank fields fail fast by name — the loop must never
|
||||
// glob for a plan or brief an implementer with the literal string "undefined".
|
||||
const requiredByMode = mode === 'mini' ? { iter_id, red_test_path, cause_summary } : { iter_id, plan_path }
|
||||
const requiredByMode = mode === 'mini' ? { iter_id, red_test_path, cause_summary } : { iter_id, plan_path, plan_sha256 }
|
||||
const missingFields = Object.entries(requiredByMode)
|
||||
.filter(([, v]) => typeof v !== 'string' || !v.trim())
|
||||
.map(([k]) => k)
|
||||
@@ -218,6 +244,21 @@ const STANDING_NONE =
|
||||
'Schema-bound mechanical stage: NO standing reading (no CLAUDE.md, no git log) — do exactly what is instructed ' +
|
||||
'below, nothing else.'
|
||||
|
||||
// Plan content-address (issue #32): appended to EVERY prompt that reads the
|
||||
// plan file (plan-index, plan-extract-all, plan-extract:<id>). The resume
|
||||
// cache keys on (prompt, opts); with only the plan PATH in the prompt, an
|
||||
// edit-and-resume replays the pre-edit task text from cache and the repair
|
||||
// silently no-ops. Hashing the plan's bytes into each reading prompt makes a
|
||||
// plan edit re-run the extraction — and, because the extracted text flows
|
||||
// verbatim into the downstream per-task prompts, a task replays from cache
|
||||
// iff its re-extracted text is byte-identical (typically every unchanged
|
||||
// task; the failure mode is a redundant live re-run, never stale text).
|
||||
const PLAN_ADDRESS =
|
||||
mode === 'standard'
|
||||
? `\n\nPlan content-address: sha256 ${plan_sha256} — binds this extraction to the plan's current bytes for ` +
|
||||
'resume caching; informational only, nothing to verify or act on.'
|
||||
: ''
|
||||
|
||||
// Working-directory anchor (issue #25): in worktree sessions a dispatched
|
||||
// agent's cwd can resolve to the PRIMARY checkout instead of the worktree the
|
||||
// loop operates on — observed as a per-dispatch race producing false
|
||||
@@ -508,7 +549,7 @@ const STD_VERIFY_SCHEMA = {
|
||||
const SNAP_SCHEMA = {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['snapshot_sha', 'modified_paths'],
|
||||
required: ['snapshot_sha', 'modified_paths', 'head_sha'],
|
||||
properties: {
|
||||
snapshot_sha: {
|
||||
type: 'string',
|
||||
@@ -524,6 +565,12 @@ const SNAP_SCHEMA = {
|
||||
'`--no-renames` keeps a renamed file\'s old path listed as a deletion; untracked files are invisible ' +
|
||||
'here by design)',
|
||||
},
|
||||
head_sha: {
|
||||
type: 'string',
|
||||
description:
|
||||
'output of `git rev-parse HEAD` — the baseline modified_paths was diffed against; the guard compares two ' +
|
||||
'boundaries only when their baselines match (issue #32)',
|
||||
},
|
||||
},
|
||||
}
|
||||
// ---- Phase 0 — clean-tree preflight (mini mode only) --------------------
|
||||
@@ -614,7 +661,7 @@ if (mode === 'mini') {
|
||||
: 'ids of EVERY task') +
|
||||
' that the plan actually defines, each with its one-line title. Return ONLY ids and titles — do NOT include ' +
|
||||
'any task body or code block. Do not summarise, invent, reorder, or pad the list. Separately report ' +
|
||||
'`total_tasks`: the count of ALL tasks the entire plan defines, ignoring any range filter. Do not edit anything.',
|
||||
`\`total_tasks\`: the count of ALL tasks the entire plan defines, ignoring any range filter. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: 'plan-index', phase: 'Per-task loop', schema: TASK_INDEX_SCHEMA },
|
||||
)
|
||||
if (!index) {
|
||||
@@ -728,7 +775,7 @@ if (mode === 'mini') {
|
||||
const all = await agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRead the plan at ${plan_path}. Extract EXACTLY these tasks (1-based ids): ${expectedIds.join(', ')} — ` +
|
||||
'EACH as its VERBATIM block: the full task text as written, including any code blocks. Do not summarise, ' +
|
||||
'truncate, reorder, or merge tasks into each other. Return exactly those tasks, nothing else. Do not edit anything.',
|
||||
`truncate, reorder, or merge tasks into each other. Return exactly those tasks, nothing else. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: 'plan-extract-all', phase: 'Per-task loop', schema: TASK_TEXTS_SCHEMA },
|
||||
)
|
||||
const returned = all && Array.isArray(all.tasks) ? all.tasks : []
|
||||
@@ -746,7 +793,7 @@ if (mode === 'mini') {
|
||||
agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRead the plan at ${plan_path}. Extract EXACTLY task ${id} (1-based) as its VERBATIM block — ` +
|
||||
'the full task text as written, including any code blocks. Do not summarise, truncate, reorder, or merge in ' +
|
||||
'any other task. Return that one task only. Do not edit anything.',
|
||||
`any other task. Return that one task only. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: `plan-extract:${id}`, phase: 'Per-task loop', schema: TASK_TEXT_SCHEMA },
|
||||
).then((t) => ({ reqId: id, t })),
|
||||
),
|
||||
@@ -1011,37 +1058,63 @@ for (const task of tasks) {
|
||||
results.push(r)
|
||||
if (tasks.length > 1) {
|
||||
const snap = await agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRecord a working-tree snapshot boundary WITHOUT changing anything. Run exactly these two ` +
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRecord a working-tree snapshot boundary WITHOUT changing anything. Run exactly these three ` +
|
||||
'commands and report their output:\n' +
|
||||
`1. \`git stash create "iter ${iter_id} post-task ${task.id}"\` — prints a sha (a dangling commit; it ` +
|
||||
'touches neither HEAD, nor the index, nor the working tree). Report it as snapshot_sha; empty string if ' +
|
||||
'it printed nothing.\n' +
|
||||
'2. `git diff HEAD --name-only --no-renames` — report every printed path as modified_paths.\n' +
|
||||
'3. `git rev-parse HEAD` — report it as head_sha.\n' +
|
||||
'NEVER run bare `git stash` or `git stash push/pop/apply/drop` — those MOVE the changes. Do NOT edit, ' +
|
||||
'stage, or commit anything.',
|
||||
{ model: 'sonnet', effort: 'medium', label: `snapshot:${task.id}`, phase: 'Per-task loop', schema: SNAP_SCHEMA },
|
||||
)
|
||||
if (snap && prevBoundary) {
|
||||
const now = new Set(snap.modified_paths || [])
|
||||
const lost = [...prevBoundary.paths].filter((p) => !now.has(p))
|
||||
if (lost.length) {
|
||||
// A path that was HEAD-modified at the previous boundary is
|
||||
// HEAD-identical now: a file-level checkout/restore discarded
|
||||
// sibling-task work mid-run. Hard stop — the per-task DONE reports
|
||||
// and the tree have diverged, so nothing downstream is trustworthy.
|
||||
discard = {
|
||||
lost_paths: lost,
|
||||
modified_through_task: prevBoundary.taskId,
|
||||
discarded_during_task: task.id,
|
||||
recovery_snapshot: prevBoundary.sha || null,
|
||||
// HEAD-aware baseline (issue #32): each boundary's path set is a diff
|
||||
// AGAINST HEAD, so two boundaries are comparable only when they were
|
||||
// measured against the SAME HEAD. The sanctioned PARTIAL repair
|
||||
// (commit the known-good subset, then resume) moves HEAD between the
|
||||
// replayed boundaries and the fresh ones — comparing across that move
|
||||
// would report the entire committed subset as "lost" and hard-BLOCK
|
||||
// the first live task. On a baseline move the guard resets instead of
|
||||
// comparing, and surfaces the move: the loop itself never commits, so
|
||||
// a HEAD move inside a single live run means some agent committed
|
||||
// mid-run — worth eyes either way. A blank/missing sha falls through
|
||||
// to the comparison — a flaky report must not quietly disarm the guard.
|
||||
const prevHead = typeof prevBoundary.head === 'string' ? prevBoundary.head.trim() : ''
|
||||
const nowHead = typeof snap.head_sha === 'string' ? snap.head_sha.trim() : ''
|
||||
if (prevHead && nowHead && prevHead !== nowHead) {
|
||||
guardConcerns.push(
|
||||
`discard-guard baseline moved between task #${prevBoundary.taskId} and task #${task.id} ` +
|
||||
`(HEAD ${prevHead} -> ${nowHead}) — expected only on a resume after the orchestrator committed a ` +
|
||||
'known-good subset (the sanctioned PARTIAL repair); in a single live run with no interim commit it ' +
|
||||
'means an agent moved HEAD mid-run — investigate. The guard reset its baseline; this boundary pair ' +
|
||||
'went uncompared.',
|
||||
)
|
||||
} else {
|
||||
const now = new Set(snap.modified_paths || [])
|
||||
const lost = [...prevBoundary.paths].filter((p) => !now.has(p))
|
||||
if (lost.length) {
|
||||
// A path that was HEAD-modified at the previous boundary is
|
||||
// HEAD-identical now: a file-level checkout/restore discarded
|
||||
// sibling-task work mid-run. Hard stop — the per-task DONE reports
|
||||
// and the tree have diverged, so nothing downstream is trustworthy.
|
||||
discard = {
|
||||
lost_paths: lost,
|
||||
modified_through_task: prevBoundary.taskId,
|
||||
discarded_during_task: task.id,
|
||||
recovery_snapshot: prevBoundary.sha || null,
|
||||
}
|
||||
break
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!snap) guardConcerns.push(`discard-guard snapshot after task #${task.id} did not return — that boundary went unchecked`)
|
||||
// A non-returning snapshot agent skips the NEXT comparison (prevBoundary
|
||||
// null) rather than false-positiving against a stale boundary.
|
||||
prevBoundary = snap ? { taskId: task.id, sha: snap.snapshot_sha || null, paths: new Set(snap.modified_paths || []) } : null
|
||||
prevBoundary = snap
|
||||
? { taskId: task.id, sha: snap.snapshot_sha || null, paths: new Set(snap.modified_paths || []), head: snap.head_sha || null }
|
||||
: null
|
||||
}
|
||||
if (r.outcome === 'BLOCKED') break // do not skip ahead — task ordering dependencies are unknown
|
||||
}
|
||||
|
||||
@@ -1,228 +0,0 @@
|
||||
---
|
||||
name: postmortem
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
---
|
||||
|
||||
# postmortem — grade the run
|
||||
|
||||
> **Violating the letter of these rules is violating the spirit.**
|
||||
|
||||
## Overview
|
||||
|
||||
Every session leaves a detailed flight recorder behind — the
|
||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`,
|
||||
one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`, and for every Workflow run a per-agent
|
||||
summary in `<session>/workflows/wf_*.json` with raw per-agent
|
||||
transcripts under `<session>/subagents/workflows/<runId>/`. Nobody
|
||||
reads it. So the same expensive
|
||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||||
loop with a 30% error rate, a cache-cold run that re-sends the full
|
||||
input on every turn. This skill turns that recorder into a graded
|
||||
retrospective: **how well did the toolchain work, how effective
|
||||
were the agents, and how many tokens did it spend.**
|
||||
|
||||
It is descriptive, not corrective. It produces a report and, where
|
||||
warranted, recommends filing tracker issues — it does not change
|
||||
code, baselines, or the pipeline.
|
||||
|
||||
## When to Use / Skipping
|
||||
|
||||
Invoke:
|
||||
|
||||
- After a substantial interactive session, when you want to know
|
||||
where the time and tokens went.
|
||||
- At the close of a `/boss` autonomous run, to grade the
|
||||
orchestrator's dispatch choices.
|
||||
- When a run *felt* token-heavy or thrashy and you want the numbers.
|
||||
|
||||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
||||
blocks a cycle.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
||||
REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE
|
||||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
||||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
||||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
||||
```
|
||||
|
||||
## What the data does and does not contain
|
||||
|
||||
Read before interpreting, so the report never overclaims:
|
||||
|
||||
- **Present:** per-request `usage` (input / output / cache-creation /
|
||||
cache-read / server-tool counts), `model`, `requestId`,
|
||||
timestamps, every `tool_use` with its input, every `tool_result`
|
||||
with the authoritative `is_error` flag, skill invocations,
|
||||
slash commands, and a full separate transcript + `agentType` /
|
||||
`description` for each subagent. Workflow runs additionally carry a
|
||||
per-agent summary (stage label, model, a live token counter) in
|
||||
`workflows/wf_*.json`; the counter under-tracks vs. the deduped raw
|
||||
transcripts (~11% in the corpus behind issue #28), so the script
|
||||
treats the raw transcripts as authoritative and reports the counter
|
||||
only as `reported_total_tokens` for cross-checking.
|
||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||||
`null`. There is no metered cost in the log, so this skill does
|
||||
not compute one — it reports **raw token counts** (kept split by
|
||||
class: input / output / cache-creation / cache-read, plus a
|
||||
`total_tokens` sum). **Wall-clock is derived** from timestamp
|
||||
deltas and the report says so.
|
||||
|
||||
The streamed transcript repeats the same `requestId` across
|
||||
several assistant lines with an identical `usage` object. The
|
||||
helper script dedups per `requestId`; never sum the raw lines.
|
||||
|
||||
## The Process
|
||||
|
||||
### Step 1 — Run the aggregator
|
||||
|
||||
```
|
||||
python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
||||
```
|
||||
|
||||
Defaults to the newest transcript in the current project's log
|
||||
dir — i.e. the session you are in (it will be flagged `active`,
|
||||
meaning totals are partial). To grade a finished run, pass its
|
||||
`--session <uuid>`.
|
||||
|
||||
The default log dir is derived from cwd (the path with `/` replaced
|
||||
by `-`), so a session run inside a git worktree resolves to a
|
||||
*different* slug than the primary clone, and after exiting a worktree
|
||||
the default again follows the current cwd. To grade a worktree run
|
||||
from the primary clone, pass `--project-dir
|
||||
~/.claude/projects/<worktree-slug>` (or `--file <path>` to bypass slug
|
||||
derivation entirely).
|
||||
|
||||
The script emits one JSON object on stdout. That JSON — not the
|
||||
raw log — is your evidence base.
|
||||
|
||||
### Step 2 — Read the JSON and grade three axes
|
||||
|
||||
Grade each axis on the cited numbers. A grade with no number
|
||||
behind it is an opinion; drop it.
|
||||
|
||||
1. **Token spend & efficiency** — `totals.total_tokens`, the
|
||||
main-vs-subagent split, the token breakdown (output share is the
|
||||
write-heavy signal), `cache_hit_ratio` (the dominant efficiency
|
||||
lever: below ~0.7 on a long session means cache-cold turns
|
||||
re-sending the full input), `server_tools` (web_search /
|
||||
web_fetch counts).
|
||||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
||||
(the authoritative `is_error` signal), the tool mix (a Read /
|
||||
Edit / Bash imbalance hints at thrashing or re-reading),
|
||||
`slash_commands` such as `/compact` (context pressure).
|
||||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
||||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
||||
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
|
||||
fleet that burned real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare each agent's token spend to what it delivered.
|
||||
For Workflow runs, grade per stage instead: `workflow_totals` and
|
||||
each run's `stages` table (label-prefix grouping, e.g. all
|
||||
`impl:N` under `impl`) show where the run's spend concentrated;
|
||||
`agents_untracked > 0` flags retried/dropped attempts. Schema-bound
|
||||
workflow stages carry no `terminal_status` — the workflow script's
|
||||
own aggregation already judged them.
|
||||
|
||||
### Step 3 — Write the report
|
||||
|
||||
Write a Markdown report to the configured destination (see
|
||||
**Output destination**). Structure it as **Output format** below.
|
||||
Lead with the scorecard, support every grade with a number from
|
||||
the JSON, and keep the prose tight — this is a retrospective, not
|
||||
a transcript replay.
|
||||
|
||||
### Step 4 — Surface, don't fix
|
||||
|
||||
Translate the sharpest findings into recommendations. Where a
|
||||
finding is a concrete, trackable defect (a subagent that reliably
|
||||
bounces `BLOCKED`, a tool loop with a runaway error rate),
|
||||
recommend filing it via the `issue` skill — name it, but do not
|
||||
file it yourself unless the user asks. This skill ends at the
|
||||
report.
|
||||
|
||||
## Output destination
|
||||
|
||||
Resolve in this order:
|
||||
|
||||
1. `--out <path>` if the invoker passed one.
|
||||
2. The project's post-mortem directory, if it has one (its CLAUDE.md
|
||||
project facts).
|
||||
3. Default: `docs/postmortems/<session-id-short>-<yyyy-mm-dd>.md`
|
||||
under the project root, creating the directory if needed.
|
||||
|
||||
Also print a ≤200-word summary to the chat so the headline grade
|
||||
and token spend are visible without opening the file.
|
||||
|
||||
## Output format
|
||||
|
||||
```
|
||||
# Post-mortem — <session-id-short> (<date>)
|
||||
|
||||
**Scorecard**
|
||||
| Axis | Grade | Headline metric |
|
||||
|------|-------|-----------------|
|
||||
| Token spend & efficiency | A–F | <total_tokens> total, cache hit <ratio> |
|
||||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, <tokens> on subagents |
|
||||
|
||||
**Token spend**
|
||||
<main vs subagent split; token breakdown by class (input / output /
|
||||
cache-creation / cache-read); the one class that dominates and why>
|
||||
|
||||
**Toolchain**
|
||||
<tool mix, error ratio + what the errors were, context-pressure
|
||||
signals like /compact>
|
||||
|
||||
**Agents**
|
||||
<per-subagent line: type — status — <total_tokens> — one-clause
|
||||
verdict. Omit if no subagents were dispatched.>
|
||||
|
||||
**Recommendations**
|
||||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
||||
an `issue`.>
|
||||
|
||||
_Token counts are raw usage. Wall-clock is derived from timestamp
|
||||
deltas, not metered._
|
||||
```
|
||||
|
||||
If the session is flagged `active`, add one line at the top:
|
||||
"⚠️ Session still active — figures are partial."
|
||||
|
||||
## Common Rationalisations
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. |
|
||||
| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. |
|
||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. |
|
||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. |
|
||||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
||||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
||||
|
||||
## Red Flags — STOP
|
||||
|
||||
- Quoting any token number that did not come out of the script's
|
||||
JSON.
|
||||
- Converting token counts into a cost or dollar figure — report
|
||||
raw counts only.
|
||||
- Reading any file under `~/.claude` other than the session
|
||||
transcript and its subagent logs — and never any credential or
|
||||
token file.
|
||||
- Editing code, baselines, or pipeline config "while you're in
|
||||
there". Report only.
|
||||
- Grading an axis with adjectives instead of the cited metric.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Helper script:** `scripts/postmortem.py` — the deterministic
|
||||
aggregator. All numbers originate here.
|
||||
- **Hand-off target:** the `issue` skill, when a finding is a
|
||||
trackable defect worth filing.
|
||||
- **Report destination (optional):** the project's post-mortem
|
||||
directory, if its CLAUDE.md project facts name one.
|
||||
- **Data-source note:** this skill reads only the current
|
||||
session's own transcript and subagent logs. It does not crawl
|
||||
other projects or other sessions (single-session by design).
|
||||
@@ -1,501 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic aggregator for a single Claude Code session.
|
||||
|
||||
Reads the main transcript JSONL for one session plus its subagent
|
||||
sidechain logs and its Workflow-substrate run logs (workflows/wf_*.json
|
||||
+ subagents/workflows/<runId>/agent-*.jsonl), and emits a compact JSON
|
||||
summary on stdout that the post-mortem SKILL interprets into a written
|
||||
report.
|
||||
|
||||
Why a script and not skill prose: transcripts run to >1 MB and a
|
||||
session can spawn dozens of subagent logs. Token accounting has two
|
||||
traps that only deterministic code gets right every time:
|
||||
|
||||
1. Streamed assistant lines repeat the SAME requestId with the SAME
|
||||
usage object. Counting per-line double-counts tokens. We dedup
|
||||
per requestId (max per field within a request, then sum across
|
||||
requests).
|
||||
2. The transcript carries NO durationMs (always null), so wall-clock
|
||||
is derived from timestamp deltas. The numbers reported here are
|
||||
raw token counts, not a dollar estimate — different token classes
|
||||
(input / output / cache) are reported separately rather than
|
||||
collapsed into a single priced figure.
|
||||
|
||||
Stdlib only. No network. Read-only against ~/.claude.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
_TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first
|
||||
# Authoritative signal: the implement end-report leads with a structured
|
||||
# "Status: DONE" line (optionally wrapped in markdown bold or a list
|
||||
# marker). Anchoring here is what stops the scan from flipping a shipped
|
||||
# DONE run to BLOCKED on the strength of the report's own template lines
|
||||
# (`BLOCKED file: BLOCKED.md`, `Blocked detail:`) or prose narrating a
|
||||
# surmounted blocker. See issue #3.
|
||||
STATUS_LINE_RE = re.compile(
|
||||
r"^[\s>*#\-]*\**\s*status\s*\**\s*[:\-]\s*\**\s*(" + _TOK_ALT + r")\b",
|
||||
re.IGNORECASE | re.MULTILINE)
|
||||
# Fallback for agents that emit a bare status token as their final line
|
||||
# (no "Status:" prefix). Requires the token to BE the line — excludes
|
||||
# substrings like "BLOCKED.md" or "BLOCKED file:" buried in prose.
|
||||
STATUS_STANDALONE_RE = re.compile(
|
||||
r"^[\s>*#\-]*\**\s*(" + _TOK_ALT + r")\**\s*$",
|
||||
re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def parse_ts(s):
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(s.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def empty_tokens():
|
||||
return {"input": 0, "output": 0, "cache_creation": 0, "cache_read": 0,
|
||||
"cache_creation_5m": 0, "cache_creation_1h": 0}
|
||||
|
||||
|
||||
def token_total(tokens):
|
||||
"""Raw count of every token the session moved. cache_creation_5m/1h are
|
||||
a sub-split of cache_creation, so they are excluded to avoid double
|
||||
counting."""
|
||||
return (tokens["input"] + tokens["output"]
|
||||
+ tokens["cache_creation"] + tokens["cache_read"])
|
||||
|
||||
|
||||
def add_tokens(acc, usage):
|
||||
"""Add one request's deduped usage into an accumulator."""
|
||||
acc["input"] += usage.get("input_tokens", 0) or 0
|
||||
acc["output"] += usage.get("output_tokens", 0) or 0
|
||||
cc = usage.get("cache_creation_input_tokens", 0) or 0
|
||||
acc["cache_creation"] += cc
|
||||
acc["cache_read"] += usage.get("cache_read_input_tokens", 0) or 0
|
||||
detail = usage.get("cache_creation") or {}
|
||||
f5 = detail.get("ephemeral_5m_input_tokens")
|
||||
f1 = detail.get("ephemeral_1h_input_tokens")
|
||||
if f5 is None and f1 is None:
|
||||
acc["cache_creation_5m"] += cc # no split available -> assume 5m
|
||||
else:
|
||||
acc["cache_creation_5m"] += f5 or 0
|
||||
acc["cache_creation_1h"] += f1 or 0
|
||||
|
||||
|
||||
def iter_lines(path):
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
|
||||
def aggregate_transcript(path):
|
||||
"""Return token totals, model mix, and per-request count for one JSONL
|
||||
transcript (main or subagent). Dedups usage by requestId."""
|
||||
# requestId -> {field: max seen}, plus model per request.
|
||||
req_usage = {}
|
||||
req_model = {}
|
||||
for rec in iter_lines(path):
|
||||
if rec.get("type") != "assistant":
|
||||
continue
|
||||
msg = rec.get("message") or {}
|
||||
usage = msg.get("usage")
|
||||
if not usage:
|
||||
continue
|
||||
key = rec.get("requestId") or rec.get("uuid")
|
||||
model = msg.get("model")
|
||||
if model:
|
||||
req_model[key] = model
|
||||
cur = req_usage.setdefault(key, {})
|
||||
# Max-per-field guards against both identical-repeat and
|
||||
# cumulative-within-request streaming shapes.
|
||||
for f in ("input_tokens", "output_tokens",
|
||||
"cache_creation_input_tokens", "cache_read_input_tokens"):
|
||||
v = usage.get(f, 0) or 0
|
||||
if v > cur.get(f, 0):
|
||||
cur[f] = v
|
||||
det = usage.get("cache_creation") or {}
|
||||
cd = cur.setdefault("cache_creation", {})
|
||||
for f in ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"):
|
||||
v = det.get(f, 0) or 0
|
||||
if v > cd.get(f, 0):
|
||||
cd[f] = v
|
||||
st = usage.get("server_tool_use") or {}
|
||||
sc = cur.setdefault("_server", {})
|
||||
for f in ("web_search_requests", "web_fetch_requests"):
|
||||
v = st.get(f, 0) or 0
|
||||
if v > sc.get(f, 0):
|
||||
sc[f] = v
|
||||
|
||||
tokens = empty_tokens()
|
||||
server = {"web_search": 0, "web_fetch": 0}
|
||||
models = {}
|
||||
for key, usage in req_usage.items():
|
||||
add_tokens(tokens, usage)
|
||||
model = req_model.get(key)
|
||||
models[model] = models.get(model, 0) + 1
|
||||
sv = usage.get("_server", {})
|
||||
server["web_search"] += sv.get("web_search_requests", 0)
|
||||
server["web_fetch"] += sv.get("web_fetch_requests", 0)
|
||||
|
||||
return {
|
||||
"requests": len(req_usage),
|
||||
"tokens": tokens,
|
||||
"total_tokens": token_total(tokens),
|
||||
"models": models,
|
||||
"server_tools": server,
|
||||
}
|
||||
|
||||
|
||||
def cache_hit_ratio(tokens):
|
||||
denom = tokens["input"] + tokens["cache_read"] + tokens["cache_creation"]
|
||||
return round(tokens["cache_read"] / denom, 4) if denom else None
|
||||
|
||||
|
||||
def analyze_main(path):
|
||||
"""Tool mix, error rate, skills, slash commands, prompts, wall-clock."""
|
||||
tool_calls = {}
|
||||
error_total = 0
|
||||
interrupted = 0
|
||||
skills = []
|
||||
slash = []
|
||||
tasks = []
|
||||
user_prompts = 0
|
||||
first_ts = last_ts = None
|
||||
cwd = version = None
|
||||
|
||||
for rec in iter_lines(path):
|
||||
ts = parse_ts(rec.get("timestamp"))
|
||||
if ts:
|
||||
if first_ts is None or ts < first_ts:
|
||||
first_ts = ts
|
||||
if last_ts is None or ts > last_ts:
|
||||
last_ts = ts
|
||||
cwd = rec.get("cwd") or cwd
|
||||
version = rec.get("version") or version
|
||||
|
||||
typ = rec.get("type")
|
||||
msg = rec.get("message") or {}
|
||||
content = msg.get("content")
|
||||
|
||||
if typ == "assistant" and isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") != "tool_use":
|
||||
continue
|
||||
name = block.get("name", "?")
|
||||
tool_calls[name] = tool_calls.get(name, 0) + 1
|
||||
if name == "Skill":
|
||||
skills.append({"skill": (block.get("input") or {}).get("skill"),
|
||||
"ts": rec.get("timestamp")})
|
||||
elif name in ("Task", "Agent"):
|
||||
inp = block.get("input") or {}
|
||||
tasks.append({"subagent_type": inp.get("subagent_type"),
|
||||
"description": inp.get("description"),
|
||||
"ts": rec.get("timestamp")})
|
||||
|
||||
if typ == "user":
|
||||
# tool_result blocks carry the authoritative is_error flag
|
||||
if isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") == "tool_result":
|
||||
if block.get("is_error"):
|
||||
error_total += 1
|
||||
elif block.get("type") == "text":
|
||||
for m in re.findall(r"<command-name>(.*?)</command-name>",
|
||||
block.get("text", "")):
|
||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
||||
elif isinstance(content, str):
|
||||
for m in re.findall(r"<command-name>(.*?)</command-name>", content):
|
||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
||||
# genuine user turns (not tool results, not slash plumbing)
|
||||
is_tool_result = (isinstance(content, list)
|
||||
and any(b.get("type") == "tool_result" for b in content))
|
||||
if not is_tool_result and rec.get("toolUseResult") is None:
|
||||
user_prompts += 1
|
||||
tur = rec.get("toolUseResult")
|
||||
if isinstance(tur, dict) and tur.get("interrupted"):
|
||||
interrupted += 1
|
||||
|
||||
return {
|
||||
"tool_calls": tool_calls,
|
||||
"total_tool_calls": sum(tool_calls.values()),
|
||||
"tool_error_total": error_total,
|
||||
"interrupted": interrupted,
|
||||
"skills_invoked": skills,
|
||||
"slash_commands": slash,
|
||||
"tasks_dispatched": tasks,
|
||||
"user_prompts": user_prompts,
|
||||
"wall_clock_sec": round((last_ts - first_ts).total_seconds(), 1)
|
||||
if first_ts and last_ts else None,
|
||||
"started": first_ts.isoformat() if first_ts else None,
|
||||
"ended": last_ts.isoformat() if last_ts else None,
|
||||
"cwd": cwd,
|
||||
"version": version,
|
||||
}
|
||||
|
||||
|
||||
def subagent_terminal_status(path):
|
||||
last_text = ""
|
||||
for rec in iter_lines(path):
|
||||
if rec.get("type") != "assistant":
|
||||
continue
|
||||
content = (rec.get("message") or {}).get("content")
|
||||
if isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") == "text" and block.get("text", "").strip():
|
||||
last_text = block["text"]
|
||||
# Primary: the structured "Status:" line. Take the LAST one so a
|
||||
# restated final status wins over any earlier mention.
|
||||
structured = STATUS_LINE_RE.findall(last_text)
|
||||
if structured:
|
||||
return structured[-1].upper()
|
||||
# Fallback: a status token standing alone on its own line.
|
||||
standalone = STATUS_STANDALONE_RE.findall(last_text)
|
||||
if standalone:
|
||||
return standalone[-1].upper()
|
||||
return "unknown"
|
||||
|
||||
|
||||
def analyze_workflows(session_dir):
|
||||
"""Aggregate Workflow-substrate runs: workflows/wf_*.json (per-agent
|
||||
summary: stage label, model, live token counter) joined with the raw
|
||||
per-agent transcripts under subagents/workflows/<runId>/agent-*.jsonl.
|
||||
|
||||
The raw transcripts are authoritative for token counts (deduped by
|
||||
requestId, same as every other transcript here); the wf_*.json
|
||||
'tokens' counters under-track vs. the deduped raw usage (~11% in the
|
||||
206-run corpus that motivated this path, issue #28) and are reported
|
||||
only as a cross-check. Stage = the label up to the first ':'
|
||||
('impl:3' -> 'impl'), so loop stages aggregate across tasks/rounds."""
|
||||
runs = []
|
||||
for wf_path in sorted(glob.glob(os.path.join(session_dir, "workflows",
|
||||
"wf_*.json"))):
|
||||
try:
|
||||
with open(wf_path) as fh:
|
||||
data = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
continue
|
||||
run_id = data.get("runId") or os.path.basename(wf_path)[:-len(".json")]
|
||||
raw_dir = os.path.join(session_dir, "subagents", "workflows", run_id)
|
||||
meta_by_agent = {}
|
||||
for entry in data.get("workflowProgress") or []:
|
||||
if entry.get("type") != "workflow_agent" or not entry.get("agentId"):
|
||||
continue
|
||||
meta_by_agent[entry["agentId"]] = entry
|
||||
|
||||
run_tokens = empty_tokens()
|
||||
stages = {}
|
||||
agent_count = 0
|
||||
untracked = 0
|
||||
for jl in sorted(glob.glob(os.path.join(raw_dir, "agent-*.jsonl"))):
|
||||
aid = os.path.basename(jl)[len("agent-"):-len(".jsonl")]
|
||||
agg = aggregate_transcript(jl)
|
||||
entry = meta_by_agent.get(aid)
|
||||
if entry is None:
|
||||
# Transcript with no progress entry (retried/dropped
|
||||
# attempt): keep it counted, grouped under '?', never
|
||||
# silently folded into a named stage.
|
||||
untracked += 1
|
||||
label = (entry or {}).get("label") or "?"
|
||||
stage = label.split(":", 1)[0]
|
||||
st = stages.setdefault(stage, {"count": 0, "tokens": empty_tokens(),
|
||||
"models": {}})
|
||||
st["count"] += 1
|
||||
agent_count += 1
|
||||
for k in run_tokens:
|
||||
run_tokens[k] += agg["tokens"][k]
|
||||
st["tokens"][k] += agg["tokens"][k]
|
||||
fallback_model = (entry or {}).get("model")
|
||||
for m, n in agg["models"].items():
|
||||
key = m or fallback_model
|
||||
st["models"][key] = st["models"].get(key, 0) + n
|
||||
|
||||
for st in stages.values():
|
||||
st["total_tokens"] = token_total(st["tokens"])
|
||||
|
||||
runs.append({
|
||||
"run_id": run_id,
|
||||
"name": data.get("workflowName"),
|
||||
"status": data.get("status"),
|
||||
"agents": agent_count,
|
||||
"agents_untracked": untracked,
|
||||
"reported_total_tokens": data.get("totalTokens"),
|
||||
"tokens": run_tokens,
|
||||
"total_tokens": token_total(run_tokens),
|
||||
"stages": stages,
|
||||
})
|
||||
return runs
|
||||
|
||||
|
||||
def analyze_subagents(session_dir):
|
||||
out = []
|
||||
sub_dir = os.path.join(session_dir, "subagents")
|
||||
for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))):
|
||||
meta = {}
|
||||
mp = jl.replace(".jsonl", ".meta.json")
|
||||
if os.path.exists(mp):
|
||||
try:
|
||||
with open(mp) as fh:
|
||||
meta = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
agg = aggregate_transcript(jl)
|
||||
out.append({
|
||||
"agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")],
|
||||
"agent_type": meta.get("agentType"),
|
||||
"description": meta.get("description"),
|
||||
"dispatched_by_tool": meta.get("toolUseId"),
|
||||
"models": agg["models"],
|
||||
"requests": agg["requests"],
|
||||
"tokens": agg["tokens"],
|
||||
"total_tokens": agg["total_tokens"],
|
||||
"terminal_status": subagent_terminal_status(jl),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def resolve_session(args):
|
||||
"""Return (main_jsonl_path, session_dir_or_None)."""
|
||||
if args.file:
|
||||
path = os.path.abspath(args.file)
|
||||
return path, path[:-len(".jsonl")] if path.endswith(".jsonl") else None
|
||||
|
||||
cwd = args.cwd or os.getcwd()
|
||||
proj_dir = args.project_dir or os.path.join(
|
||||
os.path.expanduser("~/.claude/projects"), cwd.replace("/", "-"))
|
||||
if not os.path.isdir(proj_dir):
|
||||
raise SystemExit(f"no project log dir for cwd {cwd}: {proj_dir} not found")
|
||||
|
||||
if args.session:
|
||||
path = os.path.join(proj_dir, args.session + ".jsonl")
|
||||
if not os.path.exists(path):
|
||||
raise SystemExit(f"session {args.session} not found in {proj_dir}")
|
||||
return path, os.path.join(proj_dir, args.session)
|
||||
|
||||
# default: newest .jsonl in the project dir = current/last session
|
||||
candidates = sorted(glob.glob(os.path.join(proj_dir, "*.jsonl")),
|
||||
key=os.path.getmtime, reverse=True)
|
||||
if not candidates:
|
||||
raise SystemExit(f"no session transcripts in {proj_dir}")
|
||||
path = candidates[0]
|
||||
return path, path[:-len(".jsonl")]
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Single-session post-mortem aggregator.")
|
||||
ap.add_argument("--session", help="explicit session id (UUID)")
|
||||
ap.add_argument("--file", help="explicit path to a main transcript .jsonl")
|
||||
ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)")
|
||||
ap.add_argument("--project-dir", help="explicit ~/.claude/projects/<slug> dir")
|
||||
args = ap.parse_args()
|
||||
|
||||
main_path, session_dir = resolve_session(args)
|
||||
session_id = os.path.basename(main_path)[:-len(".jsonl")]
|
||||
|
||||
main_agg = aggregate_transcript(main_path)
|
||||
main_meta = analyze_main(main_path)
|
||||
|
||||
subagents = []
|
||||
workflows = []
|
||||
if session_dir and os.path.isdir(session_dir):
|
||||
subagents = analyze_subagents(session_dir)
|
||||
workflows = analyze_workflows(session_dir)
|
||||
|
||||
sub_tokens = empty_tokens()
|
||||
for s in subagents:
|
||||
for k in sub_tokens:
|
||||
sub_tokens[k] += s["tokens"][k]
|
||||
|
||||
wf_tokens = empty_tokens()
|
||||
wf_agents = 0
|
||||
for w in workflows:
|
||||
wf_agents += w["agents"]
|
||||
for k in wf_tokens:
|
||||
wf_tokens[k] += w["tokens"][k]
|
||||
|
||||
total_tokens = empty_tokens()
|
||||
for k in total_tokens:
|
||||
total_tokens[k] = (main_agg["tokens"][k] + sub_tokens[k]
|
||||
+ wf_tokens[k])
|
||||
|
||||
warnings = []
|
||||
is_active = False
|
||||
if main_meta["ended"]:
|
||||
ended = parse_ts(main_meta["ended"])
|
||||
if ended and (datetime.now(timezone.utc) - ended).total_seconds() < 120:
|
||||
is_active = True
|
||||
warnings.append("Session appears ACTIVE (last event <2 min ago); "
|
||||
"totals are partial and will grow.")
|
||||
warnings.append("Token counts are raw usage; wall-clock is derived from "
|
||||
"timestamp deltas (the transcript carries no durationMs).")
|
||||
|
||||
report = {
|
||||
"session": {
|
||||
"id": session_id,
|
||||
"transcript": main_path,
|
||||
"cwd": main_meta["cwd"],
|
||||
"version": main_meta["version"],
|
||||
"started": main_meta["started"],
|
||||
"ended": main_meta["ended"],
|
||||
"wall_clock_sec": main_meta["wall_clock_sec"],
|
||||
"active": is_active,
|
||||
},
|
||||
"main": {
|
||||
"requests": main_agg["requests"],
|
||||
"models": main_agg["models"],
|
||||
"tokens": main_agg["tokens"],
|
||||
"total_tokens": main_agg["total_tokens"],
|
||||
"cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]),
|
||||
"server_tools": main_agg["server_tools"],
|
||||
"user_prompts": main_meta["user_prompts"],
|
||||
},
|
||||
"tools": {
|
||||
"calls": main_meta["tool_calls"],
|
||||
"total": main_meta["total_tool_calls"],
|
||||
"error_total": main_meta["tool_error_total"],
|
||||
"error_ratio": round(main_meta["tool_error_total"]
|
||||
/ main_meta["total_tool_calls"], 4)
|
||||
if main_meta["total_tool_calls"] else None,
|
||||
"interrupted": main_meta["interrupted"],
|
||||
},
|
||||
"skills_invoked": main_meta["skills_invoked"],
|
||||
"slash_commands": main_meta["slash_commands"],
|
||||
"tasks_dispatched": main_meta["tasks_dispatched"],
|
||||
"subagents": subagents,
|
||||
"subagent_totals": {
|
||||
"count": len(subagents),
|
||||
"tokens": sub_tokens,
|
||||
"total_tokens": token_total(sub_tokens),
|
||||
},
|
||||
"workflows": workflows,
|
||||
"workflow_totals": {
|
||||
"runs": len(workflows),
|
||||
"agents": wf_agents,
|
||||
"tokens": wf_tokens,
|
||||
"total_tokens": token_total(wf_tokens),
|
||||
},
|
||||
"totals": {
|
||||
"tokens": total_tokens,
|
||||
"total_tokens": token_total(total_tokens),
|
||||
"cache_hit_ratio": cache_hit_ratio(total_tokens),
|
||||
},
|
||||
"warnings": warnings,
|
||||
}
|
||||
json.dump(report, sys.stdout, indent=2)
|
||||
sys.stdout.write("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,125 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Executable spec for postmortem.subagent_terminal_status.
|
||||
|
||||
Run directly: `python3 postmortem/tests/test_terminal_status.py`.
|
||||
Exits 0 on pass, 1 on failure. No pytest dependency — the plugin has
|
||||
no test harness, and a status-classifier is small enough to pin with
|
||||
plain asserts.
|
||||
|
||||
Pins issue #3: the terminal-status text-scan over-reports BLOCKED for
|
||||
shipped implement runs, because `\\bBLOCKED\\b` fires on the end-report's
|
||||
own template lines (`BLOCKED file: BLOCKED.md`, `Blocked detail:`)
|
||||
and on prose narrating a surmounted blocker. The authoritative signal is
|
||||
the structured `Status:` line of the end-report, not any keyword
|
||||
occurrence in the body.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
SCRIPT = os.path.join(HERE, "..", "scripts", "postmortem.py")
|
||||
|
||||
spec = importlib.util.spec_from_file_location("postmortem", SCRIPT)
|
||||
pm = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(pm)
|
||||
|
||||
|
||||
def write_subagent_log(final_text):
|
||||
"""Write a one-assistant-turn subagent jsonl whose last text block is
|
||||
`final_text`, return its path. Caller cleans up."""
|
||||
fd, path = tempfile.mkstemp(suffix=".jsonl")
|
||||
rec = {"type": "assistant",
|
||||
"message": {"role": "assistant",
|
||||
"content": [{"type": "text", "text": final_text}]}}
|
||||
with os.fdopen(fd, "w") as fh:
|
||||
fh.write(json.dumps(rec) + "\n")
|
||||
return path
|
||||
|
||||
|
||||
def status_of(final_text):
|
||||
path = write_subagent_log(final_text)
|
||||
try:
|
||||
return pm.subagent_terminal_status(path)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
# A realistic plain-text iteration end-report that SHIPPED (Status: DONE)
|
||||
# but whose body carries the template's BLOCKED-bearing lines. This is the
|
||||
# exact shape that issue #3 reports as misclassified.
|
||||
SHIPPED_DONE_REPORT = """\
|
||||
Status: DONE
|
||||
Iter: 0126-leg3
|
||||
Started from: a1b2c3d
|
||||
Tasks completed: 3 of 3
|
||||
- poly RawBuf elem-var mono fix
|
||||
- series kernel module
|
||||
- owned-param tail-recursion leak
|
||||
Working tree: dirty (7 files changed)
|
||||
BLOCKED file: BLOCKED.md (uncommitted; only on PARTIAL/BLOCKED)
|
||||
Stats: .claude/stats/0126.json (uncommitted)
|
||||
Files touched: 7
|
||||
Tests: 12 green, 0 red
|
||||
E2E coverage: none (mini mode)
|
||||
Blocked detail: (only if BLOCKED or PARTIAL — also written to BLOCKED.md)
|
||||
|
||||
Note: task 2 hit a blocker early (missing fixture) which was resolved
|
||||
in-phase; no BLOCKED.md was written.
|
||||
"""
|
||||
|
||||
GENUINE_BLOCKED_REPORT = """\
|
||||
Status: BLOCKED
|
||||
Iter: 0127
|
||||
Started from: d4e5f6a
|
||||
Tasks completed: 1 of 3
|
||||
Working tree: dirty (2 files changed)
|
||||
BLOCKED file: BLOCKED.md (uncommitted)
|
||||
Blocked detail: Task: 2
|
||||
Reason: review-loop-exhausted
|
||||
Worker says: spec-compliance never reached approved
|
||||
"""
|
||||
|
||||
PARTIAL_REPORT = """\
|
||||
Status: PARTIAL
|
||||
Iter: 0128
|
||||
Tasks completed: 2 of 3
|
||||
BLOCKED file: BLOCKED.md (uncommitted)
|
||||
"""
|
||||
|
||||
CASES = [
|
||||
# (name, final_text, expected)
|
||||
("shipped DONE with BLOCKED template lines (issue #3)",
|
||||
SHIPPED_DONE_REPORT, "DONE"),
|
||||
("genuine BLOCKED end-report",
|
||||
GENUINE_BLOCKED_REPORT, "BLOCKED"),
|
||||
("PARTIAL end-report mentioning BLOCKED.md",
|
||||
PARTIAL_REPORT, "PARTIAL"),
|
||||
("narrating agent whose final line is a bare status token",
|
||||
"Work complete, tests green.\n\nDONE", "DONE"),
|
||||
("prose mentioning a resolved blocker, no status line",
|
||||
"I was briefly BLOCKED by a missing import but fixed it. All good.",
|
||||
"unknown"),
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
failures = []
|
||||
for name, text, expected in CASES:
|
||||
got = status_of(text)
|
||||
ok = got == expected
|
||||
print(f"[{'PASS' if ok else 'FAIL'}] {name}: expected {expected!r}, got {got!r}")
|
||||
if not ok:
|
||||
failures.append(name)
|
||||
if failures:
|
||||
print(f"\n{len(failures)} failure(s).")
|
||||
return 1
|
||||
print(f"\nAll {len(CASES)} cases passed.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Executable spec for postmortem.analyze_workflows.
|
||||
|
||||
Run directly: `python3 postmortem/tests/test_workflow_aggregation.py`.
|
||||
Exits 0 on pass, 1 on failure. No pytest dependency — same plain-assert
|
||||
convention as test_terminal_status.py.
|
||||
|
||||
Pins issue #28: sessions that ran Workflow-substrate pipelines
|
||||
(workflows/wf_*.json + subagents/workflows/<runId>/agent-*.jsonl) must
|
||||
not read as zero subagent spend. The raw transcripts are authoritative
|
||||
(dedup by requestId); the summary supplies stage labels; a transcript
|
||||
missing from workflowProgress groups under '?' instead of being folded
|
||||
into a named stage or dropped.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
|
||||
from postmortem import analyze_workflows # noqa: E402
|
||||
|
||||
|
||||
def _write(path, obj_lines):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w") as fh:
|
||||
for obj in obj_lines:
|
||||
fh.write(json.dumps(obj) + "\n")
|
||||
|
||||
|
||||
def _assistant_line(request_id, in_tok, out_tok, model="claude-sonnet-5"):
|
||||
return {
|
||||
"type": "assistant",
|
||||
"requestId": request_id,
|
||||
"message": {
|
||||
"model": model,
|
||||
"usage": {"input_tokens": in_tok, "output_tokens": out_tok,
|
||||
"cache_creation_input_tokens": 0,
|
||||
"cache_read_input_tokens": 0},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
with tempfile.TemporaryDirectory() as session_dir:
|
||||
run_id = "wf_test-1"
|
||||
wf = {
|
||||
"runId": run_id,
|
||||
"workflowName": "spec-fixture",
|
||||
"status": "completed",
|
||||
"totalTokens": 999, # live counter: deliberately wrong
|
||||
"defaultModel": "claude-sonnet-5",
|
||||
"workflowProgress": [
|
||||
{"type": "workflow_phase", "index": 1, "title": "P"},
|
||||
{"type": "workflow_agent", "agentId": "a1",
|
||||
"label": "impl:1", "model": "claude-sonnet-5",
|
||||
"state": "done", "tokens": 1},
|
||||
{"type": "workflow_agent", "agentId": "a2",
|
||||
"label": "impl:2", "model": "claude-sonnet-5",
|
||||
"state": "done", "tokens": 1},
|
||||
{"type": "workflow_agent", "agentId": "a4",
|
||||
"label": "qual:1", "model": "claude-opus-4-8",
|
||||
"state": "error", "tokens": 0}, # no transcript on disk
|
||||
],
|
||||
}
|
||||
os.makedirs(os.path.join(session_dir, "workflows"))
|
||||
with open(os.path.join(session_dir, "workflows",
|
||||
run_id + ".json"), "w") as fh:
|
||||
json.dump(wf, fh)
|
||||
|
||||
raw = os.path.join(session_dir, "subagents", "workflows", run_id)
|
||||
# a1: streamed repeat of the SAME requestId — must dedup to
|
||||
# max-per-field (input 10, output 7), not sum to 20/12.
|
||||
_write(os.path.join(raw, "agent-a1.jsonl"),
|
||||
[_assistant_line("req-1", 10, 5),
|
||||
_assistant_line("req-1", 10, 7)])
|
||||
# a2: two distinct requests — must sum.
|
||||
_write(os.path.join(raw, "agent-a2.jsonl"),
|
||||
[_assistant_line("req-2", 3, 2),
|
||||
_assistant_line("req-3", 4, 1)])
|
||||
# a3: transcript with NO workflowProgress entry — counts under '?'.
|
||||
_write(os.path.join(raw, "agent-a3.jsonl"),
|
||||
[_assistant_line("req-4", 100, 50)])
|
||||
|
||||
runs = analyze_workflows(session_dir)
|
||||
|
||||
assert len(runs) == 1, runs
|
||||
run = runs[0]
|
||||
assert run["run_id"] == run_id
|
||||
assert run["name"] == "spec-fixture"
|
||||
assert run["reported_total_tokens"] == 999
|
||||
|
||||
# Three transcripts on disk, one untracked.
|
||||
assert run["agents"] == 3, run
|
||||
assert run["agents_untracked"] == 1, run
|
||||
|
||||
stages = run["stages"]
|
||||
assert set(stages) == {"impl", "?"}, stages
|
||||
|
||||
impl = stages["impl"]
|
||||
assert impl["count"] == 2, impl
|
||||
# a1 deduped (10 in / 7 out) + a2 summed (7 in / 3 out)
|
||||
assert impl["tokens"]["input"] == 17, impl
|
||||
assert impl["tokens"]["output"] == 10, impl
|
||||
assert impl["total_tokens"] == 27, impl
|
||||
assert impl["models"] == {"claude-sonnet-5": 3}, impl
|
||||
|
||||
unk = stages["?"]
|
||||
assert unk["count"] == 1, unk
|
||||
assert unk["tokens"]["input"] == 100 and unk["tokens"]["output"] == 50, unk
|
||||
|
||||
# Run totals = authoritative raw aggregation, not the live counter.
|
||||
assert run["tokens"]["input"] == 117, run
|
||||
assert run["tokens"]["output"] == 60, run
|
||||
assert run["total_tokens"] == 177, run
|
||||
|
||||
print("ok: analyze_workflows aggregates raw transcripts per stage, "
|
||||
"dedups by requestId, and isolates untracked agents under '?'")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user