Compare commits
5 Commits
4996f2d919
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 8ad396116e | |||
| fe5ca8f8c0 | |||
| b2c92db725 | |||
| 0951d1f14c | |||
| 5d1cffab76 |
+3
-1
@@ -5,5 +5,7 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# postmortem skill output — reports are per-run artifacts, not tracked
|
||||
# local report archive of the retired postmortem skill (issue #27) —
|
||||
# reports were per-run artifacts, never tracked; the dir may still
|
||||
# exist locally
|
||||
docs/postmortems/
|
||||
|
||||
@@ -177,6 +177,13 @@ new feature work with no spec or plan yet, run the **Entry-path
|
||||
reflection** below before dispatching anything. Read each skill's
|
||||
`SKILL.md` trigger section if unsure.
|
||||
|
||||
**Pre-dispatch grounding stays light.** Queue triage reads issue
|
||||
bodies and `git log`; it does not fan out. A code-territory read an
|
||||
item needs belongs to the dispatched phase skill's own recon step —
|
||||
the cycle recon (`../docs/pipeline.md` § Cycle recon) — and is
|
||||
never a broad Explore-type sweep fired before, or instead of, that
|
||||
step.
|
||||
|
||||
**Entry-path reflection (feature work).** Entry paths form an **ordered
|
||||
cascade**, and the choice is made by reflection every time, not by
|
||||
habit. The discriminator is no longer a single "design line": it adds a
|
||||
|
||||
+7
-4
@@ -152,8 +152,9 @@ unconditional default.
|
||||
### Step 3 — Populate the space with the synthetic-user swarm
|
||||
|
||||
Dispatch the read-only `agents/synthetic-user.md` agent **in parallel, one per
|
||||
stance, in a single turn** — the same ad-hoc fan-out brainstorm already uses
|
||||
for `plan-recon` and `specify` uses for `spec-skeptic`. No Workflow wrapper: a
|
||||
stance, in a single turn** — the same ad-hoc fan-out `specify` uses
|
||||
for `spec-skeptic`. (brainstorm's ad-hoc `plan-recon` is a SINGLE dispatch,
|
||||
never a fan-out — `../docs/pipeline.md` § Cycle recon.) No Workflow wrapper: a
|
||||
one-shot stateless fan-out has no re-loop, no ordering, and no in-code verdict
|
||||
to enforce, so it is a direct dispatch, not an `implement-loop`-style script.
|
||||
|
||||
@@ -343,8 +344,10 @@ reflection for how this folds `brainstorm` into the autonomous cascade.
|
||||
- **Autonomous behaviour:** `../boss/SKILL.md` § Direction freedom (trigger 4)
|
||||
and the Entry-path reflection — how the autonomous swarm-brainstorm decides
|
||||
convergent forks and bounces only divergent ones.
|
||||
- **Ad-hoc dispatch.** The orchestrator MAY also ad-hoc dispatch
|
||||
- **Ad-hoc dispatch.** The orchestrator MAY ad-hoc dispatch
|
||||
`../planner/agents/plan-recon.md` during Step 1 when the cycle enters code
|
||||
territory not recently read; ad-hoc, not part of the standard process.
|
||||
territory not recently read. Subject to the coverage condition in
|
||||
`../docs/pipeline.md` § Cycle recon, that dispatch then IS the cycle's one
|
||||
full recon; `specify` and `planner` reuse its report.
|
||||
- **Project feature-acceptance criterion:** declared in the project's
|
||||
`CLAUDE.md` — applied prospectively by `specify` (its Step 2), not here.
|
||||
|
||||
@@ -136,7 +136,10 @@ Assignment rule, in priority order:
|
||||
gating a cycle (adversarial lenses, plan recon, field/bench
|
||||
evidence, the in-loop quality review) — run `opus`. Opus is
|
||||
the documented strength for code review, debugging, and deep
|
||||
reasoning.
|
||||
reasoning. This rule covers the fieldtest's probe tier too:
|
||||
a probe narrows *scope*, never the model (a sonnet probe
|
||||
override was ratified and same-day retracted on review
|
||||
grounds — issue #35).
|
||||
3. **Mechanical scope** — tightly-scoped execution of a pre-made
|
||||
plan, recon, extraction, compliance-diffing run `sonnet`
|
||||
(near-opus coding quality at lower latency and cost).
|
||||
|
||||
@@ -104,6 +104,7 @@ The pipeline's nouns are fixed:
|
||||
| **cycle** | One round in the pipeline graph (`brainstorm → specify → planner → implement → audit → [fieldtest]`). NOT the top-level container. |
|
||||
| **iteration** | A sub-unit of a cycle. |
|
||||
| **milestone** | Tracker container spanning many cycles; closes only when complete AND functional (see `pipeline.md` § Milestone-close gate). |
|
||||
| **harvest sweep** | Batch shape draining accumulated settled single-issue backlog in one pass (see `pipeline.md` § Harvest sweep). |
|
||||
| **contract** | A single design-ledger entry. |
|
||||
|
||||
When a project declares a glossary (in its CLAUDE.md project facts),
|
||||
|
||||
+126
-8
@@ -92,7 +92,10 @@ hold:
|
||||
`bug` findings; `friction` / `spec_gap` findings resolved or
|
||||
ratified into the design ledger. A `clean` roll-up is honoured
|
||||
only with positive evidence the test actually ran — at least
|
||||
2 examples in the working tree (`examples_added`). A `clean`
|
||||
2 examples in the working tree (`examples_added`). A milestone
|
||||
fieldtest always runs **full tier** (the probe tier is
|
||||
per-cycle only — `../fieldtest/SKILL.md` § Two tiers: probe
|
||||
and full), so this floor is unconditional. A `clean`
|
||||
with fewer, over a milestone that touched user-visible surface,
|
||||
is itself a `spec_gap`, not a clean close (the agent owes the
|
||||
same floor — `fieldtest/agents/fieldtester.md` "What you DO NOT
|
||||
@@ -114,6 +117,111 @@ close stays a deliberate human / orchestrator act — the
|
||||
tracker's own milestone-close action (on Gitea, `tea milestone
|
||||
close`); no skill performs it automatically.
|
||||
|
||||
## Harvest sweep
|
||||
|
||||
A recognised batch shape for draining settled single-issue
|
||||
backlog. When small, settled items accumulate on the tracker
|
||||
(bundled fieldtest findings, absorbed leftovers, mechanical
|
||||
one-offs), one full pipeline cycle per item is structurally
|
||||
uneconomical; a harvest sweep drains them in one batch:
|
||||
|
||||
- the settled issue bodies stand in for spec and plan — each
|
||||
item is already ratified and bite-sized, so `specify` and
|
||||
`planner` are skipped by design (the exception is registered
|
||||
in § Skip rules);
|
||||
- the implementer works in sequential batches of 2-3 items —
|
||||
direct `implementer` dispatches (the `implement-loop` carrier
|
||||
needs a plan the sweep deliberately lacks), with no per-task
|
||||
review pair; the whole-diff review below replaces it;
|
||||
- the sweep closes with one **independent diff review over the
|
||||
whole sweep** — an ad-hoc fresh-context reviewer over the
|
||||
sweep's full diff, independent of the orchestrator and the
|
||||
implementers (the per-task reviewer carriers do not fit a
|
||||
whole-sweep diff) — then the mandatory `audit` close, and one
|
||||
compact `fieldtest` under its normal skip rules (a
|
||||
zero-surface sweep skips it).
|
||||
|
||||
The whole-diff review is **not removable**. Unlike a planned
|
||||
cycle, where the task is the unit of review (`planner`'s
|
||||
decomposition discipline), the sweep amortises review over the
|
||||
batch — and in the consumer-project sweep this shape was
|
||||
ratified from (recorded on issue #36), that one review caught
|
||||
the only substantive error, which originated from orchestrator
|
||||
+ fieldtest consensus, not from the implementers.
|
||||
|
||||
A sweep is orchestrator-decided at queue selection — a
|
||||
judgement call like the `fieldtest` / `docwriter` dispatches,
|
||||
not an arm of the selector cascade; the trigger is observed
|
||||
accumulation of settled single-issue items.
|
||||
|
||||
## Cycle recon
|
||||
|
||||
Recon fan-out is budgeted **per cycle, not per phase**. The field
|
||||
evidence behind the budget (issue #35, one bounded cycle): a broad
|
||||
Explore-type sweep grounding `specify` (151k tokens) plus the
|
||||
`plan-recon` dispatch grounding `planner` (99k tokens) overlapped
|
||||
~40 % — two full recon passes over the same territory, the single
|
||||
biggest obligatory per-cycle cost. This section is the single
|
||||
source of the budget, reuse, and delta rules; the skills point
|
||||
here and add only their own dispatch mechanics.
|
||||
|
||||
- **At most ONE full recon dispatch while a valid cycle recon
|
||||
exists** — the **cycle recon**. It is a `plan-recon` dispatch
|
||||
(`../planner/agents/plan-recon.md`) in the pre-spec carrier
|
||||
form, normally fired at `specify` Step 1 when the work enters
|
||||
code territory the orchestrator has not recently read; an
|
||||
ad-hoc dispatch `brainstorm` fired earlier in the same cycle is
|
||||
the cycle recon instead (subject to the coverage condition
|
||||
below). One report serves every downstream consumer: the spec's
|
||||
concrete code shapes and the plan's file-map. Normally that is
|
||||
one full dispatch per cycle; a further full dispatch is
|
||||
legitimate only via the fallback below.
|
||||
- **Broad exploration sweeps are not recon.** No Explore-type
|
||||
fan-out for spec, plan, or pre-dispatch grounding — grounding
|
||||
fan-out goes through the one cycle-recon dispatch. This binds
|
||||
the orchestrator in every mode, `/boss` included.
|
||||
- **Reuse over re-dispatch.** A downstream phase reuses the
|
||||
cycle-recon report while ALL of these hold:
|
||||
- *Freshness* — no commit since the dispatch has touched the
|
||||
paths the report maps (`git log <since-recon> -- <mapped
|
||||
paths>` is empty; commits elsewhere — fixtures, tracker
|
||||
side-effects — do not invalidate it). In a multi-iteration
|
||||
cycle, iteration 1's commits to mapped paths invalidate the
|
||||
report for iteration 2's planning.
|
||||
- *Quotability* — the orchestrator can still quote the report's
|
||||
Modify entries verbatim. A summarized memory of the report is
|
||||
not the report. (The designed reuse window is the standard
|
||||
same-session `specify` → `planner` handoff; across a session
|
||||
break the fallback below applies.)
|
||||
- *Coverage* (brainstorm-fired recons only) — the ratified
|
||||
design's territory appears in the report's file-map or
|
||||
existence-table rows, not merely as a one-line out-of-scope
|
||||
acknowledgement. Missing territory is a delta below; a design
|
||||
that pivoted wholly outside the mapped territory voids the
|
||||
dispatch as cycle recon, and `specify` fires the cycle recon
|
||||
afresh.
|
||||
- **Delta dispatches do not count against the budget.** A delta
|
||||
is a narrow re-dispatch whose `focus_hint` names the gap; its
|
||||
bounded contract is **Delta dispatch** in
|
||||
`../planner/agents/plan-recon.md`. Triggers:
|
||||
- the spec names a path or symbol that appears in neither the
|
||||
cycle recon's file-map nor its existence table;
|
||||
- MANDATORY: the iteration scope is a signature / variant /
|
||||
removal or content-pin-perturbing scope and the cycle recon
|
||||
carries no compile-driven site set for it — `planner` MUST
|
||||
delta-dispatch before planning that scope. (`specify` avoids
|
||||
this delta by naming the anticipated change shape in
|
||||
`recon_scope`, arming the compile-driven enumeration inside
|
||||
the cycle recon itself.)
|
||||
|
||||
Two deltas in one planner run is the smell that the iteration
|
||||
scope is too broad — re-read the spec.
|
||||
- **A full re-dispatch is the bounded fallback, not a failure**:
|
||||
no cycle recon exists (a direct tidy dispatch that skipped
|
||||
`specify`; a session break), or a reuse condition failed. The
|
||||
cost regression is capped at the pre-consolidation status quo —
|
||||
one full dispatch.
|
||||
|
||||
## Phase descriptions
|
||||
|
||||
### brainstorm
|
||||
@@ -147,7 +255,9 @@ the `grounding-check` gate, and takes sign-off — with review but no
|
||||
interview. Under the bold stance it decides every load-bearing fork it
|
||||
can *derive* an answer for and records the decision in the run's
|
||||
reference issue; it bounces to `brainstorm` only when a fork hangs on a
|
||||
pure user preference no source settles. A core node — the spec-production
|
||||
pure user preference no source settles. Its code grounding comes from
|
||||
the single cycle-recon dispatch (§ Cycle recon), never a broad
|
||||
exploration sweep. A core node — the spec-production
|
||||
gate before `planner` on every design path.
|
||||
|
||||
Outside `/boss` the sign-off is the user's. Under `/boss` the autonomous
|
||||
@@ -173,8 +283,10 @@ Hard-gate before implement. Produces a placeholder-free,
|
||||
bite-sized implementation plan in `docs/plans` (an ephemeral,
|
||||
git-ignored working file, shell-`rm`'d alongside the spec at cycle
|
||||
close — see `conventions.md` § Lifecycle) that the implement skill can
|
||||
execute task-by-task. Dispatches
|
||||
the plan-recon agent for read-only file-structure mapping.
|
||||
execute task-by-task. Its file-map normally comes from reusing the
|
||||
cycle-recon report (§ Cycle recon); it dispatches `plan-recon` only
|
||||
for that section's delta triggers, or in full when no valid cycle
|
||||
recon exists.
|
||||
|
||||
### implement
|
||||
|
||||
@@ -245,7 +357,9 @@ practices inside `implement`.
|
||||
|
||||
Optional. Orchestrator-dispatched after the audit closes clean
|
||||
on a cycle that touched user-visible surface. Picks 2-4 real-
|
||||
world tasks within the cycle's scope, implements them using
|
||||
world tasks within the cycle's scope (probe tier, for a
|
||||
single-axis surface delta: 1-2 tasks on that axis — see
|
||||
`../fieldtest/SKILL.md` § Two tiers: probe and full), implements them using
|
||||
only the design ledger and public examples (never the language's
|
||||
own implementation), runs the results, and writes a friction-
|
||||
and-bug spec.
|
||||
@@ -285,11 +399,15 @@ documents what the skill skips and under what conditions:
|
||||
gate before `planner`. `brainstorm` is the *optional* discovery stage
|
||||
before it: skipped when the design is already settled in the sources
|
||||
(the work enters through `specify` directly), run when a load-bearing
|
||||
decision is still open.
|
||||
decision is still open. The one batch exception is the harvest sweep
|
||||
(§ Harvest sweep): its settled issue bodies are the ratified spec
|
||||
source, so no new spec is written.
|
||||
- `planner` is never skipped at iteration start, except for the
|
||||
side paths that carry no prose plan: the bug-driven
|
||||
`debug → implement (mini)` path and the `compiler-driven` path
|
||||
(whose "plan" is the type-checker's enumeration of edit sites).
|
||||
`debug → implement (mini)` path, the `compiler-driven` path
|
||||
(whose "plan" is the type-checker's enumeration of edit sites),
|
||||
and the harvest sweep (whose items are already bite-sized —
|
||||
§ Harvest sweep).
|
||||
- `implement` is the iteration body; not skippable.
|
||||
- `audit` is mandatory at cycle close.
|
||||
- `debug` is mandatory RED-first for any observable bug — first in
|
||||
|
||||
+75
-11
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: fieldtest
|
||||
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope, implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
|
||||
description: Orchestrator-dispatched only, after audit closes clean (or with ratified drift only), when the orchestrator judges the iteration is complete and wants a field test. Picks 2-4 real-world tasks within the cycle's scope (probe tier, for a single-axis surface delta: 1-2 tasks on that axis), implements each as a downstream consumer of the project, runs them, and writes a friction-and-bug spec. The spec feeds the next plan as a reference. Implementer simulates a downstream user who has only the public interface — never the project's implementation source.
|
||||
---
|
||||
|
||||
# fieldtest — usability field test for a shipped cycle
|
||||
@@ -24,8 +24,8 @@ next to cycle-design specs under `docs/specs`, with a
|
||||
`fieldtest-` prefix in the slug.
|
||||
|
||||
The substantive process — read the design ledger + cycle spec
|
||||
+ recent iter/audit commit bodies, pick 2-4 real-world tasks
|
||||
per cycle axis, implement each as a downstream consumer, run
|
||||
+ recent iter/audit commit bodies, pick the dispatched tier's
|
||||
examples, implement each as a downstream consumer, run
|
||||
the result, classify findings, write the spec — lives in
|
||||
`agents/fieldtester.md`. That file also carries the spec
|
||||
template, the source-isolation discipline (no reading under
|
||||
@@ -61,6 +61,42 @@ here. The milestone fieldtest is skippable on the same terms as
|
||||
the per-cycle one, lifted to milestone level: a milestone whose
|
||||
entire scope is internal (no user-visible surface) is exempt.
|
||||
|
||||
## Two tiers: probe and full
|
||||
|
||||
Orthogonal to scope, the **per-cycle** fieldtest runs at one of
|
||||
two tiers. The orchestrator picks the tier at dispatch and
|
||||
carries it in the carrier's `tier` field — the field, its
|
||||
example bounds, and its report cap are defined authoritatively
|
||||
under **Carrier contract** in `agents/fieldtester.md` and
|
||||
deliberately not restated here:
|
||||
|
||||
- **Full** (the default; `tier` absent): one example per axis
|
||||
the cycle touched — the shape the rest of this file describes.
|
||||
- **Probe**: for a cycle whose user-visible delta is a **single
|
||||
narrow axis** — one flag, one diagnostic, one configuration
|
||||
key. The carrier's `axis_hints` then names exactly that one
|
||||
axis. Field evidence (issue #35): a full fieldtest cost ~89k
|
||||
tokens on a one-flag cycle even at minimal example count, yet
|
||||
that same dispatch found a real self-description bug — the
|
||||
probe tier keeps that signal at a fraction of the cost.
|
||||
|
||||
**Probe is a tier, not a skip.** The skip rules below are
|
||||
unchanged: a surface-touching cycle still runs its fieldtest;
|
||||
probe merely right-sizes the dispatch when the surface delta is
|
||||
one axis. A multi-axis cycle takes the full tier — when in
|
||||
doubt, full.
|
||||
|
||||
**Both tiers run on the agent's frontmatter model.** A probe
|
||||
dispatch changes scope, not model: its economy comes from the
|
||||
one-axis cut and the capped report, while finding classification
|
||||
stays on the opus judgement tier. (A `model: sonnet` probe
|
||||
override was ratified and same-day retracted on review grounds —
|
||||
field evidence is a judgement role per
|
||||
`../docs/agent-template.md` § model rule 2, and the probe cut
|
||||
lacks the deterministic gate the issue-#30 override rests on;
|
||||
recorded on issue #35.) The **milestone fieldtest always runs
|
||||
full tier** — its carrier takes no `tier` field.
|
||||
|
||||
## When to Use / Skipping
|
||||
|
||||
Triggers:
|
||||
@@ -116,11 +152,12 @@ carrier.
|
||||
## Dispatch
|
||||
|
||||
Dispatch `fieldtester` with the carrier from the Handoff
|
||||
Contract below. The agent picks 2-4 examples (one per axis
|
||||
the cycle touched; fewer loses the signal on variation, more
|
||||
overflows one readable report), implements them as downstream
|
||||
consumers, runs them, classifies findings, and writes the
|
||||
spec. The two artefact kinds part ways at commit: the **fixtures are
|
||||
Contract below. The agent picks its examples per the dispatched
|
||||
tier (bounds in the `tier` carrier row; on full, one per axis
|
||||
the cycle touched — fewer loses the signal on variation, more
|
||||
overflows one readable report), implements them as
|
||||
downstream consumers, runs them, classifies findings, and
|
||||
writes the spec. The two artefact kinds part ways at commit: the **fixtures are
|
||||
committed as normal code** (they are consumer test assets — suggested
|
||||
subject `fieldtest: <cycle> — <N> examples, <K> findings`), while the
|
||||
**fieldtest spec is a git-ignored working file that is never committed**.
|
||||
@@ -137,10 +174,10 @@ tree as the next cycle's input; it plays no part in the delete decision
|
||||
## Handoff Contract
|
||||
|
||||
`fieldtest` consumes (from orchestrator at cycle close) the carrier
|
||||
fields `cycle_id`, `cycle_scope`, `axis_hints`, and `commit_range`,
|
||||
defined authoritatively under **Carrier contract** in
|
||||
fields `cycle_id`, `cycle_scope`, `axis_hints`, `commit_range`, and
|
||||
`tier`, defined authoritatively under **Carrier contract** in
|
||||
`agents/fieldtester.md` (which also specifies the empty-`axis_hints`
|
||||
fallback).
|
||||
fallback and the absent-`tier` default).
|
||||
|
||||
`fieldtest` produces, for the orchestrator, the fields `spec_path`,
|
||||
`examples_added`, and `findings`, defined authoritatively under
|
||||
@@ -165,6 +202,30 @@ The orchestrator drives downstream:
|
||||
|
||||
`fieldtest` does NOT self-resolve.
|
||||
|
||||
**Tracker filing — bundle by default.** A fieldtest typically
|
||||
yields more findings than a cycle closes issues (3-5 vs 1-2 in
|
||||
the field evidence on issue #36), so filing each finding as its
|
||||
own issue grows the tracker structurally with every fieldtested
|
||||
cycle. When findings are filed onto the tracker rather than
|
||||
routed inline:
|
||||
|
||||
- Consolidate before filing: before creating **any** new issue,
|
||||
check whether an open issue on the same line of work can
|
||||
carry the scope — absorb via a comment there instead.
|
||||
- Related `friction` / `spec_gap` findings of one fieldtest go
|
||||
into **one collective issue**, not one issue per finding.
|
||||
- A real `bug` keeps its own issue — it is a forward-queue work
|
||||
item. A low-severity bug may wait in the collective issue
|
||||
instead.
|
||||
- The fieldtester's report stays per-finding — classification
|
||||
is the agent's contract and does not change. Bundling is this
|
||||
orchestrator-side filing rule only. The writes themselves
|
||||
follow `../issue/SKILL.md`, as every tracker write does.
|
||||
|
||||
When settled single-issue items accumulate anyway, the
|
||||
orchestrator drains them as a **harvest sweep** — the batch
|
||||
shape defined in `../docs/pipeline.md` § Harvest sweep.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Agent dispatched:** `agents/fieldtester.md` — carries the
|
||||
@@ -176,6 +237,9 @@ The orchestrator drives downstream:
|
||||
- **Closing gate it feeds:** `../docs/pipeline.md`
|
||||
§ Milestone-close gate — the milestone fieldtest's green
|
||||
roll-up is the functional leg of that gate.
|
||||
- **Batch drain it feeds:** `../docs/pipeline.md`
|
||||
§ Harvest sweep — the batch shape that drains accumulated
|
||||
settled single-issue items (bundled findings included).
|
||||
- **Cadence ordering:** fieldtest runs *before*
|
||||
`../docwriter/SKILL.md`; docwriter happens at a later,
|
||||
longer stability window.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: fieldtester
|
||||
description: Implements 2-4 real-world tasks against the project as a downstream consumer would, runs them, and reports friction, bugs, and spec gaps as a structured spec. Simulates a downstream user who has only the public interface (the project's design ledger, READMEs, public docs, and example corpus) — never the implementation source. Does NOT fix bugs.
|
||||
description: Implements 2-4 real-world tasks (probe tier: 1-2 on a single axis) against the project as a downstream consumer would, runs them, and reports friction, bugs, and spec gaps as a structured spec. Simulates a downstream user who has only the public interface (the project's design ledger, READMEs, public docs, and example corpus) — never the implementation source. Does NOT fix bugs.
|
||||
tools: Read, Edit, Write, Bash, Glob, Grep
|
||||
model: opus
|
||||
effort: xhigh
|
||||
@@ -71,10 +71,13 @@ skill references it rather than restating it.
|
||||
| `cycle_scope` | 1-3 sentences naming what shipped |
|
||||
| `axis_hints` | bullet list, one per axis to probe |
|
||||
| `commit_range` | `<prev-cycle-close>..HEAD` |
|
||||
| `tier` | `probe`, or absent (= full). Full: 2-4 examples, one per axis. Probe: the cycle's user-visible delta is a single narrow axis — 1-2 examples on exactly that axis, report capped at ~200 words; `axis_hints` MUST name exactly that one axis, so a probe carrier with zero or several `axis_hints` entries is malformed — return `NEEDS_CONTEXT`. Per-cycle dispatches only; the milestone variant below takes no `tier`. |
|
||||
|
||||
If `axis_hints` is empty, infer from the cycle's spec under
|
||||
`docs/specs` and the most recent iter commit bodies; if
|
||||
both are also empty, return `NEEDS_CONTEXT`.
|
||||
both are also empty, return `NEEDS_CONTEXT`. (This inference
|
||||
fallback is full-tier only — on a probe dispatch an empty
|
||||
`axis_hints` is malformed, per the `tier` row.)
|
||||
|
||||
### Milestone-scope variant
|
||||
|
||||
@@ -97,7 +100,9 @@ scenarios as the union of per-cycle axes; the per-cycle
|
||||
fieldtests already covered axis-local surface. There is no
|
||||
`axis_hints` fallback in this mode: if `milestone_promise` is
|
||||
empty, infer it from the milestone's spec / tracker entry, and
|
||||
if that is also empty, return `NEEDS_CONTEXT`.
|
||||
if that is also empty, return `NEEDS_CONTEXT`. The milestone
|
||||
carrier takes no `tier` field — a milestone fieldtest is always
|
||||
full tier.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
@@ -143,7 +148,10 @@ Each phase completes before the next starts.
|
||||
transformation pipeline. Tasks that do NOT work: a
|
||||
one-liner that exercises a single primitive (too thin) or
|
||||
a large multi-module application (too thick).
|
||||
3. Total: 2-4 examples.
|
||||
3. Total: 2-4 examples (full tier). On a `tier: probe` dispatch:
|
||||
1-2 examples, on exactly the single delta axis the carrier
|
||||
names — the tier bound comes from the carrier, never from
|
||||
you trimming a full dispatch.
|
||||
|
||||
### Phase 2 — Implement each example as a downstream consumer
|
||||
|
||||
@@ -280,7 +288,7 @@ End every report with exactly one of:
|
||||
|
||||
## Output format
|
||||
|
||||
At most 350 words, structured:
|
||||
At most 350 words (probe tier: 200), structured:
|
||||
|
||||
- **Status:** one of the four above.
|
||||
- **Examples:** one bullet per example: path + 1-line task
|
||||
@@ -321,7 +329,8 @@ fields are:
|
||||
- A `friction` finding without a 1-line recommendation.
|
||||
Every finding is actionable or it isn't a finding.
|
||||
- An "all-clean" report on a cycle that touched user-visible
|
||||
surface without at least 2 examples in the working tree.
|
||||
surface without at least 2 examples in the working tree
|
||||
(`tier: probe`: at least 1, on the delta axis).
|
||||
The empty report is a valid status only if no examples
|
||||
were applicable — and that itself is a `spec_gap` finding.
|
||||
|
||||
@@ -336,7 +345,7 @@ fields are:
|
||||
| "The design ledger is fuzzy on this corner, I'll pick the natural reading and proceed" | Pick the reading, RUN the example, AND record `spec_gap` with the reading you picked and why another reading was equally plausible. |
|
||||
| "Bug found — I'll just fix it now, faster than handing off to debug" | Fix-in-place violates the skill split. The fix lands in a separate, RED-tested commit via `debug` → `implement`. |
|
||||
| "Two examples both ran clean, no findings — short report" | A clean run is itself a finding (`working`). Record what was reached for, what diagnostic showed up when wrong, what was easy. Wins protect the feature from drift. |
|
||||
| "Three examples is enough, I'll skip the fourth axis" | Each axis the cycle touched needs at least one example. Skipping an axis silently turns the field test into a partial signal, which is worse than no signal because the orchestrator will read it as full coverage. |
|
||||
| "Three examples is enough, I'll skip the fourth axis" | On a full-tier dispatch, each axis the cycle touched needs at least one example. Skipping an axis silently turns the field test into a partial signal, which is worse than no signal because the orchestrator will read it as full coverage. (A probe dispatch is scoped to its single axis by the carrier — that is the orchestrator's cut, not yours to imitate on a full dispatch.) |
|
||||
|
||||
## Red Flags — STOP and re-read the public interface
|
||||
|
||||
|
||||
+27
-2
@@ -200,7 +200,13 @@ verdict means *adjudicate against the snapshot*, not proven malice.
|
||||
Renames do not trip it (the boundary diff runs `--no-renames`, so a
|
||||
renamed file's old path stays listed as a deletion). Untracked files
|
||||
sit outside the threat model (`git checkout -- <path>` cannot discard
|
||||
them). Cost: one extra sonnet/medium snapshot call per task of a
|
||||
them). Boundaries are HEAD-aware (issue #32): each snapshot records
|
||||
the HEAD its diff ran against, and a boundary pair straddling a HEAD
|
||||
move — the sanctioned commit-the-subset-then-resume repair — resets
|
||||
the guard with a concern instead of comparing, so a replayed
|
||||
pre-commit boundary never counts the committed subset as lost paths;
|
||||
in a single live run the same concern flags an agent that moved HEAD
|
||||
mid-run. Cost: one extra sonnet/medium snapshot call per task of a
|
||||
multi-task run; single-task and mini runs skip the guard. The
|
||||
detection details live in `workflows/implement-loop.js`, the single
|
||||
source.
|
||||
@@ -220,6 +226,7 @@ Workflow({ name: "implement-loop", args: {
|
||||
mode: "standard",
|
||||
iter_id: "<iter_id>",
|
||||
plan_path: "<path under docs/plans>",
|
||||
plan_sha256: "<sha256sum of the plan file>", // recompute at EVERY (re-)invocation (issue #32)
|
||||
task_range: [3, 8], // optional
|
||||
repo_root: "<abs path of the working tree>" // strongly recommended; effectively required in worktree sessions (issue #25)
|
||||
}})
|
||||
@@ -238,6 +245,14 @@ Workflow({ name: "implement-loop", args: {
|
||||
}})
|
||||
```
|
||||
|
||||
`plan_sha256` content-addresses the plan (issue #32): a resume replays
|
||||
agents cached per (prompt, opts), and with only the plan *path* in the
|
||||
extraction prompts an edit-and-resume would replay the stale pre-edit
|
||||
task text — the repair silently no-ops. Compute the hash fresh
|
||||
(`sha256sum <plan_path>`) at every invocation, including a
|
||||
`resumeFromRunId` re-invocation after a plan repair; never reuse the
|
||||
old value.
|
||||
|
||||
`repo_root` is stamped into every stage prompt as a `git -C` anchor:
|
||||
in worktree sessions a dispatched agent's cwd can resolve to the
|
||||
primary checkout and review the wrong tree (false `infra_blocked`
|
||||
@@ -332,7 +347,17 @@ orchestrator decides what to do with the dirty working tree:
|
||||
before re-running the workflow — its preflight clean-tree check
|
||||
counts the file as dirt. Either stash everything and re-run on a
|
||||
clean tree, or commit the known-good subset, then `rm BLOCKED.md`,
|
||||
then re-run.
|
||||
then re-run. The re-run is either a fresh invocation with
|
||||
`task_range` covering the remaining tasks, or a resume
|
||||
(`resumeFromRunId` on the same script) — for a resume, recompute
|
||||
`plan_sha256` from the edited plan so the extraction prompts
|
||||
change: the repaired task runs live, and a completed task replays
|
||||
from cache when its re-extracted text is byte-identical (the
|
||||
usual case; worst case it re-runs redundantly, never stale —
|
||||
issue #32). The discard guard
|
||||
tolerates the interim subset commit: boundaries are HEAD-aware,
|
||||
so the moved baseline resets the guard with a concern instead of
|
||||
a false lost-paths `BLOCKED`.
|
||||
- **Discard:** `git checkout -- .` to drop unstaged file changes;
|
||||
`git clean -fd` / `rm BLOCKED.md` plus anything else new. main HEAD
|
||||
does NOT move.
|
||||
|
||||
@@ -73,7 +73,18 @@
|
||||
// `stash create` (a bare `git stash` would move the tree; the following
|
||||
// boundary would catch that wipe, but only after the fact). Untracked
|
||||
// files sit outside the threat model (`git checkout -- <path>` cannot
|
||||
// discard them).
|
||||
// discard them). Boundaries are HEAD-aware (issue #32): each snapshot
|
||||
// records the HEAD its diff ran against, and a boundary pair straddling
|
||||
// a HEAD move — the sanctioned commit-the-subset-then-resume repair — is
|
||||
// reset-and-surfaced as a concern instead of compared, so a replayed
|
||||
// pre-commit boundary can never count the committed subset as lost paths.
|
||||
// • Resume content-addressing (issue #32): the carrier's plan_sha256 rides
|
||||
// every plan-reading prompt, so the documented PARTIAL repair (edit the
|
||||
// plan, commit the good subset, resume) re-runs the extraction instead
|
||||
// of replaying the stale pre-edit plan from cache. Downstream, a task
|
||||
// replays iff its re-extracted text is byte-identical — typically every
|
||||
// unchanged task; the failure mode is a redundant live re-run, never
|
||||
// stale text.
|
||||
// • The script has no filesystem or shell access of its own — every
|
||||
// working-tree mutation (code, tests) happens inside an agent() call;
|
||||
// the stats.json / BLOCKED.md contents are built in-script and ride
|
||||
@@ -99,6 +110,20 @@
|
||||
// mode: 'standard' | 'mini'
|
||||
// iter_id: e.g. 'ct.2.3' (standard) or 'bugfix-<symptom>' / 'feat-<behaviour>' (mini)
|
||||
// plan_path: (standard) path under docs/plans
|
||||
// plan_sha256: (standard) sha256 hex of the plan file's CURRENT bytes
|
||||
// (`sha256sum <plan_path>`). Stamped into every plan-reading
|
||||
// prompt as a content-address (issue #32): a resume replays
|
||||
// agents cached per (prompt, opts), and a prompt that
|
||||
// carries only the plan PATH replays a stale extraction
|
||||
// after a plan edit — the documented edit-and-resume repair
|
||||
// silently no-ops. With the bytes hashed into the prompt, a
|
||||
// plan edit re-runs the extraction; downstream, a task
|
||||
// replays from cache iff its re-extracted text is
|
||||
// byte-identical — typically every unchanged task, worst
|
||||
// case a redundant live re-run, never stale text. An
|
||||
// unchanged plan replays fully from cache. Recompute it at
|
||||
// EVERY (re-)invocation — a stale or constant value
|
||||
// re-opens the no-op resume.
|
||||
// task_range: (standard, optional) [from, to] inclusive, 1-based
|
||||
// red_test_path: (mini) absolute path to the RED test from debug / tdd
|
||||
// cause_summary: (mini) 1–2 sentences (debugger cause, or tdd spec_summary)
|
||||
@@ -160,6 +185,7 @@ const {
|
||||
mode = 'standard',
|
||||
iter_id,
|
||||
plan_path,
|
||||
plan_sha256,
|
||||
task_range = null,
|
||||
red_test_path,
|
||||
cause_summary,
|
||||
@@ -180,7 +206,7 @@ if (mode !== 'standard' && mode !== 'mini') {
|
||||
// The required carrier fields per mode (the header comment above is the
|
||||
// contract). Missing or blank fields fail fast by name — the loop must never
|
||||
// glob for a plan or brief an implementer with the literal string "undefined".
|
||||
const requiredByMode = mode === 'mini' ? { iter_id, red_test_path, cause_summary } : { iter_id, plan_path }
|
||||
const requiredByMode = mode === 'mini' ? { iter_id, red_test_path, cause_summary } : { iter_id, plan_path, plan_sha256 }
|
||||
const missingFields = Object.entries(requiredByMode)
|
||||
.filter(([, v]) => typeof v !== 'string' || !v.trim())
|
||||
.map(([k]) => k)
|
||||
@@ -218,6 +244,21 @@ const STANDING_NONE =
|
||||
'Schema-bound mechanical stage: NO standing reading (no CLAUDE.md, no git log) — do exactly what is instructed ' +
|
||||
'below, nothing else.'
|
||||
|
||||
// Plan content-address (issue #32): appended to EVERY prompt that reads the
|
||||
// plan file (plan-index, plan-extract-all, plan-extract:<id>). The resume
|
||||
// cache keys on (prompt, opts); with only the plan PATH in the prompt, an
|
||||
// edit-and-resume replays the pre-edit task text from cache and the repair
|
||||
// silently no-ops. Hashing the plan's bytes into each reading prompt makes a
|
||||
// plan edit re-run the extraction — and, because the extracted text flows
|
||||
// verbatim into the downstream per-task prompts, a task replays from cache
|
||||
// iff its re-extracted text is byte-identical (typically every unchanged
|
||||
// task; the failure mode is a redundant live re-run, never stale text).
|
||||
const PLAN_ADDRESS =
|
||||
mode === 'standard'
|
||||
? `\n\nPlan content-address: sha256 ${plan_sha256} — binds this extraction to the plan's current bytes for ` +
|
||||
'resume caching; informational only, nothing to verify or act on.'
|
||||
: ''
|
||||
|
||||
// Working-directory anchor (issue #25): in worktree sessions a dispatched
|
||||
// agent's cwd can resolve to the PRIMARY checkout instead of the worktree the
|
||||
// loop operates on — observed as a per-dispatch race producing false
|
||||
@@ -508,7 +549,7 @@ const STD_VERIFY_SCHEMA = {
|
||||
const SNAP_SCHEMA = {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['snapshot_sha', 'modified_paths'],
|
||||
required: ['snapshot_sha', 'modified_paths', 'head_sha'],
|
||||
properties: {
|
||||
snapshot_sha: {
|
||||
type: 'string',
|
||||
@@ -524,6 +565,12 @@ const SNAP_SCHEMA = {
|
||||
'`--no-renames` keeps a renamed file\'s old path listed as a deletion; untracked files are invisible ' +
|
||||
'here by design)',
|
||||
},
|
||||
head_sha: {
|
||||
type: 'string',
|
||||
description:
|
||||
'output of `git rev-parse HEAD` — the baseline modified_paths was diffed against; the guard compares two ' +
|
||||
'boundaries only when their baselines match (issue #32)',
|
||||
},
|
||||
},
|
||||
}
|
||||
// ---- Phase 0 — clean-tree preflight (mini mode only) --------------------
|
||||
@@ -614,7 +661,7 @@ if (mode === 'mini') {
|
||||
: 'ids of EVERY task') +
|
||||
' that the plan actually defines, each with its one-line title. Return ONLY ids and titles — do NOT include ' +
|
||||
'any task body or code block. Do not summarise, invent, reorder, or pad the list. Separately report ' +
|
||||
'`total_tasks`: the count of ALL tasks the entire plan defines, ignoring any range filter. Do not edit anything.',
|
||||
`\`total_tasks\`: the count of ALL tasks the entire plan defines, ignoring any range filter. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: 'plan-index', phase: 'Per-task loop', schema: TASK_INDEX_SCHEMA },
|
||||
)
|
||||
if (!index) {
|
||||
@@ -728,7 +775,7 @@ if (mode === 'mini') {
|
||||
const all = await agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRead the plan at ${plan_path}. Extract EXACTLY these tasks (1-based ids): ${expectedIds.join(', ')} — ` +
|
||||
'EACH as its VERBATIM block: the full task text as written, including any code blocks. Do not summarise, ' +
|
||||
'truncate, reorder, or merge tasks into each other. Return exactly those tasks, nothing else. Do not edit anything.',
|
||||
`truncate, reorder, or merge tasks into each other. Return exactly those tasks, nothing else. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: 'plan-extract-all', phase: 'Per-task loop', schema: TASK_TEXTS_SCHEMA },
|
||||
)
|
||||
const returned = all && Array.isArray(all.tasks) ? all.tasks : []
|
||||
@@ -746,7 +793,7 @@ if (mode === 'mini') {
|
||||
agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRead the plan at ${plan_path}. Extract EXACTLY task ${id} (1-based) as its VERBATIM block — ` +
|
||||
'the full task text as written, including any code blocks. Do not summarise, truncate, reorder, or merge in ' +
|
||||
'any other task. Return that one task only. Do not edit anything.',
|
||||
`any other task. Return that one task only. Do not edit anything.${PLAN_ADDRESS}`,
|
||||
{ model: 'sonnet', effort: 'medium', label: `plan-extract:${id}`, phase: 'Per-task loop', schema: TASK_TEXT_SCHEMA },
|
||||
).then((t) => ({ reqId: id, t })),
|
||||
),
|
||||
@@ -1011,37 +1058,63 @@ for (const task of tasks) {
|
||||
results.push(r)
|
||||
if (tasks.length > 1) {
|
||||
const snap = await agent(
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRecord a working-tree snapshot boundary WITHOUT changing anything. Run exactly these two ` +
|
||||
`${STANDING_NONE}${ANCHOR}\n\nRecord a working-tree snapshot boundary WITHOUT changing anything. Run exactly these three ` +
|
||||
'commands and report their output:\n' +
|
||||
`1. \`git stash create "iter ${iter_id} post-task ${task.id}"\` — prints a sha (a dangling commit; it ` +
|
||||
'touches neither HEAD, nor the index, nor the working tree). Report it as snapshot_sha; empty string if ' +
|
||||
'it printed nothing.\n' +
|
||||
'2. `git diff HEAD --name-only --no-renames` — report every printed path as modified_paths.\n' +
|
||||
'3. `git rev-parse HEAD` — report it as head_sha.\n' +
|
||||
'NEVER run bare `git stash` or `git stash push/pop/apply/drop` — those MOVE the changes. Do NOT edit, ' +
|
||||
'stage, or commit anything.',
|
||||
{ model: 'sonnet', effort: 'medium', label: `snapshot:${task.id}`, phase: 'Per-task loop', schema: SNAP_SCHEMA },
|
||||
)
|
||||
if (snap && prevBoundary) {
|
||||
const now = new Set(snap.modified_paths || [])
|
||||
const lost = [...prevBoundary.paths].filter((p) => !now.has(p))
|
||||
if (lost.length) {
|
||||
// A path that was HEAD-modified at the previous boundary is
|
||||
// HEAD-identical now: a file-level checkout/restore discarded
|
||||
// sibling-task work mid-run. Hard stop — the per-task DONE reports
|
||||
// and the tree have diverged, so nothing downstream is trustworthy.
|
||||
discard = {
|
||||
lost_paths: lost,
|
||||
modified_through_task: prevBoundary.taskId,
|
||||
discarded_during_task: task.id,
|
||||
recovery_snapshot: prevBoundary.sha || null,
|
||||
// HEAD-aware baseline (issue #32): each boundary's path set is a diff
|
||||
// AGAINST HEAD, so two boundaries are comparable only when they were
|
||||
// measured against the SAME HEAD. The sanctioned PARTIAL repair
|
||||
// (commit the known-good subset, then resume) moves HEAD between the
|
||||
// replayed boundaries and the fresh ones — comparing across that move
|
||||
// would report the entire committed subset as "lost" and hard-BLOCK
|
||||
// the first live task. On a baseline move the guard resets instead of
|
||||
// comparing, and surfaces the move: the loop itself never commits, so
|
||||
// a HEAD move inside a single live run means some agent committed
|
||||
// mid-run — worth eyes either way. A blank/missing sha falls through
|
||||
// to the comparison — a flaky report must not quietly disarm the guard.
|
||||
const prevHead = typeof prevBoundary.head === 'string' ? prevBoundary.head.trim() : ''
|
||||
const nowHead = typeof snap.head_sha === 'string' ? snap.head_sha.trim() : ''
|
||||
if (prevHead && nowHead && prevHead !== nowHead) {
|
||||
guardConcerns.push(
|
||||
`discard-guard baseline moved between task #${prevBoundary.taskId} and task #${task.id} ` +
|
||||
`(HEAD ${prevHead} -> ${nowHead}) — expected only on a resume after the orchestrator committed a ` +
|
||||
'known-good subset (the sanctioned PARTIAL repair); in a single live run with no interim commit it ' +
|
||||
'means an agent moved HEAD mid-run — investigate. The guard reset its baseline; this boundary pair ' +
|
||||
'went uncompared.',
|
||||
)
|
||||
} else {
|
||||
const now = new Set(snap.modified_paths || [])
|
||||
const lost = [...prevBoundary.paths].filter((p) => !now.has(p))
|
||||
if (lost.length) {
|
||||
// A path that was HEAD-modified at the previous boundary is
|
||||
// HEAD-identical now: a file-level checkout/restore discarded
|
||||
// sibling-task work mid-run. Hard stop — the per-task DONE reports
|
||||
// and the tree have diverged, so nothing downstream is trustworthy.
|
||||
discard = {
|
||||
lost_paths: lost,
|
||||
modified_through_task: prevBoundary.taskId,
|
||||
discarded_during_task: task.id,
|
||||
recovery_snapshot: prevBoundary.sha || null,
|
||||
}
|
||||
break
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!snap) guardConcerns.push(`discard-guard snapshot after task #${task.id} did not return — that boundary went unchecked`)
|
||||
// A non-returning snapshot agent skips the NEXT comparison (prevBoundary
|
||||
// null) rather than false-positiving against a stale boundary.
|
||||
prevBoundary = snap ? { taskId: task.id, sha: snap.snapshot_sha || null, paths: new Set(snap.modified_paths || []) } : null
|
||||
prevBoundary = snap
|
||||
? { taskId: task.id, sha: snap.snapshot_sha || null, paths: new Set(snap.modified_paths || []), head: snap.head_sha || null }
|
||||
: null
|
||||
}
|
||||
if (r.outcome === 'BLOCKED') break // do not skip ahead — task ordering dependencies are unknown
|
||||
}
|
||||
|
||||
+38
-13
@@ -46,6 +46,11 @@ May be skipped when:
|
||||
design path — `brainstorm`, `specify`, and `planner`.
|
||||
- The work is a trivial mechanical edit (per the project's
|
||||
CLAUDE.md "trivial mechanical edits" carve-out).
|
||||
- The work is a harvest sweep — settled single-issue backlog
|
||||
drained as a batch. The settled issue bodies stand in for the
|
||||
plan, and the per-task review pair is replaced by one
|
||||
whole-sweep diff review. See `../docs/pipeline.md`
|
||||
§ Harvest sweep.
|
||||
|
||||
**Never skipped** for a standard iteration within an active
|
||||
cycle. "I know the cycle, plan from memory" is exactly the
|
||||
@@ -75,20 +80,37 @@ of one iteration's worth of work onto a task list.
|
||||
|
||||
### Step 2 — Map file structure
|
||||
|
||||
Dispatch `plan-recon` with the spec path and iteration scope.
|
||||
Read the returned file-map. Lift the relevant entries into
|
||||
the plan's "Files this plan creates or modifies" section; the
|
||||
The file-map normally already exists: the **cycle recon** report
|
||||
(`../docs/pipeline.md` § Cycle recon), dispatched during `specify`
|
||||
Step 1 or ad-hoc by `brainstorm`. Reuse it while that section's
|
||||
reuse conditions — freshness, quotability, coverage — hold. Lift
|
||||
the relevant entries
|
||||
into the plan's "Files this plan creates or modifies" section; the
|
||||
orchestrator may add or amend entries that recon missed.
|
||||
|
||||
Dispatch carrier: see the **Carrier contract** table in
|
||||
`agents/plan-recon.md` (single source for `spec_path`,
|
||||
`iteration_scope`, `focus_hint`).
|
||||
Dispatch `plan-recon` only:
|
||||
|
||||
Recon does NOT decide decomposition. Two recon dispatches per
|
||||
planner run is unusual but legitimate (e.g. one for the main
|
||||
spec sections, one with a sharper `focus_hint` for a tricky
|
||||
subsystem). Three or more is a smell — re-read the spec; the
|
||||
iteration scope is probably too broad.
|
||||
- as a **delta**, on the § Cycle recon delta triggers — a narrow
|
||||
dispatch whose `focus_hint` names the gap, bounded by the
|
||||
**Delta dispatch** contract in `agents/plan-recon.md`;
|
||||
- in **full**, when no valid cycle recon exists (a direct tidy
|
||||
dispatch that skipped `specify`, a session break, or a reuse
|
||||
condition failed).
|
||||
|
||||
Either way the orchestrator does NOT do recon in-context: reuse
|
||||
means reusing a prior *dispatch's* report, never the
|
||||
orchestrator's own tree-walking.
|
||||
|
||||
Dispatch carrier: see the **Carrier contract** tables in
|
||||
`agents/plan-recon.md` (single source for `spec_path`,
|
||||
`iteration_scope`, `focus_hint`; the pre-spec `sources` form is
|
||||
`specify`'s dispatch, not planner's).
|
||||
|
||||
Recon does NOT decide decomposition. Two *delta* dispatches in
|
||||
one planner run is the smell that the iteration scope is too
|
||||
broad — re-read the spec. A second *full* dispatch in a cycle
|
||||
that already has a valid cycle recon is a § Cycle recon budget
|
||||
violation, not a judgement call.
|
||||
|
||||
The plan's "Files this plan creates or modifies" section
|
||||
keeps the same shape:
|
||||
@@ -277,5 +299,8 @@ shell-`rm`'d alongside the spec at cycle close by `audit` (see
|
||||
task-by-task.
|
||||
- **Agents dispatched:**
|
||||
- `agents/plan-recon.md` — Step 2, file-structure mapping
|
||||
(read-only). Mandatory; the orchestrator does NOT do
|
||||
recon in-context.
|
||||
(read-only). Normally reused from the cycle recon
|
||||
(`../docs/pipeline.md` § Cycle recon); dispatched here only
|
||||
for deltas, or in full when no valid cycle recon exists. The
|
||||
file-map always comes from a dispatch's report; the
|
||||
orchestrator does NOT do recon in-context.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: plan-recon
|
||||
description: Read-only code-and-doc-recon agent for plan writing. Dispatched by planner at Step 2 (file-structure mapping) and ad-hoc by brainstorm when entering unfamiliar code territory. Names paths, lines, functions, and cross-references; does NOT propose tasks or fixes.
|
||||
description: Read-only code-and-doc-recon agent. Normally dispatched once per cycle as the cycle recon (specify Step 1 in pre-spec form, or ad-hoc by brainstorm) and reused downstream; planner re-dispatches only for deltas or in full when no valid cycle recon exists. Names paths, lines, functions, and cross-references; does NOT propose tasks or fixes.
|
||||
tools: Read, Glob, Grep, Bash
|
||||
model: opus
|
||||
effort: xhigh
|
||||
@@ -11,11 +11,15 @@ effort: xhigh
|
||||
> **Violating the letter of these rules is violating the spirit.**
|
||||
|
||||
You are the **plan-recon agent** for this project. You are
|
||||
dispatched by the `planner` skill at Step 2 of every
|
||||
iteration plan, and ad-hoc by the `brainstorm` skill when a
|
||||
cycle enters code territory the orchestrator has not recently
|
||||
read. **You do not write tasks. You do not propose fixes.
|
||||
You produce a file-map.**
|
||||
normally dispatched **once per cycle** — the cycle recon
|
||||
(`../../docs/pipeline.md` § Cycle recon): by the `specify`
|
||||
skill at Step 1 in the pre-spec form below, or ad-hoc by the
|
||||
`brainstorm` skill, whenever the cycle enters code territory
|
||||
the orchestrator has not recently read. Your report is then
|
||||
reused downstream; the `planner` skill re-dispatches you only
|
||||
for a narrow delta (a sharp `focus_hint` naming a gap) or in
|
||||
full when no valid cycle recon exists. **You do not write
|
||||
tasks. You do not propose fixes. You produce a file-map.**
|
||||
|
||||
## What this role is for
|
||||
|
||||
@@ -26,7 +30,10 @@ naming which functions and cross-references. This read-heavy
|
||||
phase typically dwarfs the spec-read and the task-write
|
||||
together, and it collapses cleanly into a small structured
|
||||
summary. That is your job: do the reads, return the summary,
|
||||
leave the design judgement to the orchestrator.
|
||||
leave the design judgement to the orchestrator. In the
|
||||
pre-spec form the same read runs one phase earlier — mapping
|
||||
the design sources — and its report serves `specify`'s
|
||||
concrete code shapes first, then `planner`'s file-map.
|
||||
|
||||
The temptation is to also draft the tasks. Do not. The
|
||||
orchestrator owns decomposition; your authority ends at
|
||||
@@ -42,8 +49,8 @@ In addition:
|
||||
|
||||
- If the project has a design ledger (its CLAUDE.md project
|
||||
facts), walk it to the contracts the carrier
|
||||
flags or that the spec touches; do not skim sections the
|
||||
spec does not touch.
|
||||
flags or that the spec (pre-spec: the sources) touches; do
|
||||
not skim sections it does not touch.
|
||||
- `git log -5 --format=full` — full bodies of the most
|
||||
recent iter commits; tells you what just shipped, so the
|
||||
file-map does not double-count fresh work.
|
||||
@@ -53,19 +60,78 @@ In addition:
|
||||
|
||||
## Carrier contract — what the controller hands you
|
||||
|
||||
This table is the authoritative definition of the carrier fields
|
||||
(including which are mandatory and the BLOCKED-on-missing rule
|
||||
below); the dispatching skill references it rather than restating
|
||||
it.
|
||||
These tables are the authoritative definition of the carrier fields
|
||||
(including which are mandatory and the BLOCKED-on-missing rules
|
||||
below); the dispatching skills reference them rather than restating
|
||||
them. A dispatch carries **exactly one** of `spec_path` (the
|
||||
spec-anchored form) or `sources` (the pre-spec form below).
|
||||
|
||||
| Field | Content |
|
||||
|-------|---------|
|
||||
| `spec_path` | Path to the spec under `docs/specs` (mandatory) |
|
||||
| `iteration_scope` | Which sections of the spec this dispatch covers (mandatory) |
|
||||
| `focus_hint` | Optional: orchestrator may flag a specific subsystem or symbol to prioritise |
|
||||
| `spec_path` | Path to the spec under `docs/specs` (mandatory in the spec-anchored form) |
|
||||
| `iteration_scope` | Which sections of the spec this dispatch covers (mandatory with `spec_path`) |
|
||||
| `focus_hint` | Optional: orchestrator may flag a specific subsystem or symbol to prioritise. On a delta dispatch it is mandatory and names the gap (see **Delta dispatch** below). |
|
||||
|
||||
If `spec_path` does not resolve or `iteration_scope` is empty,
|
||||
return `BLOCKED` with a one-line reason.
|
||||
In the spec-anchored form: if `spec_path` does not resolve or
|
||||
`iteration_scope` is empty, return `BLOCKED` with a one-line
|
||||
reason.
|
||||
|
||||
### Pre-spec dispatch (cycle recon)
|
||||
|
||||
When dispatched **before the spec exists** — by `specify` Step 1
|
||||
or ad-hoc by `brainstorm`, as the cycle recon
|
||||
(`../../docs/pipeline.md` § Cycle recon) — the carrier is anchored
|
||||
to the design sources instead:
|
||||
|
||||
| Field | Content |
|
||||
|-------|---------|
|
||||
| `sources` | The design sources: workspace paths (design docs, ledger contracts) plus prose the dispatching skill pastes verbatim (issue bodies, the ratified design narrative). You read the named paths and the pasted prose; you do NOT open the tracker or fish for further context. |
|
||||
| `recon_scope` | Which territory of the sources this dispatch covers (mandatory) — the pre-spec analogue of `iteration_scope`. When the sources settle the anticipated change shape (a signature change, a variant add / remove, a symbol removal, a content-pin perturbation), the dispatching skill names it here — that arms Step 5's compile-driven enumeration in this dispatch, so `planner` needs no mandatory delta for it later. |
|
||||
| `focus_hint` | Optional, as above |
|
||||
|
||||
If `sources` is empty (no path resolves and no prose was pasted)
|
||||
or `recon_scope` is empty, return `BLOCKED` with a one-line
|
||||
reason.
|
||||
|
||||
In this form, every rule below that reads "the spec" applies to
|
||||
the sources — specifically:
|
||||
|
||||
- Process step 2 ("Read the spec in full") → read the pasted
|
||||
prose in full and every named source path in full.
|
||||
- Process steps 4, 5, and 7 ("the spec references / implies") →
|
||||
what the sources reference or imply.
|
||||
- The scope triggers in step 5 and the omission rule after the
|
||||
output format key on `recon_scope` instead of
|
||||
`iteration_scope`.
|
||||
- Step 6's "Spec-named-path existence table" becomes the
|
||||
**source-named**-path existence table — same verification
|
||||
commands, same output section (keep the section heading; note
|
||||
`(pre-spec: source-named)` under it).
|
||||
- Step 8's priority filter: source territory in `recon_scope`
|
||||
gets full coverage; territory outside it, a one-line
|
||||
acknowledgement.
|
||||
- The `NEEDS_CONTEXT` status row reads: `recon_scope` names
|
||||
territory the sources do not contain.
|
||||
- The Rationalisations rows "Spec is ambiguous on point X" and
|
||||
"The spec says path X does not exist" and the Red-Flag entries
|
||||
that name the spec apply to the sources unchanged.
|
||||
|
||||
### Delta dispatch
|
||||
|
||||
Either form may arrive as a **delta** — the dispatching skill
|
||||
marks it so in the carrier, and `focus_hint` (mandatory on a
|
||||
delta) names the gap: a path or symbol the cycle recon missed, or
|
||||
a compile-driven site set it lacks
|
||||
(`../../docs/pipeline.md` § Cycle recon). A delta inverts the
|
||||
priority filter:
|
||||
|
||||
- Read the spec / sources only as far as needed to anchor the
|
||||
gap; do NOT re-map territory the cycle recon already covered.
|
||||
- Produce only the output sections the gap touches: the file-map
|
||||
rows, the compile-driven set, and existence-table rows for the
|
||||
gap's paths. The existence table's always-required rule applies
|
||||
to the gap's paths only.
|
||||
- Output budget ≤500 tokens.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
@@ -81,11 +147,13 @@ never to fix or write.
|
||||
|
||||
## The Process
|
||||
|
||||
1. Read the carrier. Confirm `spec_path` resolves and
|
||||
`iteration_scope` is non-empty. If either fails, return
|
||||
1. Read the carrier. Confirm the mandatory fields of its form
|
||||
(spec-anchored: `spec_path` resolves and `iteration_scope` is
|
||||
non-empty; pre-spec: `sources` and `recon_scope` per the
|
||||
Pre-spec dispatch section). If a check fails, return
|
||||
`BLOCKED`.
|
||||
2. Read the spec in full. Note every reference to a path,
|
||||
type, function, or invariant.
|
||||
2. Read the spec in full (pre-spec: the sources in full). Note
|
||||
every reference to a path, type, function, or invariant.
|
||||
3. Read the standing list in order, plus the design ledger's
|
||||
relevant contracts (if the project has one).
|
||||
4. For each path or symbol the spec references, run
|
||||
|
||||
@@ -1,228 +0,0 @@
|
||||
---
|
||||
name: postmortem
|
||||
description: Use after a working session (or a `/boss` run) to grade how well the toolchain and the dispatched agents actually performed, and how many tokens the run spent. Reads the session's own Claude Code transcript + subagent logs + Workflow run logs, aggregates tokens/tool-health deterministically via a helper script, and writes a scored retrospective. Utility skill, user-invoked or orchestrator-invoked at run close; never a pipeline gate.
|
||||
---
|
||||
|
||||
# postmortem — grade the run
|
||||
|
||||
> **Violating the letter of these rules is violating the spirit.**
|
||||
|
||||
## Overview
|
||||
|
||||
Every session leaves a detailed flight recorder behind — the
|
||||
JSONL transcript under `~/.claude/projects/<slug>/<session>.jsonl`,
|
||||
one sidechain log per dispatched subagent under
|
||||
`<session>/subagents/`, and for every Workflow run a per-agent
|
||||
summary in `<session>/workflows/wf_*.json` with raw per-agent
|
||||
transcripts under `<session>/subagents/workflows/<runId>/`. Nobody
|
||||
reads it. So the same expensive
|
||||
habits repeat: a subagent fleet that bounces `BLOCKED`, a tool
|
||||
loop with a 30% error rate, a cache-cold run that re-sends the full
|
||||
input on every turn. This skill turns that recorder into a graded
|
||||
retrospective: **how well did the toolchain work, how effective
|
||||
were the agents, and how many tokens did it spend.**
|
||||
|
||||
It is descriptive, not corrective. It produces a report and, where
|
||||
warranted, recommends filing tracker issues — it does not change
|
||||
code, baselines, or the pipeline.
|
||||
|
||||
## When to Use / Skipping
|
||||
|
||||
Invoke:
|
||||
|
||||
- After a substantial interactive session, when you want to know
|
||||
where the time and tokens went.
|
||||
- At the close of a `/boss` autonomous run, to grade the
|
||||
orchestrator's dispatch choices.
|
||||
- When a run *felt* token-heavy or thrashy and you want the numbers.
|
||||
|
||||
Not a pipeline phase. Never a hard-gate. It reads logs; it never
|
||||
blocks a cycle.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NUMBERS COME FROM THE SCRIPT, NEVER FROM MEMORY OR EYEBALLING THE LOG
|
||||
REPORT TOKENS AS RAW COUNTS — NEVER CONVERT THEM TO A COST OR PRICE
|
||||
NEVER READ ~/.ionos_token, .credentials.json, OR ANY SECRET FILE INTO CONTEXT
|
||||
A GRADE WITHOUT A CITED METRIC IS AN OPINION — DROP IT
|
||||
DEDUP TOKEN USAGE BY requestId — THE SCRIPT DOES THIS; DO NOT RE-SUM THE RAW LOG BY HAND
|
||||
```
|
||||
|
||||
## What the data does and does not contain
|
||||
|
||||
Read before interpreting, so the report never overclaims:
|
||||
|
||||
- **Present:** per-request `usage` (input / output / cache-creation /
|
||||
cache-read / server-tool counts), `model`, `requestId`,
|
||||
timestamps, every `tool_use` with its input, every `tool_result`
|
||||
with the authoritative `is_error` flag, skill invocations,
|
||||
slash commands, and a full separate transcript + `agentType` /
|
||||
`description` for each subagent. Workflow runs additionally carry a
|
||||
per-agent summary (stage label, model, a live token counter) in
|
||||
`workflows/wf_*.json`; the counter under-tracks vs. the deduped raw
|
||||
transcripts (~11% in the corpus behind issue #28), so the script
|
||||
treats the raw transcripts as authoritative and reports the counter
|
||||
only as `reported_total_tokens` for cross-checking.
|
||||
- **Absent:** `costUSD` is always `null`; `durationMs` is always
|
||||
`null`. There is no metered cost in the log, so this skill does
|
||||
not compute one — it reports **raw token counts** (kept split by
|
||||
class: input / output / cache-creation / cache-read, plus a
|
||||
`total_tokens` sum). **Wall-clock is derived** from timestamp
|
||||
deltas and the report says so.
|
||||
|
||||
The streamed transcript repeats the same `requestId` across
|
||||
several assistant lines with an identical `usage` object. The
|
||||
helper script dedups per `requestId`; never sum the raw lines.
|
||||
|
||||
## The Process
|
||||
|
||||
### Step 1 — Run the aggregator
|
||||
|
||||
```
|
||||
python3 <skill-dir>/scripts/postmortem.py [--session <id>] [--cwd <project>]
|
||||
```
|
||||
|
||||
Defaults to the newest transcript in the current project's log
|
||||
dir — i.e. the session you are in (it will be flagged `active`,
|
||||
meaning totals are partial). To grade a finished run, pass its
|
||||
`--session <uuid>`.
|
||||
|
||||
The default log dir is derived from cwd (the path with `/` replaced
|
||||
by `-`), so a session run inside a git worktree resolves to a
|
||||
*different* slug than the primary clone, and after exiting a worktree
|
||||
the default again follows the current cwd. To grade a worktree run
|
||||
from the primary clone, pass `--project-dir
|
||||
~/.claude/projects/<worktree-slug>` (or `--file <path>` to bypass slug
|
||||
derivation entirely).
|
||||
|
||||
The script emits one JSON object on stdout. That JSON — not the
|
||||
raw log — is your evidence base.
|
||||
|
||||
### Step 2 — Read the JSON and grade three axes
|
||||
|
||||
Grade each axis on the cited numbers. A grade with no number
|
||||
behind it is an opinion; drop it.
|
||||
|
||||
1. **Token spend & efficiency** — `totals.total_tokens`, the
|
||||
main-vs-subagent split, the token breakdown (output share is the
|
||||
write-heavy signal), `cache_hit_ratio` (the dominant efficiency
|
||||
lever: below ~0.7 on a long session means cache-cold turns
|
||||
re-sending the full input), `server_tools` (web_search /
|
||||
web_fetch counts).
|
||||
2. **Toolchain health** — `tools.error_ratio` and `interrupted`
|
||||
(the authoritative `is_error` signal), the tool mix (a Read /
|
||||
Edit / Bash imbalance hints at thrashing or re-reading),
|
||||
`slash_commands` such as `/compact` (context pressure).
|
||||
3. **Agent effectiveness** — per subagent: `terminal_status`
|
||||
(`DONE` / `DONE_WITH_CONCERNS` / `PARTIAL` / `BLOCKED` /
|
||||
`NEEDS_CONTEXT` / `unknown`), `total_tokens`, and `requests`. A
|
||||
fleet that burned real tokens to return `BLOCKED` is the headline
|
||||
finding. Compare each agent's token spend to what it delivered.
|
||||
For Workflow runs, grade per stage instead: `workflow_totals` and
|
||||
each run's `stages` table (label-prefix grouping, e.g. all
|
||||
`impl:N` under `impl`) show where the run's spend concentrated;
|
||||
`agents_untracked > 0` flags retried/dropped attempts. Schema-bound
|
||||
workflow stages carry no `terminal_status` — the workflow script's
|
||||
own aggregation already judged them.
|
||||
|
||||
### Step 3 — Write the report
|
||||
|
||||
Write a Markdown report to the configured destination (see
|
||||
**Output destination**). Structure it as **Output format** below.
|
||||
Lead with the scorecard, support every grade with a number from
|
||||
the JSON, and keep the prose tight — this is a retrospective, not
|
||||
a transcript replay.
|
||||
|
||||
### Step 4 — Surface, don't fix
|
||||
|
||||
Translate the sharpest findings into recommendations. Where a
|
||||
finding is a concrete, trackable defect (a subagent that reliably
|
||||
bounces `BLOCKED`, a tool loop with a runaway error rate),
|
||||
recommend filing it via the `issue` skill — name it, but do not
|
||||
file it yourself unless the user asks. This skill ends at the
|
||||
report.
|
||||
|
||||
## Output destination
|
||||
|
||||
Resolve in this order:
|
||||
|
||||
1. `--out <path>` if the invoker passed one.
|
||||
2. The project's post-mortem directory, if it has one (its CLAUDE.md
|
||||
project facts).
|
||||
3. Default: `docs/postmortems/<session-id-short>-<yyyy-mm-dd>.md`
|
||||
under the project root, creating the directory if needed.
|
||||
|
||||
Also print a ≤200-word summary to the chat so the headline grade
|
||||
and token spend are visible without opening the file.
|
||||
|
||||
## Output format
|
||||
|
||||
```
|
||||
# Post-mortem — <session-id-short> (<date>)
|
||||
|
||||
**Scorecard**
|
||||
| Axis | Grade | Headline metric |
|
||||
|------|-------|-----------------|
|
||||
| Token spend & efficiency | A–F | <total_tokens> total, cache hit <ratio> |
|
||||
| Toolchain health | A–F | <error_ratio>, <n> interrupts |
|
||||
| Agent effectiveness | A–F | <k>/<n> agents DONE, <tokens> on subagents |
|
||||
|
||||
**Token spend**
|
||||
<main vs subagent split; token breakdown by class (input / output /
|
||||
cache-creation / cache-read); the one class that dominates and why>
|
||||
|
||||
**Toolchain**
|
||||
<tool mix, error ratio + what the errors were, context-pressure
|
||||
signals like /compact>
|
||||
|
||||
**Agents**
|
||||
<per-subagent line: type — status — <total_tokens> — one-clause
|
||||
verdict. Omit if no subagents were dispatched.>
|
||||
|
||||
**Recommendations**
|
||||
<≤5 bullets. Each ties to a cited metric. Mark any that warrant
|
||||
an `issue`.>
|
||||
|
||||
_Token counts are raw usage. Wall-clock is derived from timestamp
|
||||
deltas, not metered._
|
||||
```
|
||||
|
||||
If the session is flagged `active`, add one line at the top:
|
||||
"⚠️ Session still active — figures are partial."
|
||||
|
||||
## Common Rationalisations
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "I'll just skim the transcript and tally the tokens" | The transcript repeats usage per streamed line under a shared requestId. Hand-summing double-counts. Run the script. |
|
||||
| "Let me convert the tokens to a rough dollar figure" | Don't. There is no metered cost in the log and prices vary by plan; report raw token counts and let the reader price them if they want. |
|
||||
| "Cache hit ratio of 0.5 is fine" | On a long session it means half the input was re-sent uncached. That is the single biggest efficiency lever — grade it. |
|
||||
| "All agents returned, so effectiveness is an A" | "Returned" ≠ "succeeded". A `BLOCKED` agent that burned tens of thousands of tokens is a failure that still spent the budget. Read `terminal_status`. |
|
||||
| "Let me also peek at the IONOS token to check the endpoint" | Never. Secret files are out of bounds — see the Iron Law and the user's global rules. |
|
||||
| "I found a broken agent, I'll fix its prompt now" | Out of scope. This skill reports and recommends; the fix is a separate, tracked change. |
|
||||
|
||||
## Red Flags — STOP
|
||||
|
||||
- Quoting any token number that did not come out of the script's
|
||||
JSON.
|
||||
- Converting token counts into a cost or dollar figure — report
|
||||
raw counts only.
|
||||
- Reading any file under `~/.claude` other than the session
|
||||
transcript and its subagent logs — and never any credential or
|
||||
token file.
|
||||
- Editing code, baselines, or pipeline config "while you're in
|
||||
there". Report only.
|
||||
- Grading an axis with adjectives instead of the cited metric.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Helper script:** `scripts/postmortem.py` — the deterministic
|
||||
aggregator. All numbers originate here.
|
||||
- **Hand-off target:** the `issue` skill, when a finding is a
|
||||
trackable defect worth filing.
|
||||
- **Report destination (optional):** the project's post-mortem
|
||||
directory, if its CLAUDE.md project facts name one.
|
||||
- **Data-source note:** this skill reads only the current
|
||||
session's own transcript and subagent logs. It does not crawl
|
||||
other projects or other sessions (single-session by design).
|
||||
@@ -1,501 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic aggregator for a single Claude Code session.
|
||||
|
||||
Reads the main transcript JSONL for one session plus its subagent
|
||||
sidechain logs and its Workflow-substrate run logs (workflows/wf_*.json
|
||||
+ subagents/workflows/<runId>/agent-*.jsonl), and emits a compact JSON
|
||||
summary on stdout that the post-mortem SKILL interprets into a written
|
||||
report.
|
||||
|
||||
Why a script and not skill prose: transcripts run to >1 MB and a
|
||||
session can spawn dozens of subagent logs. Token accounting has two
|
||||
traps that only deterministic code gets right every time:
|
||||
|
||||
1. Streamed assistant lines repeat the SAME requestId with the SAME
|
||||
usage object. Counting per-line double-counts tokens. We dedup
|
||||
per requestId (max per field within a request, then sum across
|
||||
requests).
|
||||
2. The transcript carries NO durationMs (always null), so wall-clock
|
||||
is derived from timestamp deltas. The numbers reported here are
|
||||
raw token counts, not a dollar estimate — different token classes
|
||||
(input / output / cache) are reported separately rather than
|
||||
collapsed into a single priced figure.
|
||||
|
||||
Stdlib only. No network. Read-only against ~/.claude.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
_TOK_ALT = "DONE_WITH_CONCERNS|NEEDS_CONTEXT|BLOCKED|PARTIAL|DONE" # longest-first
|
||||
# Authoritative signal: the implement end-report leads with a structured
|
||||
# "Status: DONE" line (optionally wrapped in markdown bold or a list
|
||||
# marker). Anchoring here is what stops the scan from flipping a shipped
|
||||
# DONE run to BLOCKED on the strength of the report's own template lines
|
||||
# (`BLOCKED file: BLOCKED.md`, `Blocked detail:`) or prose narrating a
|
||||
# surmounted blocker. See issue #3.
|
||||
STATUS_LINE_RE = re.compile(
|
||||
r"^[\s>*#\-]*\**\s*status\s*\**\s*[:\-]\s*\**\s*(" + _TOK_ALT + r")\b",
|
||||
re.IGNORECASE | re.MULTILINE)
|
||||
# Fallback for agents that emit a bare status token as their final line
|
||||
# (no "Status:" prefix). Requires the token to BE the line — excludes
|
||||
# substrings like "BLOCKED.md" or "BLOCKED file:" buried in prose.
|
||||
STATUS_STANDALONE_RE = re.compile(
|
||||
r"^[\s>*#\-]*\**\s*(" + _TOK_ALT + r")\**\s*$",
|
||||
re.IGNORECASE | re.MULTILINE)
|
||||
|
||||
|
||||
def parse_ts(s):
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(s.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def empty_tokens():
|
||||
return {"input": 0, "output": 0, "cache_creation": 0, "cache_read": 0,
|
||||
"cache_creation_5m": 0, "cache_creation_1h": 0}
|
||||
|
||||
|
||||
def token_total(tokens):
|
||||
"""Raw count of every token the session moved. cache_creation_5m/1h are
|
||||
a sub-split of cache_creation, so they are excluded to avoid double
|
||||
counting."""
|
||||
return (tokens["input"] + tokens["output"]
|
||||
+ tokens["cache_creation"] + tokens["cache_read"])
|
||||
|
||||
|
||||
def add_tokens(acc, usage):
|
||||
"""Add one request's deduped usage into an accumulator."""
|
||||
acc["input"] += usage.get("input_tokens", 0) or 0
|
||||
acc["output"] += usage.get("output_tokens", 0) or 0
|
||||
cc = usage.get("cache_creation_input_tokens", 0) or 0
|
||||
acc["cache_creation"] += cc
|
||||
acc["cache_read"] += usage.get("cache_read_input_tokens", 0) or 0
|
||||
detail = usage.get("cache_creation") or {}
|
||||
f5 = detail.get("ephemeral_5m_input_tokens")
|
||||
f1 = detail.get("ephemeral_1h_input_tokens")
|
||||
if f5 is None and f1 is None:
|
||||
acc["cache_creation_5m"] += cc # no split available -> assume 5m
|
||||
else:
|
||||
acc["cache_creation_5m"] += f5 or 0
|
||||
acc["cache_creation_1h"] += f1 or 0
|
||||
|
||||
|
||||
def iter_lines(path):
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
|
||||
def aggregate_transcript(path):
|
||||
"""Return token totals, model mix, and per-request count for one JSONL
|
||||
transcript (main or subagent). Dedups usage by requestId."""
|
||||
# requestId -> {field: max seen}, plus model per request.
|
||||
req_usage = {}
|
||||
req_model = {}
|
||||
for rec in iter_lines(path):
|
||||
if rec.get("type") != "assistant":
|
||||
continue
|
||||
msg = rec.get("message") or {}
|
||||
usage = msg.get("usage")
|
||||
if not usage:
|
||||
continue
|
||||
key = rec.get("requestId") or rec.get("uuid")
|
||||
model = msg.get("model")
|
||||
if model:
|
||||
req_model[key] = model
|
||||
cur = req_usage.setdefault(key, {})
|
||||
# Max-per-field guards against both identical-repeat and
|
||||
# cumulative-within-request streaming shapes.
|
||||
for f in ("input_tokens", "output_tokens",
|
||||
"cache_creation_input_tokens", "cache_read_input_tokens"):
|
||||
v = usage.get(f, 0) or 0
|
||||
if v > cur.get(f, 0):
|
||||
cur[f] = v
|
||||
det = usage.get("cache_creation") or {}
|
||||
cd = cur.setdefault("cache_creation", {})
|
||||
for f in ("ephemeral_5m_input_tokens", "ephemeral_1h_input_tokens"):
|
||||
v = det.get(f, 0) or 0
|
||||
if v > cd.get(f, 0):
|
||||
cd[f] = v
|
||||
st = usage.get("server_tool_use") or {}
|
||||
sc = cur.setdefault("_server", {})
|
||||
for f in ("web_search_requests", "web_fetch_requests"):
|
||||
v = st.get(f, 0) or 0
|
||||
if v > sc.get(f, 0):
|
||||
sc[f] = v
|
||||
|
||||
tokens = empty_tokens()
|
||||
server = {"web_search": 0, "web_fetch": 0}
|
||||
models = {}
|
||||
for key, usage in req_usage.items():
|
||||
add_tokens(tokens, usage)
|
||||
model = req_model.get(key)
|
||||
models[model] = models.get(model, 0) + 1
|
||||
sv = usage.get("_server", {})
|
||||
server["web_search"] += sv.get("web_search_requests", 0)
|
||||
server["web_fetch"] += sv.get("web_fetch_requests", 0)
|
||||
|
||||
return {
|
||||
"requests": len(req_usage),
|
||||
"tokens": tokens,
|
||||
"total_tokens": token_total(tokens),
|
||||
"models": models,
|
||||
"server_tools": server,
|
||||
}
|
||||
|
||||
|
||||
def cache_hit_ratio(tokens):
|
||||
denom = tokens["input"] + tokens["cache_read"] + tokens["cache_creation"]
|
||||
return round(tokens["cache_read"] / denom, 4) if denom else None
|
||||
|
||||
|
||||
def analyze_main(path):
|
||||
"""Tool mix, error rate, skills, slash commands, prompts, wall-clock."""
|
||||
tool_calls = {}
|
||||
error_total = 0
|
||||
interrupted = 0
|
||||
skills = []
|
||||
slash = []
|
||||
tasks = []
|
||||
user_prompts = 0
|
||||
first_ts = last_ts = None
|
||||
cwd = version = None
|
||||
|
||||
for rec in iter_lines(path):
|
||||
ts = parse_ts(rec.get("timestamp"))
|
||||
if ts:
|
||||
if first_ts is None or ts < first_ts:
|
||||
first_ts = ts
|
||||
if last_ts is None or ts > last_ts:
|
||||
last_ts = ts
|
||||
cwd = rec.get("cwd") or cwd
|
||||
version = rec.get("version") or version
|
||||
|
||||
typ = rec.get("type")
|
||||
msg = rec.get("message") or {}
|
||||
content = msg.get("content")
|
||||
|
||||
if typ == "assistant" and isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") != "tool_use":
|
||||
continue
|
||||
name = block.get("name", "?")
|
||||
tool_calls[name] = tool_calls.get(name, 0) + 1
|
||||
if name == "Skill":
|
||||
skills.append({"skill": (block.get("input") or {}).get("skill"),
|
||||
"ts": rec.get("timestamp")})
|
||||
elif name in ("Task", "Agent"):
|
||||
inp = block.get("input") or {}
|
||||
tasks.append({"subagent_type": inp.get("subagent_type"),
|
||||
"description": inp.get("description"),
|
||||
"ts": rec.get("timestamp")})
|
||||
|
||||
if typ == "user":
|
||||
# tool_result blocks carry the authoritative is_error flag
|
||||
if isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") == "tool_result":
|
||||
if block.get("is_error"):
|
||||
error_total += 1
|
||||
elif block.get("type") == "text":
|
||||
for m in re.findall(r"<command-name>(.*?)</command-name>",
|
||||
block.get("text", "")):
|
||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
||||
elif isinstance(content, str):
|
||||
for m in re.findall(r"<command-name>(.*?)</command-name>", content):
|
||||
slash.append({"cmd": m, "ts": rec.get("timestamp")})
|
||||
# genuine user turns (not tool results, not slash plumbing)
|
||||
is_tool_result = (isinstance(content, list)
|
||||
and any(b.get("type") == "tool_result" for b in content))
|
||||
if not is_tool_result and rec.get("toolUseResult") is None:
|
||||
user_prompts += 1
|
||||
tur = rec.get("toolUseResult")
|
||||
if isinstance(tur, dict) and tur.get("interrupted"):
|
||||
interrupted += 1
|
||||
|
||||
return {
|
||||
"tool_calls": tool_calls,
|
||||
"total_tool_calls": sum(tool_calls.values()),
|
||||
"tool_error_total": error_total,
|
||||
"interrupted": interrupted,
|
||||
"skills_invoked": skills,
|
||||
"slash_commands": slash,
|
||||
"tasks_dispatched": tasks,
|
||||
"user_prompts": user_prompts,
|
||||
"wall_clock_sec": round((last_ts - first_ts).total_seconds(), 1)
|
||||
if first_ts and last_ts else None,
|
||||
"started": first_ts.isoformat() if first_ts else None,
|
||||
"ended": last_ts.isoformat() if last_ts else None,
|
||||
"cwd": cwd,
|
||||
"version": version,
|
||||
}
|
||||
|
||||
|
||||
def subagent_terminal_status(path):
|
||||
last_text = ""
|
||||
for rec in iter_lines(path):
|
||||
if rec.get("type") != "assistant":
|
||||
continue
|
||||
content = (rec.get("message") or {}).get("content")
|
||||
if isinstance(content, list):
|
||||
for block in content:
|
||||
if block.get("type") == "text" and block.get("text", "").strip():
|
||||
last_text = block["text"]
|
||||
# Primary: the structured "Status:" line. Take the LAST one so a
|
||||
# restated final status wins over any earlier mention.
|
||||
structured = STATUS_LINE_RE.findall(last_text)
|
||||
if structured:
|
||||
return structured[-1].upper()
|
||||
# Fallback: a status token standing alone on its own line.
|
||||
standalone = STATUS_STANDALONE_RE.findall(last_text)
|
||||
if standalone:
|
||||
return standalone[-1].upper()
|
||||
return "unknown"
|
||||
|
||||
|
||||
def analyze_workflows(session_dir):
|
||||
"""Aggregate Workflow-substrate runs: workflows/wf_*.json (per-agent
|
||||
summary: stage label, model, live token counter) joined with the raw
|
||||
per-agent transcripts under subagents/workflows/<runId>/agent-*.jsonl.
|
||||
|
||||
The raw transcripts are authoritative for token counts (deduped by
|
||||
requestId, same as every other transcript here); the wf_*.json
|
||||
'tokens' counters under-track vs. the deduped raw usage (~11% in the
|
||||
206-run corpus that motivated this path, issue #28) and are reported
|
||||
only as a cross-check. Stage = the label up to the first ':'
|
||||
('impl:3' -> 'impl'), so loop stages aggregate across tasks/rounds."""
|
||||
runs = []
|
||||
for wf_path in sorted(glob.glob(os.path.join(session_dir, "workflows",
|
||||
"wf_*.json"))):
|
||||
try:
|
||||
with open(wf_path) as fh:
|
||||
data = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
continue
|
||||
run_id = data.get("runId") or os.path.basename(wf_path)[:-len(".json")]
|
||||
raw_dir = os.path.join(session_dir, "subagents", "workflows", run_id)
|
||||
meta_by_agent = {}
|
||||
for entry in data.get("workflowProgress") or []:
|
||||
if entry.get("type") != "workflow_agent" or not entry.get("agentId"):
|
||||
continue
|
||||
meta_by_agent[entry["agentId"]] = entry
|
||||
|
||||
run_tokens = empty_tokens()
|
||||
stages = {}
|
||||
agent_count = 0
|
||||
untracked = 0
|
||||
for jl in sorted(glob.glob(os.path.join(raw_dir, "agent-*.jsonl"))):
|
||||
aid = os.path.basename(jl)[len("agent-"):-len(".jsonl")]
|
||||
agg = aggregate_transcript(jl)
|
||||
entry = meta_by_agent.get(aid)
|
||||
if entry is None:
|
||||
# Transcript with no progress entry (retried/dropped
|
||||
# attempt): keep it counted, grouped under '?', never
|
||||
# silently folded into a named stage.
|
||||
untracked += 1
|
||||
label = (entry or {}).get("label") or "?"
|
||||
stage = label.split(":", 1)[0]
|
||||
st = stages.setdefault(stage, {"count": 0, "tokens": empty_tokens(),
|
||||
"models": {}})
|
||||
st["count"] += 1
|
||||
agent_count += 1
|
||||
for k in run_tokens:
|
||||
run_tokens[k] += agg["tokens"][k]
|
||||
st["tokens"][k] += agg["tokens"][k]
|
||||
fallback_model = (entry or {}).get("model")
|
||||
for m, n in agg["models"].items():
|
||||
key = m or fallback_model
|
||||
st["models"][key] = st["models"].get(key, 0) + n
|
||||
|
||||
for st in stages.values():
|
||||
st["total_tokens"] = token_total(st["tokens"])
|
||||
|
||||
runs.append({
|
||||
"run_id": run_id,
|
||||
"name": data.get("workflowName"),
|
||||
"status": data.get("status"),
|
||||
"agents": agent_count,
|
||||
"agents_untracked": untracked,
|
||||
"reported_total_tokens": data.get("totalTokens"),
|
||||
"tokens": run_tokens,
|
||||
"total_tokens": token_total(run_tokens),
|
||||
"stages": stages,
|
||||
})
|
||||
return runs
|
||||
|
||||
|
||||
def analyze_subagents(session_dir):
|
||||
out = []
|
||||
sub_dir = os.path.join(session_dir, "subagents")
|
||||
for jl in sorted(glob.glob(os.path.join(sub_dir, "agent-*.jsonl"))):
|
||||
meta = {}
|
||||
mp = jl.replace(".jsonl", ".meta.json")
|
||||
if os.path.exists(mp):
|
||||
try:
|
||||
with open(mp) as fh:
|
||||
meta = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
pass
|
||||
agg = aggregate_transcript(jl)
|
||||
out.append({
|
||||
"agent_id": os.path.basename(jl)[len("agent-"):-len(".jsonl")],
|
||||
"agent_type": meta.get("agentType"),
|
||||
"description": meta.get("description"),
|
||||
"dispatched_by_tool": meta.get("toolUseId"),
|
||||
"models": agg["models"],
|
||||
"requests": agg["requests"],
|
||||
"tokens": agg["tokens"],
|
||||
"total_tokens": agg["total_tokens"],
|
||||
"terminal_status": subagent_terminal_status(jl),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def resolve_session(args):
|
||||
"""Return (main_jsonl_path, session_dir_or_None)."""
|
||||
if args.file:
|
||||
path = os.path.abspath(args.file)
|
||||
return path, path[:-len(".jsonl")] if path.endswith(".jsonl") else None
|
||||
|
||||
cwd = args.cwd or os.getcwd()
|
||||
proj_dir = args.project_dir or os.path.join(
|
||||
os.path.expanduser("~/.claude/projects"), cwd.replace("/", "-"))
|
||||
if not os.path.isdir(proj_dir):
|
||||
raise SystemExit(f"no project log dir for cwd {cwd}: {proj_dir} not found")
|
||||
|
||||
if args.session:
|
||||
path = os.path.join(proj_dir, args.session + ".jsonl")
|
||||
if not os.path.exists(path):
|
||||
raise SystemExit(f"session {args.session} not found in {proj_dir}")
|
||||
return path, os.path.join(proj_dir, args.session)
|
||||
|
||||
# default: newest .jsonl in the project dir = current/last session
|
||||
candidates = sorted(glob.glob(os.path.join(proj_dir, "*.jsonl")),
|
||||
key=os.path.getmtime, reverse=True)
|
||||
if not candidates:
|
||||
raise SystemExit(f"no session transcripts in {proj_dir}")
|
||||
path = candidates[0]
|
||||
return path, path[:-len(".jsonl")]
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Single-session post-mortem aggregator.")
|
||||
ap.add_argument("--session", help="explicit session id (UUID)")
|
||||
ap.add_argument("--file", help="explicit path to a main transcript .jsonl")
|
||||
ap.add_argument("--cwd", help="project cwd to resolve logs for (default: $PWD)")
|
||||
ap.add_argument("--project-dir", help="explicit ~/.claude/projects/<slug> dir")
|
||||
args = ap.parse_args()
|
||||
|
||||
main_path, session_dir = resolve_session(args)
|
||||
session_id = os.path.basename(main_path)[:-len(".jsonl")]
|
||||
|
||||
main_agg = aggregate_transcript(main_path)
|
||||
main_meta = analyze_main(main_path)
|
||||
|
||||
subagents = []
|
||||
workflows = []
|
||||
if session_dir and os.path.isdir(session_dir):
|
||||
subagents = analyze_subagents(session_dir)
|
||||
workflows = analyze_workflows(session_dir)
|
||||
|
||||
sub_tokens = empty_tokens()
|
||||
for s in subagents:
|
||||
for k in sub_tokens:
|
||||
sub_tokens[k] += s["tokens"][k]
|
||||
|
||||
wf_tokens = empty_tokens()
|
||||
wf_agents = 0
|
||||
for w in workflows:
|
||||
wf_agents += w["agents"]
|
||||
for k in wf_tokens:
|
||||
wf_tokens[k] += w["tokens"][k]
|
||||
|
||||
total_tokens = empty_tokens()
|
||||
for k in total_tokens:
|
||||
total_tokens[k] = (main_agg["tokens"][k] + sub_tokens[k]
|
||||
+ wf_tokens[k])
|
||||
|
||||
warnings = []
|
||||
is_active = False
|
||||
if main_meta["ended"]:
|
||||
ended = parse_ts(main_meta["ended"])
|
||||
if ended and (datetime.now(timezone.utc) - ended).total_seconds() < 120:
|
||||
is_active = True
|
||||
warnings.append("Session appears ACTIVE (last event <2 min ago); "
|
||||
"totals are partial and will grow.")
|
||||
warnings.append("Token counts are raw usage; wall-clock is derived from "
|
||||
"timestamp deltas (the transcript carries no durationMs).")
|
||||
|
||||
report = {
|
||||
"session": {
|
||||
"id": session_id,
|
||||
"transcript": main_path,
|
||||
"cwd": main_meta["cwd"],
|
||||
"version": main_meta["version"],
|
||||
"started": main_meta["started"],
|
||||
"ended": main_meta["ended"],
|
||||
"wall_clock_sec": main_meta["wall_clock_sec"],
|
||||
"active": is_active,
|
||||
},
|
||||
"main": {
|
||||
"requests": main_agg["requests"],
|
||||
"models": main_agg["models"],
|
||||
"tokens": main_agg["tokens"],
|
||||
"total_tokens": main_agg["total_tokens"],
|
||||
"cache_hit_ratio": cache_hit_ratio(main_agg["tokens"]),
|
||||
"server_tools": main_agg["server_tools"],
|
||||
"user_prompts": main_meta["user_prompts"],
|
||||
},
|
||||
"tools": {
|
||||
"calls": main_meta["tool_calls"],
|
||||
"total": main_meta["total_tool_calls"],
|
||||
"error_total": main_meta["tool_error_total"],
|
||||
"error_ratio": round(main_meta["tool_error_total"]
|
||||
/ main_meta["total_tool_calls"], 4)
|
||||
if main_meta["total_tool_calls"] else None,
|
||||
"interrupted": main_meta["interrupted"],
|
||||
},
|
||||
"skills_invoked": main_meta["skills_invoked"],
|
||||
"slash_commands": main_meta["slash_commands"],
|
||||
"tasks_dispatched": main_meta["tasks_dispatched"],
|
||||
"subagents": subagents,
|
||||
"subagent_totals": {
|
||||
"count": len(subagents),
|
||||
"tokens": sub_tokens,
|
||||
"total_tokens": token_total(sub_tokens),
|
||||
},
|
||||
"workflows": workflows,
|
||||
"workflow_totals": {
|
||||
"runs": len(workflows),
|
||||
"agents": wf_agents,
|
||||
"tokens": wf_tokens,
|
||||
"total_tokens": token_total(wf_tokens),
|
||||
},
|
||||
"totals": {
|
||||
"tokens": total_tokens,
|
||||
"total_tokens": token_total(total_tokens),
|
||||
"cache_hit_ratio": cache_hit_ratio(total_tokens),
|
||||
},
|
||||
"warnings": warnings,
|
||||
}
|
||||
json.dump(report, sys.stdout, indent=2)
|
||||
sys.stdout.write("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,125 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Executable spec for postmortem.subagent_terminal_status.
|
||||
|
||||
Run directly: `python3 postmortem/tests/test_terminal_status.py`.
|
||||
Exits 0 on pass, 1 on failure. No pytest dependency — the plugin has
|
||||
no test harness, and a status-classifier is small enough to pin with
|
||||
plain asserts.
|
||||
|
||||
Pins issue #3: the terminal-status text-scan over-reports BLOCKED for
|
||||
shipped implement runs, because `\\bBLOCKED\\b` fires on the end-report's
|
||||
own template lines (`BLOCKED file: BLOCKED.md`, `Blocked detail:`)
|
||||
and on prose narrating a surmounted blocker. The authoritative signal is
|
||||
the structured `Status:` line of the end-report, not any keyword
|
||||
occurrence in the body.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
SCRIPT = os.path.join(HERE, "..", "scripts", "postmortem.py")
|
||||
|
||||
spec = importlib.util.spec_from_file_location("postmortem", SCRIPT)
|
||||
pm = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(pm)
|
||||
|
||||
|
||||
def write_subagent_log(final_text):
|
||||
"""Write a one-assistant-turn subagent jsonl whose last text block is
|
||||
`final_text`, return its path. Caller cleans up."""
|
||||
fd, path = tempfile.mkstemp(suffix=".jsonl")
|
||||
rec = {"type": "assistant",
|
||||
"message": {"role": "assistant",
|
||||
"content": [{"type": "text", "text": final_text}]}}
|
||||
with os.fdopen(fd, "w") as fh:
|
||||
fh.write(json.dumps(rec) + "\n")
|
||||
return path
|
||||
|
||||
|
||||
def status_of(final_text):
|
||||
path = write_subagent_log(final_text)
|
||||
try:
|
||||
return pm.subagent_terminal_status(path)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
# A realistic plain-text iteration end-report that SHIPPED (Status: DONE)
|
||||
# but whose body carries the template's BLOCKED-bearing lines. This is the
|
||||
# exact shape that issue #3 reports as misclassified.
|
||||
SHIPPED_DONE_REPORT = """\
|
||||
Status: DONE
|
||||
Iter: 0126-leg3
|
||||
Started from: a1b2c3d
|
||||
Tasks completed: 3 of 3
|
||||
- poly RawBuf elem-var mono fix
|
||||
- series kernel module
|
||||
- owned-param tail-recursion leak
|
||||
Working tree: dirty (7 files changed)
|
||||
BLOCKED file: BLOCKED.md (uncommitted; only on PARTIAL/BLOCKED)
|
||||
Stats: .claude/stats/0126.json (uncommitted)
|
||||
Files touched: 7
|
||||
Tests: 12 green, 0 red
|
||||
E2E coverage: none (mini mode)
|
||||
Blocked detail: (only if BLOCKED or PARTIAL — also written to BLOCKED.md)
|
||||
|
||||
Note: task 2 hit a blocker early (missing fixture) which was resolved
|
||||
in-phase; no BLOCKED.md was written.
|
||||
"""
|
||||
|
||||
GENUINE_BLOCKED_REPORT = """\
|
||||
Status: BLOCKED
|
||||
Iter: 0127
|
||||
Started from: d4e5f6a
|
||||
Tasks completed: 1 of 3
|
||||
Working tree: dirty (2 files changed)
|
||||
BLOCKED file: BLOCKED.md (uncommitted)
|
||||
Blocked detail: Task: 2
|
||||
Reason: review-loop-exhausted
|
||||
Worker says: spec-compliance never reached approved
|
||||
"""
|
||||
|
||||
PARTIAL_REPORT = """\
|
||||
Status: PARTIAL
|
||||
Iter: 0128
|
||||
Tasks completed: 2 of 3
|
||||
BLOCKED file: BLOCKED.md (uncommitted)
|
||||
"""
|
||||
|
||||
CASES = [
|
||||
# (name, final_text, expected)
|
||||
("shipped DONE with BLOCKED template lines (issue #3)",
|
||||
SHIPPED_DONE_REPORT, "DONE"),
|
||||
("genuine BLOCKED end-report",
|
||||
GENUINE_BLOCKED_REPORT, "BLOCKED"),
|
||||
("PARTIAL end-report mentioning BLOCKED.md",
|
||||
PARTIAL_REPORT, "PARTIAL"),
|
||||
("narrating agent whose final line is a bare status token",
|
||||
"Work complete, tests green.\n\nDONE", "DONE"),
|
||||
("prose mentioning a resolved blocker, no status line",
|
||||
"I was briefly BLOCKED by a missing import but fixed it. All good.",
|
||||
"unknown"),
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
failures = []
|
||||
for name, text, expected in CASES:
|
||||
got = status_of(text)
|
||||
ok = got == expected
|
||||
print(f"[{'PASS' if ok else 'FAIL'}] {name}: expected {expected!r}, got {got!r}")
|
||||
if not ok:
|
||||
failures.append(name)
|
||||
if failures:
|
||||
print(f"\n{len(failures)} failure(s).")
|
||||
return 1
|
||||
print(f"\nAll {len(CASES)} cases passed.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Executable spec for postmortem.analyze_workflows.
|
||||
|
||||
Run directly: `python3 postmortem/tests/test_workflow_aggregation.py`.
|
||||
Exits 0 on pass, 1 on failure. No pytest dependency — same plain-assert
|
||||
convention as test_terminal_status.py.
|
||||
|
||||
Pins issue #28: sessions that ran Workflow-substrate pipelines
|
||||
(workflows/wf_*.json + subagents/workflows/<runId>/agent-*.jsonl) must
|
||||
not read as zero subagent spend. The raw transcripts are authoritative
|
||||
(dedup by requestId); the summary supplies stage labels; a transcript
|
||||
missing from workflowProgress groups under '?' instead of being folded
|
||||
into a named stage or dropped.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
|
||||
from postmortem import analyze_workflows # noqa: E402
|
||||
|
||||
|
||||
def _write(path, obj_lines):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w") as fh:
|
||||
for obj in obj_lines:
|
||||
fh.write(json.dumps(obj) + "\n")
|
||||
|
||||
|
||||
def _assistant_line(request_id, in_tok, out_tok, model="claude-sonnet-5"):
|
||||
return {
|
||||
"type": "assistant",
|
||||
"requestId": request_id,
|
||||
"message": {
|
||||
"model": model,
|
||||
"usage": {"input_tokens": in_tok, "output_tokens": out_tok,
|
||||
"cache_creation_input_tokens": 0,
|
||||
"cache_read_input_tokens": 0},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
with tempfile.TemporaryDirectory() as session_dir:
|
||||
run_id = "wf_test-1"
|
||||
wf = {
|
||||
"runId": run_id,
|
||||
"workflowName": "spec-fixture",
|
||||
"status": "completed",
|
||||
"totalTokens": 999, # live counter: deliberately wrong
|
||||
"defaultModel": "claude-sonnet-5",
|
||||
"workflowProgress": [
|
||||
{"type": "workflow_phase", "index": 1, "title": "P"},
|
||||
{"type": "workflow_agent", "agentId": "a1",
|
||||
"label": "impl:1", "model": "claude-sonnet-5",
|
||||
"state": "done", "tokens": 1},
|
||||
{"type": "workflow_agent", "agentId": "a2",
|
||||
"label": "impl:2", "model": "claude-sonnet-5",
|
||||
"state": "done", "tokens": 1},
|
||||
{"type": "workflow_agent", "agentId": "a4",
|
||||
"label": "qual:1", "model": "claude-opus-4-8",
|
||||
"state": "error", "tokens": 0}, # no transcript on disk
|
||||
],
|
||||
}
|
||||
os.makedirs(os.path.join(session_dir, "workflows"))
|
||||
with open(os.path.join(session_dir, "workflows",
|
||||
run_id + ".json"), "w") as fh:
|
||||
json.dump(wf, fh)
|
||||
|
||||
raw = os.path.join(session_dir, "subagents", "workflows", run_id)
|
||||
# a1: streamed repeat of the SAME requestId — must dedup to
|
||||
# max-per-field (input 10, output 7), not sum to 20/12.
|
||||
_write(os.path.join(raw, "agent-a1.jsonl"),
|
||||
[_assistant_line("req-1", 10, 5),
|
||||
_assistant_line("req-1", 10, 7)])
|
||||
# a2: two distinct requests — must sum.
|
||||
_write(os.path.join(raw, "agent-a2.jsonl"),
|
||||
[_assistant_line("req-2", 3, 2),
|
||||
_assistant_line("req-3", 4, 1)])
|
||||
# a3: transcript with NO workflowProgress entry — counts under '?'.
|
||||
_write(os.path.join(raw, "agent-a3.jsonl"),
|
||||
[_assistant_line("req-4", 100, 50)])
|
||||
|
||||
runs = analyze_workflows(session_dir)
|
||||
|
||||
assert len(runs) == 1, runs
|
||||
run = runs[0]
|
||||
assert run["run_id"] == run_id
|
||||
assert run["name"] == "spec-fixture"
|
||||
assert run["reported_total_tokens"] == 999
|
||||
|
||||
# Three transcripts on disk, one untracked.
|
||||
assert run["agents"] == 3, run
|
||||
assert run["agents_untracked"] == 1, run
|
||||
|
||||
stages = run["stages"]
|
||||
assert set(stages) == {"impl", "?"}, stages
|
||||
|
||||
impl = stages["impl"]
|
||||
assert impl["count"] == 2, impl
|
||||
# a1 deduped (10 in / 7 out) + a2 summed (7 in / 3 out)
|
||||
assert impl["tokens"]["input"] == 17, impl
|
||||
assert impl["tokens"]["output"] == 10, impl
|
||||
assert impl["total_tokens"] == 27, impl
|
||||
assert impl["models"] == {"claude-sonnet-5": 3}, impl
|
||||
|
||||
unk = stages["?"]
|
||||
assert unk["count"] == 1, unk
|
||||
assert unk["tokens"]["input"] == 100 and unk["tokens"]["output"] == 50, unk
|
||||
|
||||
# Run totals = authoritative raw aggregation, not the live counter.
|
||||
assert run["tokens"]["input"] == 117, run
|
||||
assert run["tokens"]["output"] == 60, run
|
||||
assert run["total_tokens"] == 177, run
|
||||
|
||||
print("ok: analyze_workflows aggregates raw transcripts per stage, "
|
||||
"dedups by requestId, and isolates untracked agents under '?'")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+63
-4
@@ -35,6 +35,8 @@ the decision as a *source* rather than making it inline.
|
||||
|
||||
`specify` is a core pipeline node: it is the gate before `planner` on
|
||||
every design path that is not a bug fix or a test-specifiable feature.
|
||||
The side paths that carry no fresh prose design — `compiler-driven`,
|
||||
the harvest sweep — bypass it (`../docs/pipeline.md` § Skip rules).
|
||||
Where `tdd` is one entry path among three (chosen when behaviour is
|
||||
test-specifiable), `specify` sits on every prose-design path — it is
|
||||
not bypassed for those.
|
||||
@@ -59,6 +61,9 @@ Triggers:
|
||||
- An observed bug — use `debug` directly (RED-first).
|
||||
- A tidy iteration — use `audit` directly.
|
||||
- A trivial mechanical edit — per the project's CLAUDE.md carve-out.
|
||||
- Settled single-issue backlog drained as a batch — the harvest
|
||||
sweep's issue bodies are the ratified spec source; no new spec is
|
||||
written. See `../docs/pipeline.md` § Harvest sweep.
|
||||
|
||||
**Skipping a gate is never permitted.** The interview is `brainstorm`'s,
|
||||
and its absence here is the whole point — but the *production gates*
|
||||
@@ -108,6 +113,26 @@ Before producing anything, establish what the sources actually say:
|
||||
walk to the contracts the work touches. Note any `status:
|
||||
aspirational` source: its code is a target, not verified fact.
|
||||
|
||||
**Grounding fan-out — the cycle recon.** When the work enters code
|
||||
territory the orchestrator has not recently read, dispatch
|
||||
`../planner/agents/plan-recon.md` **once**, in its pre-spec form —
|
||||
carrier fields (`sources`, `recon_scope`, optional `focus_hint`)
|
||||
defined authoritatively under its **Pre-spec dispatch (cycle
|
||||
recon)** heading. When the sources already settle the change shape
|
||||
(a signature change, a variant add / remove, a symbol removal, a
|
||||
content-pin perturbation), name it in `recon_scope`: that arms the
|
||||
compile-driven enumeration inside the cycle recon and saves
|
||||
`planner` a mandatory delta later. The returned file-map grounds
|
||||
Step 2's criterion evidence and Step 3's `## Concrete code
|
||||
shapes`, and is reused downstream by `planner` — budget, reuse
|
||||
conditions, and delta triggers are defined once in
|
||||
`../docs/pipeline.md` § Cycle recon. A `brainstorm`-fired dispatch
|
||||
that meets that section's coverage condition IS the cycle recon —
|
||||
do not dispatch again. Do NOT ground a spec with a broad
|
||||
Explore-type sweep (§ Cycle recon's ban; field evidence on issue
|
||||
#35: a 151k-token Explore sweep here, ~40 % duplicated by the
|
||||
later plan-recon).
|
||||
|
||||
### Step 1.5 — Precondition gate (bounce-back)
|
||||
|
||||
Enumerate the **load-bearing design decisions** the spec must encode
|
||||
@@ -307,6 +332,24 @@ follows as an explicitly secondary subsection, never first. Prose may
|
||||
*explain* code, never *replace* it. Exact bytes / line-numbers remain
|
||||
the planner's job — the spec owns the *shape*, shown.
|
||||
|
||||
**Assumption economy — fewer restatements, never fewer reliances.**
|
||||
Every assertion the spec makes about *current* codebase behaviour is
|
||||
a load-bearing-assumption candidate the Step-5 `grounding-check`
|
||||
must extract and ratify — the gate's cost scales with the count, and
|
||||
it BLOCKs an over-broad spec outright (`agents/grounding-check.md`).
|
||||
So: state a current-behaviour claim only where the change actually
|
||||
relies on it (the gate's test: its falsity would make the change
|
||||
fail or cost additional work), and render context-only mentions of
|
||||
existing mechanisms as a design-ledger contract citation instead of
|
||||
restating their behaviour. This trims *restatements*, never *reliances*: a
|
||||
mechanism the change builds on remains an assumption — explicit, or
|
||||
the implicit form the gate extracts anyway — and writing it as a
|
||||
bare citation does not exempt it; hiding a reliance is exactly the
|
||||
failure the gate exists to catch. The same economy applies to scope:
|
||||
a tightly-cut iteration makes fewer claims than a sprawling one
|
||||
(field evidence on issue #35: gate cost tracks the spec's assumption
|
||||
count near-linearly).
|
||||
|
||||
### Step 4 — Self-review
|
||||
|
||||
Inline checklist (not a subagent dispatch):
|
||||
@@ -323,6 +366,16 @@ Inline checklist (not a subagent dispatch):
|
||||
carry before → after code for every load-bearing change (plus the
|
||||
worked user-facing example for a surface cycle)? A load-bearing
|
||||
change described only in prose is a self-review failure to fix.
|
||||
6. **Assumption economy:** scan every claim about current codebase
|
||||
behaviour. The gate's own criterion decides which stay: a claim
|
||||
is a *reliance* iff its being false would make the proposed
|
||||
change fail or require additional work
|
||||
(`agents/grounding-check.md` § What "load-bearing assumption"
|
||||
means). Non-reliances are gate cost with no return — delete
|
||||
them, or demote them to a citation that carries no behaviour
|
||||
claim. Never demote a reliance: a bare citation does not ratify
|
||||
it, and the gate BLOCKs on it unratified (Step 3's
|
||||
restatements-vs-reliances line).
|
||||
|
||||
Fix issues inline.
|
||||
|
||||
@@ -492,6 +545,7 @@ is narrow by design: only the user's-call forks bounce, not every fork.
|
||||
|-----------|---------|
|
||||
| user / issue (design settled) → `specify` | the sources: issue body, in-context design discussion, design docs |
|
||||
| `brainstorm` → `specify` | ratified design narrative (approaches chosen, constraints, sections approved) — in-context; brainstorm writes no spec file |
|
||||
| `specify` → `plan-recon` (Step 1, the cycle recon — skipped when a brainstorm-fired dispatch meets the coverage condition) | `sources` + `recon_scope` + optional `focus_hint`; defined authoritatively under **Pre-spec dispatch (cycle recon)** in `../planner/agents/plan-recon.md` |
|
||||
| `specify` → `grounding-check` (Step 5) | `spec_path` (absolute) + `iteration_scope` |
|
||||
| `specify` → `spec-skeptic` bias-breaker (Step 1.5, optional, orchestrator's call) | `spec_path` (absolute) + `iteration_scope` + one `lens` + `seeding_issue` (or `none`); dispatched ad hoc for a fork the orchestrator is deciding, never as a gate |
|
||||
| `specify` → reference issue (Step 1.5) | a derived-decision comment (rationale) or user-decision comment (provenance); a freshly created seeding issue if the cycle had none |
|
||||
@@ -510,6 +564,7 @@ is narrow by design: only the user's-call forks bounce, not every fork.
|
||||
| "The shape is clear from the prose, I don't need to paste the code" | If it's clear, pasting it is free; if pasting it is hard, it wasn't clear. The acceptance criterion is unjudgeable without the worked code. |
|
||||
| "Just polishing a wording after PASS, no need to re-dispatch" | The grounding-check report attests to specific bytes. A polish edit changes the bytes; the attestation no longer covers them. Re-dispatch is cheap. |
|
||||
| "It's in a model doc / the design ledger, so it's canonical" | A `models` / RFC / proposal doc holds aspirational code — a target, not verified fact. Flag code lifted from a `status: aspirational` source as such; it is not ratified contract. |
|
||||
| "Fewer assumptions = a cheaper grounding-check, so I'll phrase the reliances as vague citations" | Economy cuts *restatements*, not *reliances*. A mechanism the change builds on stays a claim — vague phrasing merely converts it to the implicit form the gate extracts anyway, or worse, hides a false premise until a downstream iteration BLOCKs on it. Trim what the change does not rest on; never what it does. |
|
||||
| "In `/boss` this spec looks solid — I'll sign and move on" | Your sense that it looks solid is not the signature. Under `/boss` the signature is the Step-5 `grounding-check` `PASS` — an independent agent's verdict, not your confidence. Run the grounding-check; a no-override `BLOCK` routes to the human, never auto-signed over. |
|
||||
| "A grounding `BLOCK` looks like a false alarm — I'll sign over it" | With no human in the loop there is no override. A no-override grounding `BLOCK` is the one objective denial of an autonomous sign; it routes to the human sign-off pause. The whole point of trusting grounding-PASS as the signature is that you also honour its `BLOCK`. |
|
||||
|
||||
@@ -538,6 +593,9 @@ is narrow by design: only the user's-call forks bounce, not every fork.
|
||||
- Jumping straight from spec to `implement` (must go via `planner`)
|
||||
- Running an interview here (that is `brainstorm`'s job — if the work
|
||||
needs one, bounce)
|
||||
- Grounding the spec with a broad Explore-type sweep — or a second
|
||||
full recon dispatch when a cycle recon already exists
|
||||
(`../docs/pipeline.md` § Cycle recon)
|
||||
|
||||
## Cross-references
|
||||
|
||||
@@ -561,9 +619,10 @@ is narrow by design: only the user's-call forks bounce, not every fork.
|
||||
unanimous five-lens auto-sign panel is retired (baseline tag
|
||||
`pre-autosign-rework`); the autonomous signature is now the Step-5
|
||||
`grounding-check` `PASS`.
|
||||
- **Ad-hoc dispatch.** The orchestrator MAY also ad-hoc dispatch
|
||||
`../planner/agents/plan-recon.md` during Step 1 when the work enters
|
||||
code territory not recently read; discretionary, not part of the
|
||||
standard process.
|
||||
- **Cycle recon (Step 1).** `../planner/agents/plan-recon.md` in its
|
||||
pre-spec form is the cycle's ONE full recon dispatch when the work
|
||||
enters code territory not recently read; its report also serves
|
||||
`planner` downstream. Budget, reuse conditions, and delta triggers:
|
||||
`../docs/pipeline.md` § Cycle recon.
|
||||
- **Project feature-acceptance criterion:** declared in the project's
|
||||
`CLAUDE.md`. Applied prospectively in Step 2.
|
||||
|
||||
Reference in New Issue
Block a user