From 640e06fc60621f0d8a22d309efbab0db7ba42aa3 Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 11 May 2026 19:34:32 +0200 Subject: [PATCH] =?UTF-8?q?spec:=20brainstorm=20grounding-check=20agent=20?= =?UTF-8?q?=E2=80=94=20close=20spec-defect=20loophole?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-05-11-brainstorm-grounding-check.md | 254 ++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 docs/specs/2026-05-11-brainstorm-grounding-check.md diff --git a/docs/specs/2026-05-11-brainstorm-grounding-check.md b/docs/specs/2026-05-11-brainstorm-grounding-check.md new file mode 100644 index 0000000..ee03c05 --- /dev/null +++ b/docs/specs/2026-05-11-brainstorm-grounding-check.md @@ -0,0 +1,254 @@ +# Brainstorm Grounding-Check — Design Spec + +**Date:** 2026-05-11 +**Status:** Draft — awaiting user spec review +**Authors:** Brummel (orchestrator) + Claude + +## Goal + +Close the spec-defect loophole that surfaced on 2026-05-11, when iter +23.4 BLOCKED three times on architectural prerequisites the parent +spec had not anticipated. The root cause was a load-bearing +half-sentence in `docs/specs/2026-05-10-23-eq-ord-prelude.md` — +"polymorphic helpers ... work the same way: the typeclass-elaboration +in 22b.3 inserts the right monomorphised compare__T symbol per use +site" — that was never verified against any existing green test. + +Today the `brainstorm` skill verifies specs only linguistically (Step +7 self-review) and via user approval (Step 8). No step grounds the +spec's assumptions against the actual codebase. This spec adds one: +a new agent, dispatched between Step 7 and Step 8, that reads the +spec with fresh context, extracts its load-bearing assumptions, and +for each one searches for a currently-green test that ratifies it. +On miss: hard-discard with a roadmap entry. On miss + Boss override: +proceed. On pass: through to user approval. + +The expected outcome is that future iter-loops do not start on +unfounded spec assertions. The cost of the agent is one extra step +per brainstorm session; the cost of the failure mode it prevents is +calibrated against 2026-05-11, where a half-day was lost and four +iter dispatches discarded. + +## Architecture + +A new agent file at `skills/brainstorm/agents/ailang-grounding-check.md`, +read-only (tools: `Read, Glob, Grep, Bash`). The agent is dispatched +internally by `skills/brainstorm/SKILL.md` in a new Step 7.5; it is +not standalone-callable in normal flow (Boss MAY dispatch it ad-hoc +for retro reviews, but no skill other than `brainstorm` is wired to +invoke it). + +The pipeline becomes: + +``` +Step 6 — write the spec +Step 7 — linguistic self-review (existing) +Step 7.5 — Grounding-Check Agent (NEW; hard-gate, Boss-overridable) +Step 8 — user-approval gate (existing) +Step 9 — hand off to planner (existing) +``` + +On agent `BLOCK`: brainstorm deletes the spec file, appends a +`depends on:` entry to `docs/roadmap.md`, and terminates the session. +On agent `PASS` or on `BLOCK + Boss override` (free chat act): +brainstorm proceeds to Step 8 unchanged. + +Fresh-context discipline is the central design choice. The +brainstorm-driven orchestrator has spent 30+ minutes building the +spec and is biased toward shipping it; the agent has no such bias. +This mirrors how `ailang-architect` reviews DESIGN.md drift at +milestone close without loyalty to the implementation that produced +the drift. + +The agent does NOT run on its own spec (the spec you are reading). +Bootstrap: the first real run is on the re-brainstormed Spec-23 +(see "Application to in-flight Spec-23" below). + +## Components + +| What | Where | Action | +|------|-------|--------| +| Agent definition | `skills/brainstorm/agents/ailang-grounding-check.md` | Create | +| Skill body | `skills/brainstorm/SKILL.md` | Modify — insert Step 7.5, document failure-mode procedure | +| Skill roster | `skills/README.md` | Modify — extend the agent-roster table | +| Agent discovery | `.claude/agents/brainstorm` → `skills/brainstorm/agents` | Create symlink | +| Roadmap shape | `docs/roadmap.md` | Documentation note — `depends on:` line type stays; no schema change | +| In-flight Spec-23 | `docs/specs/2026-05-10-23-eq-ord-prelude.md` | Delete (after the agent ships) | +| Re-brainstormed Spec-23 | `docs/specs/YYYY-MM-DD-23-eq-ord-prelude.md` | Created by the first real run of the new pipeline | + +The agent file follows the standard agent template described in +`skills/README.md` ("Agent structure"): frontmatter (`name`, +`description`, `tools`), spirit-letter lead-in, role purpose, +standing reading list, carrier contract, Iron Law, the Process, +status protocol, output format, common rationalisations, red flags. + +## Data flow + +Carrier from `brainstorm` to the agent (Step 7.5 dispatch): + +| Field | Content | +|-------|---------| +| `spec_path` | path to the just-drafted spec | +| `iteration_scope` | which sections of the spec are in scope for the imminent first iteration (so the agent prioritises load-bearing assumptions over speculative future-iter content) | + +The agent then: + +1. Reads `spec_path` with no prior context from the brainstorm chat. +2. Reads the standing reading list (CLAUDE.md, DESIGN.md, latest + journals, `skills/README.md`). +3. Extracts assumptions. Heuristic: an assumption is any spec + assertion that the compiler, checker, codegen, or schema *currently* + does X. The agent flags both explicit ("the typeclass-elaboration + in 22b.3 inserts ...") and implicit ones (e.g. a spec section + that proposes adding to a table without saying the table's + existing entries cover the new shape). +4. For each assumption, searches the codebase for a green test that + ratifies it. Search is free-form: `grep` on test bodies, + `cargo test --list`, reading fixture files. The agent makes a + judgement call about what "ratifies" means. +5. Emits a report (see "Output format" below). Aggregate-status is + `PASS` (all assumptions ratified) or `BLOCK` (at least one + unratified). + +Boss interpretation: + +| Agent status | brainstorm behaviour | +|--------------|----------------------| +| `PASS` | Proceed to Step 8. Report goes into the brainstorm chat for the record. | +| `BLOCK` | Present the report to the user. User either: (a) accepts discard → brainstorm deletes the spec, appends roadmap entry, terminates; or (b) overrides in free chat ("Test `::` ratifies that") → brainstorm logs the override note and proceeds to Step 8. | +| Infra error (`cargo test --list` fails, spec file missing, etc.) | brainstorm aborts; Boss debugs out-of-band. | + +## Output format + +The agent's report is plain-text, ≤500 tokens, hybrid layout: + +``` +GROUNDING-CHECK REPORT +Status: PASS|BLOCK +Spec: +Iteration scope: + +Ratified assumptions: +| # | Assumption | Test path | Test name | +| 1 | | crates/.../foo.rs | test_bar_baz | +| 2 | | examples/qux.ail.json | (fixture: build+run green) | +... + +Unratified assumptions: +- Assumption: + Searched: + Why no ratification: + +(Repeat the unratified block per assumption.) +``` + +Ratified entries are scannable; unratified entries carry the +information the Boss needs to decide whether to override. + +## Error handling + +| Failure | Agent action | brainstorm action | +|---------|--------------|-------------------| +| No assumptions extracted (spec is trivial / pure rename / cosmetic) | Report empty ratified list, `PASS` with a note | brainstorm proceeds; the note is logged in the chat | +| Cannot read spec file | Status `INFRA_ERROR`, raw path | brainstorm aborts; Boss investigates | +| `cargo test --list` or `cargo` invocation fails (build broken in workspace) | Status `INFRA_ERROR`, raw stderr | brainstorm aborts; the spec is held — fix the workspace first | +| Ambiguous assumption (cannot decide whether it's a load-bearing claim) | Include in unratified block with note "ambiguous: judgement call deferred to Boss" | Treated as `BLOCK` by default; Boss overrides if the ambiguity is benign | +| Re-running the agent on the same spec | Allowed; idempotent (agent has no state) | Boss may re-dispatch after a clarifying spec edit; no special handling | + +The agent is read-only. No file writes, no test runs (`cargo test +--list` only enumerates). Compile errors in the workspace are +treated as infra, not as agent results. + +## Application to in-flight Spec-23 + +Spec `docs/specs/2026-05-10-23-eq-ord-prelude.md` is currently +mid-flight (iters 23.1-23.3 shipped, 23.4 BLOCKED). Iter 23.1-23.3 +are kept (they shipped working code). The spec itself is deleted +once the new agent exists. A new spec for the remaining Eq/Ord +prelude scope (free fns + the polymorphic mono-extension surfaced +on 2026-05-11) is re-brainstormed from scratch and is the first +real run of the new Step 7.5. + +The bootstrap irony: the spec you are reading does NOT run through +the new agent (the agent does not exist yet). The first real check +happens on the re-brainstormed Spec-23, not on this spec. + +## Testing strategy + +The agent is prompt-driven and read-only; it has no Rust unit tests. +The validation strategy has three layers: + +1. **First real run** is the re-brainstormed Spec-23. The agent + must catch the assumption that polymorphic free fns are + mono-specialised in the same way as class methods (the failure + mode of 2026-05-11). If it catches that, the agent is doing its + job; if it misses it, the prompt iterates. + +2. **Dry-run on past specs** is optional but useful for prompt + tuning. The agent can be invoked manually on + `docs/specs/2026-05-09-skill-system.md` and other past specs to + surface false-positive and false-negative rates before relying + on it in the live pipeline. Not a hard acceptance criterion. + +3. **Long-term feedback loop.** Future per-iter journals are the + real testbed. If a spec passes the agent and an iter BLOCKS + anyway on an unanticipated prerequisite, the journal entry + surfaces it and the agent prompt is revised. Failure modes are + captured in `feedback_spec_over_plan_patches.md`-style memory + entries. + +There is no formal "test suite" for the agent. The discipline is: +when the agent misses a real assumption, that miss is treated as a +prompt-defect to fix, not as an acceptable false negative. + +## Acceptance criteria + +The iteration that implements this spec is complete when: + +1. `skills/brainstorm/agents/ailang-grounding-check.md` exists, + follows the standard agent template, and has been committed. +2. `skills/brainstorm/SKILL.md` has a new Step 7.5 documenting the + dispatch + Boss-interpretation flow, AND a failure-mode procedure + (delete spec, append roadmap entry, terminate) for the + no-override `BLOCK` path. +3. `skills/README.md` skill table and agent roster reflect the new + agent. +4. `.claude/agents/brainstorm` symlink exists and resolves to + `skills/brainstorm/agents`. +5. `docs/specs/2026-05-10-23-eq-ord-prelude.md` is removed and a + roadmap entry exists pointing forward to the re-brainstormed + replacement (the re-brainstorm itself is a separate session and + NOT part of this iteration's acceptance — only the deletion + + roadmap pointer is). +6. A `docs/journals/2026-05-11-iter-.md` records the iteration + and is linked from `docs/journals/INDEX.md`. + +The acceptance criteria deliberately exclude "the agent has been +run successfully on a real spec." The first real run is the +re-brainstormed Spec-23, in a separate brainstorm session, after +this spec ships. Conflating the two is the same trap that broke +Spec-23 in the first place: shipping infrastructure and the test +case for that infrastructure as one undifferentiated batch. + +## Out of scope + +- A test-naming convention or ratification-marker syntax. Today's + free-form `grep + judgement` search is the agreed mechanism; no + test annotation is introduced. +- A separate `examples/grounding/` ratification corpus. Same + rationale. +- A retroactive grounding-check pass over all existing specs + (`2026-05-09-skill-system.md`, etc.). Only Spec-23 is touched, + and not via this iteration — via a separate re-brainstorm + afterwards. +- A "discarded drafts" archive (`docs/specs/discarded/`). Discarded + drafts are deleted; the roadmap entry is the only persistent + trace. +- An override-documentation mechanism (Pre-flight Ratifications + section in the spec format). Overrides stay as free chat acts; + the brainstorm chat history is the only record. + +These exclusions are explicit so future agents do not re-derive +them.