diff --git a/CLAUDE.md b/CLAUDE.md index 232127a..ab4cec1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -177,9 +177,12 @@ exception, not the rule. ## Bug fixes — TDD, always -Bug fixes are RED-first, autonomous, no orchestrator gate. Full +Bug fixes are RED-first, autonomous, no orchestrator gate. The +trigger and handoff are in `skills/debug/SKILL.md`; the full discipline (Iron Law, four phases, Phase 4.5 architecture trigger, -common rationalisations) lives in `skills/debug/SKILL.md`. Headline: +common rationalisations) lives in +`skills/debug/agents/ailang-debugger.md` — that is the file the +dispatched subagent reads. Headline: 1. **RED first** — write a failing test that pins down the symptom, commit it as `test: red for ` BEFORE any fix. diff --git a/skills/debug/SKILL.md b/skills/debug/SKILL.md index a1f318a..92df54b 100644 --- a/skills/debug/SKILL.md +++ b/skills/debug/SKILL.md @@ -15,6 +15,13 @@ skill then drives the fix to GREEN. Skipping the RED stage — even for "trivial" bugs — produces fixes that don't stick and tests that don't exist to catch the next regression. +The substantive process — root cause investigation → pattern analysis +→ RED test → handoff, plus the Phase 4.5 architecture-question trigger +after three failed hypotheses — lives in +`agents/ailang-debugger.md`. That file is the single source of truth +for the discipline; this skill file only governs trigger, dispatch, +and handoff. + ## When to Use / Skipping Trigger this skill on: @@ -40,74 +47,24 @@ Both clauses are non-negotiable. The second was formerly the CLAUDE.md "Bug fixes — TDD, always" section before the 2026-05-09 skill-system migration. -## The Process — four phases +## Dispatch -Each phase must complete before the next starts. +Dispatch `ailang-debugger` with: -### Phase 1 — Root Cause Investigation +| Carrier field | Content | +|---------------|---------| +| `symptom` | Exact error message, stack trace, wrong output, or repro command | +| `repro_known` | One-line repro if known, otherwise the agent finds one | +| `recent_iter` | Iteration that last touched the suspected area (often `git log` tail) | -1. Read the error message in full. Stack traces, line numbers, exit - codes — none of them get skimmed. -2. Reproduce with the shortest possible command. If you can't repro - reliably, gather more data — do NOT guess. -3. Check recent changes: `git log --oneline -20`, - `git diff ..HEAD`. -4. For multi-component issues (compiler → emitter → linker; or - runtime → C glue → binary), instrument every boundary. Find the - layer where the data goes wrong before deciding which layer to - fix. -5. Trace data flow backward from symptom to source. Fix at source, - not at symptom. - -### Phase 2 — Pattern Analysis - -1. Find a similar working example in the codebase. What's different? -2. If the bug is in a pattern (e.g. ADT match lowering, RC drop - emission), read the reference implementation completely. No - skimming. -3. List every difference between working and broken — however small. - -### Phase 3 — Hypothesis and Test - -1. State the hypothesis as a single falsifiable claim: - *"X is the root cause because Y."* -2. **Write the failing test FIRST**, before any fix attempt: - - smallest possible reproducer (E2E in `crates/ail/tests/e2e.rs`, - or unit test in the affected crate) - - automated, deterministic — same input always same output - - doc comment names the property the test protects, not just - what it does -3. Run it to confirm it fails on the current code, with the symptom - the bug-report described. -4. Commit the failing test as a separate commit: - `git commit -m "test: red for "`. - -### Phase 4 — Hand off to `implement` - -The fix itself is `implement`'s job, run in mini-mode. Hand it: -- path to the RED-test file -- 1-2 sentence cause summary from Phase 1 -- hard constraint: *"minimal fix, no surrounding cleanup, - no opportunistic refactor"* - -`implement` runs the GREEN side and commits `fix: `. - -### Phase 4.5 — 3+ failures = architecture question - -If the third hypothesis fails: - -``` -STOP. Do not attempt Fix #4. -``` - -Three failed fixes in a row indicate the architecture is wrong, not -that the latest hypothesis is wrong. Each fix likely revealed new -shared state or coupling in a different place. Surface the -architecture question to the user; do not guess again. +The agent commits the RED test as `test: red for ` and reports +the handoff carrier for `implement` mini-mode. The agent does NOT write +the fix — splitting RED (this skill) and GREEN (`implement` mini-mode) +across two dispatches keeps the diagnosis honest. ## Handoff Contract -The carrier `debug` hands to `implement`: +`debug` produces, for `implement` mini-mode: | Field | Content | |-------|---------| @@ -118,34 +75,12 @@ The carrier `debug` hands to `implement`: Anything else (broader refactor, doc rewrite, new feature) is OUT of scope for the bug-fix iteration and gets queued for a separate one. -## Common Rationalisations - -| Excuse | Reality | -|--------|---------| -| "The implementer says it's obviously the new tag-extract emit" | Confident guesses paper over real causes. Repro, isolate, RED test, then fix. | -| "Trivial bug, existing tests will catch it if we break something" | The existing tests already passed while this bug shipped — by definition they don't cover it. | -| "30 seconds to repro, 30 seconds to fix, skip the test" | 30 seconds to repro is also 30 seconds to capture as a regression fixture. Don't trade durable coverage for nothing. | -| "I'll write the test after the fix works" | Test-after proves nothing about whether the test would have caught the bug pre-fix. | -| "Two fixes failed; the third hypothesis is the right one" | Two failures means the hypothesis space is wrong. STOP, dispatch the debugger agent, do not guess again. | -| "End-of-day, just push the fix and write the test tomorrow" | End-of-day pressure is exactly when shotgun fixes corrupt the tree. Clean tree + open bug > three speculative half-fixes merged. | -| "Architecture question is for next milestone, fix-and-revisit" | Phase 4.5 fires NOW or never. Carrying the wrong shape forward compounds. | - -## Red Flags — STOP and follow process - -- "Quick fix for now, investigate later" -- "Just try changing X and see if it works" -- "I see the symptom, let me fix it" -- "The test is redundant, I manually verified" -- "Pattern says X but I'll adapt it differently" -- *"One more fix attempt"* (when ≥ 2 already failed) - -All of these mean: **return to Phase 1**. - ## Cross-references - **Agent dispatched:** `skills/debug/agents/ailang-debugger.md` — - carries the actual diagnostic work (read CLAUDE.md / DESIGN.md / - JOURNAL, reproduce, write RED test, propose minimal fix). + carries the four-phase process, the Phase 4.5 escalation rule, the + Common Rationalisations table, and the Red Flags list. The + orchestrator does not execute these phases directly. - **Hand-off target:** `skills/implement/SKILL.md` — runs the GREEN side after the RED test is committed. - **Project source:** former CLAUDE.md "Bug fixes — TDD, always" diff --git a/skills/debug/agents/ailang-debugger.md b/skills/debug/agents/ailang-debugger.md index 6d188d7..bd9aa94 100644 --- a/skills/debug/agents/ailang-debugger.md +++ b/skills/debug/agents/ailang-debugger.md @@ -26,7 +26,9 @@ genuinely captures the symptom, not the post-fix code path. 2. `docs/DESIGN.md` — invariants the bug may have crossed. 3. `docs/JOURNAL.md` — most recent entries; the last iteration may have introduced the bug. -4. `skills/debug/SKILL.md` — the four-phase process you follow. + +The four-phase process below is the single source of truth — the +dispatching skill file does not duplicate it. ## Carrier contract — what the controller hands you diff --git a/skills/fieldtest/SKILL.md b/skills/fieldtest/SKILL.md index 5664fe9..84606b1 100644 --- a/skills/fieldtest/SKILL.md +++ b/skills/fieldtest/SKILL.md @@ -22,6 +22,15 @@ The skill produces a friction-and-bug spec that the next iteration's `plan` consumes as a reference. The spec sits next to milestone-design specs at `docs/specs/-fieldtest-.md`. +The substantive process — read DESIGN.md + JOURNAL + milestone spec, +pick 2-4 real-world programming tasks per milestone axis, implement +each in `.ailx` Surface form, run via `ail check`/`build`/`run`, +classify findings, write the spec — lives in +`agents/ailang-fieldtester.md`. That file also carries the spec +template, the source-isolation discipline (no reading under +`crates/`, `runtime/`, `bench/`), and the per-finding classification +rules. This skill file only governs trigger, dispatch, and handoff. + ## When to Use / Skipping Triggers: @@ -53,144 +62,24 @@ EVERY EXAMPLE IS WRITTEN IN .ailx (SURFACE) FIRST. RAW .ail.json IS NEVER HAND-A EVERY FRICTION POINT AND BUG IS RECORDED. NONE IS WORKED AROUND. ``` -The first clause is the load-bearing one. The whole point of the -field test is to simulate a downstream LLM author who has only the -specification and the example corpus. If the dispatched agent reads -`crates/ailang-check/`, `crates/ailang-codegen/`, or any other -implementation source, the test contaminates: it now reflects what -the code happens to support, not what DESIGN.md promises. The carrier -contract enforces this; the agent's standing reading list re-states -it. +The first clause is load-bearing: the whole point of the field test is +to simulate a downstream LLM author who has only the specification and +the example corpus. The agent file enforces this with a hard path +allowlist; the orchestrator must trust that contract and not feed the +agent compiler-internal hints in the carrier. -The second clause exists because `.ail.json` is the canonical storage -form, but `.ailx` is the form an LLM author is meant to reach for. -Hand-writing JSON skips exactly the part of the language we want to -field-test. If the surface form makes a particular pattern painful, -that is a finding — not a reason to drop down to JSON. +## Dispatch -## The Process — five phases - -### Phase 1 — Scope the field test - -The orchestrator collects: -- the milestone identifier and the JOURNAL entries describing what - shipped, -- the milestone's design spec (`docs/specs/.md`) if one - exists, -- a 1-line scope hint per axis the milestone touched (e.g. "23 - shipped typeclass dictionaries: try a Functor and an Eq instance", - "20 shipped prose projection: try one prose-first authoring - workflow"). +Dispatch `ailang-fieldtester` with the carrier from the Handoff +Contract below. The agent picks 2-4 examples (one per axis the +milestone touched), implements them in `.ailx`, runs them through the +public `ail` CLI, classifies findings, writes the spec, and commits +both fixtures and spec in one commit +(`fieldtest: — examples, findings`). Two to four examples is the right size. One is too few (no signal on variation); five is too many for one report to stay readable. -### Phase 2 — Dispatch `ailang-fieldtester` - -Hand the agent the carrier (see Handoff Contract). The agent: -1. Reads DESIGN.md + JOURNAL tail + milestone spec + a representative - slice of `examples/` (to learn the *form* of valid AIL — not the - *content* of the compiler). -2. Picks the example tasks. Each task is a real, named programming - problem (FizzBuzz, polynomial root via Newton's method, JSON-ish - tree depth, BFS over a small graph, run-length encoder, …) chosen - so it exercises the milestone's deliveries without being a - compiler torture-test. Toy problems find toy bugs. -3. For each task: - - drafts the program in `.ailx` Surface form, - - saves it as `examples/fieldtest/__.ailx`, - - runs `ail check` and `ail build` and `ail run`, - - records the experience: what the agent reached for, what it had - to back out of, what the diagnostic said, what surprised it, - where DESIGN.md was silent or ambiguous. - -The agent has the `ail` CLI as a black box. It must NOT open -`crates/`, `runtime/`, or any internal source. It MAY read other -`.ailx`/`.ail.json` files in `examples/` to learn the form. - -### Phase 3 — Friction & bug classification - -Each finding is classified into exactly one of: - -| Class | Meaning | Downstream action | -|-------|---------|-------------------| -| `bug` | Compiler/runtime misbehaves vs. DESIGN.md (panic, wrong stdout, misleading diagnostic, schema break). | Hand off to `debug` (RED-test first). | -| `friction` | Compiles and runs, but the LLM-author had to fight the surface to express the thing. Surface clumsy, redundancy forced, mode/clause dance ugly. | Feeds next milestone's `brainstorm` or a tidy iteration's `plan`. | -| `spec_gap` | DESIGN.md does not constrain the case; the agent had to guess. The compiler picked a reading; another reading was equally plausible. | Orchestrator decides: tighten DESIGN.md, or ratify the existing reading in DESIGN.md, with a JOURNAL entry. | -| `working` | The author reached for the milestone's new surface and got it right on the first try with a clean diagnostic when wrong. Worth recording — these are the wins that protect the feature from drift attempts later. | - -### Phase 4 — Self-review the spec - -Before commit, the agent (or the orchestrator on its behalf) checks: -1. **Source-isolation check:** is any finding phrased as "the - compiler does X"? If so, replace with "running `ail run` produced - X". Findings must reference observable behaviour, not internal - mechanism. -2. **Surface-only check:** every shipped example is a `.ailx` file. - No hand-written `.ail.json` for fieldtest examples. -3. **Classification check:** every finding has exactly one class. A - friction-and-bug item gets two entries. -4. **Reproducibility check:** every `bug` carries the one-line repro - command and the observed output. - -### Phase 5 — Commit + handoff - -``` -examples/fieldtest/__.ailx (Surface form) -examples/fieldtest/__.ail.json (canonical, generated via `ail render --json` or equivalent) -docs/specs/-fieldtest-.md (the report) -``` - -Commit pattern (one commit per fieldtest run): - -```bash -git add examples/fieldtest/*.ail* docs/specs/*-fieldtest-*.md -git commit -m "fieldtest: — examples, findings" -``` - -Hand the orchestrator: -- the spec path, -- the per-finding classification, -- a one-line recommendation per finding (`debug` / `plan` / `ratify` / - `carry-on`). - -The orchestrator dispatches the follow-ups; `fieldtest` does NOT -self-resolve. - -## Spec structure - -`docs/specs/-fieldtest-.md`: - -```markdown -# Fieldtest — — - -**Status:** Draft — awaiting orchestrator triage -**Author:** ailang-fieldtester (dispatched by skills/fieldtest) - -## Scope -What the milestone shipped. One paragraph. - -## Examples -Per example, one subsection: -### `examples/fieldtest/__.ailx` — -- What it does -- Why this task fits the milestone's scope -- Outcome: compiles? runs? matches expected stdout? - -## Findings -Per finding, one entry: -### [class] -- Example(s) where it surfaced -- What happened (verbatim diagnostic / wrong output / etc.) -- Why it is friction / bug / spec_gap / working -- (for `bug`) one-line repro -- Recommended downstream action - -## Recommendation summary -A short table mapping finding → action (`debug` / `plan` / `ratify` / -`carry-on`). -``` - ## Handoff Contract `fieldtest` consumes (from orchestrator at milestone close): @@ -208,44 +97,25 @@ A short table mapping finding → action (`debug` / `plan` / `ratify` / |-------|---------| | `spec_path` | `docs/specs/-fieldtest-.md` | | `examples_added` | list of `.ailx` paths committed | -| `findings` | list, each with class + recommendation | +| `findings` | list, each with class (`bug` / `friction` / `spec_gap` / `working`) + recommendation | | `status` | `clean` / `friction_found` / `bugs_found` / `infra_blocked` | -The orchestrator drives downstream: bugs → `debug`; friction or -spec_gap → next `brainstorm` or a tidy iteration via `plan`. +The orchestrator drives downstream: +- `bug` → `debug` (RED-first; GREEN via `implement` mini-mode) +- `friction` → next `brainstorm` or a tidy iteration via `plan` +- `spec_gap` → ratify in JOURNAL + DESIGN.md, or tighten DESIGN.md +- `working` → carry-on (worth recording, no follow-up) -## Common Rationalisations - -| Excuse | Reality | -|--------|---------| -| "I'll just peek at `crates/ailang-check/` to understand the new typeclass coherence rule" | Peeking ends the test. The whole skill exists because the downstream LLM author cannot peek. If DESIGN.md does not answer the question, that is a `spec_gap` finding — the most valuable kind. | -| "Hand-writing JSON is faster, the surface is just sugar" | The surface is the field-test target. JSON-direct skips it. If you want to skip the surface, you have skipped the field test. | -| "Diagnostic is unclear but I figured out the fix; the example is fine" | The diagnostic being unclear *is* the friction. Record it verbatim. | -| "Two examples is enough, both worked" | Two working examples on the milestone's primary axis is a thin signal. Pick at least one example per axis the milestone touched. | -| "DESIGN.md does not say what should happen here, I'll just pick the obvious reading" | The "obvious" reading IS the spec gap. Record it. The orchestrator decides whether to ratify or tighten. | -| "The example would be cleaner if I added one helper, even though the milestone didn't ship it" | The field test runs the language as shipped. Helpers you wish existed go in the friction list, not into the example. | -| "Bug found, I'll just fix it while I'm here, faster than handing off" | Fix-in-place violates the skill split. Record as `bug`, hand to `debug` (RED-first). The fix lands in a separate, tested commit. | -| "audit said clean, I'm sure the field test will too — skip" | audit and fieldtest answer different questions. audit: does code match DESIGN? fieldtest: can a downstream LLM use what shipped? Both gates exist for a reason. | - -## Red Flags — STOP - -- About to open any file under `crates/`, `runtime/`, or `bench/`'s - internals -- About to hand-write `.ail.json` for a fieldtest example -- About to silently swap an example to a smaller form because the - intended one didn't compile (the failure-to-compile IS the finding) -- About to merge a `bug` finding into a `friction` finding "because - it's borderline" — pick one class -- About to skip Phase 4 self-review because "I wrote it carefully" -- About to attempt the bug fix in this dispatch (the skill is RED-by- - observation; GREEN goes to `debug` → `implement`) -- About to recommend more than one downstream action per finding +`fieldtest` does NOT self-resolve. ## Cross-references - **Agent dispatched:** `skills/fieldtest/agents/ailang-fieldtester.md` + — carries the five-phase process, the Iron Law in operational form, + the spec template, the per-finding classification rules, the Common + Rationalisations table, and the Red Flags list. - **Upstream gate:** `skills/audit/SKILL.md` runs first; fieldtest - comes after a clean (or ratified) audit + comes after a clean (or ratified) audit. - **Downstream targets:** - `skills/debug/SKILL.md` — for each `bug` finding - `skills/brainstorm/SKILL.md` or `skills/plan/SKILL.md` — for diff --git a/skills/fieldtest/agents/ailang-fieldtester.md b/skills/fieldtest/agents/ailang-fieldtester.md index 0eb3008..a929dda 100644 --- a/skills/fieldtest/agents/ailang-fieldtester.md +++ b/skills/fieldtest/agents/ailang-fieldtester.md @@ -164,8 +164,7 @@ merged. ### Phase 5 — Write the spec, commit, hand back Write `docs/specs/-fieldtest-.md` using the -structure documented in `skills/fieldtest/SKILL.md` ("Spec -structure"). Then: +spec structure below. Then: ```bash git add examples/fieldtest/*.ail* docs/specs/*-fieldtest-*.md @@ -174,6 +173,40 @@ git commit -m "fieldtest: — examples, findings" Report `DONE` with the carrier (see Output format). +## Spec structure + +`docs/specs/-fieldtest-.md`: + +```markdown +# Fieldtest — — + +**Status:** Draft — awaiting orchestrator triage +**Author:** ailang-fieldtester (dispatched by skills/fieldtest) + +## Scope +What the milestone shipped. One paragraph. + +## Examples +Per example, one subsection: +### `examples/fieldtest/__.ailx` — +- What it does +- Why this task fits the milestone's scope +- Outcome: compiles? runs? matches expected stdout? + +## Findings +Per finding, one entry: +### [class] +- Example(s) where it surfaced +- What happened (verbatim diagnostic / wrong output / etc.) +- Why it is friction / bug / spec_gap / working +- (for `bug`) one-line repro +- Recommended downstream action + +## Recommendation summary +A short table mapping finding → action (`debug` / `plan` / `ratify` / +`carry-on`). +``` + ## Status protocol End every report with exactly one of: