From 0ee5af9ef1e972adaca2bd570f05736cbe247395 Mon Sep 17 00:00:00 2001 From: Brummel Date: Mon, 11 May 2026 13:27:30 +0200 Subject: [PATCH] spec: au.1 audit-orchestrator agent --- .../2026-05-11-audit-orchestrator-agent.md | 476 ++++++++++++++++++ 1 file changed, 476 insertions(+) create mode 100644 docs/specs/2026-05-11-audit-orchestrator-agent.md diff --git a/docs/specs/2026-05-11-audit-orchestrator-agent.md b/docs/specs/2026-05-11-audit-orchestrator-agent.md new file mode 100644 index 0000000..546c5dd --- /dev/null +++ b/docs/specs/2026-05-11-audit-orchestrator-agent.md @@ -0,0 +1,476 @@ +# `ailang-audit-orchestrator` — Design Spec + +**Date:** 2026-05-11 +**Status:** Draft — awaiting user spec review +**Authors:** Brummel (orchestrator) + Claude + +## Goal + +Extend the Boss-context-offload pattern established in `or.1` / +corrected in `or.2` to the post-implement validation cycle. The Boss +should only carry the milestone-close context that involves a +**decision**: classify a drift item, ratify a baseline shift, dispatch +a tidy iter. Everything that's pure execution — running architect, +running the bench scripts, running docwriter, running fieldtester — +moves into a dedicated subagent context (`ailang-audit-orchestrator`) +and reaches the Boss only as a structured end-report. + +The two existing skills (`audit`, `fieldtest`) and their four worker +agents (`ailang-architect`, `ailang-bencher`, `ailang-docwriter`, +`ailang-fieldtester`) **remain unchanged in body**. What changes is +their *caller*: at milestone-close, the Boss dispatches the new +orchestrator agent, which inhales the audit + fieldtest discipline as +sequential role-switches in its own context — the same shape as the +or.2-corrected `ailang-implement-orchestrator`. + +## Background — why now + +Three observations from 2026-05-11 motivate this spec: + +1. **`or.1` proved the offload pattern.** Per-`/implement` Boss-context + cost dropped from ~100k to ~700 tokens by relocating the per-task + loop into the implement-orchestrator subagent. + +2. **`or.2` defined the constraint.** Claude Code categorically forbids + nested subagent dispatch. The corrected architecture uses inline + role-switches inside the subagent's own context. Phase-reference + files (`ailang-implementer.md`, etc.) define the discipline; the + orchestrator inhales them per phase. + +3. **Milestone close was left out.** Audit + fieldtest currently run + as direct Boss-dispatched calls: architect report → Boss-side bench + bash → docwriter dispatch → fieldtest dispatch, with each report + landing in the Boss-context. Roughly 5–15k tokens per + milestone-close. Not the 100×-hebel of `/implement`, but a + real cost and — more importantly — a structural inconsistency: the + per-iter execution path is offloaded, the post-iter validation path + is not. + +The user-stated principle that governs this spec: + +> **Boss-context is decision-context.** Everything else moves out. + +Concretely: +- Audit + fieldtest clean → Boss gets a compact end-report ("milestone + closes, nothing to decide"), writes the milestone-close JOURNAL + entry, advances. +- Audit + fieldtest find anything actionable → Boss gets a structured + findings report and decides per item: `fix` / `ratify` / `carry-on`. + The decision and the downstream dispatch (`planner` + `implement` + for `fix`, `--update-baseline` for `ratify`) remain Boss-level work, + as today. + +## Architecture + +``` +Boss-Orchestrator + │ + │ Agent("ailang-audit-orchestrator", carrier) + ▼ +ailang-audit-orchestrator ← runs in its own context + │ + │ Phase 1 — architect role-switch + │ reads skills/audit/agents/ailang-architect.md + │ performs the drift review inline + │ collects: prioritised drift list + │ + │ Phase 2 — bencher role-switch + │ runs bench/check.py && bench/compile_check.py && bench/cross_lang.py + │ collects: exit code + raw numbers + │ HALT on exit 2 (infra-broken — subsequent phases unreliable) + │ + │ Phase 3 — docwriter role-switch (conditional) + │ cargo doc --no-deps 2>&1 → warnings? + │ if yes: read skills/audit/agents/ailang-docwriter.md, run inline, + │ commit rustdoc fixes on main + │ collects: warning count, list of fixed warnings (or 0) + │ + │ Phase 4 — fieldtester role-switch (conditional, carrier flag) + │ if run_fieldtest && Phase 1 architect did not flag critical drift: + │ reads skills/fieldtest/agents/ailang-fieldtester.md + │ writes 2–4 .ailx examples in examples/, runs ail check/build/run, + │ writes docs/specs/-fieldtest-.md, + │ commits as `fieldtest: — examples, findings` + │ collects: spec_path, examples_added, findings, status + │ + │ Phase 5 — end-report assembly + │ structured findings → compact text → ≤500 tokens + │ JOURNAL draft (Boss may accept/edit) → /tmp/ail-audit//draft.md + ▼ +end-report → Boss + +Boss-Orchestrator (after reading end-report): + │ + │ SUCCESS (status: clean): + │ read /tmp/ail-audit/.../draft.md (optional) + │ write docs/journals/-milestone--close.md + │ append docs/journals/INDEX.md + │ (no further dispatches — milestone closes) + │ + │ FAILURE (status: findings_*): + │ classify each drift_item / bench finding / fieldtest finding + │ → fix → dispatch planner + implement-orchestrator (tidy iter) + │ → ratify → run bench/.py --update-baseline + JOURNAL ratify entry + │ → carry-on → record in JOURNAL, no commit + │ on bug from fieldtest: dispatch debug skill +``` + +The Boss-Orchestrator sees: one dispatch out, one compressed report +back. Per-phase work — drift list assembly, bench bash output, rustdoc +fix loops, fieldtest implementation cycles — all happen inside the +orchestrator-agent's context and never reach the Boss. + +Same structural pattern as `or.2`: the four phase-reference files +(`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`, +`ailang-fieldtester.md`) stay in their current locations. The +orchestrator reads each one verbatim at its role-switch and applies +the discipline inline. **No nested subagent dispatch.** + +## Components + +### `ailang-audit-orchestrator` (new agent) + +Location: `skills/audit/agents/ailang-audit-orchestrator.md`. + +Tools: `Read, Edit, Write, Bash, Glob, Grep`. **No `Agent` tool** — +the or.2 Iron Law applies (Claude Code does not permit nested +subagent dispatch). + +Standing reading list: +1. `CLAUDE.md` — orchestrator framing. +2. `docs/DESIGN.md` — invariants the audit must hold against. +3. `docs/journals/INDEX.md` plus the last 1–3 referenced per-iter + files — recent state to inform the architect-phase scope. +4. `skills/audit/SKILL.md` — canonical audit discipline. +5. `skills/fieldtest/SKILL.md` — canonical fieldtest discipline, + conditionally relevant. +6. Phase-reference files (read per-phase, not all upfront): + - `skills/audit/agents/ailang-architect.md` + - `skills/audit/agents/ailang-bencher.md` + - `skills/audit/agents/ailang-docwriter.md` + - `skills/fieldtest/agents/ailang-fieldtester.md` + +Model: Opus 4.7. + +Iron Law (verbatim, derived from or.2): +``` +PER-PHASE WORK RUNS SEQUENTIALLY IN YOUR OWN CONTEXT — architect role-switch, +then bencher, then docwriter, then fieldtester. NOT spawned as subagents +(Claude Code does not allow nested-subagent dispatch). + +BENCH EXIT CODE 2 HALTS THE RUN. Subsequent phases would run on a broken +infrastructure and produce unreliable findings. Report `infra_blocked` and stop. + +EVERY PHASE COMPLETES (or halts on the bench-2 condition). FINDINGS AGGREGATE. +Boss decides per item; the orchestrator does NOT pre-resolve. +``` + +### `skills/audit/SKILL.md` (small additive change) + +Today's body describes the five-step process for Boss-direct +invocation. After this spec: + +- Gain a short "Standard Mode" subsection at the top, saying: + > "At milestone close: dispatch `ailang-audit-orchestrator`. The + > orchestrator runs Steps 1–3 + the conditional fieldtest phase + > inline, returns a structured end-report. The Boss applies Steps 4 + > (Classify) and 5 (Resolve) on the report — these remain Boss + > work." + +- The existing Steps 1–5 are retained verbatim. They now serve two + audiences: + - The orchestrator-agent reads them as **phase-reference** + (alongside the per-agent files). + - The Boss reads them when invoking audit in **Manual Mode** (drift + suspected outside milestone close, debugging an isolated phase, + etc.). Manual Mode dispatch shape unchanged. + +- The Cross-references section gains: + - Standard-mode dispatcher: `skills/audit/agents/ailang-audit-orchestrator.md` + - "Why inline phases, not nested subagents" pointer to the or.2 + journal. + +### `skills/fieldtest/SKILL.md` (cross-reference update only) + +Today's body is unchanged. Fieldtest's standard trigger expands from +"orchestrator dispatches `ailang-fieldtester` after audit closes" to +"either the audit-orchestrator runs phase 4 inline, OR the Boss +dispatches `ailang-fieldtester` directly in Manual Mode". The agent +file (`ailang-fieldtester.md`) is unchanged. + +Manual Mode for fieldtest covers the genuine edge case: audit found +drift, a tidy-iter landed, re-running fieldtest in isolation makes +sense, no point re-running the whole audit-orchestrator. + +### Worker agents — unchanged in body, expanded role + +`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`, +`ailang-fieldtester.md` keep their current frontmatter, tools, +standing reading lists, status protocols, output formats. + +What changes is their role-in-the-system: they serve as **phase +references** when invoked by the audit-orchestrator (Standard Mode), +AND as **directly-dispatchable subagents** when invoked by the Boss +(Manual Mode). The dual duty mirrors how or.2 left +`ailang-implementer.md` and siblings in the implement skill. + +Same caveat applies as in or.2 known-debt: these files frame +themselves as "dispatched subagents receiving a carrier from a +dispatcher". In Standard Mode the orchestrator IS the dispatcher and +the worker. If the first real audit-orchestrator run shows confusion +during phase reads, a follow-up cleanup iter rewrites the framing. +Not pre-emptive. + +### `skills/README.md` (skill table + roster + pipeline) + +- Skill table row for `audit`: gains a one-line note about + Standard-Mode dispatcher. +- Agent roster: gains row for `ailang-audit-orchestrator` (dispatched + by `audit`, Standard Mode). Existing rows for architect / bencher / + docwriter / fieldtester gain "and serves as phase reference for + ailang-audit-orchestrator" suffix, analogous to or.2. +- Pipeline diagram: the `audit -> fieldtest` chain at milestone-close + is replaced by a single `audit (via ailang-audit-orchestrator)` + node; the conditional fieldtest is now phase 4 of that single + dispatch. + +## Data flow + +### Carrier — Boss → `ailang-audit-orchestrator` + +| Field | Content | +|-------|---------| +| `milestone_id` | e.g. `"ct.2"` or `"22"` (string, free-form, used in commit messages and filenames) | +| `commit_range` | e.g. `"..HEAD"` — the architect's scope | +| `run_fieldtest` | bool — set by Boss based on milestone spec. `true` if milestone touched user-visible surface (Decision 6, schema, effects, types, modes, diagnostics); `false` for purely internal milestones | +| `fieldtest_axis_hints` | (only if `run_fieldtest = true`) bullet list, one per milestone axis the fieldtest should probe — copied verbatim from the milestone spec | +| `milestone_scope` | (only if `run_fieldtest = true`) 1–3 sentences naming what shipped, for the fieldtester's carrier | + +### End-report — `ailang-audit-orchestrator` → Boss + +Plain-text report, ≤ 500 tokens, fixed structure: + +``` +Status: clean | findings_drift | findings_bench | findings_rustdoc + | findings_fieldtest | findings_multiple | infra_blocked +Milestone: + +Phase 1 (architect): + Drift items: + - [] : <1-line justification> + - ... + (or: none) + Recommendation: + (verbatim from architect output) + +Phase 2 (bench): + Exit code: 0 | 1 | 2 + Numbers: + +Phase 3 (rustdoc): + Warnings: (or 0) + - + - ... + Auto-fixed: yes | no | partial (orchestrator may commit fixes) + +Phase 4 (fieldtest): + Ran: yes | no (reason: run_fieldtest=false | architect critical-drift) + Status: clean | friction_found | bugs_found | (n/a) + Spec: docs/specs/-fieldtest-.md (committed) + Findings: (one line each, with class) + +Suggested classification (Boss decides): + - : fix | ratify | carry-on | needs-debug + - ... + (Orchestrator's heuristic suggestion. Boss applies audit-Step-4 + semantics and may override per item.) + +Journal draft: /tmp/ail-audit//draft.md +Commits made: +``` + +### Boss-side reading workflow (on clean) + +1. Read the end-report's `Status: clean` confirmation. +2. Optionally read `/tmp/ail-audit//draft.md`. +3. Write `docs/journals/-milestone--close.md`, accepting the + orchestrator's draft as a starting point or rewriting the summary. +4. Append a one-line entry to `docs/journals/INDEX.md`. +5. (If trigger is done-state: WhatsNew + notify per CLAUDE.md.) +6. Milestone closes; pick up the next item from `docs/roadmap.md`. + +### Boss-side workflow (on findings) + +1. Read the end-report's structured findings. +2. Per item, classify: `fix` / `ratify` / `carry-on` / `needs-debug`. +3. Dispatch downstream as today: + - `fix` → `planner` + `implement` for a tidy iter + - `ratify` → `bench/.py --update-baseline` + JOURNAL ratify + entry + - `carry-on` → record in JOURNAL, no commit + - `needs-debug` → `debug` skill (RED-first, GREEN via implement + mini-mode) +4. Write `docs/journals/-milestone--close.md` summarising + findings + classification + downstream-actions-taken. +5. If `findings_fieldtest` includes `friction` or `spec_gap` worth a + follow-up: `brainstorm` for next milestone OR add to roadmap. + +## Error handling + +### Phase-level outcomes + +| Phase | Possible outcomes | Orchestrator action | +|-------|-------------------|---------------------| +| Phase 1 (architect) | `drift_found` / `clean` | Always continue to phase 2. Drift severity is metadata in the report, NOT a halt condition. | +| Phase 2 (bench) | Exit 0 / 1 / 2 | 0 or 1: continue. **2: HALT** — report `infra_blocked`, skip phases 3+4, exit with `Status: infra_blocked`. | +| Phase 3 (docwriter) | Warnings present / absent / fix-failed | Continue regardless. If warnings present, attempt fix and commit; record outcome. If fix fails, record `Auto-fixed: no` and pass through. | +| Phase 4 (fieldtest) | clean / friction / bugs / infra_blocked | Skipped entirely if `run_fieldtest=false` or if architect flagged a `priority: highest` drift item. Skip is recorded as `Ran: no (reason: architect-highest-priority-drift)`. Always reported, never halts. | + +### Why aggregate, not abort-at-first-failure + +The four phases test independent dimensions: +- Architect: code vs. DESIGN.md (semantic drift). +- Bench: code vs. baseline (performance drift). +- Docwriter: code vs. rustdoc (documentation drift). +- Fieldtester: code vs. real-world LLM-author usage (usability drift). + +A drift in one is not predictive of a drift in another. Aborting at +phase 1 on architect-drift would force the Boss into a fix-then-re-run +loop: fix architect drift, re-run, discover bench regression, fix +that, re-run, discover fieldtest bug. Aggregating gives Boss the +whole picture in one pass. + +The ONE exception is bench exit 2: the bench infrastructure itself is +broken, which means later phases run on an unreliable system. Halting +is correct there. + +### Critical-drift skip on phase 4 + +The architect-phase output classifies drift items by `priority` +(highest / high / medium — see `skills/audit/agents/ailang-architect.md` +output format). If any item is `priority: highest` (i.e. drift against +`docs/DESIGN.md` — an invariant violation, a fundamental schema break), +phase 4 is skipped. Rationale: fieldtest writes new examples that +exercise the surface; on a codebase that violates DESIGN.md, those +examples generate noise findings that the Boss must triage instead of +acting on the actual `highest`-priority drift. + +`priority: highest` is the architect's call, not the orchestrator's. +The orchestrator just reads priority from the architect-phase output +and gates phase 4 on it. `priority: high` items (drift against the +milestone spec, not DESIGN.md) do NOT trigger the skip — the +language's invariants still hold; fieldtest can still produce useful +signal. + +### `NEEDS_CONTEXT` from orchestrator + +Returned if `milestone_id` or `commit_range` is ambiguous, or if +`run_fieldtest=true` but `fieldtest_axis_hints` / `milestone_scope` is +missing. Boss provides the missing carrier fields and re-dispatches. + +### `BLOCKED` from orchestrator + +Returned on unrecoverable internal error (e.g., a phase-reference file +missing, a bash binary not found, a git operation failing +non-trivially). Halts immediately. Boss diagnoses and either fixes +infrastructure + re-dispatches, or escalates. + +## Testing strategy + +This is a skill/agent refactor, not a language feature. No `cargo +test` coverage applies directly. Verification ladder: + +1. **Agent file parses** — `Agent("ailang-audit-orchestrator", ...)` + resolves to a runnable spec. Confirm via a no-op dispatch on a + trivial milestone fixture (e.g., commit-range with no real + changes). +2. **All four phases produce structured output** — dispatch on a real + recent milestone, verify the end-report has all five sections + populated (or `n/a` with reason). +3. **Bench exit 2 halts** — temporarily break a bench fixture, verify + the orchestrator stops after Phase 2 with `infra_blocked` and does + NOT run phases 3+4. +4. **Critical-drift skip on phase 4** — temporarily inject a fake + `severity=high` architect finding, verify phase 4 is skipped with + the documented reason. +5. **`run_fieldtest=false` skip** — dispatch on a purely-internal + milestone, verify phase 4 is skipped with the documented reason. +6. **Aggregate, not abort** — dispatch on a milestone with known + architect-drift, verify phases 2, 3, 4 still run despite the + architect finding. +7. **Boss-side workflow on clean** — manually integrate a clean run: + read draft journal, write milestone-close journal, append INDEX. + Verify linear history and INDEX coherence. +8. **Manual-Mode preserved** — invoke `audit` skill directly (without + the orchestrator) — verify the current Boss-direct dispatch path + still works for ad-hoc invocations. + +Verification steps 3, 4, 6 require synthetic fixtures and are +one-shot reproducers — they confirm the discipline once. Future audit +runs answer "is the design holding up" empirically. + +## Acceptance criteria + +The refactor is accepted when: + +1. `skills/audit/agents/ailang-audit-orchestrator.md` exists, following + the standard agent template (frontmatter, spirit-letter lead-in, + standing reading list, carrier contract, Iron Law, process, status + protocol, output format, common rationalisations, red flags). +2. `skills/audit/SKILL.md` gains a Standard-Mode subsection naming the + orchestrator-agent as the dispatcher at milestone close. Steps 1–5 + retained verbatim as phase references + Manual-Mode path. Cross- + references updated. +3. `skills/fieldtest/SKILL.md` cross-references updated to reflect + that Phase 4 of the audit-orchestrator is now the standard trigger; + Manual Mode path retained. +4. `skills/README.md` skill table, agent roster, and pipeline diagram + updated. +5. The four worker agent files (`ailang-architect.md`, + `ailang-bencher.md`, `ailang-docwriter.md`, + `ailang-fieldtester.md`) are **unchanged in body**. (The known-debt + re: subagent framing carries over from or.2.) +6. The eight verification steps in Testing strategy pass. +7. The first real milestone close after the refactor shows a + Boss-context delta consistent with the predicted offload (target: + ≤ 1k tokens per audit-orchestrator dispatch + end-report, plus the + Boss-side decision context for whatever findings emerge). + +## Migration + +Single trivial-mechanical addition. No code touched, no language +behaviour change. Two ways to land: + +**Option (a) — single direct-on-main commit.** The new agent file, +plus the additive edits to `audit/SKILL.md`, `fieldtest/SKILL.md`, +`README.md`. Analogous to `or.2`'s landing. + +**Option (b) — `iter au.1` via the planner + implement loop.** Forces +the new orchestrator-agent through its own discipline cycle (plan → +TDD → review). Adds about a day of latency for a refactor that has +no executable surface to test against. + +**Recommendation: (a).** Same reasoning as `or.2`: doc-only revision, +no behaviour to test, the first real audit at the next milestone +close IS the verification step. Option (b) is process-theatre for +this case. + +## Out of scope (deferred to other specs) + +- **Brainstorm offload** — the `/brainstorm` usage statistic was 2% + pre-or.1; not cost-effective to encapsulate yet. +- **Planner offload** — the `pr.1` recon-subagent already addresses + the read-heavy phase of planning; further offload is premature. +- **Audit-result auto-resolution** — the Boss's `fix / ratify / + carry-on` decision is semantic, not mechanical. Pushing this into + the orchestrator would violate the user-stated principle that + Boss-context is decision-context. Out of scope on principle, not + on effort. +- **Per-milestone-close branch isolation** — implement-orchestrator + uses iter-branches; audit-orchestrator commits direct-on-main + because per-phase changes (docwriter rustdoc fixes, fieldtest + fixtures) are individually safe and don't benefit from + iter-isolation. May be reconsidered if a future audit produces + larger-scale changes that warrant branching. +- **Worker model downgrade** — all roles stay at Opus 4.7, as in + or.1.