# `ailang-audit-orchestrator` — Design Spec **Date:** 2026-05-11 **Status:** Draft — awaiting user spec review **Authors:** Brummel (orchestrator) + Claude ## Goal Extend the Boss-context-offload pattern established in `or.1` / corrected in `or.2` to the post-implement validation cycle. The Boss should only carry the milestone-close context that involves a **decision**: classify a drift item, ratify a baseline shift, dispatch a tidy iter. Everything that's pure execution — running architect, running the bench scripts, running docwriter, running fieldtester — moves into a dedicated subagent context (`ailang-audit-orchestrator`) and reaches the Boss only as a structured end-report. The two existing skills (`audit`, `fieldtest`) and their four worker agents (`ailang-architect`, `ailang-bencher`, `ailang-docwriter`, `ailang-fieldtester`) **remain unchanged in body**. What changes is their *caller*: at milestone-close, the Boss dispatches the new orchestrator agent, which inhales the audit + fieldtest discipline as sequential role-switches in its own context — the same shape as the or.2-corrected `ailang-implement-orchestrator`. ## Background — why now Three observations from 2026-05-11 motivate this spec: 1. **`or.1` proved the offload pattern.** Per-`/implement` Boss-context cost dropped from ~100k to ~700 tokens by relocating the per-task loop into the implement-orchestrator subagent. 2. **`or.2` defined the constraint.** Claude Code categorically forbids nested subagent dispatch. The corrected architecture uses inline role-switches inside the subagent's own context. Phase-reference files (`ailang-implementer.md`, etc.) define the discipline; the orchestrator inhales them per phase. 3. **Milestone close was left out.** Audit + fieldtest currently run as direct Boss-dispatched calls: architect report → Boss-side bench bash → docwriter dispatch → fieldtest dispatch, with each report landing in the Boss-context. Roughly 5–15k tokens per milestone-close. Not the 100×-hebel of `/implement`, but a real cost and — more importantly — a structural inconsistency: the per-iter execution path is offloaded, the post-iter validation path is not. The user-stated principle that governs this spec: > **Boss-context is decision-context.** Everything else moves out. Concretely: - Audit + fieldtest clean → Boss gets a compact end-report ("milestone closes, nothing to decide"), writes the milestone-close JOURNAL entry, advances. - Audit + fieldtest find anything actionable → Boss gets a structured findings report and decides per item: `fix` / `ratify` / `carry-on`. The decision and the downstream dispatch (`planner` + `implement` for `fix`, `--update-baseline` for `ratify`) remain Boss-level work, as today. ## Architecture ``` Boss-Orchestrator │ │ Agent("ailang-audit-orchestrator", carrier) ▼ ailang-audit-orchestrator ← runs in its own context │ │ Phase 1 — architect role-switch │ reads skills/audit/agents/ailang-architect.md │ performs the drift review inline │ collects: prioritised drift list │ │ Phase 2 — bencher role-switch │ runs bench/check.py && bench/compile_check.py && bench/cross_lang.py │ collects: exit code + raw numbers │ HALT on exit 2 (infra-broken — subsequent phases unreliable) │ │ Phase 3 — docwriter role-switch (conditional) │ cargo doc --no-deps 2>&1 → warnings? │ if yes: read skills/audit/agents/ailang-docwriter.md, run inline, │ commit rustdoc fixes on main │ collects: warning count, list of fixed warnings (or 0) │ │ Phase 4 — fieldtester role-switch (conditional, carrier flag) │ if run_fieldtest && Phase 1 architect did not flag critical drift: │ reads skills/fieldtest/agents/ailang-fieldtester.md │ writes 2–4 .ailx examples in examples/, runs ail check/build/run, │ writes docs/specs/-fieldtest-.md, │ commits as `fieldtest: — examples, findings` │ collects: spec_path, examples_added, findings, status │ │ Phase 5 — end-report assembly │ structured findings → compact text → ≤500 tokens │ JOURNAL draft (Boss may accept/edit) → /tmp/ail-audit//draft.md ▼ end-report → Boss Boss-Orchestrator (after reading end-report): │ │ SUCCESS (status: clean): │ read /tmp/ail-audit/.../draft.md (optional) │ write docs/journals/-milestone--close.md │ append docs/journals/INDEX.md │ (no further dispatches — milestone closes) │ │ FAILURE (status: findings_*): │ classify each drift_item / bench finding / fieldtest finding │ → fix → dispatch planner + implement-orchestrator (tidy iter) │ → ratify → run bench/.py --update-baseline + JOURNAL ratify entry │ → carry-on → record in JOURNAL, no commit │ on bug from fieldtest: dispatch debug skill ``` The Boss-Orchestrator sees: one dispatch out, one compressed report back. Per-phase work — drift list assembly, bench bash output, rustdoc fix loops, fieldtest implementation cycles — all happen inside the orchestrator-agent's context and never reach the Boss. Same structural pattern as `or.2`: the four phase-reference files (`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`, `ailang-fieldtester.md`) stay in their current locations. The orchestrator reads each one verbatim at its role-switch and applies the discipline inline. **No nested subagent dispatch.** ## Components ### `ailang-audit-orchestrator` (new agent) Location: `skills/audit/agents/ailang-audit-orchestrator.md`. Tools: `Read, Edit, Write, Bash, Glob, Grep`. **No `Agent` tool** — the or.2 Iron Law applies (Claude Code does not permit nested subagent dispatch). Standing reading list: 1. `CLAUDE.md` — orchestrator framing. 2. `docs/DESIGN.md` — invariants the audit must hold against. 3. `docs/journals/INDEX.md` plus the last 1–3 referenced per-iter files — recent state to inform the architect-phase scope. 4. `skills/audit/SKILL.md` — canonical audit discipline. 5. `skills/fieldtest/SKILL.md` — canonical fieldtest discipline, conditionally relevant. 6. Phase-reference files (read per-phase, not all upfront): - `skills/audit/agents/ailang-architect.md` - `skills/audit/agents/ailang-bencher.md` - `skills/audit/agents/ailang-docwriter.md` - `skills/fieldtest/agents/ailang-fieldtester.md` Model: Opus 4.7. Iron Law (verbatim, derived from or.2): ``` PER-PHASE WORK RUNS SEQUENTIALLY IN YOUR OWN CONTEXT — architect role-switch, then bencher, then docwriter, then fieldtester. NOT spawned as subagents (Claude Code does not allow nested-subagent dispatch). BENCH EXIT CODE 2 HALTS THE RUN. Subsequent phases would run on a broken infrastructure and produce unreliable findings. Report `infra_blocked` and stop. EVERY PHASE COMPLETES (or halts on the bench-2 condition). FINDINGS AGGREGATE. Boss decides per item; the orchestrator does NOT pre-resolve. ``` ### `skills/audit/SKILL.md` (small additive change) Today's body describes the five-step process for Boss-direct invocation. After this spec: - Gain a short "Standard Mode" subsection at the top, saying: > "At milestone close: dispatch `ailang-audit-orchestrator`. The > orchestrator runs Steps 1–3 + the conditional fieldtest phase > inline, returns a structured end-report. The Boss applies Steps 4 > (Classify) and 5 (Resolve) on the report — these remain Boss > work." - The existing Steps 1–5 are retained verbatim. They now serve two audiences: - The orchestrator-agent reads them as **phase-reference** (alongside the per-agent files). - The Boss reads them when invoking audit in **Manual Mode** (drift suspected outside milestone close, debugging an isolated phase, etc.). Manual Mode dispatch shape unchanged. - The Cross-references section gains: - Standard-mode dispatcher: `skills/audit/agents/ailang-audit-orchestrator.md` - "Why inline phases, not nested subagents" pointer to the or.2 journal. ### `skills/fieldtest/SKILL.md` (cross-reference update only) Today's body is unchanged. Fieldtest's standard trigger expands from "orchestrator dispatches `ailang-fieldtester` after audit closes" to "either the audit-orchestrator runs phase 4 inline, OR the Boss dispatches `ailang-fieldtester` directly in Manual Mode". The agent file (`ailang-fieldtester.md`) is unchanged. Manual Mode for fieldtest covers the genuine edge case: audit found drift, a tidy-iter landed, re-running fieldtest in isolation makes sense, no point re-running the whole audit-orchestrator. ### Worker agents — unchanged in body, expanded role `ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`, `ailang-fieldtester.md` keep their current frontmatter, tools, standing reading lists, status protocols, output formats. What changes is their role-in-the-system: they serve as **phase references** when invoked by the audit-orchestrator (Standard Mode), AND as **directly-dispatchable subagents** when invoked by the Boss (Manual Mode). The dual duty mirrors how or.2 left `ailang-implementer.md` and siblings in the implement skill. Same caveat applies as in or.2 known-debt: these files frame themselves as "dispatched subagents receiving a carrier from a dispatcher". In Standard Mode the orchestrator IS the dispatcher and the worker. If the first real audit-orchestrator run shows confusion during phase reads, a follow-up cleanup iter rewrites the framing. Not pre-emptive. ### `skills/README.md` (skill table + roster + pipeline) - Skill table row for `audit`: gains a one-line note about Standard-Mode dispatcher. - Agent roster: gains row for `ailang-audit-orchestrator` (dispatched by `audit`, Standard Mode). Existing rows for architect / bencher / docwriter / fieldtester gain "and serves as phase reference for ailang-audit-orchestrator" suffix, analogous to or.2. - Pipeline diagram: the `audit -> fieldtest` chain at milestone-close is replaced by a single `audit (via ailang-audit-orchestrator)` node; the conditional fieldtest is now phase 4 of that single dispatch. ## Data flow ### Carrier — Boss → `ailang-audit-orchestrator` | Field | Content | |-------|---------| | `milestone_id` | e.g. `"ct.2"` or `"22"` (string, free-form, used in commit messages and filenames) | | `commit_range` | e.g. `"..HEAD"` — the architect's scope | | `run_fieldtest` | bool — set by Boss based on milestone spec. `true` if milestone touched user-visible surface (Decision 6, schema, effects, types, modes, diagnostics); `false` for purely internal milestones | | `fieldtest_axis_hints` | (only if `run_fieldtest = true`) bullet list, one per milestone axis the fieldtest should probe — copied verbatim from the milestone spec | | `milestone_scope` | (only if `run_fieldtest = true`) 1–3 sentences naming what shipped, for the fieldtester's carrier | ### End-report — `ailang-audit-orchestrator` → Boss Plain-text report, ≤ 500 tokens, fixed structure: ``` Status: clean | findings_drift | findings_bench | findings_rustdoc | findings_fieldtest | findings_multiple | infra_blocked Milestone: Phase 1 (architect): Drift items: - [] : <1-line justification> - ... (or: none) Recommendation: (verbatim from architect output) Phase 2 (bench): Exit code: 0 | 1 | 2 Numbers: Phase 3 (rustdoc): Warnings: (or 0) - - ... Auto-fixed: yes | no | partial (orchestrator may commit fixes) Phase 4 (fieldtest): Ran: yes | no (reason: run_fieldtest=false | architect critical-drift) Status: clean | friction_found | bugs_found | (n/a) Spec: docs/specs/-fieldtest-.md (committed) Findings: (one line each, with class) Suggested classification (Boss decides): - : fix | ratify | carry-on | needs-debug - ... (Orchestrator's heuristic suggestion. Boss applies audit-Step-4 semantics and may override per item.) Journal draft: /tmp/ail-audit//draft.md Commits made: ``` ### Boss-side reading workflow (on clean) 1. Read the end-report's `Status: clean` confirmation. 2. Optionally read `/tmp/ail-audit//draft.md`. 3. Write `docs/journals/-milestone--close.md`, accepting the orchestrator's draft as a starting point or rewriting the summary. 4. Append a one-line entry to `docs/journals/INDEX.md`. 5. (If trigger is done-state: WhatsNew + notify per CLAUDE.md.) 6. Milestone closes; pick up the next item from `docs/roadmap.md`. ### Boss-side workflow (on findings) 1. Read the end-report's structured findings. 2. Per item, classify: `fix` / `ratify` / `carry-on` / `needs-debug`. 3. Dispatch downstream as today: - `fix` → `planner` + `implement` for a tidy iter - `ratify` → `bench/.py --update-baseline` + JOURNAL ratify entry - `carry-on` → record in JOURNAL, no commit - `needs-debug` → `debug` skill (RED-first, GREEN via implement mini-mode) 4. Write `docs/journals/-milestone--close.md` summarising findings + classification + downstream-actions-taken. 5. If `findings_fieldtest` includes `friction` or `spec_gap` worth a follow-up: `brainstorm` for next milestone OR add to roadmap. ## Error handling ### Phase-level outcomes | Phase | Possible outcomes | Orchestrator action | |-------|-------------------|---------------------| | Phase 1 (architect) | `drift_found` / `clean` | Always continue to phase 2. Drift severity is metadata in the report, NOT a halt condition. | | Phase 2 (bench) | Exit 0 / 1 / 2 | 0 or 1: continue. **2: HALT** — report `infra_blocked`, skip phases 3+4, exit with `Status: infra_blocked`. | | Phase 3 (docwriter) | Warnings present / absent / fix-failed | Continue regardless. If warnings present, attempt fix and commit; record outcome. If fix fails, record `Auto-fixed: no` and pass through. | | Phase 4 (fieldtest) | clean / friction / bugs / infra_blocked | Skipped entirely if `run_fieldtest=false` or if architect flagged a `priority: highest` drift item. Skip is recorded as `Ran: no (reason: architect-highest-priority-drift)`. Always reported, never halts. | ### Why aggregate, not abort-at-first-failure The four phases test independent dimensions: - Architect: code vs. DESIGN.md (semantic drift). - Bench: code vs. baseline (performance drift). - Docwriter: code vs. rustdoc (documentation drift). - Fieldtester: code vs. real-world LLM-author usage (usability drift). A drift in one is not predictive of a drift in another. Aborting at phase 1 on architect-drift would force the Boss into a fix-then-re-run loop: fix architect drift, re-run, discover bench regression, fix that, re-run, discover fieldtest bug. Aggregating gives Boss the whole picture in one pass. The ONE exception is bench exit 2: the bench infrastructure itself is broken, which means later phases run on an unreliable system. Halting is correct there. ### Critical-drift skip on phase 4 The architect-phase output classifies drift items by `priority` (highest / high / medium — see `skills/audit/agents/ailang-architect.md` output format). If any item is `priority: highest` (i.e. drift against `docs/DESIGN.md` — an invariant violation, a fundamental schema break), phase 4 is skipped. Rationale: fieldtest writes new examples that exercise the surface; on a codebase that violates DESIGN.md, those examples generate noise findings that the Boss must triage instead of acting on the actual `highest`-priority drift. `priority: highest` is the architect's call, not the orchestrator's. The orchestrator just reads priority from the architect-phase output and gates phase 4 on it. `priority: high` items (drift against the milestone spec, not DESIGN.md) do NOT trigger the skip — the language's invariants still hold; fieldtest can still produce useful signal. ### `NEEDS_CONTEXT` from orchestrator Returned if `milestone_id` or `commit_range` is ambiguous, or if `run_fieldtest=true` but `fieldtest_axis_hints` / `milestone_scope` is missing. Boss provides the missing carrier fields and re-dispatches. ### `BLOCKED` from orchestrator Returned on unrecoverable internal error (e.g., a phase-reference file missing, a bash binary not found, a git operation failing non-trivially). Halts immediately. Boss diagnoses and either fixes infrastructure + re-dispatches, or escalates. ## Testing strategy This is a skill/agent refactor, not a language feature. No `cargo test` coverage applies directly. Verification ladder: 1. **Agent file parses** — `Agent("ailang-audit-orchestrator", ...)` resolves to a runnable spec. Confirm via a no-op dispatch on a trivial milestone fixture (e.g., commit-range with no real changes). 2. **All four phases produce structured output** — dispatch on a real recent milestone, verify the end-report has all five sections populated (or `n/a` with reason). 3. **Bench exit 2 halts** — temporarily break a bench fixture, verify the orchestrator stops after Phase 2 with `infra_blocked` and does NOT run phases 3+4. 4. **Critical-drift skip on phase 4** — temporarily inject a fake `severity=high` architect finding, verify phase 4 is skipped with the documented reason. 5. **`run_fieldtest=false` skip** — dispatch on a purely-internal milestone, verify phase 4 is skipped with the documented reason. 6. **Aggregate, not abort** — dispatch on a milestone with known architect-drift, verify phases 2, 3, 4 still run despite the architect finding. 7. **Boss-side workflow on clean** — manually integrate a clean run: read draft journal, write milestone-close journal, append INDEX. Verify linear history and INDEX coherence. 8. **Manual-Mode preserved** — invoke `audit` skill directly (without the orchestrator) — verify the current Boss-direct dispatch path still works for ad-hoc invocations. Verification steps 3, 4, 6 require synthetic fixtures and are one-shot reproducers — they confirm the discipline once. Future audit runs answer "is the design holding up" empirically. ## Acceptance criteria The refactor is accepted when: 1. `skills/audit/agents/ailang-audit-orchestrator.md` exists, following the standard agent template (frontmatter, spirit-letter lead-in, standing reading list, carrier contract, Iron Law, process, status protocol, output format, common rationalisations, red flags). 2. `skills/audit/SKILL.md` gains a Standard-Mode subsection naming the orchestrator-agent as the dispatcher at milestone close. Steps 1–5 retained verbatim as phase references + Manual-Mode path. Cross- references updated. 3. `skills/fieldtest/SKILL.md` cross-references updated to reflect that Phase 4 of the audit-orchestrator is now the standard trigger; Manual Mode path retained. 4. `skills/README.md` skill table, agent roster, and pipeline diagram updated. 5. The four worker agent files (`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`, `ailang-fieldtester.md`) are **unchanged in body**. (The known-debt re: subagent framing carries over from or.2.) 6. The eight verification steps in Testing strategy pass. 7. The first real milestone close after the refactor shows a Boss-context delta consistent with the predicted offload (target: ≤ 1k tokens per audit-orchestrator dispatch + end-report, plus the Boss-side decision context for whatever findings emerge). ## Migration Single trivial-mechanical addition. No code touched, no language behaviour change. Two ways to land: **Option (a) — single direct-on-main commit.** The new agent file, plus the additive edits to `audit/SKILL.md`, `fieldtest/SKILL.md`, `README.md`. Analogous to `or.2`'s landing. **Option (b) — `iter au.1` via the planner + implement loop.** Forces the new orchestrator-agent through its own discipline cycle (plan → TDD → review). Adds about a day of latency for a refactor that has no executable surface to test against. **Recommendation: (a).** Same reasoning as `or.2`: doc-only revision, no behaviour to test, the first real audit at the next milestone close IS the verification step. Option (b) is process-theatre for this case. ## Out of scope (deferred to other specs) - **Brainstorm offload** — the `/brainstorm` usage statistic was 2% pre-or.1; not cost-effective to encapsulate yet. - **Planner offload** — the `pr.1` recon-subagent already addresses the read-heavy phase of planning; further offload is premature. - **Audit-result auto-resolution** — the Boss's `fix / ratify / carry-on` decision is semantic, not mechanical. Pushing this into the orchestrator would violate the user-stated principle that Boss-context is decision-context. Out of scope on principle, not on effort. - **Per-milestone-close branch isolation** — implement-orchestrator uses iter-branches; audit-orchestrator commits direct-on-main because per-phase changes (docwriter rustdoc fixes, fieldtest fixtures) are individually safe and don't benefit from iter-isolation. May be reconsidered if a future audit produces larger-scale changes that warrant branching. - **Worker model downgrade** — all roles stay at Opus 4.7, as in or.1.