AILang/docs/specs/0011-audit-orchestrator-agent.md

ailang-audit-orchestrator — Design Spec

Date: 2026-05-11 Status: Draft — awaiting user spec review Authors: Brummel (orchestrator) + Claude

Goal

Extend the Boss-context-offload pattern established in or.1 / corrected in or.2 to the post-implement validation cycle. The Boss should only carry the milestone-close context that involves a decision: classify a drift item, ratify a baseline shift, dispatch a tidy iter. Everything that's pure execution — running architect, running the bench scripts, running docwriter, running fieldtester — moves into a dedicated subagent context (ailang-audit-orchestrator) and reaches the Boss only as a structured end-report.

The two existing skills (audit, fieldtest) and their four worker agents (ailang-architect, ailang-bencher, ailang-docwriter, ailang-fieldtester) remain unchanged in body. What changes is their caller: at milestone-close, the Boss dispatches the new orchestrator agent, which inhales the audit + fieldtest discipline as sequential role-switches in its own context — the same shape as the or.2-corrected ailang-implement-orchestrator.

Background — why now

Three observations from 2026-05-11 motivate this spec:

1. or.1 proved the offload pattern. Per-/implement Boss-context cost dropped from ~100k to ~700 tokens by relocating the per-task loop into the implement-orchestrator subagent.

2. or.2 defined the constraint. Claude Code categorically forbids nested subagent dispatch. The corrected architecture uses inline role-switches inside the subagent's own context. Phase-reference files (ailang-implementer.md, etc.) define the discipline; the orchestrator inhales them per phase.

3. Milestone close was left out. Audit + fieldtest currently run as direct Boss-dispatched calls: architect report → Boss-side bench bash → docwriter dispatch → fieldtest dispatch, with each report landing in the Boss-context. Roughly 5–15k tokens per milestone-close. Not the 100×-hebel of /implement, but a real cost and — more importantly — a structural inconsistency: the per-iter execution path is offloaded, the post-iter validation path is not.

The user-stated principle that governs this spec:

Boss-context is decision-context. Everything else moves out.

Concretely:

• Audit + fieldtest clean → Boss gets a compact end-report ("milestone closes, nothing to decide"), writes the milestone-close JOURNAL entry, advances.
• Audit + fieldtest find anything actionable → Boss gets a structured findings report and decides per item: fix / ratify / carry-on. The decision and the downstream dispatch (planner + implement for fix, --update-baseline for ratify) remain Boss-level work, as today.

Architecture

Boss-Orchestrator
    │
    │ Agent("ailang-audit-orchestrator", carrier)
    ▼
ailang-audit-orchestrator        ← runs in its own context
    │
    │   Phase 1 — architect role-switch
    │       reads skills/audit/agents/ailang-architect.md
    │       performs the drift review inline
    │       collects: prioritised drift list
    │
    │   Phase 2 — bencher role-switch
    │       runs bench/check.py && bench/compile_check.py && bench/cross_lang.py
    │       collects: exit code + raw numbers
    │       HALT on exit 2 (infra-broken — subsequent phases unreliable)
    │
    │   Phase 3 — docwriter role-switch (conditional)
    │       cargo doc --no-deps 2>&1 → warnings?
    │       if yes: read skills/audit/agents/ailang-docwriter.md, run inline,
    │               commit rustdoc fixes on main
    │       collects: warning count, list of fixed warnings (or 0)
    │
    │   Phase 4 — fieldtester role-switch (conditional, carrier flag)
    │       if run_fieldtest && Phase 1 architect did not flag critical drift:
    │           reads skills/fieldtest/agents/ailang-fieldtester.md
    │           writes 2–4 .ailx examples in examples/, runs ail check/build/run,
    │           writes docs/specs/<date>-fieldtest-<milestone>.md,
    │           commits as `fieldtest: <milestone> — <N> examples, <K> findings`
    │       collects: spec_path, examples_added, findings, status
    │
    │   Phase 5 — end-report assembly
    │       structured findings → compact text → ≤500 tokens
    │       JOURNAL draft (Boss may accept/edit) → /tmp/ail-audit/<milestone>/draft.md
    ▼
end-report → Boss

Boss-Orchestrator (after reading end-report):
    │
    │   SUCCESS (status: clean):
    │       read /tmp/ail-audit/.../draft.md (optional)
    │       write docs/journals/<date>-milestone-<X>-close.md
    │       append docs/journals/INDEX.md
    │       (no further dispatches — milestone closes)
    │
    │   FAILURE (status: findings_*):
    │       classify each drift_item / bench finding / fieldtest finding
    │           → fix    → dispatch planner + implement-orchestrator (tidy iter)
    │           → ratify → run bench/<x>.py --update-baseline + JOURNAL ratify entry
    │           → carry-on → record in JOURNAL, no commit
    │       on bug from fieldtest: dispatch debug skill

The Boss-Orchestrator sees: one dispatch out, one compressed report back. Per-phase work — drift list assembly, bench bash output, rustdoc fix loops, fieldtest implementation cycles — all happen inside the orchestrator-agent's context and never reach the Boss.

Same structural pattern as or.2: the four phase-reference files (ailang-architect.md, ailang-bencher.md, ailang-docwriter.md, ailang-fieldtester.md) stay in their current locations. The orchestrator reads each one verbatim at its role-switch and applies the discipline inline. No nested subagent dispatch.

Components

ailang-audit-orchestrator (new agent)

Location: skills/audit/agents/ailang-audit-orchestrator.md.

Tools: Read, Edit, Write, Bash, Glob, Grep. No Agent tool — the or.2 Iron Law applies (Claude Code does not permit nested subagent dispatch).

Standing reading list:

1. CLAUDE.md — orchestrator framing.
2. docs/DESIGN.md — invariants the audit must hold against.
3. docs/journals/INDEX.md plus the last 1–3 referenced per-iter files — recent state to inform the architect-phase scope.
4. skills/audit/SKILL.md — canonical audit discipline.
5. skills/fieldtest/SKILL.md — canonical fieldtest discipline, conditionally relevant.
6. Phase-reference files (read per-phase, not all upfront):
   • skills/audit/agents/ailang-architect.md
   • skills/audit/agents/ailang-bencher.md
   • skills/audit/agents/ailang-docwriter.md
   • skills/fieldtest/agents/ailang-fieldtester.md

Model: Opus 4.7.

Iron Law (verbatim, derived from or.2):

PER-PHASE WORK RUNS SEQUENTIALLY IN YOUR OWN CONTEXT — architect role-switch,
then bencher, then docwriter, then fieldtester. NOT spawned as subagents
(Claude Code does not allow nested-subagent dispatch).

BENCH EXIT CODE 2 HALTS THE RUN. Subsequent phases would run on a broken
infrastructure and produce unreliable findings. Report `infra_blocked` and stop.

EVERY PHASE COMPLETES (or halts on the bench-2 condition). FINDINGS AGGREGATE.
Boss decides per item; the orchestrator does NOT pre-resolve.

skills/audit/SKILL.md (small additive change)

Today's body describes the five-step process for Boss-direct invocation. After this spec:

• Gain a short "Standard Mode" subsection at the top, saying:

  "At milestone close: dispatch ailang-audit-orchestrator. The orchestrator runs Steps 1–3 + the conditional fieldtest phase inline, returns a structured end-report. The Boss applies Steps 4 (Classify) and 5 (Resolve) on the report — these remain Boss work."

• The existing Steps 1–5 are retained verbatim. They now serve two audiences:

  • The orchestrator-agent reads them as phase-reference (alongside the per-agent files).
  • The Boss reads them when invoking audit in Manual Mode (drift suspected outside milestone close, debugging an isolated phase, etc.). Manual Mode dispatch shape unchanged.

• The Cross-references section gains:

  • Standard-mode dispatcher: skills/audit/agents/ailang-audit-orchestrator.md
  • "Why inline phases, not nested subagents" pointer to the or.2 journal.

skills/fieldtest/SKILL.md (cross-reference update only)

Today's body is unchanged. Fieldtest's standard trigger expands from "orchestrator dispatches ailang-fieldtester after audit closes" to "either the audit-orchestrator runs phase 4 inline, OR the Boss dispatches ailang-fieldtester directly in Manual Mode". The agent file (ailang-fieldtester.md) is unchanged.

Manual Mode for fieldtest covers the genuine edge case: audit found drift, a tidy-iter landed, re-running fieldtest in isolation makes sense, no point re-running the whole audit-orchestrator.

Worker agents — unchanged in body, expanded role

ailang-architect.md, ailang-bencher.md, ailang-docwriter.md, ailang-fieldtester.md keep their current frontmatter, tools, standing reading lists, status protocols, output formats.

What changes is their role-in-the-system: they serve as phase references when invoked by the audit-orchestrator (Standard Mode), AND as directly-dispatchable subagents when invoked by the Boss (Manual Mode). The dual duty mirrors how or.2 left ailang-implementer.md and siblings in the implement skill.

Same caveat applies as in or.2 known-debt: these files frame themselves as "dispatched subagents receiving a carrier from a dispatcher". In Standard Mode the orchestrator IS the dispatcher and the worker. If the first real audit-orchestrator run shows confusion during phase reads, a follow-up cleanup iter rewrites the framing. Not pre-emptive.

skills/README.md (skill table + roster + pipeline)

• Skill table row for audit: gains a one-line note about Standard-Mode dispatcher.
• Agent roster: gains row for ailang-audit-orchestrator (dispatched by audit, Standard Mode). Existing rows for architect / bencher / docwriter / fieldtester gain "and serves as phase reference for ailang-audit-orchestrator" suffix, analogous to or.2.
• Pipeline diagram: the audit -> fieldtest chain at milestone-close is replaced by a single audit (via ailang-audit-orchestrator) node; the conditional fieldtest is now phase 4 of that single dispatch.

Data flow

Carrier — Boss → ailang-audit-orchestrator

Field	Content
milestone_id	e.g. "ct.2" or "22" (string, free-form, used in commit messages and filenames)
commit_range	e.g. "<prev-milestone-close-sha>..HEAD" — the architect's scope
run_fieldtest	bool — set by Boss based on milestone spec. true if milestone touched user-visible surface (Decision 6, schema, effects, types, modes, diagnostics); false for purely internal milestones
fieldtest_axis_hints	(only if run_fieldtest = true) bullet list, one per milestone axis the fieldtest should probe — copied verbatim from the milestone spec
milestone_scope	(only if run_fieldtest = true) 1–3 sentences naming what shipped, for the fieldtester's carrier

End-report — ailang-audit-orchestrator → Boss

Plain-text report, ≤ 500 tokens, fixed structure:

Status:           clean | findings_drift | findings_bench | findings_rustdoc
                 | findings_fieldtest | findings_multiple | infra_blocked
Milestone:        <milestone_id>

Phase 1 (architect):
  Drift items:    <N>
                  - [<priority>] <path>: <1-line justification>
                  - ...
                  (or: none)
  Recommendation: <architect's single next-iter recommendation>
                  (verbatim from architect output)

Phase 2 (bench):
  Exit code:      0 | 1 | 2
  Numbers:        <raw figures from the three scripts, verbatim>

Phase 3 (rustdoc):
  Warnings:       <count> (or 0)
                  - <warning-snippet 1>
                  - ...
                  Auto-fixed: yes | no | partial (orchestrator may commit fixes)

Phase 4 (fieldtest):
  Ran:            yes | no (reason: run_fieldtest=false | architect critical-drift)
  Status:         clean | friction_found | bugs_found | (n/a)
  Spec:           docs/specs/<date>-fieldtest-<milestone>.md (committed)
  Findings:       <N> (one line each, with class)

Suggested classification (Boss decides):
  - <item ref>: fix | ratify | carry-on | needs-debug
  - ...
  (Orchestrator's heuristic suggestion. Boss applies audit-Step-4
  semantics and may override per item.)

Journal draft:    /tmp/ail-audit/<milestone>/draft.md
Commits made:     <SHA range or "none">

Boss-side reading workflow (on clean)

1. Read the end-report's Status: clean confirmation.
2. Optionally read /tmp/ail-audit/<milestone>/draft.md.
3. Write docs/journals/<date>-milestone-<X>-close.md, accepting the orchestrator's draft as a starting point or rewriting the summary.
4. Append a one-line entry to docs/journals/INDEX.md.
5. (If trigger is done-state: WhatsNew + notify per CLAUDE.md.)
6. Milestone closes; pick up the next item from docs/roadmap.md.

Boss-side workflow (on findings)

1. Read the end-report's structured findings.
2. Per item, classify: fix / ratify / carry-on / needs-debug.
3. Dispatch downstream as today:
   • fix → planner + implement for a tidy iter
   • ratify → bench/<x>.py --update-baseline + JOURNAL ratify entry
   • carry-on → record in JOURNAL, no commit
   • needs-debug → debug skill (RED-first, GREEN via implement mini-mode)
4. Write docs/journals/<date>-milestone-<X>-close.md summarising findings + classification + downstream-actions-taken.
5. If findings_fieldtest includes friction or spec_gap worth a follow-up: brainstorm for next milestone OR add to roadmap.

Error handling

Phase-level outcomes

Phase	Possible outcomes	Orchestrator action
Phase 1 (architect)	drift_found / clean	Always continue to phase 2. Drift severity is metadata in the report, NOT a halt condition.
Phase 2 (bench)	Exit 0 / 1 / 2	0 or 1: continue. 2: HALT — report infra_blocked, skip phases 3+4, exit with Status: infra_blocked.
Phase 3 (docwriter)	Warnings present / absent / fix-failed	Continue regardless. If warnings present, attempt fix and commit; record outcome. If fix fails, record Auto-fixed: no and pass through.
Phase 4 (fieldtest)	clean / friction / bugs / infra_blocked	Skipped entirely if run_fieldtest=false or if architect flagged a priority: highest drift item. Skip is recorded as Ran: no (reason: architect-highest-priority-drift). Always reported, never halts.

Why aggregate, not abort-at-first-failure

The four phases test independent dimensions:

• Architect: code vs. DESIGN.md (semantic drift).
• Bench: code vs. baseline (performance drift).
• Docwriter: code vs. rustdoc (documentation drift).
• Fieldtester: code vs. real-world LLM-author usage (usability drift).

A drift in one is not predictive of a drift in another. Aborting at phase 1 on architect-drift would force the Boss into a fix-then-re-run loop: fix architect drift, re-run, discover bench regression, fix that, re-run, discover fieldtest bug. Aggregating gives Boss the whole picture in one pass.

The ONE exception is bench exit 2: the bench infrastructure itself is broken, which means later phases run on an unreliable system. Halting is correct there.

Critical-drift skip on phase 4

The architect-phase output classifies drift items by priority (highest / high / medium — see skills/audit/agents/ailang-architect.md output format). If any item is priority: highest (i.e. drift against docs/DESIGN.md — an invariant violation, a fundamental schema break), phase 4 is skipped. Rationale: fieldtest writes new examples that exercise the surface; on a codebase that violates DESIGN.md, those examples generate noise findings that the Boss must triage instead of acting on the actual highest-priority drift.

priority: highest is the architect's call, not the orchestrator's. The orchestrator just reads priority from the architect-phase output and gates phase 4 on it. priority: high items (drift against the milestone spec, not DESIGN.md) do NOT trigger the skip — the language's invariants still hold; fieldtest can still produce useful signal.

NEEDS_CONTEXT from orchestrator

Returned if milestone_id or commit_range is ambiguous, or if run_fieldtest=true but fieldtest_axis_hints / milestone_scope is missing. Boss provides the missing carrier fields and re-dispatches.

BLOCKED from orchestrator

Returned on unrecoverable internal error (e.g., a phase-reference file missing, a bash binary not found, a git operation failing non-trivially). Halts immediately. Boss diagnoses and either fixes infrastructure + re-dispatches, or escalates.

Testing strategy

This is a skill/agent refactor, not a language feature. No cargo test coverage applies directly. Verification ladder:

1. Agent file parses — Agent("ailang-audit-orchestrator", ...) resolves to a runnable spec. Confirm via a no-op dispatch on a trivial milestone fixture (e.g., commit-range with no real changes).
2. All four phases produce structured output — dispatch on a real recent milestone, verify the end-report has all five sections populated (or n/a with reason).
3. Bench exit 2 halts — temporarily break a bench fixture, verify the orchestrator stops after Phase 2 with infra_blocked and does NOT run phases 3+4.
4. Critical-drift skip on phase 4 — temporarily inject a fake severity=high architect finding, verify phase 4 is skipped with the documented reason.
5. run_fieldtest=false skip — dispatch on a purely-internal milestone, verify phase 4 is skipped with the documented reason.
6. Aggregate, not abort — dispatch on a milestone with known architect-drift, verify phases 2, 3, 4 still run despite the architect finding.
7. Boss-side workflow on clean — manually integrate a clean run: read draft journal, write milestone-close journal, append INDEX. Verify linear history and INDEX coherence.
8. Manual-Mode preserved — invoke audit skill directly (without the orchestrator) — verify the current Boss-direct dispatch path still works for ad-hoc invocations.

Verification steps 3, 4, 6 require synthetic fixtures and are one-shot reproducers — they confirm the discipline once. Future audit runs answer "is the design holding up" empirically.

Acceptance criteria

The refactor is accepted when:

1. skills/audit/agents/ailang-audit-orchestrator.md exists, following the standard agent template (frontmatter, spirit-letter lead-in, standing reading list, carrier contract, Iron Law, process, status protocol, output format, common rationalisations, red flags).
2. skills/audit/SKILL.md gains a Standard-Mode subsection naming the orchestrator-agent as the dispatcher at milestone close. Steps 1–5 retained verbatim as phase references + Manual-Mode path. Cross- references updated.
3. skills/fieldtest/SKILL.md cross-references updated to reflect that Phase 4 of the audit-orchestrator is now the standard trigger; Manual Mode path retained.
4. skills/README.md skill table, agent roster, and pipeline diagram updated.
5. The four worker agent files (ailang-architect.md, ailang-bencher.md, ailang-docwriter.md, ailang-fieldtester.md) are unchanged in body. (The known-debt re: subagent framing carries over from or.2.)
6. The eight verification steps in Testing strategy pass.
7. The first real milestone close after the refactor shows a Boss-context delta consistent with the predicted offload (target: ≤ 1k tokens per audit-orchestrator dispatch + end-report, plus the Boss-side decision context for whatever findings emerge).

Migration

Single trivial-mechanical addition. No code touched, no language behaviour change. Two ways to land:

Option (a) — single direct-on-main commit. The new agent file, plus the additive edits to audit/SKILL.md, fieldtest/SKILL.md, README.md. Analogous to or.2's landing.

Option (b) — iter au.1 via the planner + implement loop. Forces the new orchestrator-agent through its own discipline cycle (plan → TDD → review). Adds about a day of latency for a refactor that has no executable surface to test against.

Recommendation: (a). Same reasoning as or.2: doc-only revision, no behaviour to test, the first real audit at the next milestone close IS the verification step. Option (b) is process-theatre for this case.

Out of scope (deferred to other specs)

• Brainstorm offload — the /brainstorm usage statistic was 2% pre-or.1; not cost-effective to encapsulate yet.
• Planner offload — the pr.1 recon-subagent already addresses the read-heavy phase of planning; further offload is premature.
• Audit-result auto-resolution — the Boss's fix / ratify / carry-on decision is semantic, not mechanical. Pushing this into the orchestrator would violate the user-stated principle that Boss-context is decision-context. Out of scope on principle, not on effort.
• Per-milestone-close branch isolation — implement-orchestrator uses iter-branches; audit-orchestrator commits direct-on-main because per-phase changes (docwriter rustdoc fixes, fieldtest fixtures) are individually safe and don't benefit from iter-isolation. May be reconsidered if a future audit produces larger-scale changes that warrant branching.
• Worker model downgrade — all roles stay at Opus 4.7, as in or.1.