AILang/docs/specs/0011-audit-orchestrator-agent.md

# `ailang-audit-orchestrator` — Design Spec

**Date:** 2026-05-11
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude

## Goal

Extend the Boss-context-offload pattern established in `or.1` /
corrected in `or.2` to the post-implement validation cycle. The Boss
should only carry the milestone-close context that involves a
**decision**: classify a drift item, ratify a baseline shift, dispatch
a tidy iter. Everything that's pure execution — running architect,
running the bench scripts, running docwriter, running fieldtester —
moves into a dedicated subagent context (`ailang-audit-orchestrator`)
and reaches the Boss only as a structured end-report.

The two existing skills (`audit`, `fieldtest`) and their four worker
agents (`ailang-architect`, `ailang-bencher`, `ailang-docwriter`,
`ailang-fieldtester`) **remain unchanged in body**. What changes is
their *caller*: at milestone-close, the Boss dispatches the new
orchestrator agent, which inhales the audit + fieldtest discipline as
sequential role-switches in its own context — the same shape as the
or.2-corrected `ailang-implement-orchestrator`.

## Background — why now

Three observations from 2026-05-11 motivate this spec:

1. **`or.1` proved the offload pattern.** Per-`/implement` Boss-context
   cost dropped from ~100k to ~700 tokens by relocating the per-task
   loop into the implement-orchestrator subagent.

2. **`or.2` defined the constraint.** Claude Code categorically forbids
   nested subagent dispatch. The corrected architecture uses inline
   role-switches inside the subagent's own context. Phase-reference
   files (`ailang-implementer.md`, etc.) define the discipline; the
   orchestrator inhales them per phase.

3. **Milestone close was left out.** Audit + fieldtest currently run
   as direct Boss-dispatched calls: architect report → Boss-side bench
   bash → docwriter dispatch → fieldtest dispatch, with each report
   landing in the Boss-context. Roughly 5–15k tokens per
   milestone-close. Not the 100×-hebel of `/implement`, but a
   real cost and — more importantly — a structural inconsistency: the
   per-iter execution path is offloaded, the post-iter validation path
   is not.

The user-stated principle that governs this spec:

> **Boss-context is decision-context.** Everything else moves out.

Concretely:
- Audit + fieldtest clean → Boss gets a compact end-report ("milestone
  closes, nothing to decide"), writes the milestone-close JOURNAL
  entry, advances.
- Audit + fieldtest find anything actionable → Boss gets a structured
  findings report and decides per item: `fix` / `ratify` / `carry-on`.
  The decision and the downstream dispatch (`planner` + `implement`
  for `fix`, `--update-baseline` for `ratify`) remain Boss-level work,
  as today.

## Architecture

```
Boss-Orchestrator
    │
    │ Agent("ailang-audit-orchestrator", carrier)
    ▼
ailang-audit-orchestrator        ← runs in its own context
    │
    │   Phase 1 — architect role-switch
    │       reads skills/audit/agents/ailang-architect.md
    │       performs the drift review inline
    │       collects: prioritised drift list
    │
    │   Phase 2 — bencher role-switch
    │       runs bench/check.py && bench/compile_check.py && bench/cross_lang.py
    │       collects: exit code + raw numbers
    │       HALT on exit 2 (infra-broken — subsequent phases unreliable)
    │
    │   Phase 3 — docwriter role-switch (conditional)
    │       cargo doc --no-deps 2>&1 → warnings?
    │       if yes: read skills/audit/agents/ailang-docwriter.md, run inline,
    │               commit rustdoc fixes on main
    │       collects: warning count, list of fixed warnings (or 0)
    │
    │   Phase 4 — fieldtester role-switch (conditional, carrier flag)
    │       if run_fieldtest && Phase 1 architect did not flag critical drift:
    │           reads skills/fieldtest/agents/ailang-fieldtester.md
    │           writes 2–4 .ailx examples in examples/, runs ail check/build/run,
    │           writes docs/specs/<date>-fieldtest-<milestone>.md,
    │           commits as `fieldtest: <milestone> — <N> examples, <K> findings`
    │       collects: spec_path, examples_added, findings, status
    │
    │   Phase 5 — end-report assembly
    │       structured findings → compact text → ≤500 tokens
    │       JOURNAL draft (Boss may accept/edit) → /tmp/ail-audit/<milestone>/draft.md
    ▼
end-report → Boss

Boss-Orchestrator (after reading end-report):
    │
    │   SUCCESS (status: clean):
    │       read /tmp/ail-audit/.../draft.md (optional)
    │       write docs/journals/<date>-milestone-<X>-close.md
    │       append docs/journals/INDEX.md
    │       (no further dispatches — milestone closes)
    │
    │   FAILURE (status: findings_*):
    │       classify each drift_item / bench finding / fieldtest finding
    │           → fix    → dispatch planner + implement-orchestrator (tidy iter)
    │           → ratify → run bench/<x>.py --update-baseline + JOURNAL ratify entry
    │           → carry-on → record in JOURNAL, no commit
    │       on bug from fieldtest: dispatch debug skill
```

The Boss-Orchestrator sees: one dispatch out, one compressed report
back. Per-phase work — drift list assembly, bench bash output, rustdoc
fix loops, fieldtest implementation cycles — all happen inside the
orchestrator-agent's context and never reach the Boss.

Same structural pattern as `or.2`: the four phase-reference files
(`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`,
`ailang-fieldtester.md`) stay in their current locations. The
orchestrator reads each one verbatim at its role-switch and applies
the discipline inline. **No nested subagent dispatch.**

## Components

### `ailang-audit-orchestrator` (new agent)

Location: `skills/audit/agents/ailang-audit-orchestrator.md`.

Tools: `Read, Edit, Write, Bash, Glob, Grep`. **No `Agent` tool** —
the or.2 Iron Law applies (Claude Code does not permit nested
subagent dispatch).

Standing reading list:
1. `CLAUDE.md` — orchestrator framing.
2. `docs/DESIGN.md` — invariants the audit must hold against.
3. `docs/journals/INDEX.md` plus the last 1–3 referenced per-iter
   files — recent state to inform the architect-phase scope.
4. `skills/audit/SKILL.md` — canonical audit discipline.
5. `skills/fieldtest/SKILL.md` — canonical fieldtest discipline,
   conditionally relevant.
6. Phase-reference files (read per-phase, not all upfront):
   - `skills/audit/agents/ailang-architect.md`
   - `skills/audit/agents/ailang-bencher.md`
   - `skills/audit/agents/ailang-docwriter.md`
   - `skills/fieldtest/agents/ailang-fieldtester.md`

Model: Opus 4.7.

Iron Law (verbatim, derived from or.2):
```
PER-PHASE WORK RUNS SEQUENTIALLY IN YOUR OWN CONTEXT — architect role-switch,
then bencher, then docwriter, then fieldtester. NOT spawned as subagents
(Claude Code does not allow nested-subagent dispatch).

BENCH EXIT CODE 2 HALTS THE RUN. Subsequent phases would run on a broken
infrastructure and produce unreliable findings. Report `infra_blocked` and stop.

EVERY PHASE COMPLETES (or halts on the bench-2 condition). FINDINGS AGGREGATE.
Boss decides per item; the orchestrator does NOT pre-resolve.
```

### `skills/audit/SKILL.md` (small additive change)

Today's body describes the five-step process for Boss-direct
invocation. After this spec:

- Gain a short "Standard Mode" subsection at the top, saying:
  > "At milestone close: dispatch `ailang-audit-orchestrator`. The
  > orchestrator runs Steps 1–3 + the conditional fieldtest phase
  > inline, returns a structured end-report. The Boss applies Steps 4
  > (Classify) and 5 (Resolve) on the report — these remain Boss
  > work."

- The existing Steps 1–5 are retained verbatim. They now serve two
  audiences:
  - The orchestrator-agent reads them as **phase-reference**
    (alongside the per-agent files).
  - The Boss reads them when invoking audit in **Manual Mode** (drift
    suspected outside milestone close, debugging an isolated phase,
    etc.). Manual Mode dispatch shape unchanged.

- The Cross-references section gains:
  - Standard-mode dispatcher: `skills/audit/agents/ailang-audit-orchestrator.md`
  - "Why inline phases, not nested subagents" pointer to the or.2
    journal.

### `skills/fieldtest/SKILL.md` (cross-reference update only)

Today's body is unchanged. Fieldtest's standard trigger expands from
"orchestrator dispatches `ailang-fieldtester` after audit closes" to
"either the audit-orchestrator runs phase 4 inline, OR the Boss
dispatches `ailang-fieldtester` directly in Manual Mode". The agent
file (`ailang-fieldtester.md`) is unchanged.

Manual Mode for fieldtest covers the genuine edge case: audit found
drift, a tidy-iter landed, re-running fieldtest in isolation makes
sense, no point re-running the whole audit-orchestrator.

### Worker agents — unchanged in body, expanded role

`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`,
`ailang-fieldtester.md` keep their current frontmatter, tools,
standing reading lists, status protocols, output formats.

What changes is their role-in-the-system: they serve as **phase
references** when invoked by the audit-orchestrator (Standard Mode),
AND as **directly-dispatchable subagents** when invoked by the Boss
(Manual Mode). The dual duty mirrors how or.2 left
`ailang-implementer.md` and siblings in the implement skill.

Same caveat applies as in or.2 known-debt: these files frame
themselves as "dispatched subagents receiving a carrier from a
dispatcher". In Standard Mode the orchestrator IS the dispatcher and
the worker. If the first real audit-orchestrator run shows confusion
during phase reads, a follow-up cleanup iter rewrites the framing.
Not pre-emptive.

### `skills/README.md` (skill table + roster + pipeline)

- Skill table row for `audit`: gains a one-line note about
  Standard-Mode dispatcher.
- Agent roster: gains row for `ailang-audit-orchestrator` (dispatched
  by `audit`, Standard Mode). Existing rows for architect / bencher /
  docwriter / fieldtester gain "and serves as phase reference for
  ailang-audit-orchestrator" suffix, analogous to or.2.
- Pipeline diagram: the `audit -> fieldtest` chain at milestone-close
  is replaced by a single `audit (via ailang-audit-orchestrator)`
  node; the conditional fieldtest is now phase 4 of that single
  dispatch.

## Data flow

### Carrier — Boss → `ailang-audit-orchestrator`

| Field | Content |
|-------|---------|
| `milestone_id` | e.g. `"ct.2"` or `"22"` (string, free-form, used in commit messages and filenames) |
| `commit_range` | e.g. `"<prev-milestone-close-sha>..HEAD"` — the architect's scope |
| `run_fieldtest` | bool — set by Boss based on milestone spec. `true` if milestone touched user-visible surface (Decision 6, schema, effects, types, modes, diagnostics); `false` for purely internal milestones |
| `fieldtest_axis_hints` | (only if `run_fieldtest = true`) bullet list, one per milestone axis the fieldtest should probe — copied verbatim from the milestone spec |
| `milestone_scope` | (only if `run_fieldtest = true`) 1–3 sentences naming what shipped, for the fieldtester's carrier |

### End-report — `ailang-audit-orchestrator` → Boss

Plain-text report, ≤ 500 tokens, fixed structure:

```
Status:           clean | findings_drift | findings_bench | findings_rustdoc
                 | findings_fieldtest | findings_multiple | infra_blocked
Milestone:        <milestone_id>

Phase 1 (architect):
  Drift items:    <N>
                  - [<priority>] <path>: <1-line justification>
                  - ...
                  (or: none)
  Recommendation: <architect's single next-iter recommendation>
                  (verbatim from architect output)

Phase 2 (bench):
  Exit code:      0 | 1 | 2
  Numbers:        <raw figures from the three scripts, verbatim>

Phase 3 (rustdoc):
  Warnings:       <count> (or 0)
                  - <warning-snippet 1>
                  - ...
                  Auto-fixed: yes | no | partial (orchestrator may commit fixes)

Phase 4 (fieldtest):
  Ran:            yes | no (reason: run_fieldtest=false | architect critical-drift)
  Status:         clean | friction_found | bugs_found | (n/a)
  Spec:           docs/specs/<date>-fieldtest-<milestone>.md (committed)
  Findings:       <N> (one line each, with class)

Suggested classification (Boss decides):
  - <item ref>: fix | ratify | carry-on | needs-debug
  - ...
  (Orchestrator's heuristic suggestion. Boss applies audit-Step-4
  semantics and may override per item.)

Journal draft:    /tmp/ail-audit/<milestone>/draft.md
Commits made:     <SHA range or "none">
```

### Boss-side reading workflow (on clean)

1. Read the end-report's `Status: clean` confirmation.
2. Optionally read `/tmp/ail-audit/<milestone>/draft.md`.
3. Write `docs/journals/<date>-milestone-<X>-close.md`, accepting the
   orchestrator's draft as a starting point or rewriting the summary.
4. Append a one-line entry to `docs/journals/INDEX.md`.
5. (If trigger is done-state: WhatsNew + notify per CLAUDE.md.)
6. Milestone closes; pick up the next item from `docs/roadmap.md`.

### Boss-side workflow (on findings)

1. Read the end-report's structured findings.
2. Per item, classify: `fix` / `ratify` / `carry-on` / `needs-debug`.
3. Dispatch downstream as today:
   - `fix` → `planner` + `implement` for a tidy iter
   - `ratify` → `bench/<x>.py --update-baseline` + JOURNAL ratify
     entry
   - `carry-on` → record in JOURNAL, no commit
   - `needs-debug` → `debug` skill (RED-first, GREEN via implement
     mini-mode)
4. Write `docs/journals/<date>-milestone-<X>-close.md` summarising
   findings + classification + downstream-actions-taken.
5. If `findings_fieldtest` includes `friction` or `spec_gap` worth a
   follow-up: `brainstorm` for next milestone OR add to roadmap.

## Error handling

### Phase-level outcomes

| Phase | Possible outcomes | Orchestrator action |
|-------|-------------------|---------------------|
| Phase 1 (architect) | `drift_found` / `clean` | Always continue to phase 2. Drift severity is metadata in the report, NOT a halt condition. |
| Phase 2 (bench) | Exit 0 / 1 / 2 | 0 or 1: continue. **2: HALT** — report `infra_blocked`, skip phases 3+4, exit with `Status: infra_blocked`. |
| Phase 3 (docwriter) | Warnings present / absent / fix-failed | Continue regardless. If warnings present, attempt fix and commit; record outcome. If fix fails, record `Auto-fixed: no` and pass through. |
| Phase 4 (fieldtest) | clean / friction / bugs / infra_blocked | Skipped entirely if `run_fieldtest=false` or if architect flagged a `priority: highest` drift item. Skip is recorded as `Ran: no (reason: architect-highest-priority-drift)`. Always reported, never halts. |

### Why aggregate, not abort-at-first-failure

The four phases test independent dimensions:
- Architect: code vs. DESIGN.md (semantic drift).
- Bench: code vs. baseline (performance drift).
- Docwriter: code vs. rustdoc (documentation drift).
- Fieldtester: code vs. real-world LLM-author usage (usability drift).

A drift in one is not predictive of a drift in another. Aborting at
phase 1 on architect-drift would force the Boss into a fix-then-re-run
loop: fix architect drift, re-run, discover bench regression, fix
that, re-run, discover fieldtest bug. Aggregating gives Boss the
whole picture in one pass.

The ONE exception is bench exit 2: the bench infrastructure itself is
broken, which means later phases run on an unreliable system. Halting
is correct there.

### Critical-drift skip on phase 4

The architect-phase output classifies drift items by `priority`
(highest / high / medium — see `skills/audit/agents/ailang-architect.md`
output format). If any item is `priority: highest` (i.e. drift against
`docs/DESIGN.md` — an invariant violation, a fundamental schema break),
phase 4 is skipped. Rationale: fieldtest writes new examples that
exercise the surface; on a codebase that violates DESIGN.md, those
examples generate noise findings that the Boss must triage instead of
acting on the actual `highest`-priority drift.

`priority: highest` is the architect's call, not the orchestrator's.
The orchestrator just reads priority from the architect-phase output
and gates phase 4 on it. `priority: high` items (drift against the
milestone spec, not DESIGN.md) do NOT trigger the skip — the
language's invariants still hold; fieldtest can still produce useful
signal.

### `NEEDS_CONTEXT` from orchestrator

Returned if `milestone_id` or `commit_range` is ambiguous, or if
`run_fieldtest=true` but `fieldtest_axis_hints` / `milestone_scope` is
missing. Boss provides the missing carrier fields and re-dispatches.

### `BLOCKED` from orchestrator

Returned on unrecoverable internal error (e.g., a phase-reference file
missing, a bash binary not found, a git operation failing
non-trivially). Halts immediately. Boss diagnoses and either fixes
infrastructure + re-dispatches, or escalates.

## Testing strategy

This is a skill/agent refactor, not a language feature. No `cargo
test` coverage applies directly. Verification ladder:

1. **Agent file parses** — `Agent("ailang-audit-orchestrator", ...)`
   resolves to a runnable spec. Confirm via a no-op dispatch on a
   trivial milestone fixture (e.g., commit-range with no real
   changes).
2. **All four phases produce structured output** — dispatch on a real
   recent milestone, verify the end-report has all five sections
   populated (or `n/a` with reason).
3. **Bench exit 2 halts** — temporarily break a bench fixture, verify
   the orchestrator stops after Phase 2 with `infra_blocked` and does
   NOT run phases 3+4.
4. **Critical-drift skip on phase 4** — temporarily inject a fake
   `severity=high` architect finding, verify phase 4 is skipped with
   the documented reason.
5. **`run_fieldtest=false` skip** — dispatch on a purely-internal
   milestone, verify phase 4 is skipped with the documented reason.
6. **Aggregate, not abort** — dispatch on a milestone with known
   architect-drift, verify phases 2, 3, 4 still run despite the
   architect finding.
7. **Boss-side workflow on clean** — manually integrate a clean run:
   read draft journal, write milestone-close journal, append INDEX.
   Verify linear history and INDEX coherence.
8. **Manual-Mode preserved** — invoke `audit` skill directly (without
   the orchestrator) — verify the current Boss-direct dispatch path
   still works for ad-hoc invocations.

Verification steps 3, 4, 6 require synthetic fixtures and are
one-shot reproducers — they confirm the discipline once. Future audit
runs answer "is the design holding up" empirically.

## Acceptance criteria

The refactor is accepted when:

1. `skills/audit/agents/ailang-audit-orchestrator.md` exists, following
   the standard agent template (frontmatter, spirit-letter lead-in,
   standing reading list, carrier contract, Iron Law, process, status
   protocol, output format, common rationalisations, red flags).
2. `skills/audit/SKILL.md` gains a Standard-Mode subsection naming the
   orchestrator-agent as the dispatcher at milestone close. Steps 1–5
   retained verbatim as phase references + Manual-Mode path. Cross-
   references updated.
3. `skills/fieldtest/SKILL.md` cross-references updated to reflect
   that Phase 4 of the audit-orchestrator is now the standard trigger;
   Manual Mode path retained.
4. `skills/README.md` skill table, agent roster, and pipeline diagram
   updated.
5. The four worker agent files (`ailang-architect.md`,
   `ailang-bencher.md`, `ailang-docwriter.md`,
   `ailang-fieldtester.md`) are **unchanged in body**. (The known-debt
   re: subagent framing carries over from or.2.)
6. The eight verification steps in Testing strategy pass.
7. The first real milestone close after the refactor shows a
   Boss-context delta consistent with the predicted offload (target:
   ≤ 1k tokens per audit-orchestrator dispatch + end-report, plus the
   Boss-side decision context for whatever findings emerge).

## Migration

Single trivial-mechanical addition. No code touched, no language
behaviour change. Two ways to land:

**Option (a) — single direct-on-main commit.** The new agent file,
plus the additive edits to `audit/SKILL.md`, `fieldtest/SKILL.md`,
`README.md`. Analogous to `or.2`'s landing.

**Option (b) — `iter au.1` via the planner + implement loop.** Forces
the new orchestrator-agent through its own discipline cycle (plan →
TDD → review). Adds about a day of latency for a refactor that has
no executable surface to test against.

**Recommendation: (a).** Same reasoning as `or.2`: doc-only revision,
no behaviour to test, the first real audit at the next milestone
close IS the verification step. Option (b) is process-theatre for
this case.

## Out of scope (deferred to other specs)

- **Brainstorm offload** — the `/brainstorm` usage statistic was 2%
  pre-or.1; not cost-effective to encapsulate yet.
- **Planner offload** — the `pr.1` recon-subagent already addresses
  the read-heavy phase of planning; further offload is premature.
- **Audit-result auto-resolution** — the Boss's `fix / ratify /
  carry-on` decision is semantic, not mechanical. Pushing this into
  the orchestrator would violate the user-stated principle that
  Boss-context is decision-context. Out of scope on principle, not
  on effort.
- **Per-milestone-close branch isolation** — implement-orchestrator
  uses iter-branches; audit-orchestrator commits direct-on-main
  because per-phase changes (docwriter rustdoc fixes, fieldtest
  fixtures) are individually safe and don't benefit from
  iter-isolation. May be reconsidered if a future audit produces
  larger-scale changes that warrant branching.
- **Worker model downgrade** — all roles stay at Opus 4.7, as in
  or.1.