spec: au.1 audit-orchestrator agent

docs/specs/2026-05-11-audit-orchestrator-agent.md

@@ -0,0 +1,476 @@
+# `ailang-audit-orchestrator` — Design Spec
+
+**Date:** 2026-05-11
+**Status:** Draft — awaiting user spec review
+**Authors:** Brummel (orchestrator) + Claude
+
+## Goal
+
+Extend the Boss-context-offload pattern established in `or.1` /
+corrected in `or.2` to the post-implement validation cycle. The Boss
+should only carry the milestone-close context that involves a
+**decision**: classify a drift item, ratify a baseline shift, dispatch
+a tidy iter. Everything that's pure execution — running architect,
+running the bench scripts, running docwriter, running fieldtester —
+moves into a dedicated subagent context (`ailang-audit-orchestrator`)
+and reaches the Boss only as a structured end-report.
+
+The two existing skills (`audit`, `fieldtest`) and their four worker
+agents (`ailang-architect`, `ailang-bencher`, `ailang-docwriter`,
+`ailang-fieldtester`) **remain unchanged in body**. What changes is
+their *caller*: at milestone-close, the Boss dispatches the new
+orchestrator agent, which inhales the audit + fieldtest discipline as
+sequential role-switches in its own context — the same shape as the
+or.2-corrected `ailang-implement-orchestrator`.
+
+## Background — why now
+
+Three observations from 2026-05-11 motivate this spec:
+
+1. **`or.1` proved the offload pattern.** Per-`/implement` Boss-context
+   cost dropped from ~100k to ~700 tokens by relocating the per-task
+   loop into the implement-orchestrator subagent.
+
+2. **`or.2` defined the constraint.** Claude Code categorically forbids
+   nested subagent dispatch. The corrected architecture uses inline
+   role-switches inside the subagent's own context. Phase-reference
+   files (`ailang-implementer.md`, etc.) define the discipline; the
+   orchestrator inhales them per phase.
+
+3. **Milestone close was left out.** Audit + fieldtest currently run
+   as direct Boss-dispatched calls: architect report → Boss-side bench
+   bash → docwriter dispatch → fieldtest dispatch, with each report
+   landing in the Boss-context. Roughly 5–15k tokens per
+   milestone-close. Not the 100×-hebel of `/implement`, but a
+   real cost and — more importantly — a structural inconsistency: the
+   per-iter execution path is offloaded, the post-iter validation path
+   is not.
+
+The user-stated principle that governs this spec:
+
+> **Boss-context is decision-context.** Everything else moves out.
+
+Concretely:
+- Audit + fieldtest clean → Boss gets a compact end-report ("milestone
+  closes, nothing to decide"), writes the milestone-close JOURNAL
+  entry, advances.
+- Audit + fieldtest find anything actionable → Boss gets a structured
+  findings report and decides per item: `fix` / `ratify` / `carry-on`.
+  The decision and the downstream dispatch (`planner` + `implement`
+  for `fix`, `--update-baseline` for `ratify`) remain Boss-level work,
+  as today.
+
+## Architecture
+
+```
+Boss-Orchestrator
+    │
+    │ Agent("ailang-audit-orchestrator", carrier)
+    ▼
+ailang-audit-orchestrator        ← runs in its own context
+    │
+    │   Phase 1 — architect role-switch
+    │       reads skills/audit/agents/ailang-architect.md
+    │       performs the drift review inline
+    │       collects: prioritised drift list
+    │
+    │   Phase 2 — bencher role-switch
+    │       runs bench/check.py && bench/compile_check.py && bench/cross_lang.py
+    │       collects: exit code + raw numbers
+    │       HALT on exit 2 (infra-broken — subsequent phases unreliable)
+    │
+    │   Phase 3 — docwriter role-switch (conditional)
+    │       cargo doc --no-deps 2>&1 → warnings?
+    │       if yes: read skills/audit/agents/ailang-docwriter.md, run inline,
+    │               commit rustdoc fixes on main
+    │       collects: warning count, list of fixed warnings (or 0)
+    │
+    │   Phase 4 — fieldtester role-switch (conditional, carrier flag)
+    │       if run_fieldtest && Phase 1 architect did not flag critical drift:
+    │           reads skills/fieldtest/agents/ailang-fieldtester.md
+    │           writes 2–4 .ailx examples in examples/, runs ail check/build/run,
+    │           writes docs/specs/<date>-fieldtest-<milestone>.md,
+    │           commits as `fieldtest: <milestone> — <N> examples, <K> findings`
+    │       collects: spec_path, examples_added, findings, status
+    │
+    │   Phase 5 — end-report assembly
+    │       structured findings → compact text → ≤500 tokens
+    │       JOURNAL draft (Boss may accept/edit) → /tmp/ail-audit/<milestone>/draft.md
+    ▼
+end-report → Boss
+
+Boss-Orchestrator (after reading end-report):
+    │
+    │   SUCCESS (status: clean):
+    │       read /tmp/ail-audit/.../draft.md (optional)
+    │       write docs/journals/<date>-milestone-<X>-close.md
+    │       append docs/journals/INDEX.md
+    │       (no further dispatches — milestone closes)
+    │
+    │   FAILURE (status: findings_*):
+    │       classify each drift_item / bench finding / fieldtest finding
+    │           → fix    → dispatch planner + implement-orchestrator (tidy iter)
+    │           → ratify → run bench/<x>.py --update-baseline + JOURNAL ratify entry
+    │           → carry-on → record in JOURNAL, no commit
+    │       on bug from fieldtest: dispatch debug skill
+```
+
+The Boss-Orchestrator sees: one dispatch out, one compressed report
+back. Per-phase work — drift list assembly, bench bash output, rustdoc
+fix loops, fieldtest implementation cycles — all happen inside the
+orchestrator-agent's context and never reach the Boss.
+
+Same structural pattern as `or.2`: the four phase-reference files
+(`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`,
+`ailang-fieldtester.md`) stay in their current locations. The
+orchestrator reads each one verbatim at its role-switch and applies
+the discipline inline. **No nested subagent dispatch.**
+
+## Components
+
+### `ailang-audit-orchestrator` (new agent)
+
+Location: `skills/audit/agents/ailang-audit-orchestrator.md`.
+
+Tools: `Read, Edit, Write, Bash, Glob, Grep`. **No `Agent` tool** —
+the or.2 Iron Law applies (Claude Code does not permit nested
+subagent dispatch).
+
+Standing reading list:
+1. `CLAUDE.md` — orchestrator framing.
+2. `docs/DESIGN.md` — invariants the audit must hold against.
+3. `docs/journals/INDEX.md` plus the last 1–3 referenced per-iter
+   files — recent state to inform the architect-phase scope.
+4. `skills/audit/SKILL.md` — canonical audit discipline.
+5. `skills/fieldtest/SKILL.md` — canonical fieldtest discipline,
+   conditionally relevant.
+6. Phase-reference files (read per-phase, not all upfront):
+   - `skills/audit/agents/ailang-architect.md`
+   - `skills/audit/agents/ailang-bencher.md`
+   - `skills/audit/agents/ailang-docwriter.md`
+   - `skills/fieldtest/agents/ailang-fieldtester.md`
+
+Model: Opus 4.7.
+
+Iron Law (verbatim, derived from or.2):
+```
+PER-PHASE WORK RUNS SEQUENTIALLY IN YOUR OWN CONTEXT — architect role-switch,
+then bencher, then docwriter, then fieldtester. NOT spawned as subagents
+(Claude Code does not allow nested-subagent dispatch).
+
+BENCH EXIT CODE 2 HALTS THE RUN. Subsequent phases would run on a broken
+infrastructure and produce unreliable findings. Report `infra_blocked` and stop.
+
+EVERY PHASE COMPLETES (or halts on the bench-2 condition). FINDINGS AGGREGATE.
+Boss decides per item; the orchestrator does NOT pre-resolve.
+```
+
+### `skills/audit/SKILL.md` (small additive change)
+
+Today's body describes the five-step process for Boss-direct
+invocation. After this spec:
+
+- Gain a short "Standard Mode" subsection at the top, saying:
+  > "At milestone close: dispatch `ailang-audit-orchestrator`. The
+  > orchestrator runs Steps 1–3 + the conditional fieldtest phase
+  > inline, returns a structured end-report. The Boss applies Steps 4
+  > (Classify) and 5 (Resolve) on the report — these remain Boss
+  > work."
+
+- The existing Steps 1–5 are retained verbatim. They now serve two
+  audiences:
+  - The orchestrator-agent reads them as **phase-reference**
+    (alongside the per-agent files).
+  - The Boss reads them when invoking audit in **Manual Mode** (drift
+    suspected outside milestone close, debugging an isolated phase,
+    etc.). Manual Mode dispatch shape unchanged.
+
+- The Cross-references section gains:
+  - Standard-mode dispatcher: `skills/audit/agents/ailang-audit-orchestrator.md`
+  - "Why inline phases, not nested subagents" pointer to the or.2
+    journal.
+
+### `skills/fieldtest/SKILL.md` (cross-reference update only)
+
+Today's body is unchanged. Fieldtest's standard trigger expands from
+"orchestrator dispatches `ailang-fieldtester` after audit closes" to
+"either the audit-orchestrator runs phase 4 inline, OR the Boss
+dispatches `ailang-fieldtester` directly in Manual Mode". The agent
+file (`ailang-fieldtester.md`) is unchanged.
+
+Manual Mode for fieldtest covers the genuine edge case: audit found
+drift, a tidy-iter landed, re-running fieldtest in isolation makes
+sense, no point re-running the whole audit-orchestrator.
+
+### Worker agents — unchanged in body, expanded role
+
+`ailang-architect.md`, `ailang-bencher.md`, `ailang-docwriter.md`,
+`ailang-fieldtester.md` keep their current frontmatter, tools,
+standing reading lists, status protocols, output formats.
+
+What changes is their role-in-the-system: they serve as **phase
+references** when invoked by the audit-orchestrator (Standard Mode),
+AND as **directly-dispatchable subagents** when invoked by the Boss
+(Manual Mode). The dual duty mirrors how or.2 left
+`ailang-implementer.md` and siblings in the implement skill.
+
+Same caveat applies as in or.2 known-debt: these files frame
+themselves as "dispatched subagents receiving a carrier from a
+dispatcher". In Standard Mode the orchestrator IS the dispatcher and
+the worker. If the first real audit-orchestrator run shows confusion
+during phase reads, a follow-up cleanup iter rewrites the framing.
+Not pre-emptive.
+
+### `skills/README.md` (skill table + roster + pipeline)
+
+- Skill table row for `audit`: gains a one-line note about
+  Standard-Mode dispatcher.
+- Agent roster: gains row for `ailang-audit-orchestrator` (dispatched
+  by `audit`, Standard Mode). Existing rows for architect / bencher /
+  docwriter / fieldtester gain "and serves as phase reference for
+  ailang-audit-orchestrator" suffix, analogous to or.2.
+- Pipeline diagram: the `audit -> fieldtest` chain at milestone-close
+  is replaced by a single `audit (via ailang-audit-orchestrator)`
+  node; the conditional fieldtest is now phase 4 of that single
+  dispatch.
+
+## Data flow
+
+### Carrier — Boss → `ailang-audit-orchestrator`
+
+| Field | Content |
+|-------|---------|
+| `milestone_id` | e.g. `"ct.2"` or `"22"` (string, free-form, used in commit messages and filenames) |
+| `commit_range` | e.g. `"<prev-milestone-close-sha>..HEAD"` — the architect's scope |
+| `run_fieldtest` | bool — set by Boss based on milestone spec. `true` if milestone touched user-visible surface (Decision 6, schema, effects, types, modes, diagnostics); `false` for purely internal milestones |
+| `fieldtest_axis_hints` | (only if `run_fieldtest = true`) bullet list, one per milestone axis the fieldtest should probe — copied verbatim from the milestone spec |
+| `milestone_scope` | (only if `run_fieldtest = true`) 1–3 sentences naming what shipped, for the fieldtester's carrier |
+
+### End-report — `ailang-audit-orchestrator` → Boss
+
+Plain-text report, ≤ 500 tokens, fixed structure:
+
+```
+Status:           clean | findings_drift | findings_bench | findings_rustdoc
+                 | findings_fieldtest | findings_multiple | infra_blocked
+Milestone:        <milestone_id>
+
+Phase 1 (architect):
+  Drift items:    <N>
+                  - [<priority>] <path>: <1-line justification>
+                  - ...
+                  (or: none)
+  Recommendation: <architect's single next-iter recommendation>
+                  (verbatim from architect output)
+
+Phase 2 (bench):
+  Exit code:      0 | 1 | 2
+  Numbers:        <raw figures from the three scripts, verbatim>
+
+Phase 3 (rustdoc):
+  Warnings:       <count> (or 0)
+                  - <warning-snippet 1>
+                  - ...
+                  Auto-fixed: yes | no | partial (orchestrator may commit fixes)
+
+Phase 4 (fieldtest):
+  Ran:            yes | no (reason: run_fieldtest=false | architect critical-drift)
+  Status:         clean | friction_found | bugs_found | (n/a)
+  Spec:           docs/specs/<date>-fieldtest-<milestone>.md (committed)
+  Findings:       <N> (one line each, with class)
+
+Suggested classification (Boss decides):
+  - <item ref>: fix | ratify | carry-on | needs-debug
+  - ...
+  (Orchestrator's heuristic suggestion. Boss applies audit-Step-4
+  semantics and may override per item.)
+
+Journal draft:    /tmp/ail-audit/<milestone>/draft.md
+Commits made:     <SHA range or "none">
+```
+
+### Boss-side reading workflow (on clean)
+
+1. Read the end-report's `Status: clean` confirmation.
+2. Optionally read `/tmp/ail-audit/<milestone>/draft.md`.
+3. Write `docs/journals/<date>-milestone-<X>-close.md`, accepting the
+   orchestrator's draft as a starting point or rewriting the summary.
+4. Append a one-line entry to `docs/journals/INDEX.md`.
+5. (If trigger is done-state: WhatsNew + notify per CLAUDE.md.)
+6. Milestone closes; pick up the next item from `docs/roadmap.md`.
+
+### Boss-side workflow (on findings)
+
+1. Read the end-report's structured findings.
+2. Per item, classify: `fix` / `ratify` / `carry-on` / `needs-debug`.
+3. Dispatch downstream as today:
+   - `fix` → `planner` + `implement` for a tidy iter
+   - `ratify` → `bench/<x>.py --update-baseline` + JOURNAL ratify
+     entry
+   - `carry-on` → record in JOURNAL, no commit
+   - `needs-debug` → `debug` skill (RED-first, GREEN via implement
+     mini-mode)
+4. Write `docs/journals/<date>-milestone-<X>-close.md` summarising
+   findings + classification + downstream-actions-taken.
+5. If `findings_fieldtest` includes `friction` or `spec_gap` worth a
+   follow-up: `brainstorm` for next milestone OR add to roadmap.
+
+## Error handling
+
+### Phase-level outcomes
+
+| Phase | Possible outcomes | Orchestrator action |
+|-------|-------------------|---------------------|
+| Phase 1 (architect) | `drift_found` / `clean` | Always continue to phase 2. Drift severity is metadata in the report, NOT a halt condition. |
+| Phase 2 (bench) | Exit 0 / 1 / 2 | 0 or 1: continue. **2: HALT** — report `infra_blocked`, skip phases 3+4, exit with `Status: infra_blocked`. |
+| Phase 3 (docwriter) | Warnings present / absent / fix-failed | Continue regardless. If warnings present, attempt fix and commit; record outcome. If fix fails, record `Auto-fixed: no` and pass through. |
+| Phase 4 (fieldtest) | clean / friction / bugs / infra_blocked | Skipped entirely if `run_fieldtest=false` or if architect flagged a `priority: highest` drift item. Skip is recorded as `Ran: no (reason: architect-highest-priority-drift)`. Always reported, never halts. |
+
+### Why aggregate, not abort-at-first-failure
+
+The four phases test independent dimensions:
+- Architect: code vs. DESIGN.md (semantic drift).
+- Bench: code vs. baseline (performance drift).
+- Docwriter: code vs. rustdoc (documentation drift).
+- Fieldtester: code vs. real-world LLM-author usage (usability drift).
+
+A drift in one is not predictive of a drift in another. Aborting at
+phase 1 on architect-drift would force the Boss into a fix-then-re-run
+loop: fix architect drift, re-run, discover bench regression, fix
+that, re-run, discover fieldtest bug. Aggregating gives Boss the
+whole picture in one pass.
+
+The ONE exception is bench exit 2: the bench infrastructure itself is
+broken, which means later phases run on an unreliable system. Halting
+is correct there.
+
+### Critical-drift skip on phase 4
+
+The architect-phase output classifies drift items by `priority`
+(highest / high / medium — see `skills/audit/agents/ailang-architect.md`
+output format). If any item is `priority: highest` (i.e. drift against
+`docs/DESIGN.md` — an invariant violation, a fundamental schema break),
+phase 4 is skipped. Rationale: fieldtest writes new examples that
+exercise the surface; on a codebase that violates DESIGN.md, those
+examples generate noise findings that the Boss must triage instead of
+acting on the actual `highest`-priority drift.
+
+`priority: highest` is the architect's call, not the orchestrator's.
+The orchestrator just reads priority from the architect-phase output
+and gates phase 4 on it. `priority: high` items (drift against the
+milestone spec, not DESIGN.md) do NOT trigger the skip — the
+language's invariants still hold; fieldtest can still produce useful
+signal.
+
+### `NEEDS_CONTEXT` from orchestrator
+
+Returned if `milestone_id` or `commit_range` is ambiguous, or if
+`run_fieldtest=true` but `fieldtest_axis_hints` / `milestone_scope` is
+missing. Boss provides the missing carrier fields and re-dispatches.
+
+### `BLOCKED` from orchestrator
+
+Returned on unrecoverable internal error (e.g., a phase-reference file
+missing, a bash binary not found, a git operation failing
+non-trivially). Halts immediately. Boss diagnoses and either fixes
+infrastructure + re-dispatches, or escalates.
+
+## Testing strategy
+
+This is a skill/agent refactor, not a language feature. No `cargo
+test` coverage applies directly. Verification ladder:
+
+1. **Agent file parses** — `Agent("ailang-audit-orchestrator", ...)`
+   resolves to a runnable spec. Confirm via a no-op dispatch on a
+   trivial milestone fixture (e.g., commit-range with no real
+   changes).
+2. **All four phases produce structured output** — dispatch on a real
+   recent milestone, verify the end-report has all five sections
+   populated (or `n/a` with reason).
+3. **Bench exit 2 halts** — temporarily break a bench fixture, verify
+   the orchestrator stops after Phase 2 with `infra_blocked` and does
+   NOT run phases 3+4.
+4. **Critical-drift skip on phase 4** — temporarily inject a fake
+   `severity=high` architect finding, verify phase 4 is skipped with
+   the documented reason.
+5. **`run_fieldtest=false` skip** — dispatch on a purely-internal
+   milestone, verify phase 4 is skipped with the documented reason.
+6. **Aggregate, not abort** — dispatch on a milestone with known
+   architect-drift, verify phases 2, 3, 4 still run despite the
+   architect finding.
+7. **Boss-side workflow on clean** — manually integrate a clean run:
+   read draft journal, write milestone-close journal, append INDEX.
+   Verify linear history and INDEX coherence.
+8. **Manual-Mode preserved** — invoke `audit` skill directly (without
+   the orchestrator) — verify the current Boss-direct dispatch path
+   still works for ad-hoc invocations.
+
+Verification steps 3, 4, 6 require synthetic fixtures and are
+one-shot reproducers — they confirm the discipline once. Future audit
+runs answer "is the design holding up" empirically.
+
+## Acceptance criteria
+
+The refactor is accepted when:
+
+1. `skills/audit/agents/ailang-audit-orchestrator.md` exists, following
+   the standard agent template (frontmatter, spirit-letter lead-in,
+   standing reading list, carrier contract, Iron Law, process, status
+   protocol, output format, common rationalisations, red flags).
+2. `skills/audit/SKILL.md` gains a Standard-Mode subsection naming the
+   orchestrator-agent as the dispatcher at milestone close. Steps 1–5
+   retained verbatim as phase references + Manual-Mode path. Cross-
+   references updated.
+3. `skills/fieldtest/SKILL.md` cross-references updated to reflect
+   that Phase 4 of the audit-orchestrator is now the standard trigger;
+   Manual Mode path retained.
+4. `skills/README.md` skill table, agent roster, and pipeline diagram
+   updated.
+5. The four worker agent files (`ailang-architect.md`,
+   `ailang-bencher.md`, `ailang-docwriter.md`,
+   `ailang-fieldtester.md`) are **unchanged in body**. (The known-debt
+   re: subagent framing carries over from or.2.)
+6. The eight verification steps in Testing strategy pass.
+7. The first real milestone close after the refactor shows a
+   Boss-context delta consistent with the predicted offload (target:
+   ≤ 1k tokens per audit-orchestrator dispatch + end-report, plus the
+   Boss-side decision context for whatever findings emerge).
+
+## Migration
+
+Single trivial-mechanical addition. No code touched, no language
+behaviour change. Two ways to land:
+
+**Option (a) — single direct-on-main commit.** The new agent file,
+plus the additive edits to `audit/SKILL.md`, `fieldtest/SKILL.md`,
+`README.md`. Analogous to `or.2`'s landing.
+
+**Option (b) — `iter au.1` via the planner + implement loop.** Forces
+the new orchestrator-agent through its own discipline cycle (plan →
+TDD → review). Adds about a day of latency for a refactor that has
+no executable surface to test against.
+
+**Recommendation: (a).** Same reasoning as `or.2`: doc-only revision,
+no behaviour to test, the first real audit at the next milestone
+close IS the verification step. Option (b) is process-theatre for
+this case.
+
+## Out of scope (deferred to other specs)
+
+- **Brainstorm offload** — the `/brainstorm` usage statistic was 2%
+  pre-or.1; not cost-effective to encapsulate yet.
+- **Planner offload** — the `pr.1` recon-subagent already addresses
+  the read-heavy phase of planning; further offload is premature.
+- **Audit-result auto-resolution** — the Boss's `fix / ratify /
+  carry-on` decision is semantic, not mechanical. Pushing this into
+  the orchestrator would violate the user-stated principle that
+  Boss-context is decision-context. Out of scope on principle, not
+  on effort.
+- **Per-milestone-close branch isolation** — implement-orchestrator
+  uses iter-branches; audit-orchestrator commits direct-on-main
+  because per-phase changes (docwriter rustdoc fixes, fieldtest
+  fixtures) are individually safe and don't benefit from
+  iter-isolation. May be reconsidered if a future audit produces
+  larger-scale changes that warrant branching.
+- **Worker model downgrade** — all roles stay at Opus 4.7, as in
+  or.1.