AILang/docs/specs/0008-implement-orchestrator-agent.md

# `ailang-implement-orchestrator` — Design Spec

**Date:** 2026-05-11
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude

## Goal

Three coupled changes to `/implement`, each individually motivated,
together resolving the dominant cost and workflow-coordination
problems observed on 2026-05-11:

1. **Boss-context offload.** Reduce the Boss-Orchestrator's
   per-`/implement` context cost from ~100k tokens to ~700 tokens
   (factor ~100×) by delegating the entire per-task loop to a single
   subagent — `ailang-implement-orchestrator` — running in its own
   context.

2. **Iter isolation via branches.** Every `/implement` run owns a
   dedicated branch (`iter/<iter_id>`). The Boss-Orchestrator
   decides per run whether and when to integrate. Eliminates the
   "parallel implementer pushed 37 commits while I tried to commit
   a spec"-class of problem.

3. **Per-iter journals replace the JOURNAL.md monolith.** Iteration
   reports move from a single 14k-line `docs/JOURNAL.md` to per-iter
   files in `docs/journals/<YYYY-MM-DD>-iter-<id>.md`. The existing
   JOURNAL.md is archived as `docs/journal-archive.md`. A short
   `docs/journals/INDEX.md` provides chronological navigation, edited
   by the Boss as part of the merge step.

Quality of decisions is non-negotiable: model selection stays at
Opus 4.7 for every role; the saving comes from token-volume
relocation, not from model downgrades.

## Background — what the current `/implement` actually costs

Anthropic-side usage statistics for the project (2026-05-11):

```
30% /implement (skill)
26% subagents under "implement"
12% /plan
50% of usage at >150k context
49% of usage in 8h+ sessions
```

A per-task breakdown of today's `/implement` flow in the Boss-context:

| Step | Tokens (typical) |
|------|------------------|
| Plan loaded once (Step 1) — 478–1883 lines via `Read`  | 12k–25k *per iter* |
| Per-task implementer dispatch — `task_text` verbatim   | 2k–7k |
| Implementer report                                     | ~300  |
| Per-task spec-reviewer dispatch — `task_text` AGAIN    | 2k–7k *(duplicate)* |
| Spec-reviewer report                                   | ~350  |
| Per-task quality-reviewer dispatch — SHAs only         | ~100  |
| Quality-reviewer report                                | ~450  |
| Re-loop on `non_compliant` (~50% rate)                 | × 1.5 |

For an 8-task iteration: ~12–25k (plan) + ~88k (per-task ×8 ×1.5) ≈
**100k tokens in the Boss-context**. Combined with multi-iter
sessions this explains the >150k tail and the cost concentration the
statistics flag.

In parallel, two structural issues surfaced:
- Concurrent `/implement` activity (separate session pushing to
  `main` while the Boss was working) leaves the working tree in a
  detached-HEAD state with no clean spot to commit a small artefact
  like a spec file.
- `docs/JOURNAL.md` is 14309 lines. Skill steps that "read tail"
  routinely pull thousands of tokens of which 90% is irrelevant to
  the current iter; cross-iter searches require scrolling rather
  than file-system tools.

## Architecture

```
Boss-Orchestrator
    │
    │ Agent("ailang-implement-orchestrator", carrier)
    ▼
ailang-implement-orchestrator         ← runs in its own context
    │
    │   git fetch origin main
    │   git switch -c iter/<iter_id> origin/main
    │
    │   reads plan (once, in its context)
    │   per task K = 1..N:
    │       Agent("ailang-implementer", task_carrier_K)        ── carrier points to /tmp/ail-iter/<iter_id>/task-K.md
    │       on DONE:   Agent("ailang-spec-reviewer", review_carrier_K)
    │       on compliant: Agent("ailang-quality-reviewer", quality_carrier_K)
    │       loop on non_compliant / NEEDS_CONTEXT (≤ 2 retries)
    │       on BLOCKED or 3rd retry: stop, emit PARTIAL bundle
    │   Agent("ailang-tester", coverage_carrier)      ── E2E coverage
    │   write docs/journals/<YYYY-MM-DD>-iter-<id>.md   ── per-iter journal
    │   commit journal file on the branch
    │   write bench/orchestrator-stats/<...>.json
    │   commit stats file on the branch
    ▼
compressed end-report (~500 tokens) — pointer to journal file + branch name

then Boss-Orchestrator (separate, after reading end-report):
    │
    │   reviews docs/journals/<...>.md on the branch
    │   updates docs/journals/INDEX.md with one line
    │   git rebase iter/<iter_id> onto main   ── linear history
    │   git branch -D iter/<iter_id>          ── optional cleanup
```

The Boss-Orchestrator sees: one dispatch out, one compressed report
back, then a Boss-side merge step that touches one INDEX line.
Per-task loops, plan reads, review re-loops, journal writing, stats
writing — all happen inside the orchestrator-agent's context and
never reach the Boss.

`skills/README.md` currently states "Agents do not call other
agents." `ailang-implement-orchestrator` becomes the documented
exception, named explicitly. The rule remains otherwise binding —
no other agent gains the `Agent` tool.

## Components

### `ailang-implement-orchestrator` (new agent)

Location: `skills/implement/agents/ailang-implement-orchestrator.md`.

Tools: `Read, Edit, Write, Bash, Glob, Grep, Agent`. The `Agent`
tool is the explicit exception — it dispatches the existing four
worker agents (implementer, spec-reviewer, quality-reviewer,
tester).

Standing reading list:
1. `CLAUDE.md` — orchestrator framing.
2. `docs/DESIGN.md` — invariants the iteration must respect.
3. `docs/journals/INDEX.md` + the last 1–3 per-iter journal files
   linked from there — recent state. (After the migration window,
   no agent reads `docs/journal-archive.md` directly; long-tail
   context lives there but is referenced by date if needed.)
4. `skills/implement/SKILL.md` — **canonical discipline** (Iron
   Law, Step-2 sub-status mechanics, per-task scope rules). The
   orchestrator reads this verbatim every dispatch; it does not
   paraphrase.

Model: Opus 4.7.

First and last actions in every dispatch:
- **First**: `git fetch origin main && git switch -c iter/<iter_id> origin/main`.
  Ensures a clean base from current Remote regardless of what state
  the local working tree was in.
- **Last (on DONE)**: write per-iter journal file, commit it on the
  branch, write stats file, commit it on the branch, return
  end-report. Do not merge — that's Boss-side.
- **Last (on BLOCKED)**: write per-iter journal file with the
  partial status, commit it on the branch, write stats file with
  `outcome: PARTIAL` or `outcome: BLOCKED`, commit it, return
  end-report. Branch is left for the Boss to inspect, repair, or
  delete.

### `skills/implement/SKILL.md` (rewritten)

Today: ~210 lines covering Step 1 (plan load), Step 2.1–2.5
(per-task dispatch), Step 3 (tester), Step 4 (JOURNAL), Mini-Mode
handoff, Common Rationalisations, Red Flags.

After: ~100 lines. Both Standard-Mode and Mini-Mode collapse to:

> 1. Dispatch `ailang-implement-orchestrator` with the carrier
>    below.
> 2. Read the returned end-report.
> 3. **Boss merge step (on DONE):** open the per-iter journal at
>    `docs/journals/<...>.md` on the branch. Accept as-is or
>    condense; append a one-line entry to `docs/journals/INDEX.md`;
>    `git rebase iter/<iter_id> onto main`; optionally delete
>    branch.
> 4. **Boss handling (on BLOCKED):** inspect the partial branch,
>    repair (re-dispatch with adjusted plan / extended context),
>    or discard (`git branch -D iter/<iter_id>`).

The Iron Law, per-task sub-status table, and Common Rationalisations
stay in this file — they remain the canonical discipline source,
now read by the orchestrator-agent. What goes away from the skill
body is the *procedural Boss-side* of each per-task step (no
Boss-side per-task step remains).

### Worker agents — unchanged in role, minor carrier change

`ailang-implementer`, `ailang-spec-reviewer`,
`ailang-quality-reviewer`, `ailang-tester` keep their current
frontmatter, tools, standing reading lists, and status protocols.
Only their *caller* changes from the Boss-Orchestrator to the
`ailang-implement-orchestrator`.

One mechanical change to carriers: instead of `task_text` as a
verbatim string, implementer and spec-reviewer receive
`task_text_path` (`/tmp/ail-iter/<iter_id>/task-<N>.md`) and read
the file themselves. Eliminates per-task duplication of the task
text between the two agents.

### `docs/journals/` — new subdir

Replaces `docs/JOURNAL.md` going forward.

Naming: `docs/journals/<YYYY-MM-DD>-iter-<id>.md`. Example:
`docs/journals/2026-05-11-iter-ct.2.3.md`.

File content, written by the orchestrator-agent:

```markdown
# iter <id> — <title>

**Date:** YYYY-MM-DD
**Branch:** iter/<id>
**Status:** DONE | PARTIAL | BLOCKED
**Tasks completed:** <N> of <total>

## Summary

<1-paragraph, Boss may rewrite during merge>

## Per-task subjects

- iter <id>.1: <subject>
- iter <id>.2: <subject>
- ...

## Concerns

(DONE_WITH_CONCERNS aggregate, one line each)

## Known debt

(one-liner each, with why-not-touched)

## Blocked detail

(only if BLOCKED / PARTIAL: task N, reason, worker-quote, suggested next step)

## Commits

<pre-iter SHA>..<head SHA>

## Stats

bench/orchestrator-stats/<file>.json
```

The Summary section is what the Boss may rewrite when integrating —
the agent's draft is a starting point. The rest is factual and
preserved verbatim.

### `docs/journals/INDEX.md` — new file

Boss-maintained chronological index. Append-only, one line per iter:

```markdown
# Journal index

- 2026-05-11 — iter ct.2.3: typecheck cleanup → 2026-05-11-iter-ct.2.3.md
- 2026-05-12 — bugfix sigsegv-pattern-ctor → 2026-05-12-iter-bugfix-sigsegv-pattern-ctor.md
- ...
```

Written by the Boss as part of the merge step, never by the
orchestrator-agent. Filenames are relative to `docs/journals/`.

### `docs/journal-archive.md` — renamed historical JOURNAL.md

At spec-acceptance time, `docs/JOURNAL.md` is renamed to
`docs/journal-archive.md`. Content is preserved verbatim — it
remains the canonical record for everything pre-migration.

A single header note is added at the top:

```markdown
> **Status:** archived 2026-05-11. New iter journals live under
> `docs/journals/`. See `docs/journals/INDEX.md` for the
> chronological pointer list. This file remains the historical
> record for entries prior to that date.
```

`CLAUDE.md` is updated in the same migration commit to reflect the
new layout. Existing cross-references to "JOURNAL.md" in skill
files and agent files are updated to point at the new layout
(`docs/journals/INDEX.md` + recent entries).

### `bench/orchestrator-stats/<YYYY-MM-DD>-iter-<X>.json` — unchanged

Per-iter JSON, written by the orchestrator before the journal
commit. Pointer in the end-report. No aggregation script ships
with this spec; once a few iters accumulate, a separate utility
can compute distributions.

### `skills/README.md` (one-line update)

The "Agents do not call other agents" subsection gains a sentence
naming `ailang-implement-orchestrator` as the dispatch-and-coordinate
exception.

## Data flow

### Carrier — Boss → `ailang-implement-orchestrator`

| Field | Content |
|-------|---------|
| `mode` | `"standard"` or `"mini"` |
| `iter_id` | e.g. `"ct.2.3"` (standard) or `"bugfix-<short-symptom>"` (mini) — used for branch name, paths (stats file, scratch dir, journal file), JOURNAL header |
| `plan_path` | (standard only) `docs/plans/<file>.md` |
| `task_range` | (standard, optional) e.g. `[3, 8]` to run only Tasks 3..8 |
| `red_test_path` | (mini only) absolute path to the failing test from `debug` |
| `cause_summary` | (mini only) 1–2 sentences from the debugger agent |
| `constraint` | (mini only) `"minimal fix, no surrounding cleanup"` |

### Carrier — `ailang-implement-orchestrator` → worker agents

Implementer (per task):

| Field | Content |
|-------|---------|
| `task_text_path` | `/tmp/ail-iter/<iter_id>/task-<N>.md` — orchestrator extracts the verbatim task block here on first reference |
| `scene_set` | parent milestone, position in iter, dependencies |
| `cross_task_context` | shared types, naming conventions for the iter |
| `mode` | `standard` or `mini` |

Spec-reviewer:

| Field | Content |
|-------|---------|
| `task_text_path` | same file as the implementer received |
| `pre_task_sha`, `head_sha` | for `git diff <pre>..<head>` |
| `status_from_implementer` | `DONE` or `DONE_WITH_CONCERNS` (with concerns) |

Quality-reviewer:

| Field | Content |
|-------|---------|
| `pre_task_sha`, `head_sha` | for the diff |
| `spec_review_status` | must be `compliant` |
| `task_subject` | one-line title, NOT the full task text |

Tester (once per iter, at end, on DONE only):

| Field | Content |
|-------|---------|
| `iteration_scope` | what shipped — feature name, commit range, invariants |
| `coverage_gap` | if known |
| `mode` | `e2e_after_iter` |

### End-report — `ailang-implement-orchestrator` → Boss

Plain-text report, ≤ 500 tokens, fixed structure:

```
Status:           DONE | PARTIAL | BLOCKED
Iter:             <iter_id>
Branch:           iter/<iter_id>
Tasks completed:  <N> of <total>
                  - <commit subject 1>
                  - <commit subject 2>
                  ...
Journal file:     docs/journals/<file>.md (on branch)
Stats:            bench/orchestrator-stats/<file>.json (on branch)
Commits:          <pre-iter SHA>..<head SHA>
Tests:            <count> green, <count> red
E2E coverage:     <new fixture paths>
Blocked detail:   (only if BLOCKED)
                  Task: <N>
                  Reason: context-exhausted | review-loop-exhausted | worker-blocked
                  Worker says: <verbatim reason>
                  Suggested next step: <one sentence>
```

The Boss reads this directly; the per-iter journal file is read
selectively only when integrating or when investigating BLOCKED.

### Boss-side merge workflow (on DONE)

1. Read `docs/journals/<file>.md` from the branch
   (`git show iter/<iter_id>:docs/journals/<file>.md` or check out
   if a closer look is needed).
2. Accept the agent's summary as-is, or rewrite the Summary section
   to reflect Boss-level framing.
3. Append a one-line entry to `docs/journals/INDEX.md`.
4. `git rebase iter/<iter_id> onto main` for linear history.
5. Optionally `git branch -D iter/<iter_id>`.
6. If trigger is done-state and user-away: write `WhatsNew.md` entry
   and `notify.sh` per CLAUDE.md.

### Boss-side handling (on BLOCKED / PARTIAL)

1. Read the per-iter journal on the branch — the `Blocked detail:`
   section names the failure mode.
2. Decide:
   - **Repair**: adjust plan, re-dispatch the orchestrator with the
     same `iter_id` and `task_range` covering the remaining tasks.
     The orchestrator picks up the existing branch (`git switch
     iter/<iter_id>`) instead of creating a new one.
   - **Discard**: `git branch -D iter/<iter_id>`. The per-iter
     journal file disappears with the branch; INDEX.md is not
     touched (no entry was added).
   - **Escalate**: ask the user. The branch + journal file sit until
     the conversation resumes.

## Error handling — per-task sub-status mechanics

The orchestrator follows this table verbatim. The same table is the
canonical reference in `skills/implement/SKILL.md`.

| Sub-status | Orchestrator action |
|------------|---------------------|
| `DONE` | next task / next phase |
| `DONE_WITH_CONCERNS` | accumulate concern, next |
| `NEEDS_CONTEXT` (1st–2nd) | re-dispatch with expanded carrier |
| `NEEDS_CONTEXT` (3rd) | stop → `BLOCKED` to Boss, reason `context-exhausted` |
| `non_compliant` / `changes_requested` (1st–2nd) | implementer re-dispatch with the reviewer's report as repair brief |
| `non_compliant` / `changes_requested` (3rd) | stop → `BLOCKED` to Boss, reason `review-loop-exhausted` |
| `BLOCKED` (worker) | stop → `BLOCKED` to Boss, reason verbatim |
| Tool / infra error | stop → `BLOCKED` to Boss, reason `infra` + raw error |

On `BLOCKED`:
- already-committed tasks stay on the branch
- the stats file is written with `outcome: PARTIAL` and the partial
  per-task list
- the per-iter journal file IS written (with `Status: PARTIAL` or
  `Status: BLOCKED`), committed on the branch
- end-report carries `Status: BLOCKED` with detail
- `docs/journals/INDEX.md` is **not** touched (Boss decides on
  integration when handling the branch)

`skip task K, continue` is intentionally NOT a mode — tasks have
implicit ordering dependencies and the orchestrator does not know
the plan dependency graph. Skip is a Boss decision.

## Testing strategy

This is a skill/agent refactor, not a language feature. No
`cargo test` coverage applies directly. Verification ladder:

1. **Stand-alone agent file parses** — `Agent("ailang-implement-orchestrator", ...)`
   resolves to a runnable spec. Check via a no-op dispatch on a
   trivial fixture iter.
2. **Branch isolation** — dispatch the orchestrator; verify
   `git branch` after run shows `iter/<test-iter>` distinct from
   main; main is unchanged.
3. **Single-task standard-mode run** — dispatch the orchestrator on
   a one-task plan. Expect: green build, green tests, journal file
   on branch, stats file on branch, end-report follows the fixed
   format.
4. **Multi-task standard-mode run** — same on a 3+ task plan.
5. **Mini-mode run** — dispatch from `debug` handoff. Expect: same
   end-report shape, single-task stats, branch named `iter/bugfix-<>`.
6. **Re-loop run** — feed an intentionally non-compliant task.
   Expect: 2 re-dispatches, then `BLOCKED` with reason
   `review-loop-exhausted`. Branch preserved with partial commits.
7. **BLOCKED run** — feed an intentionally impossible task. Expect:
   immediate `BLOCKED`, partial commits preserved on branch.
8. **Boss-merge step** — manually integrate a successful test run:
   read journal file, write INDEX line, rebase, delete branch.
   Verify linear history and INDEX coherence.

Verification steps 6, 7, 8 are one-shot reproducers — they confirm
the discipline once. Future stats data answers "is the re-loop
limit too tight / too loose" empirically.

## Acceptance criteria

The refactor is accepted when:

1. `skills/implement/agents/ailang-implement-orchestrator.md` exists,
   following the standard agent template (frontmatter, spirit-letter
   lead-in, standing reading list, carrier contract, Iron Law,
   process, status protocol, output format, common rationalisations,
   red flags). Contains the branch-creation step as Process Step 1
   and the journal-file-write step as a late-process action.
2. `skills/implement/SKILL.md` is rewritten to delegate both modes
   to the orchestrator-agent and to document the Boss-side merge
   step. The canonical discipline (Iron Law, sub-status table,
   common rationalisations) remains in this file.
3. `skills/README.md` documents the orchestrator-agent as the named
   exception to "agents do not call other agents".
4. `skills/implement/agents/ailang-implementer.md` and
   `skills/implement/agents/ailang-spec-reviewer.md` accept
   `task_text_path` in place of `task_text` in their carriers.
5. `docs/JOURNAL.md` is renamed to `docs/journal-archive.md` with
   the archived-status header. `docs/journals/INDEX.md` exists
   (initially with a single line linking back to
   `journal-archive.md` for pre-migration context).
6. Skill files and agent files that reference `docs/JOURNAL.md` are
   updated to point at the new layout. `CLAUDE.md` reflects the new
   roles of `docs/journals/`, `docs/journals/INDEX.md`, and
   `docs/journal-archive.md`.
7. The eight verification steps in Testing strategy pass.
8. The first real `/implement` run after the refactor lands shows a
   Boss-context delta consistent with the predicted factor (target:
   ≤ 1k tokens per `/implement` in the Boss-context for the agent
   dispatch + end-report, plus ≤ 1k tokens for the Boss merge step).

## Migration

The migration touches three categories of files. All happen in a
single commit (or one commit per category, atomically).

**1. Filesystem.**
- `git mv docs/JOURNAL.md docs/journal-archive.md`
- Add archived-status header to `docs/journal-archive.md`
- Create empty `docs/journals/` directory with `.gitkeep` (or with
  the initial INDEX.md)
- Create `docs/journals/INDEX.md` with the header and a single
  pointer line referencing the archive for pre-migration entries

**2. References.**
- Scan all files for `JOURNAL.md` and update each reference:
  - `CLAUDE.md` "Roles of …" section: new descriptions for
    `docs/journals/`, `docs/journals/INDEX.md`,
    `docs/journal-archive.md`
  - `skills/*/SKILL.md` files: "read JOURNAL.md tail" → "read
    `docs/journals/INDEX.md` and the latest 1–3 referenced files"
  - `skills/*/agents/*.md` standing reading lists: same update
  - `docs/DESIGN.md` if it references JOURNAL.md: same update

**3. Workflow.**
- Existing plans in `docs/plans/` work unchanged — the plan format
  is unchanged.
- Existing Iter-22+ plans can be re-run end-to-end through the new
  orchestrator without rewriting; only the journal-write target
  differs (per-iter file instead of appending to JOURNAL.md).
- WhatsNew.md and `notify.sh` workflow unchanged.

## Out of scope (deferred to other specs)

- **Worker model downgrade** (cheap models for simple subagents).
  User has explicitly excluded this; all roles stay at Opus 4.7.
- **Stats aggregation tool** — written once empirical data exists.
- **Plan-drafter sub-agent** for the `/planner` skill. Separate hebel,
  separate spec.
- **`/brainstorm` spec-drafter** — usage statistic shows
  `/brainstorm` at 2%, not cost-effective.
- **`docs/DESIGN.md` changes**. The refactor is below the language
  spec; the canonical AILang language definition is untouched.
- **External-platform integration** (Gitea PR via `tea`, GitHub
  spiegelung). Decided 2026-05-11: the Per-Iter-File pattern in the
  Git repo is the single source of truth; external platform UI is
  redundant because Gitea's native file-browser already serves
  these files read-only once pushed. May be reconsidered if
  team-review workflow appears later.