# audit-ms — Milestone close: Multi-subject Authoring-Form Test (CodeLlama Replication)

**Date:** 2026-05-12
**Milestone:** Multi-subject Authoring-Form Test — CodeLlama Replication (ms.1 + ms.2)
**Status:** Closed, clean (no tidy fixes needed)

## Architect drift review

`ailang-architect` reported `clean` over the working tree
(`8689f7a..HEAD`, no commits yet). Working tree matches DESIGN.md
(the §Decision-6 addendum extension is the only DESIGN.md touch,
intentional and spec-mandated), matches the spec's seven acceptance
criteria, and matches the boss-only-commit / append-only-INDEX /
main-sacrosanct discipline.

Two `[low]` items flagged, both expected:

- The spec mandates an `audit-ms` INDEX entry on milestone close;
  this audit pass is the one that produces it — written below.
- `docs/specs/2026-05-12-multi-subject-codellama.md:154` references
  "the cma.3 baseline" (`runs/2026-05-12-df7531/`) which DESIGN.md
  no longer cites after the addendum rewrite to a two-subject
  view. The spec text is correct (the new runs are distinct from
  cma.3) and the baseline dir is still on disk under the cma run
  tree, so the asymmetry is purely cosmetic — ratified, no edit.

Architect recommendation: carry on as planned.

## Bench-regression check

`bench/check.py && bench/compile_check.py && bench/cross_lang.py`:

- `check.py`: exit 0; **63 metrics; 0 regressed, 5 improved beyond
  tolerance, 58 stable**.
- `compile_check.py`: exit 0; 24 metrics; 0 regressed, 0 improved
  beyond tolerance, 24 stable.
- `cross_lang.py`: exit 0; 25 metrics; 0 regressed, 0 improved
  beyond tolerance, 25 stable.

All three exit-code gates green. The five improvements on
`check.py` cluster on `latency.explicit_at_rc` (p99, p99_9, max,
p99_over_median) plus `throughput.bench_list_sum.gc_over_bump` —
the exact same cluster `audit-cma` (this morning's milestone close)
flagged and left unratified for the same reason. The ms milestone
touched only `experiments/2026-05-12-cross-model-authoring/harness/src/`
(plus docs); no in-workspace runtime / codegen / RC path changed.
The improvements therefore reflect runtime variance or
system-level drift uncorrelated with the milestone.

**Decision: no baseline update (carry-on).** Same rationale as
audit-cma: ratifying an unexplained improvement would lock it into
the baseline and lose the ability to detect a future regression
back to the prior level. If the same improvement cluster persists
across the next two or three audits, it ratifies with confidence.
This is now the second audit in a row showing the same five-metric
improvement cluster; one more clean audit and the next audit can
ratify.

## Spec acceptance criteria — final check

Reviewing `docs/specs/2026-05-12-multi-subject-codellama.md`
§"Acceptance criteria" against landed work:

1. ✓ `pipeline.rs:118` reads `format!("check: {e:#}")`; RED→GREEN
   pinning unit test `check_format_preserves_anyhow_chain` present
   in the same file. (ms.1)
2. ✓ Existing 13 harness tests still pass; suite now 14/14 with
   the new pin test. (ms.1, re-confirmed at ms.2 close)
3. ✓ `meta-llama/CodeLlama-13b-Instruct-hf` ran to completion
   against IONOS (8 cells, RUN_STATUS=ok at run level; 2 cells
   `api_failure` on turn 1 disclosed in addendum). Chosen model id
   recorded in ms.2 journal. (ms.2)
4. ✓ Qwen3-Coder-Next re-run with fixed pipeline completed (8
   cells, RUN_STATUS=ok). New dir `runs/2026-05-12-080864/`
   distinct from cma.3 baseline `runs/2026-05-12-df7531/`; both new
   runs checked into the working tree. (ms.2)
5. ✓ `docs/DESIGN.md` §"Decision 6 / Empirical addendum
   (2026-05-12)" extended: framing paragraph + four-column per-cell
   table (Qwen-JSON, Qwen-AILX, CodeLlama-JSON, CodeLlama-AILX)
   covering prompt_tokens, completion_tokens, pass-rate, turn-count
   distribution, top error class; api_failure anomaly disclosed;
   scope paragraph updated to "two subjects, n=1 each". (ms.2)
6. ✓ Roadmap P3 multi-subject entry removed; one-line mirror in
   `2026-05-12-iter-ms.2.md`. (ms.2)
7. ✓ Per-iter journal entries (ms.1, ms.2) linked from INDEX.md;
   this audit-ms entry being added by this audit pass. (ms.{1,2,
   audit})

All seven criteria green.

## Outcome

Milestone closed clean — no tidy fixes needed (both architect-flagged
`[low]` items are expected by-products of the audit step itself,
not drift). The multi-subject expansion ships: pipeline-error
formatting fixed (JSON-cohort now sees the full anyhow cause
chain), Qwen retroactive re-run plus first CodeLlama-13b-Instruct
run recorded, DESIGN.md §Decision-6 addendum now spans two
subjects with consistent direction (AILX cheaper + ≥ green for
both), roadmap P3 entry closed. Token spend across the milestone:
prompt 501,985 + completion 23,391 = 525,376.

Decision 6 still pending universal ratification — the spec was
explicitly scoped to two subjects, not the ≥3-subjects + statistics
that a ratification would need. A future iteration may re-open the
queue if additional subjects are wanted; for now the second data
point is recorded and the queue is back to user-direction.
