# iter ms.2 — Multi-subject run + DESIGN.md addendum extension

**Date:** 2026-05-12
**Started from:** 8689f7a2386e0d7993609c374d91984bc28d215c
**Status:** DONE
**Tasks completed:** 6 of 6

## Summary

Two live IONOS harness invocations executed with the ms.1 fix in
place: a Qwen3-Coder-Next retroactive re-run
(`runs/2026-05-12-080864/`, RUN_STATUS=ok) and a first-ever
CodeLlama-13b-Instruct run (`runs/2026-05-12-9197fd/`, RUN_STATUS=ok
at the run level; 2 of 8 cells terminated as api_failure on turn 1
with zero tokens — transient IONOS-side terminal error, disclosed
in the addendum and below). DESIGN.md §"Decision 6 / Empirical
addendum (2026-05-12)" replaced with a four-column per-cell table
plus a framing paragraph; roadmap P3 "Multi-subject expansion"
entry removed.

Both subjects point the same way on every measured axis: AILX
cohort is cheaper than JSON cohort on prompt tokens (Qwen 61%,
CodeLlama 80%) and on completion tokens (Qwen 23%, CodeLlama 82%),
and AILX reached-green ≥ JSON reached-green for each subject (Qwen
ties at 1/4; CodeLlama strictly dominates 2/4 vs 0/4). Three of the
four first-attempt-green cells across the two subjects are AILX.
Decision 6 is not ratified — n=1 per subject across two subjects is
not statistical robustness — but the second data point points
consistently. The roadmap entry was removed because the brainstorm
spec was scoped to two subjects, not because the universal claim is
closed; a future ratification iteration would re-open it.

Anomaly disclosure: the Qwen re-run shifted by one cell vs the cma.3
baseline (AILX 2/4 → 1/4, JSON 1/4 → 1/4) at identical
temperature=0; this is IONOS-side state-of-day noise, not an effect
of the ms.1 feedback fix, which only changes JSON-cohort prompt
content. The two CodeLlama JSON api_failure cells (`t2_length`,
`t4_count_zeros`) were not re-run — the carrier constraint forbids
exploratory token spend, and the addendum discloses the partial
sample directly.

Roadmap P3 "Multi-subject expansion" entry removed in this iter.

## Per-task notes

- iter ms.2.1: Qwen retroactive re-run via harness — new run dir
  `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/`,
  RUN_STATUS=ok, 8/8 cells populated. AIL_BIN set to
  `target/release/ail` (no `~/.local/bin/ail`); harness picks it up
  via env-var resolution at `pipeline.rs:42`.
- iter ms.2.2: CodeLlama-13b-Instruct first run via harness — new
  run dir
  `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/`,
  model id `meta-llama/CodeLlama-13b-Instruct-hf` (Boss-pre-flighted
  IONOS catalogue id), RUN_STATUS=ok. 6/8 cells informative; 2/8
  json cells (`t2_length`, `t4_count_zeros`) `api_failure` on turn 1
  zero tokens.
- iter ms.2.3: Per-cell metrics computed via direct scores.csv
  read — see addendum table. Total milestone token spend: prompt
  501,985 + completion 23,391 = 525,376.
- iter ms.2.4: DESIGN.md lines 604-636 replaced with new
  four-column addendum block (~50 lines including the framing
  paragraph). Subjects named with canonical IONOS model ids and
  run-dir pointers; framing paragraph reports direction agreement,
  failure-class symmetry, first-attempt tally, api_failure anomaly,
  and Qwen-rerun noise. Scope paragraph extended to "two subjects,
  n=1 each".
- iter ms.2.5: roadmap.md lines 253-268 removed (P3 multi-subject
  entry + context note); journal file written. INDEX.md is NOT
  edited by this agent per the Iron Law; the Boss appends the
  ms.2 line at commit time. Suggested INDEX line (per plan Task 5
  Step 3):
  `- 2026-05-12 — iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run; DESIGN.md §Decision-6 addendum extended to four cells; roadmap P3 removed → 2026-05-12-iter-ms.2.md`
- iter ms.2.6: Spec acceptance criteria sweep (1-7 all green); 14
  harness tests still pass.

## Spec acceptance sweep

1. ms.1 `pipeline.rs:118` reads `format!("check: {e:#}")` + RED→GREEN unit test → green (ms.1 working tree).
2. ms.1 existing 14 harness tests still pass → green (verified Task 6 Step 2).
3. CodeLlama-13b-Instruct-hf runs to completion against IONOS endpoint, RUN_STATUS=ok at run level → green; model id recorded above.
4. Qwen3-Coder-Next re-run completes, RUN_STATUS=ok, 8 cells; distinct dir from cma.3 baseline → green.
5. DESIGN.md §"Decision 6 / Empirical addendum" extended with follow-up paragraph + four-cell-shaped table → green.
6. Roadmap P3 entry removed; one-line mirror above → green.
7. Per-iter journal (this file) written → green. INDEX.md ms.2 entry is Boss-side per the Iron Law; suggested line is in the end-report.

## Concerns

- 2 of 4 CodeLlama JSON cells are `api_failure` (terminal IONOS
  error on turn 1). Per the Plan's branch (c) and the carrier's
  "no exploratory re-runs" constraint, not re-attempted. The
  addendum and journal disclose this directly; the
  CodeLlama-JSON column's prompt-token average is therefore over 2
  informative cells, not 4. Effect on the AILX-cheaper conclusion:
  none in direction; the prompt-token gap is preserved over the
  two informative JSON cells. If a future iteration re-runs
  CodeLlama at lower IONOS load, this should resolve.
- Qwen-re-run vs cma.3-baseline drift (1 cell) at identical
  temperature=0 is IONOS noise on a single deterministic call. A
  future ratification iteration should add ≥3 repetitions per
  subject to bound this noise, not rely on a single
  temperature-zero call.

## Known debt

- The pre-existing 3 warnings in `harness/src/ionos.rs` lines 86,
  125, 137 (`last_err` assigned but never read) survived ms.1
  and ms.2. Not in scope here; cma.2 inheritance.
- Decision 6 universal-claim status is still "second data point,
  not verdict". A future ratification iteration would need ≥3
  subjects with statistical robustness. Out of scope for ms.

## Files touched

- `docs/DESIGN.md` (addendum extended)
- `docs/roadmap.md` (P3 entry removed)
- `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/` (created by Qwen run)
- `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/` (created by CodeLlama run)
- `docs/journals/2026-05-12-iter-ms.2.md` (this file)
- `bench/orchestrator-stats/2026-05-12-iter-ms.2.json` (stats)

## Stats

bench/orchestrator-stats/2026-05-12-iter-ms.2.json