AILang/docs/journals/2026-05-12-iter-ms.2.md

iter ms.2 — Multi-subject run + DESIGN.md addendum extension

Date: 2026-05-12 Started from: 8689f7a238 Status: DONE Tasks completed: 6 of 6

Summary

Two live IONOS harness invocations executed with the ms.1 fix in place: a Qwen3-Coder-Next retroactive re-run (runs/2026-05-12-080864/, RUN_STATUS=ok) and a first-ever CodeLlama-13b-Instruct run (runs/2026-05-12-9197fd/, RUN_STATUS=ok at the run level; 2 of 8 cells terminated as api_failure on turn 1 with zero tokens — transient IONOS-side terminal error, disclosed in the addendum and below). DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)" replaced with a four-column per-cell table plus a framing paragraph; roadmap P3 "Multi-subject expansion" entry removed.

Both subjects point the same way on every measured axis: AILX cohort is cheaper than JSON cohort on prompt tokens (Qwen 61%, CodeLlama 80%) and on completion tokens (Qwen 23%, CodeLlama 82%), and AILX reached-green ≥ JSON reached-green for each subject (Qwen ties at 1/4; CodeLlama strictly dominates 2/4 vs 0/4). Three of the four first-attempt-green cells across the two subjects are AILX. Decision 6 is not ratified — n=1 per subject across two subjects is not statistical robustness — but the second data point points consistently. The roadmap entry was removed because the brainstorm spec was scoped to two subjects, not because the universal claim is closed; a future ratification iteration would re-open it.

Anomaly disclosure: the Qwen re-run shifted by one cell vs the cma.3 baseline (AILX 2/4 → 1/4, JSON 1/4 → 1/4) at identical temperature=0; this is IONOS-side state-of-day noise, not an effect of the ms.1 feedback fix, which only changes JSON-cohort prompt content. The two CodeLlama JSON api_failure cells (t2_length, t4_count_zeros) were not re-run — the carrier constraint forbids exploratory token spend, and the addendum discloses the partial sample directly.

Roadmap P3 "Multi-subject expansion" entry removed in this iter.

Per-task notes

• iter ms.2.1: Qwen retroactive re-run via harness — new run dir experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/, RUN_STATUS=ok, 8/8 cells populated. AIL_BIN set to target/release/ail (no ~/.local/bin/ail); harness picks it up via env-var resolution at pipeline.rs:42.
• iter ms.2.2: CodeLlama-13b-Instruct first run via harness — new run dir experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/, model id meta-llama/CodeLlama-13b-Instruct-hf (Boss-pre-flighted IONOS catalogue id), RUN_STATUS=ok. 6/8 cells informative; 2/8 json cells (t2_length, t4_count_zeros) api_failure on turn 1 zero tokens.
• iter ms.2.3: Per-cell metrics computed via direct scores.csv read — see addendum table. Total milestone token spend: prompt 501,985 + completion 23,391 = 525,376.
• iter ms.2.4: DESIGN.md lines 604-636 replaced with new four-column addendum block (~50 lines including the framing paragraph). Subjects named with canonical IONOS model ids and run-dir pointers; framing paragraph reports direction agreement, failure-class symmetry, first-attempt tally, api_failure anomaly, and Qwen-rerun noise. Scope paragraph extended to "two subjects, n=1 each".
• iter ms.2.5: roadmap.md lines 253-268 removed (P3 multi-subject entry + context note); journal file written. INDEX.md is NOT edited by this agent per the Iron Law; the Boss appends the ms.2 line at commit time. Suggested INDEX line (per plan Task 5 Step 3): - 2026-05-12 — iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run; DESIGN.md §Decision-6 addendum extended to four cells; roadmap P3 removed → 2026-05-12-iter-ms.2.md
• iter ms.2.6: Spec acceptance criteria sweep (1-7 all green); 14 harness tests still pass.

Spec acceptance sweep

1. ms.1 pipeline.rs:118 reads format!("check: {e:#}") + RED→GREEN unit test → green (ms.1 working tree).
2. ms.1 existing 14 harness tests still pass → green (verified Task 6 Step 2).
3. CodeLlama-13b-Instruct-hf runs to completion against IONOS endpoint, RUN_STATUS=ok at run level → green; model id recorded above.
4. Qwen3-Coder-Next re-run completes, RUN_STATUS=ok, 8 cells; distinct dir from cma.3 baseline → green.
5. DESIGN.md §"Decision 6 / Empirical addendum" extended with follow-up paragraph + four-cell-shaped table → green.
6. Roadmap P3 entry removed; one-line mirror above → green.
7. Per-iter journal (this file) written → green. INDEX.md ms.2 entry is Boss-side per the Iron Law; suggested line is in the end-report.

Concerns

• 2 of 4 CodeLlama JSON cells are api_failure (terminal IONOS error on turn 1). Per the Plan's branch (c) and the carrier's "no exploratory re-runs" constraint, not re-attempted. The addendum and journal disclose this directly; the CodeLlama-JSON column's prompt-token average is therefore over 2 informative cells, not 4. Effect on the AILX-cheaper conclusion: none in direction; the prompt-token gap is preserved over the two informative JSON cells. If a future iteration re-runs CodeLlama at lower IONOS load, this should resolve.
• Qwen-re-run vs cma.3-baseline drift (1 cell) at identical temperature=0 is IONOS noise on a single deterministic call. A future ratification iteration should add ≥3 repetitions per subject to bound this noise, not rely on a single temperature-zero call.

Known debt

• The pre-existing 3 warnings in harness/src/ionos.rs lines 86, 125, 137 (last_err assigned but never read) survived ms.1 and ms.2. Not in scope here; cma.2 inheritance.
• Decision 6 universal-claim status is still "second data point, not verdict". A future ratification iteration would need ≥3 subjects with statistical robustness. Out of scope for ms.

Files touched

• docs/DESIGN.md (addendum extended)
• docs/roadmap.md (P3 entry removed)
• experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/ (created by Qwen run)
• experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/ (created by CodeLlama run)
• docs/journals/2026-05-12-iter-ms.2.md (this file)
• bench/orchestrator-stats/2026-05-12-iter-ms.2.json (stats)

Stats

bench/orchestrator-stats/2026-05-12-iter-ms.2.json