## Multi-subject Authoring-Form Test — CodeLlama Replication — Design Spec **Date:** 2026-05-12 **Status:** Draft — awaiting grounding-check **Authors:** Brummel (orchestrator) + Claude ## Goal Extend the single-subject cma baseline to a second foreign LLM (CodeLlama via IONOS), n=1 on the same four tasks, to test whether the per-cohort token-economy and pass-rate patterns observed with Qwen3-Coder-Next reproduce across another model class — or are Qwen-specific. While planning this milestone a JSON-cohort feedback asymmetry surfaced that was previously mis-attributed to the strip_locations regex: harness-side JSON-parse failures get their anyhow `Caused by` chain dropped on the way into the model prompt, leaving the JSON cohort with the uninformative top-line "parsing JSON in /tmp/..." while the AILX cohort sees the full `ail parse` diagnostic. Fix that bug first; then run both subjects with the fixed harness so the dataset is internally consistent. ## Architecture Two iterations: - **ms.1 — pipeline error-formatting fix.** Change `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118` from `format!("check: {e}")` to `format!("check: {e:#}")` so anyhow's alternate Display inlines the full cause chain. RED-first: a unit test that constructs the same anyhow chain shape the live failure produces (a `serde_json::Error` wrapped with the `parsing JSON in ` context) and asserts the formatted output contains both layers. The change is two characters; the test is the protection. The strip_locations regex already collapses `\nCaused by:` chains, so the alternate-Display form (which joins with `: ` inline, no newline) survives stripping cleanly — no regex change needed. - **ms.2 — Multi-subject run + integration.** Two `harness` invocations, both with the fixed pipeline: (a) retroactive Qwen3-Coder-Next re-run, so the original four-task dataset matches the fixed JSON-feedback behaviour and is apples-to-apples with CodeLlama; (b) first-ever CodeLlama-class run. Live IONOS catalogue (`/v1/models` GET, pre-flighted) names the model `meta-llama/CodeLlama-13b-Instruct-hf` — the Instruct-tuned HuggingFace mirror of CodeLlama-13b. Both into fresh `runs/-/` directories. Comparative summary written into `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)" as a follow-up paragraph plus an extended table spanning the four (model × cohort) cells. Roadmap P3 entry closed; one-line mirror in the iter journal. Multi-model orchestration is out of scope: the harness remains single-model-per-invocation. Two cargo runs is acceptable for two subjects. ## Components ms.1: - `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118` — change `{e}` to `{e:#}` (two-character fix). - `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs` — add `#[cfg(test)] mod tests` (or extend it if present) with a RED→GREEN test for the anyhow-chain formatter. ms.2: - `experiments/2026-05-12-cross-model-authoring/runs/-/` — populated by Qwen re-run. - `experiments/2026-05-12-cross-model-authoring/runs/-/` — populated by CodeLlama run. - `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)" — extended with one follow-up paragraph and one extended per-cell table. - `docs/roadmap.md` — P3 "Multi-subject expansion" entry removed. - `docs/journals/2026-05-12-iter-ms.1.md`, `docs/journals/2026-05-12-iter-ms.2.md`, `docs/journals/2026-05-12-audit-ms.md`, `docs/journals/INDEX.md` entries. ## Data flow ms.1: 1. Write a `#[test]` constructing an anyhow chain that mirrors the live failure mode: a `serde_json::Error` wrapped with `.context(format!("parsing JSON in {path}"))`. Assert `format!("check: {e:#}")` contains both `parsing JSON` and the underlying serde token from the chain. 2. Run the test → RED (current code uses `{e}`, drops chain). 3. Change `{e}` → `{e:#}` on line 118. 4. Run the test → GREEN. 5. Run the full existing harness test sweep — all 13 must still pass. ms.2: 1. Run the harness with `--model meta-llama/CodeLlama-13b-Instruct-hf` (the canonical IONOS catalogue entry, confirmed via pre-flight `/v1/models` GET during planning). If IONOS rejects the call on the first completion attempt despite the pre-flight, abort and bounce to the user with the captured error text. 2. Run harness against Qwen3-Coder-Next with the fixed pipeline → 8 cells (4 tasks × 2 cohorts), RUN_STATUS=ok. 3. Run harness against the chosen CodeLlama model → 8 cells, RUN_STATUS=ok. 4. Compute per-model means: prompt_tokens (Qwen-counted via the API's `usage.prompt_tokens` field — note CodeLlama's tokeniser differs, so cross-model token counts compare counter-output not identical content), completion_tokens, pass-rate, turn-count distribution. 5. Write extended addendum: one paragraph framing the second data point, plus a per-cell table replacing the current two-column shape with four columns (Qwen-JSON, Qwen-AILX, CodeLlama-JSON, CodeLlama-AILX). 6. Remove the roadmap P3 entry; mirror the line in the iter journal. ## Error handling - ms.1 test: a pure unit test, no external dependency, no I/O. - ms.2 IONOS API: existing harness retry policy (5xx exp backoff, Retry-After honouring on 429) applies unchanged. - CodeLlama model ID resolution: pre-flighted via `/v1/models` GET during planning. If IONOS unexpectedly rejects `meta-llama/CodeLlama-13b-Instruct-hf` on the first completion call despite the pre-flight, the orchestrator aborts the run and bounces back to the user with the captured error text. This is the only point in the milestone with a real user-bounce trigger; the rest is mechanical. - Token budget: 2× the cma per-run cap (the harness's existing `--max-tokens` default suffices; no override needed). The cma baseline used ~305k actual; CodeLlama may produce more retries and burn more, but the cap provides headroom. ## Testing strategy ms.1: one new unit test (RED→GREEN) pinning the anyhow-chain preservation. The existing 13 harness tests must still pass. ms.2: rely entirely on the existing harness test suite. The milestone exercises the harness in live mode, not extends it. The two new runs verify completion by RUN_STATUS=ok in `summary.md` plus 8 fully populated cells per run. ## Acceptance criteria 1. ms.1: `pipeline.rs:118` reads `format!("check: {e:#}")`; a RED→GREEN unit test pins the chain-preservation property. 2. ms.1: existing 13 harness tests still pass. 3. ms.2: `meta-llama/CodeLlama-13b-Instruct-hf` (or, on user-redirected fallback after an unexpected model-rejection bounce, whatever model the user names) runs to completion against the IONOS endpoint (8 cells, RUN_STATUS=ok). The chosen model ID is recorded in the iter journal. 4. ms.2: a Qwen3-Coder-Next re-run with the fixed pipeline completes (8 cells, RUN_STATUS=ok). Distinct `runs/-/` from the cma.3 baseline; both new runs are checked in. 5. ms.2: `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)" extended with one follow-up paragraph plus a four-cell-shaped table covering prompt_tokens, completion_tokens, pass-rate, and turn-count distribution. 6. ms.2: Roadmap P3 multi-subject entry removed; one-line mirror in `2026-05-12-iter-ms.2.md`. 7. Per-iter journal entries (ms.1, ms.2) linked from `docs/journals/INDEX.md`; audit-ms entry on milestone close.