AILang/docs/specs/0018-multi-subject-codellama.md

Multi-subject Authoring-Form Test — CodeLlama Replication — Design Spec

Date: 2026-05-12 Status: Draft — awaiting grounding-check Authors: Brummel (orchestrator) + Claude

Goal

Extend the single-subject cma baseline to a second foreign LLM (CodeLlama via IONOS), n=1 on the same four tasks, to test whether the per-cohort token-economy and pass-rate patterns observed with Qwen3-Coder-Next reproduce across another model class — or are Qwen-specific. While planning this milestone a JSON-cohort feedback asymmetry surfaced that was previously mis-attributed to the strip_locations regex: harness-side JSON-parse failures get their anyhow Caused by chain dropped on the way into the model prompt, leaving the JSON cohort with the uninformative top-line "parsing JSON in /tmp/..." while the AILX cohort sees the full ail parse diagnostic. Fix that bug first; then run both subjects with the fixed harness so the dataset is internally consistent.

Architecture

Two iterations:

• ms.1 — pipeline error-formatting fix. Change experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118 from format!("check: {e}") to format!("check: {e:#}") so anyhow's alternate Display inlines the full cause chain. RED-first: a unit test that constructs the same anyhow chain shape the live failure produces (a serde_json::Error wrapped with the parsing JSON in <path> context) and asserts the formatted output contains both layers. The change is two characters; the test is the protection. The strip_locations regex already collapses \nCaused by: chains, so the alternate-Display form (which joins with : inline, no newline) survives stripping cleanly — no regex change needed.

• ms.2 — Multi-subject run + integration. Two harness invocations, both with the fixed pipeline: (a) retroactive Qwen3-Coder-Next re-run, so the original four-task dataset matches the fixed JSON-feedback behaviour and is apples-to-apples with CodeLlama; (b) first-ever CodeLlama-class run. Live IONOS catalogue (/v1/models GET, pre-flighted) names the model meta-llama/CodeLlama-13b-Instruct-hf — the Instruct-tuned HuggingFace mirror of CodeLlama-13b. Both into fresh runs/<date>-<hash>/ directories. Comparative summary written into docs/DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)" as a follow-up paragraph plus an extended table spanning the four (model × cohort) cells. Roadmap P3 entry closed; one-line mirror in the iter journal.

Multi-model orchestration is out of scope: the harness remains single-model-per-invocation. Two cargo runs is acceptable for two subjects.

Components

ms.1:

• experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118 — change {e} to {e:#} (two-character fix).
• experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs — add #[cfg(test)] mod tests (or extend it if present) with a RED→GREEN test for the anyhow-chain formatter.

ms.2:

• experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<qwen-hash>/ — populated by Qwen re-run.
• experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<codellama-hash>/ — populated by CodeLlama run.
• docs/DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)" — extended with one follow-up paragraph and one extended per-cell table.
• docs/roadmap.md — P3 "Multi-subject expansion" entry removed.
• docs/journals/2026-05-12-iter-ms.1.md, docs/journals/2026-05-12-iter-ms.2.md, docs/journals/2026-05-12-audit-ms.md, docs/journals/INDEX.md entries.

Data flow

ms.1:

1. Write a #[test] constructing an anyhow chain that mirrors the live failure mode: a serde_json::Error wrapped with .context(format!("parsing JSON in {path}")). Assert format!("check: {e:#}") contains both parsing JSON and the underlying serde token from the chain.
2. Run the test → RED (current code uses {e}, drops chain).
3. Change {e} → {e:#} on line 118.
4. Run the test → GREEN.
5. Run the full existing harness test sweep — all 13 must still pass.

ms.2:

1. Run the harness with --model meta-llama/CodeLlama-13b-Instruct-hf (the canonical IONOS catalogue entry, confirmed via pre-flight /v1/models GET during planning). If IONOS rejects the call on the first completion attempt despite the pre-flight, abort and bounce to the user with the captured error text.
2. Run harness against Qwen3-Coder-Next with the fixed pipeline → 8 cells (4 tasks × 2 cohorts), RUN_STATUS=ok.
3. Run harness against the chosen CodeLlama model → 8 cells, RUN_STATUS=ok.
4. Compute per-model means: prompt_tokens (Qwen-counted via the API's usage.prompt_tokens field — note CodeLlama's tokeniser differs, so cross-model token counts compare counter-output not identical content), completion_tokens, pass-rate, turn-count distribution.
5. Write extended addendum: one paragraph framing the second data point, plus a per-cell table replacing the current two-column shape with four columns (Qwen-JSON, Qwen-AILX, CodeLlama-JSON, CodeLlama-AILX).
6. Remove the roadmap P3 entry; mirror the line in the iter journal.

Error handling

• ms.1 test: a pure unit test, no external dependency, no I/O.
• ms.2 IONOS API: existing harness retry policy (5xx exp backoff, Retry-After honouring on 429) applies unchanged.
• CodeLlama model ID resolution: pre-flighted via /v1/models GET during planning. If IONOS unexpectedly rejects meta-llama/CodeLlama-13b-Instruct-hf on the first completion call despite the pre-flight, the orchestrator aborts the run and bounces back to the user with the captured error text. This is the only point in the milestone with a real user-bounce trigger; the rest is mechanical.
• Token budget: 2× the cma per-run cap (the harness's existing --max-tokens default suffices; no override needed). The cma baseline used ~305k actual; CodeLlama may produce more retries and burn more, but the cap provides headroom.

Testing strategy

ms.1: one new unit test (RED→GREEN) pinning the anyhow-chain preservation. The existing 13 harness tests must still pass.

ms.2: rely entirely on the existing harness test suite. The milestone exercises the harness in live mode, not extends it. The two new runs verify completion by RUN_STATUS=ok in summary.md plus 8 fully populated cells per run.

Acceptance criteria

1. ms.1: pipeline.rs:118 reads format!("check: {e:#}"); a RED→GREEN unit test pins the chain-preservation property.
2. ms.1: existing 13 harness tests still pass.
3. ms.2: meta-llama/CodeLlama-13b-Instruct-hf (or, on user-redirected fallback after an unexpected model-rejection bounce, whatever model the user names) runs to completion against the IONOS endpoint (8 cells, RUN_STATUS=ok). The chosen model ID is recorded in the iter journal.
4. ms.2: a Qwen3-Coder-Next re-run with the fixed pipeline completes (8 cells, RUN_STATUS=ok). Distinct runs/<date>-<hash>/ from the cma.3 baseline; both new runs are checked in.
5. ms.2: docs/DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)" extended with one follow-up paragraph plus a four-cell-shaped table covering prompt_tokens, completion_tokens, pass-rate, and turn-count distribution.
6. ms.2: Roadmap P3 multi-subject entry removed; one-line mirror in 2026-05-12-iter-ms.2.md.
7. Per-iter journal entries (ms.1, ms.2) linked from docs/journals/INDEX.md; audit-ms entry on milestone close.