AILang/docs/specs/0018-multi-subject-codellama.md

## Multi-subject Authoring-Form Test — CodeLlama Replication — Design Spec

**Date:** 2026-05-12
**Status:** Draft — awaiting grounding-check
**Authors:** Brummel (orchestrator) + Claude

## Goal

Extend the single-subject cma baseline to a second foreign LLM
(CodeLlama via IONOS), n=1 on the same four tasks, to test whether
the per-cohort token-economy and pass-rate patterns observed with
Qwen3-Coder-Next reproduce across another model class — or are
Qwen-specific. While planning this milestone a JSON-cohort feedback
asymmetry surfaced that was previously mis-attributed to the
strip_locations regex: harness-side JSON-parse failures get their
anyhow `Caused by` chain dropped on the way into the model prompt,
leaving the JSON cohort with the uninformative top-line "parsing
JSON in /tmp/..." while the AILX cohort sees the full `ail parse`
diagnostic. Fix that bug first; then run both subjects with the
fixed harness so the dataset is internally consistent.

## Architecture

Two iterations:

- **ms.1 — pipeline error-formatting fix.** Change
  `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118`
  from `format!("check: {e}")` to `format!("check: {e:#}")` so
  anyhow's alternate Display inlines the full cause chain. RED-first:
  a unit test that constructs the same anyhow chain shape the live
  failure produces (a `serde_json::Error` wrapped with the
  `parsing JSON in <path>` context) and asserts the formatted
  output contains both layers. The change is two characters; the
  test is the protection. The strip_locations regex already
  collapses `\nCaused by:` chains, so the alternate-Display form
  (which joins with `: ` inline, no newline) survives stripping
  cleanly — no regex change needed.

- **ms.2 — Multi-subject run + integration.** Two `harness`
  invocations, both with the fixed pipeline:
  (a) retroactive Qwen3-Coder-Next re-run, so the original four-task
  dataset matches the fixed JSON-feedback behaviour and is
  apples-to-apples with CodeLlama;
  (b) first-ever CodeLlama-class run. Live IONOS catalogue
  (`/v1/models` GET, pre-flighted) names the model
  `meta-llama/CodeLlama-13b-Instruct-hf` — the Instruct-tuned
  HuggingFace mirror of CodeLlama-13b. Both into fresh
  `runs/<date>-<hash>/` directories. Comparative summary written
  into `docs/DESIGN.md` §"Decision 6 / Empirical addendum
  (2026-05-12)" as a follow-up paragraph plus an extended table
  spanning the four (model × cohort) cells. Roadmap P3 entry closed;
  one-line mirror in the iter journal.

Multi-model orchestration is out of scope: the harness remains
single-model-per-invocation. Two cargo runs is acceptable for two
subjects.

## Components

ms.1:
- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118`
  — change `{e}` to `{e:#}` (two-character fix).
- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs`
  — add `#[cfg(test)] mod tests` (or extend it if present) with a
  RED→GREEN test for the anyhow-chain formatter.

ms.2:
- `experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<qwen-hash>/`
  — populated by Qwen re-run.
- `experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<codellama-hash>/`
  — populated by CodeLlama run.
- `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)"
  — extended with one follow-up paragraph and one extended per-cell
  table.
- `docs/roadmap.md` — P3 "Multi-subject expansion" entry removed.
- `docs/journals/2026-05-12-iter-ms.1.md`,
  `docs/journals/2026-05-12-iter-ms.2.md`,
  `docs/journals/2026-05-12-audit-ms.md`,
  `docs/journals/INDEX.md` entries.

## Data flow

ms.1:
1. Write a `#[test]` constructing an anyhow chain that mirrors the
   live failure mode: a `serde_json::Error` wrapped with
   `.context(format!("parsing JSON in {path}"))`. Assert
   `format!("check: {e:#}")` contains both `parsing JSON` and the
   underlying serde token from the chain.
2. Run the test → RED (current code uses `{e}`, drops chain).
3. Change `{e}` → `{e:#}` on line 118.
4. Run the test → GREEN.
5. Run the full existing harness test sweep — all 13 must still pass.

ms.2:
1. Run the harness with `--model meta-llama/CodeLlama-13b-Instruct-hf`
   (the canonical IONOS catalogue entry, confirmed via pre-flight
   `/v1/models` GET during planning). If IONOS rejects the call on
   the first completion attempt despite the pre-flight, abort and
   bounce to the user with the captured error text.
2. Run harness against Qwen3-Coder-Next with the fixed pipeline →
   8 cells (4 tasks × 2 cohorts), RUN_STATUS=ok.
3. Run harness against the chosen CodeLlama model → 8 cells,
   RUN_STATUS=ok.
4. Compute per-model means: prompt_tokens (Qwen-counted via the
   API's `usage.prompt_tokens` field — note CodeLlama's tokeniser
   differs, so cross-model token counts compare counter-output not
   identical content), completion_tokens, pass-rate, turn-count
   distribution.
5. Write extended addendum: one paragraph framing the second data
   point, plus a per-cell table replacing the current two-column
   shape with four columns (Qwen-JSON, Qwen-AILX, CodeLlama-JSON,
   CodeLlama-AILX).
6. Remove the roadmap P3 entry; mirror the line in the iter journal.

## Error handling

- ms.1 test: a pure unit test, no external dependency, no I/O.
- ms.2 IONOS API: existing harness retry policy (5xx exp backoff,
  Retry-After honouring on 429) applies unchanged.
- CodeLlama model ID resolution: pre-flighted via `/v1/models` GET
  during planning. If IONOS unexpectedly rejects
  `meta-llama/CodeLlama-13b-Instruct-hf` on the first completion
  call despite the pre-flight, the orchestrator aborts the run and
  bounces back to the user with the captured error text. This is
  the only point in the milestone with a real user-bounce trigger;
  the rest is mechanical.
- Token budget: 2× the cma per-run cap (the harness's existing
  `--max-tokens` default suffices; no override needed). The cma
  baseline used ~305k actual; CodeLlama may produce more retries
  and burn more, but the cap provides headroom.

## Testing strategy

ms.1: one new unit test (RED→GREEN) pinning the anyhow-chain
preservation. The existing 13 harness tests must still pass.

ms.2: rely entirely on the existing harness test suite. The
milestone exercises the harness in live mode, not extends it. The
two new runs verify completion by RUN_STATUS=ok in `summary.md` plus
8 fully populated cells per run.

## Acceptance criteria

1. ms.1: `pipeline.rs:118` reads `format!("check: {e:#}")`; a
   RED→GREEN unit test pins the chain-preservation property.
2. ms.1: existing 13 harness tests still pass.
3. ms.2: `meta-llama/CodeLlama-13b-Instruct-hf` (or, on
   user-redirected fallback after an unexpected model-rejection
   bounce, whatever model the user names) runs to completion
   against the IONOS endpoint (8 cells, RUN_STATUS=ok). The chosen
   model ID is recorded in the iter journal.
4. ms.2: a Qwen3-Coder-Next re-run with the fixed pipeline
   completes (8 cells, RUN_STATUS=ok). Distinct
   `runs/<date>-<hash>/` from the cma.3 baseline; both new runs
   are checked in.
5. ms.2: `docs/DESIGN.md` §"Decision 6 / Empirical addendum
   (2026-05-12)" extended with one follow-up paragraph plus a
   four-cell-shaped table covering prompt_tokens, completion_tokens,
   pass-rate, and turn-count distribution.
6. ms.2: Roadmap P3 multi-subject entry removed; one-line mirror in
   `2026-05-12-iter-ms.2.md`.
7. Per-iter journal entries (ms.1, ms.2) linked from
   `docs/journals/INDEX.md`; audit-ms entry on milestone close.