spec: multi-subject CodeLlama replication — pipeline anyhow-chain fix + second subject run

docs/specs/2026-05-12-multi-subject-codellama.md

@@ -0,0 +1,163 @@
+## Multi-subject Authoring-Form Test — CodeLlama Replication — Design Spec
+
+**Date:** 2026-05-12
+**Status:** Draft — awaiting grounding-check
+**Authors:** Brummel (orchestrator) + Claude
+
+## Goal
+
+Extend the single-subject cma baseline to a second foreign LLM
+(CodeLlama via IONOS), n=1 on the same four tasks, to test whether
+the per-cohort token-economy and pass-rate patterns observed with
+Qwen3-Coder-Next reproduce across another model class — or are
+Qwen-specific. While planning this milestone a JSON-cohort feedback
+asymmetry surfaced that was previously mis-attributed to the
+strip_locations regex: harness-side JSON-parse failures get their
+anyhow `Caused by` chain dropped on the way into the model prompt,
+leaving the JSON cohort with the uninformative top-line "parsing
+JSON in /tmp/..." while the AILX cohort sees the full `ail parse`
+diagnostic. Fix that bug first; then run both subjects with the
+fixed harness so the dataset is internally consistent.
+
+## Architecture
+
+Two iterations:
+
+- **ms.1 — pipeline error-formatting fix.** Change
+  `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118`
+  from `format!("check: {e}")` to `format!("check: {e:#}")` so
+  anyhow's alternate Display inlines the full cause chain. RED-first:
+  a unit test that constructs the same anyhow chain shape the live
+  failure produces (a `serde_json::Error` wrapped with the
+  `parsing JSON in <path>` context) and asserts the formatted
+  output contains both layers. The change is two characters; the
+  test is the protection. The strip_locations regex already
+  collapses `\nCaused by:` chains, so the alternate-Display form
+  (which joins with `: ` inline, no newline) survives stripping
+  cleanly — no regex change needed.
+
+- **ms.2 — Multi-subject run + integration.** Two `harness`
+  invocations, both with the fixed pipeline:
+  (a) retroactive Qwen3-Coder-Next re-run, so the original four-task
+  dataset matches the fixed JSON-feedback behaviour and is
+  apples-to-apples with CodeLlama;
+  (b) first-ever CodeLlama-class run. Live IONOS catalogue
+  (`/v1/models` GET, pre-flighted) names the model
+  `meta-llama/CodeLlama-13b-Instruct-hf` — the Instruct-tuned
+  HuggingFace mirror of CodeLlama-13b. Both into fresh
+  `runs/<date>-<hash>/` directories. Comparative summary written
+  into `docs/DESIGN.md` §"Decision 6 / Empirical addendum
+  (2026-05-12)" as a follow-up paragraph plus an extended table
+  spanning the four (model × cohort) cells. Roadmap P3 entry closed;
+  one-line mirror in the iter journal.
+
+Multi-model orchestration is out of scope: the harness remains
+single-model-per-invocation. Two cargo runs is acceptable for two
+subjects.
+
+## Components
+
+ms.1:
+- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118`
+  — change `{e}` to `{e:#}` (two-character fix).
+- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs`
+  — add `#[cfg(test)] mod tests` (or extend it if present) with a
+  RED→GREEN test for the anyhow-chain formatter.
+
+ms.2:
+- `experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<qwen-hash>/`
+  — populated by Qwen re-run.
+- `experiments/2026-05-12-cross-model-authoring/runs/<new-date>-<codellama-hash>/`
+  — populated by CodeLlama run.
+- `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)"
+  — extended with one follow-up paragraph and one extended per-cell
+  table.
+- `docs/roadmap.md` — P3 "Multi-subject expansion" entry removed.
+- `docs/journals/2026-05-12-iter-ms.1.md`,
+  `docs/journals/2026-05-12-iter-ms.2.md`,
+  `docs/journals/2026-05-12-audit-ms.md`,
+  `docs/journals/INDEX.md` entries.
+
+## Data flow
+
+ms.1:
+1. Write a `#[test]` constructing an anyhow chain that mirrors the
+   live failure mode: a `serde_json::Error` wrapped with
+   `.context(format!("parsing JSON in {path}"))`. Assert
+   `format!("check: {e:#}")` contains both `parsing JSON` and the
+   underlying serde token from the chain.
+2. Run the test → RED (current code uses `{e}`, drops chain).
+3. Change `{e}` → `{e:#}` on line 118.
+4. Run the test → GREEN.
+5. Run the full existing harness test sweep — all 13 must still pass.
+
+ms.2:
+1. Run the harness with `--model meta-llama/CodeLlama-13b-Instruct-hf`
+   (the canonical IONOS catalogue entry, confirmed via pre-flight
+   `/v1/models` GET during planning). If IONOS rejects the call on
+   the first completion attempt despite the pre-flight, abort and
+   bounce to the user with the captured error text.
+2. Run harness against Qwen3-Coder-Next with the fixed pipeline →
+   8 cells (4 tasks × 2 cohorts), RUN_STATUS=ok.
+3. Run harness against the chosen CodeLlama model → 8 cells,
+   RUN_STATUS=ok.
+4. Compute per-model means: prompt_tokens (Qwen-counted via the
+   API's `usage.prompt_tokens` field — note CodeLlama's tokeniser
+   differs, so cross-model token counts compare counter-output not
+   identical content), completion_tokens, pass-rate, turn-count
+   distribution.
+5. Write extended addendum: one paragraph framing the second data
+   point, plus a per-cell table replacing the current two-column
+   shape with four columns (Qwen-JSON, Qwen-AILX, CodeLlama-JSON,
+   CodeLlama-AILX).
+6. Remove the roadmap P3 entry; mirror the line in the iter journal.
+
+## Error handling
+
+- ms.1 test: a pure unit test, no external dependency, no I/O.
+- ms.2 IONOS API: existing harness retry policy (5xx exp backoff,
+  Retry-After honouring on 429) applies unchanged.
+- CodeLlama model ID resolution: pre-flighted via `/v1/models` GET
+  during planning. If IONOS unexpectedly rejects
+  `meta-llama/CodeLlama-13b-Instruct-hf` on the first completion
+  call despite the pre-flight, the orchestrator aborts the run and
+  bounces back to the user with the captured error text. This is
+  the only point in the milestone with a real user-bounce trigger;
+  the rest is mechanical.
+- Token budget: 2× the cma per-run cap (the harness's existing
+  `--max-tokens` default suffices; no override needed). The cma
+  baseline used ~305k actual; CodeLlama may produce more retries
+  and burn more, but the cap provides headroom.
+
+## Testing strategy
+
+ms.1: one new unit test (RED→GREEN) pinning the anyhow-chain
+preservation. The existing 13 harness tests must still pass.
+
+ms.2: rely entirely on the existing harness test suite. The
+milestone exercises the harness in live mode, not extends it. The
+two new runs verify completion by RUN_STATUS=ok in `summary.md` plus
+8 fully populated cells per run.
+
+## Acceptance criteria
+
+1. ms.1: `pipeline.rs:118` reads `format!("check: {e:#}")`; a
+   RED→GREEN unit test pins the chain-preservation property.
+2. ms.1: existing 13 harness tests still pass.
+3. ms.2: `meta-llama/CodeLlama-13b-Instruct-hf` (or, on
+   user-redirected fallback after an unexpected model-rejection
+   bounce, whatever model the user names) runs to completion
+   against the IONOS endpoint (8 cells, RUN_STATUS=ok). The chosen
+   model ID is recorded in the iter journal.
+4. ms.2: a Qwen3-Coder-Next re-run with the fixed pipeline
+   completes (8 cells, RUN_STATUS=ok). Distinct
+   `runs/<date>-<hash>/` from the cma.3 baseline; both new runs
+   are checked in.
+5. ms.2: `docs/DESIGN.md` §"Decision 6 / Empirical addendum
+   (2026-05-12)" extended with one follow-up paragraph plus a
+   four-cell-shaped table covering prompt_tokens, completion_tokens,
+   pass-rate, and turn-count distribution.
+6. ms.2: Roadmap P3 multi-subject entry removed; one-line mirror in
+   `2026-05-12-iter-ms.2.md`.
+7. Per-iter journal entries (ms.1, ms.2) linked from
+   `docs/journals/INDEX.md`; audit-ms entry on milestone close.