diff --git a/docs/specs/2026-05-12-multi-subject-codellama.md b/docs/specs/2026-05-12-multi-subject-codellama.md new file mode 100644 index 0000000..21fc93a --- /dev/null +++ b/docs/specs/2026-05-12-multi-subject-codellama.md @@ -0,0 +1,163 @@ +## Multi-subject Authoring-Form Test — CodeLlama Replication — Design Spec + +**Date:** 2026-05-12 +**Status:** Draft — awaiting grounding-check +**Authors:** Brummel (orchestrator) + Claude + +## Goal + +Extend the single-subject cma baseline to a second foreign LLM +(CodeLlama via IONOS), n=1 on the same four tasks, to test whether +the per-cohort token-economy and pass-rate patterns observed with +Qwen3-Coder-Next reproduce across another model class — or are +Qwen-specific. While planning this milestone a JSON-cohort feedback +asymmetry surfaced that was previously mis-attributed to the +strip_locations regex: harness-side JSON-parse failures get their +anyhow `Caused by` chain dropped on the way into the model prompt, +leaving the JSON cohort with the uninformative top-line "parsing +JSON in /tmp/..." while the AILX cohort sees the full `ail parse` +diagnostic. Fix that bug first; then run both subjects with the +fixed harness so the dataset is internally consistent. + +## Architecture + +Two iterations: + +- **ms.1 — pipeline error-formatting fix.** Change + `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118` + from `format!("check: {e}")` to `format!("check: {e:#}")` so + anyhow's alternate Display inlines the full cause chain. RED-first: + a unit test that constructs the same anyhow chain shape the live + failure produces (a `serde_json::Error` wrapped with the + `parsing JSON in ` context) and asserts the formatted + output contains both layers. The change is two characters; the + test is the protection. The strip_locations regex already + collapses `\nCaused by:` chains, so the alternate-Display form + (which joins with `: ` inline, no newline) survives stripping + cleanly — no regex change needed. + +- **ms.2 — Multi-subject run + integration.** Two `harness` + invocations, both with the fixed pipeline: + (a) retroactive Qwen3-Coder-Next re-run, so the original four-task + dataset matches the fixed JSON-feedback behaviour and is + apples-to-apples with CodeLlama; + (b) first-ever CodeLlama-class run. Live IONOS catalogue + (`/v1/models` GET, pre-flighted) names the model + `meta-llama/CodeLlama-13b-Instruct-hf` — the Instruct-tuned + HuggingFace mirror of CodeLlama-13b. Both into fresh + `runs/-/` directories. Comparative summary written + into `docs/DESIGN.md` §"Decision 6 / Empirical addendum + (2026-05-12)" as a follow-up paragraph plus an extended table + spanning the four (model × cohort) cells. Roadmap P3 entry closed; + one-line mirror in the iter journal. + +Multi-model orchestration is out of scope: the harness remains +single-model-per-invocation. Two cargo runs is acceptable for two +subjects. + +## Components + +ms.1: +- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs:118` + — change `{e}` to `{e:#}` (two-character fix). +- `experiments/2026-05-12-cross-model-authoring/harness/src/pipeline.rs` + — add `#[cfg(test)] mod tests` (or extend it if present) with a + RED→GREEN test for the anyhow-chain formatter. + +ms.2: +- `experiments/2026-05-12-cross-model-authoring/runs/-/` + — populated by Qwen re-run. +- `experiments/2026-05-12-cross-model-authoring/runs/-/` + — populated by CodeLlama run. +- `docs/DESIGN.md` §"Decision 6 / Empirical addendum (2026-05-12)" + — extended with one follow-up paragraph and one extended per-cell + table. +- `docs/roadmap.md` — P3 "Multi-subject expansion" entry removed. +- `docs/journals/2026-05-12-iter-ms.1.md`, + `docs/journals/2026-05-12-iter-ms.2.md`, + `docs/journals/2026-05-12-audit-ms.md`, + `docs/journals/INDEX.md` entries. + +## Data flow + +ms.1: +1. Write a `#[test]` constructing an anyhow chain that mirrors the + live failure mode: a `serde_json::Error` wrapped with + `.context(format!("parsing JSON in {path}"))`. Assert + `format!("check: {e:#}")` contains both `parsing JSON` and the + underlying serde token from the chain. +2. Run the test → RED (current code uses `{e}`, drops chain). +3. Change `{e}` → `{e:#}` on line 118. +4. Run the test → GREEN. +5. Run the full existing harness test sweep — all 13 must still pass. + +ms.2: +1. Run the harness with `--model meta-llama/CodeLlama-13b-Instruct-hf` + (the canonical IONOS catalogue entry, confirmed via pre-flight + `/v1/models` GET during planning). If IONOS rejects the call on + the first completion attempt despite the pre-flight, abort and + bounce to the user with the captured error text. +2. Run harness against Qwen3-Coder-Next with the fixed pipeline → + 8 cells (4 tasks × 2 cohorts), RUN_STATUS=ok. +3. Run harness against the chosen CodeLlama model → 8 cells, + RUN_STATUS=ok. +4. Compute per-model means: prompt_tokens (Qwen-counted via the + API's `usage.prompt_tokens` field — note CodeLlama's tokeniser + differs, so cross-model token counts compare counter-output not + identical content), completion_tokens, pass-rate, turn-count + distribution. +5. Write extended addendum: one paragraph framing the second data + point, plus a per-cell table replacing the current two-column + shape with four columns (Qwen-JSON, Qwen-AILX, CodeLlama-JSON, + CodeLlama-AILX). +6. Remove the roadmap P3 entry; mirror the line in the iter journal. + +## Error handling + +- ms.1 test: a pure unit test, no external dependency, no I/O. +- ms.2 IONOS API: existing harness retry policy (5xx exp backoff, + Retry-After honouring on 429) applies unchanged. +- CodeLlama model ID resolution: pre-flighted via `/v1/models` GET + during planning. If IONOS unexpectedly rejects + `meta-llama/CodeLlama-13b-Instruct-hf` on the first completion + call despite the pre-flight, the orchestrator aborts the run and + bounces back to the user with the captured error text. This is + the only point in the milestone with a real user-bounce trigger; + the rest is mechanical. +- Token budget: 2× the cma per-run cap (the harness's existing + `--max-tokens` default suffices; no override needed). The cma + baseline used ~305k actual; CodeLlama may produce more retries + and burn more, but the cap provides headroom. + +## Testing strategy + +ms.1: one new unit test (RED→GREEN) pinning the anyhow-chain +preservation. The existing 13 harness tests must still pass. + +ms.2: rely entirely on the existing harness test suite. The +milestone exercises the harness in live mode, not extends it. The +two new runs verify completion by RUN_STATUS=ok in `summary.md` plus +8 fully populated cells per run. + +## Acceptance criteria + +1. ms.1: `pipeline.rs:118` reads `format!("check: {e:#}")`; a + RED→GREEN unit test pins the chain-preservation property. +2. ms.1: existing 13 harness tests still pass. +3. ms.2: `meta-llama/CodeLlama-13b-Instruct-hf` (or, on + user-redirected fallback after an unexpected model-rejection + bounce, whatever model the user names) runs to completion + against the IONOS endpoint (8 cells, RUN_STATUS=ok). The chosen + model ID is recorded in the iter journal. +4. ms.2: a Qwen3-Coder-Next re-run with the fixed pipeline + completes (8 cells, RUN_STATUS=ok). Distinct + `runs/-/` from the cma.3 baseline; both new runs + are checked in. +5. ms.2: `docs/DESIGN.md` §"Decision 6 / Empirical addendum + (2026-05-12)" extended with one follow-up paragraph plus a + four-cell-shaped table covering prompt_tokens, completion_tokens, + pass-rate, and turn-count distribution. +6. ms.2: Roadmap P3 multi-subject entry removed; one-line mirror in + `2026-05-12-iter-ms.2.md`. +7. Per-iter journal entries (ms.1, ms.2) linked from + `docs/journals/INDEX.md`; audit-ms entry on milestone close.