Brummel
65a4f0aa16
Two live IONOS harness invocations with the ms.1 fix in place:
- Qwen/Qwen3-Coder-Next → runs/2026-05-12-080864/ (RUN_STATUS=ok)
- meta-llama/CodeLlama-13b-Instruct-hf
→ runs/2026-05-12-9197fd/ (RUN_STATUS=ok;
2/8 JSON cells terminated as api_failure
on turn 1 zero-token — transient IONOS
terminal error, disclosed in addendum)
Per-cell results (reached/4, prompt tokens, completion tokens):
Qwen JSON 1/4 182,378 14,972
Qwen AILX 1/4 110,575 3,474
CodeLlama JSON 0/4 116,015 2,711 [2 cells api_failure]
CodeLlama AILX 2/4 93,017 2,234
Both subjects agree on direction: AILX cohort cheaper on prompt
tokens (Qwen 61%, CodeLlama 80% of JSON-cohort spend) and on
completion tokens (Qwen 23%, CodeLlama 82%); AILX reached-green
≥ JSON reached-green for each subject. Three of four
first-attempt-green cells across the two subjects are AILX.
Failure classes split symmetric to form: JSON cohorts fail at
typecheck, AILX cohorts at parse — each form's
front-of-pipeline check.
DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)"
replaced: 2-column single-subject view → 4-column two-subject
view, framing paragraph reports direction agreement and the
api_failure / Qwen-rerun-noise anomalies, scope paragraph
updated to 'two subjects, n=1 each, deterministic; not a
verdict — a universal claim would need ≥3 subjects with
statistical robustness'.
Roadmap P3 'Multi-subject expansion' entry removed; scoped
goal met for two subjects.
Milestone token spend: prompt 501,985 + completion 23,391
= 525,376.
30 lines
1.2 KiB
JSON
30 lines
1.2 KiB
JSON
{
|
|
"iter_id": "ms.2",
|
|
"date": "2026-05-12",
|
|
"mode": "standard",
|
|
"outcome": "DONE",
|
|
"tasks_total": 6,
|
|
"tasks_completed": 6,
|
|
"reloops_per_task": { "1": 0, "2": 0, "3": 0, "4": 0, "5": 0, "6": 0 },
|
|
"review_loops_spec": 0,
|
|
"review_loops_quality": 0,
|
|
"blocked_reason": null,
|
|
"anomalies": [
|
|
"CodeLlama JSON-cohort t2_length and t4_count_zeros terminated as api_failure on turn 1 with zero tokens (transient IONOS-side terminal error); not re-attempted per carrier no-exploratory-re-runs constraint.",
|
|
"Qwen re-run shifted one cell vs cma.3 baseline (AILX 2/4 -> 1/4, JSON 1/4 -> 1/4) at identical temperature=0; attributed to IONOS-side state-of-day noise on a single deterministic call, not the ms.1 feedback-formatting fix."
|
|
],
|
|
"milestone_token_spend": {
|
|
"qwen_prompt": 292953,
|
|
"qwen_completion": 18446,
|
|
"codellama_prompt": 209032,
|
|
"codellama_completion": 4945,
|
|
"total_prompt": 501985,
|
|
"total_completion": 23391,
|
|
"grand_total": 525376
|
|
},
|
|
"run_dirs": {
|
|
"qwen": "experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/",
|
|
"codellama": "experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/"
|
|
}
|
|
}
|