2 Commits

Author	SHA1	Message	Date
Brummel	65a4f0aa16	iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run + DESIGN.md §Decision-6 addendum extended to two subjects ... Two live IONOS harness invocations with the ms.1 fix in place: - Qwen/Qwen3-Coder-Next → runs/2026-05-12-080864/ (RUN_STATUS=ok) - meta-llama/CodeLlama-13b-Instruct-hf → runs/2026-05-12-9197fd/ (RUN_STATUS=ok; 2/8 JSON cells terminated as api_failure on turn 1 zero-token — transient IONOS terminal error, disclosed in addendum) Per-cell results (reached/4, prompt tokens, completion tokens): Qwen JSON 1/4 182,378 14,972 Qwen AILX 1/4 110,575 3,474 CodeLlama JSON 0/4 116,015 2,711 [2 cells api_failure] CodeLlama AILX 2/4 93,017 2,234 Both subjects agree on direction: AILX cohort cheaper on prompt tokens (Qwen 61%, CodeLlama 80% of JSON-cohort spend) and on completion tokens (Qwen 23%, CodeLlama 82%); AILX reached-green ≥ JSON reached-green for each subject. Three of four first-attempt-green cells across the two subjects are AILX. Failure classes split symmetric to form: JSON cohorts fail at typecheck, AILX cohorts at parse — each form's front-of-pipeline check. DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)" replaced: 2-column single-subject view → 4-column two-subject view, framing paragraph reports direction agreement and the api_failure / Qwen-rerun-noise anomalies, scope paragraph updated to 'two subjects, n=1 each, deterministic; not a verdict — a universal claim would need ≥3 subjects with statistical robustness'. Roadmap P3 'Multi-subject expansion' entry removed; scoped goal met for two subjects. Milestone token spend: prompt 501,985 + completion 23,391 = 525,376.	2026-05-12 13:29:15 +02:00
Brummel	e91e31aadb	iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + milestone close ... Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close).	2026-05-12 12:16:32 +02:00