Brummel
e91e31aadb
Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close).
4.4 KiB
iter cma.3 — Live run + DESIGN.md addendum + milestone close
Date: 2026-05-12
Started from: fe1fb6b
Status: DONE (milestone Cross-Model Authoring-Form Test closed)
Mode: Boss-direct (no implement dispatch per spec)
Summary
Live IONOS run against Qwen3-Coder-Next executed; raw dataset
under experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/.
AILX cohort reached green on 2/4 tasks at ~98k total tokens; JSON
cohort reached green on 1/4 tasks at ~207k total tokens. The only
first-attempt success was AILX on t3_main_prints. JSON-cohort
failures clustered in the typecheck stage (check×4); AILX-cohort
failures clustered in the parse stage (parse×3). DESIGN.md
§"Decision 6" gains a 27-line "Empirical addendum (2026-05-12)"
recording the data, the illustrative t3 case, and an explicit
single-subject single-run scope note. Multi-subject expansion
queued as a fresh roadmap P3 entry.
Run details
- Endpoint:
https://openai.inference.de-txl.ionos.com/v1/chat/completions - Model:
Qwen/Qwen3-Coder-Next - Parameters: temperature=0, top_p=1, max-turns=5, token-budget=500000.
- RUN_STATUS:
ok. Total tokens consumed: 305,148 of 500,000 budget (39% headroom). - 40 raw artefact files (8 runs × 5 per-turn files on average)
preserved under
runs/2026-05-12-df7531/per_cohort/<cohort>/<task>/.
Per-task notes
json/t1_add_three: turn_limit. Cohort consumed 5 turns trying to satisfy the schema constraints arounddo/seq/ effect positioning. Top error classcheckthroughout.json/t2_length: turn_limit. ADT + polymorphism stress; the model emitted plausible-shape JSON that the typechecker rejected.json/t3_main_prints: green on turn 2 (the only green in the JSON cohort). Turn 1 was ~23 lines of structured JSON with a small structural mistake the check loop pointed out (with location info stripped); turn 2 was correct.json/t4_count_zeros: turn_limit. Typeclass surface + Eq constraint shape; the model never converged.ailx/t1_add_three: green on turn 3. Got the structure right but needed two correction turns to align with the prelude function naming.ailx/t2_length: turn_limit. Errors mixedcheckandparse(model occasionally produced malformed s-expressions on correction turns).ailx/t3_main_prints: green on turn 1, ~5 lines of tagged s-expr. Only first-attempt success in the entire run.ailx/t4_count_zeros: turn_limit. Parse-stage failures on every turn; the model struggled with the typeclass-constraint shape in AILX.
DESIGN.md edit
§"Decision 6: authoring surface" gains a new "Empirical addendum (2026-05-12)" subsection (~27 lines after the existing prose, before §"Decision 8"). Records: model id, run parameters, the 6-row metric table, an illustrative paragraph naming the t3 contrast, and the explicit scope note that the data is one subject + one deterministic run and that the universal claim of Decision 6 stays pending multi-subject follow-up.
Roadmap
- Removed: P2 entry "Cross-model authoring-form test" (closed).
- Added: P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" pointing at this run as baseline.
Concerns
- AILX cohort's high parse-error count is partly model-side
(small grammar mistakes the typechecker would have forgiven)
and partly the strip_locations regex erasing
at byte Noffsets, which leaves the model without precise localisation during correction turns. Symmetric degradation is the fairness anchor; calibration trade-off is documented in the spec. - The JSON cohort's mini-spec at ~15k system tokens vs AILX's ~7k is an inherent form asymmetry (JSON is wordier per construct). Token costs are partly a measurement of the form, not just of how easy each is to author. This is part of what the experiment measures, but worth naming.
- Only 3/8 (cohort, task) pairs reached green. The dataset's signal-to-noise ratio is modest; the multi-subject expansion needs more tasks and/or more permissive prelude coverage so the failure clusters are less dominant.
Known debt
- Multi-subject expansion (≥2 additional foreign LLMs, ≥4 more tasks, repetitions for statistical robustness) — queued as the P3 roadmap entry. Not started.
- README "Total 8 passed" inherited from the cma.2 plan still understates the actual 13/13 test count; the prose was carried verbatim per cma.2 plan discipline. Cosmetic; can be fixed in a future tidy iter.