# iter cma.3 — Live run + DESIGN.md addendum + milestone close

**Date:** 2026-05-12
**Started from:** fe1fb6b
**Status:** DONE (milestone Cross-Model Authoring-Form Test closed)
**Mode:** Boss-direct (no implement dispatch per spec)

## Summary

Live IONOS run against Qwen3-Coder-Next executed; raw dataset
under `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
AILX cohort reached green on 2/4 tasks at ~98k total tokens; JSON
cohort reached green on 1/4 tasks at ~207k total tokens. The only
first-attempt success was AILX on `t3_main_prints`. JSON-cohort
failures clustered in the typecheck stage (check×4); AILX-cohort
failures clustered in the parse stage (parse×3). DESIGN.md
§"Decision 6" gains a 27-line "Empirical addendum (2026-05-12)"
recording the data, the illustrative t3 case, and an explicit
single-subject single-run scope note. Multi-subject expansion
queued as a fresh roadmap P3 entry.

## Run details

- Endpoint: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
- Model: `Qwen/Qwen3-Coder-Next`
- Parameters: temperature=0, top_p=1, max-turns=5,
  token-budget=500000.
- RUN_STATUS: `ok`. Total tokens consumed: 305,148 of 500,000
  budget (39% headroom).
- 40 raw artefact files (8 runs × 5 per-turn files on average)
  preserved under `runs/2026-05-12-df7531/per_cohort/<cohort>/<task>/`.

## Per-task notes

- `json/t1_add_three`: turn_limit. Cohort consumed 5 turns trying
  to satisfy the schema constraints around `do` / `seq` / effect
  positioning. Top error class `check` throughout.
- `json/t2_length`: turn_limit. ADT + polymorphism stress; the
  model emitted plausible-shape JSON that the typechecker
  rejected.
- `json/t3_main_prints`: green on turn 2 (the only green in the
  JSON cohort). Turn 1 was ~23 lines of structured JSON with a
  small structural mistake the check loop pointed out (with
  location info stripped); turn 2 was correct.
- `json/t4_count_zeros`: turn_limit. Typeclass surface + Eq
  constraint shape; the model never converged.
- `ailx/t1_add_three`: green on turn 3. Got the structure right
  but needed two correction turns to align with the prelude
  function naming.
- `ailx/t2_length`: turn_limit. Errors mixed `check` and `parse`
  (model occasionally produced malformed s-expressions on
  correction turns).
- `ailx/t3_main_prints`: green on turn 1, ~5 lines of tagged
  s-expr. Only first-attempt success in the entire run.
- `ailx/t4_count_zeros`: turn_limit. Parse-stage failures on
  every turn; the model struggled with the typeclass-constraint
  shape in AILX.

## DESIGN.md edit

§"Decision 6: authoring surface" gains a new
"Empirical addendum (2026-05-12)" subsection (~27 lines after the
existing prose, before §"Decision 8"). Records: model id, run
parameters, the 6-row metric table, an illustrative paragraph
naming the t3 contrast, and the explicit scope note that the
data is one subject + one deterministic run and that the
universal claim of Decision 6 stays pending multi-subject
follow-up.

## Roadmap

- Removed: P2 entry "Cross-model authoring-form test" (closed).
- Added: P3 entry "Multi-subject expansion
  (cross-model authoring-form follow-up)" pointing at this run as
  baseline.

## Concerns

- AILX cohort's high parse-error count is partly model-side
  (small grammar mistakes the typechecker would have forgiven)
  and partly the strip_locations regex erasing `at byte N`
  offsets, which leaves the model without precise localisation
  during correction turns. Symmetric degradation is the fairness
  anchor; calibration trade-off is documented in the spec.
- The JSON cohort's mini-spec at ~15k system tokens vs AILX's
  ~7k is an inherent form asymmetry (JSON is wordier per
  construct). Token costs are partly a measurement of *the form*,
  not just of how easy each is to author. This is part of what
  the experiment measures, but worth naming.
- Only 3/8 (cohort, task) pairs reached green. The dataset's
  signal-to-noise ratio is modest; the multi-subject expansion
  needs more tasks and/or more permissive prelude coverage so
  the failure clusters are less dominant.

## Known debt

- Multi-subject expansion (≥2 additional foreign LLMs, ≥4 more
  tasks, repetitions for statistical robustness) — queued as the
  P3 roadmap entry. Not started.
- README "Total 8 passed" inherited from the cma.2 plan still
  understates the actual 13/13 test count; the prose was carried
  verbatim per cma.2 plan discipline. Cosmetic; can be fixed in
  a future tidy iter.
