Brummel
e91e31aadb
Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close).
104 lines
4.4 KiB
Markdown
104 lines
4.4 KiB
Markdown
# iter cma.3 — Live run + DESIGN.md addendum + milestone close
|
||
|
||
**Date:** 2026-05-12
|
||
**Started from:** fe1fb6b
|
||
**Status:** DONE (milestone Cross-Model Authoring-Form Test closed)
|
||
**Mode:** Boss-direct (no implement dispatch per spec)
|
||
|
||
## Summary
|
||
|
||
Live IONOS run against Qwen3-Coder-Next executed; raw dataset
|
||
under `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||
AILX cohort reached green on 2/4 tasks at ~98k total tokens; JSON
|
||
cohort reached green on 1/4 tasks at ~207k total tokens. The only
|
||
first-attempt success was AILX on `t3_main_prints`. JSON-cohort
|
||
failures clustered in the typecheck stage (check×4); AILX-cohort
|
||
failures clustered in the parse stage (parse×3). DESIGN.md
|
||
§"Decision 6" gains a 27-line "Empirical addendum (2026-05-12)"
|
||
recording the data, the illustrative t3 case, and an explicit
|
||
single-subject single-run scope note. Multi-subject expansion
|
||
queued as a fresh roadmap P3 entry.
|
||
|
||
## Run details
|
||
|
||
- Endpoint: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
|
||
- Model: `Qwen/Qwen3-Coder-Next`
|
||
- Parameters: temperature=0, top_p=1, max-turns=5,
|
||
token-budget=500000.
|
||
- RUN_STATUS: `ok`. Total tokens consumed: 305,148 of 500,000
|
||
budget (39% headroom).
|
||
- 40 raw artefact files (8 runs × 5 per-turn files on average)
|
||
preserved under `runs/2026-05-12-df7531/per_cohort/<cohort>/<task>/`.
|
||
|
||
## Per-task notes
|
||
|
||
- `json/t1_add_three`: turn_limit. Cohort consumed 5 turns trying
|
||
to satisfy the schema constraints around `do` / `seq` / effect
|
||
positioning. Top error class `check` throughout.
|
||
- `json/t2_length`: turn_limit. ADT + polymorphism stress; the
|
||
model emitted plausible-shape JSON that the typechecker
|
||
rejected.
|
||
- `json/t3_main_prints`: green on turn 2 (the only green in the
|
||
JSON cohort). Turn 1 was ~23 lines of structured JSON with a
|
||
small structural mistake the check loop pointed out (with
|
||
location info stripped); turn 2 was correct.
|
||
- `json/t4_count_zeros`: turn_limit. Typeclass surface + Eq
|
||
constraint shape; the model never converged.
|
||
- `ailx/t1_add_three`: green on turn 3. Got the structure right
|
||
but needed two correction turns to align with the prelude
|
||
function naming.
|
||
- `ailx/t2_length`: turn_limit. Errors mixed `check` and `parse`
|
||
(model occasionally produced malformed s-expressions on
|
||
correction turns).
|
||
- `ailx/t3_main_prints`: green on turn 1, ~5 lines of tagged
|
||
s-expr. Only first-attempt success in the entire run.
|
||
- `ailx/t4_count_zeros`: turn_limit. Parse-stage failures on
|
||
every turn; the model struggled with the typeclass-constraint
|
||
shape in AILX.
|
||
|
||
## DESIGN.md edit
|
||
|
||
§"Decision 6: authoring surface" gains a new
|
||
"Empirical addendum (2026-05-12)" subsection (~27 lines after the
|
||
existing prose, before §"Decision 8"). Records: model id, run
|
||
parameters, the 6-row metric table, an illustrative paragraph
|
||
naming the t3 contrast, and the explicit scope note that the
|
||
data is one subject + one deterministic run and that the
|
||
universal claim of Decision 6 stays pending multi-subject
|
||
follow-up.
|
||
|
||
## Roadmap
|
||
|
||
- Removed: P2 entry "Cross-model authoring-form test" (closed).
|
||
- Added: P3 entry "Multi-subject expansion
|
||
(cross-model authoring-form follow-up)" pointing at this run as
|
||
baseline.
|
||
|
||
## Concerns
|
||
|
||
- AILX cohort's high parse-error count is partly model-side
|
||
(small grammar mistakes the typechecker would have forgiven)
|
||
and partly the strip_locations regex erasing `at byte N`
|
||
offsets, which leaves the model without precise localisation
|
||
during correction turns. Symmetric degradation is the fairness
|
||
anchor; calibration trade-off is documented in the spec.
|
||
- The JSON cohort's mini-spec at ~15k system tokens vs AILX's
|
||
~7k is an inherent form asymmetry (JSON is wordier per
|
||
construct). Token costs are partly a measurement of *the form*,
|
||
not just of how easy each is to author. This is part of what
|
||
the experiment measures, but worth naming.
|
||
- Only 3/8 (cohort, task) pairs reached green. The dataset's
|
||
signal-to-noise ratio is modest; the multi-subject expansion
|
||
needs more tasks and/or more permissive prelude coverage so
|
||
the failure clusters are less dominant.
|
||
|
||
## Known debt
|
||
|
||
- Multi-subject expansion (≥2 additional foreign LLMs, ≥4 more
|
||
tasks, repetitions for statistical robustness) — queued as the
|
||
P3 roadmap entry. Not started.
|
||
- README "Total 8 passed" inherited from the cma.2 plan still
|
||
understates the actual 13/13 test count; the prose was carried
|
||
verbatim per cma.2 plan discipline. Cosmetic; can be fixed in
|
||
a future tidy iter.
|