iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + milestone close
Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close).
This commit is contained in:
@@ -0,0 +1,103 @@
|
||||
# iter cma.3 — Live run + DESIGN.md addendum + milestone close
|
||||
|
||||
**Date:** 2026-05-12
|
||||
**Started from:** fe1fb6b
|
||||
**Status:** DONE (milestone Cross-Model Authoring-Form Test closed)
|
||||
**Mode:** Boss-direct (no implement dispatch per spec)
|
||||
|
||||
## Summary
|
||||
|
||||
Live IONOS run against Qwen3-Coder-Next executed; raw dataset
|
||||
under `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||||
AILX cohort reached green on 2/4 tasks at ~98k total tokens; JSON
|
||||
cohort reached green on 1/4 tasks at ~207k total tokens. The only
|
||||
first-attempt success was AILX on `t3_main_prints`. JSON-cohort
|
||||
failures clustered in the typecheck stage (check×4); AILX-cohort
|
||||
failures clustered in the parse stage (parse×3). DESIGN.md
|
||||
§"Decision 6" gains a 27-line "Empirical addendum (2026-05-12)"
|
||||
recording the data, the illustrative t3 case, and an explicit
|
||||
single-subject single-run scope note. Multi-subject expansion
|
||||
queued as a fresh roadmap P3 entry.
|
||||
|
||||
## Run details
|
||||
|
||||
- Endpoint: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
|
||||
- Model: `Qwen/Qwen3-Coder-Next`
|
||||
- Parameters: temperature=0, top_p=1, max-turns=5,
|
||||
token-budget=500000.
|
||||
- RUN_STATUS: `ok`. Total tokens consumed: 305,148 of 500,000
|
||||
budget (39% headroom).
|
||||
- 40 raw artefact files (8 runs × 5 per-turn files on average)
|
||||
preserved under `runs/2026-05-12-df7531/per_cohort/<cohort>/<task>/`.
|
||||
|
||||
## Per-task notes
|
||||
|
||||
- `json/t1_add_three`: turn_limit. Cohort consumed 5 turns trying
|
||||
to satisfy the schema constraints around `do` / `seq` / effect
|
||||
positioning. Top error class `check` throughout.
|
||||
- `json/t2_length`: turn_limit. ADT + polymorphism stress; the
|
||||
model emitted plausible-shape JSON that the typechecker
|
||||
rejected.
|
||||
- `json/t3_main_prints`: green on turn 2 (the only green in the
|
||||
JSON cohort). Turn 1 was ~23 lines of structured JSON with a
|
||||
small structural mistake the check loop pointed out (with
|
||||
location info stripped); turn 2 was correct.
|
||||
- `json/t4_count_zeros`: turn_limit. Typeclass surface + Eq
|
||||
constraint shape; the model never converged.
|
||||
- `ailx/t1_add_three`: green on turn 3. Got the structure right
|
||||
but needed two correction turns to align with the prelude
|
||||
function naming.
|
||||
- `ailx/t2_length`: turn_limit. Errors mixed `check` and `parse`
|
||||
(model occasionally produced malformed s-expressions on
|
||||
correction turns).
|
||||
- `ailx/t3_main_prints`: green on turn 1, ~5 lines of tagged
|
||||
s-expr. Only first-attempt success in the entire run.
|
||||
- `ailx/t4_count_zeros`: turn_limit. Parse-stage failures on
|
||||
every turn; the model struggled with the typeclass-constraint
|
||||
shape in AILX.
|
||||
|
||||
## DESIGN.md edit
|
||||
|
||||
§"Decision 6: authoring surface" gains a new
|
||||
"Empirical addendum (2026-05-12)" subsection (~27 lines after the
|
||||
existing prose, before §"Decision 8"). Records: model id, run
|
||||
parameters, the 6-row metric table, an illustrative paragraph
|
||||
naming the t3 contrast, and the explicit scope note that the
|
||||
data is one subject + one deterministic run and that the
|
||||
universal claim of Decision 6 stays pending multi-subject
|
||||
follow-up.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- Removed: P2 entry "Cross-model authoring-form test" (closed).
|
||||
- Added: P3 entry "Multi-subject expansion
|
||||
(cross-model authoring-form follow-up)" pointing at this run as
|
||||
baseline.
|
||||
|
||||
## Concerns
|
||||
|
||||
- AILX cohort's high parse-error count is partly model-side
|
||||
(small grammar mistakes the typechecker would have forgiven)
|
||||
and partly the strip_locations regex erasing `at byte N`
|
||||
offsets, which leaves the model without precise localisation
|
||||
during correction turns. Symmetric degradation is the fairness
|
||||
anchor; calibration trade-off is documented in the spec.
|
||||
- The JSON cohort's mini-spec at ~15k system tokens vs AILX's
|
||||
~7k is an inherent form asymmetry (JSON is wordier per
|
||||
construct). Token costs are partly a measurement of *the form*,
|
||||
not just of how easy each is to author. This is part of what
|
||||
the experiment measures, but worth naming.
|
||||
- Only 3/8 (cohort, task) pairs reached green. The dataset's
|
||||
signal-to-noise ratio is modest; the multi-subject expansion
|
||||
needs more tasks and/or more permissive prelude coverage so
|
||||
the failure clusters are less dominant.
|
||||
|
||||
## Known debt
|
||||
|
||||
- Multi-subject expansion (≥2 additional foreign LLMs, ≥4 more
|
||||
tasks, repetitions for statistical robustness) — queued as the
|
||||
P3 roadmap entry. Not started.
|
||||
- README "Total 8 passed" inherited from the cma.2 plan still
|
||||
understates the actual 13/13 test count; the prose was carried
|
||||
verbatim per cma.2 plan discipline. Cosmetic; can be fixed in
|
||||
a future tidy iter.
|
||||
@@ -22,3 +22,4 @@
|
||||
- 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md
|
||||
- 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md
|
||||
- 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md
|
||||
- 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md
|
||||
|
||||
Reference in New Issue
Block a user