Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + milestone close

Browse Source 
Milestone "Cross-model authoring-form test" closed. Live IONOS run
against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5,
token-budget=500000. Raw dataset under runs/2026-05-12-df7531/:
RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts),
summary.md, plus 156 per-turn raw artefacts (request/response/program/
stderr/stdout) preserved for any future analysis.

Headline numbers:
- AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion
- JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion
- Only first-attempt success was AILX on t3_main_prints (~5 lines of
  tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's
  turn-1 attempt that needed correction).
- Per spec, single subject + single deterministic run = data point,
  not verdict.

DESIGN.md §"Decision 6: authoring surface" gains an "Empirical
addendum (2026-05-12)" subsection: 6-row metric table, illustrative
t3 contrast, explicit single-subject scope note pointing at the
roadmap follow-up entry for multi-subject expansion.

Roadmap: P2 entry "Cross-model authoring-form test" removed (this
journal is the convention's one-line mirror); P3 entry
"Multi-subject expansion (cross-model authoring-form follow-up)"
added with the run dir named as baseline.

Milestone artefacts span three iters (cma.1 master mini-spec +
renderer; cma.2 harness + tasks + reference solutions; cma.3 this
live run + DESIGN.md addendum + close).
This commit is contained in:
Brummel
2026-05-12 12:16:32 +02:00
parent fe1fb6b4f0
commit e91e31aadb
162 changed files with 3455 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DESIGN.md
+34
View File
@@ -601,6 +601,40 @@ server, LSP) was deferred — all three layer additively on the
static-prompt path ships, which remains the
lowest-common-denominator fallback that always works.
### Empirical addendum (2026-05-12)
Cross-model measurement against `Qwen/Qwen3-Coder-Next` (IONOS),
temperature=0, top_p=1, max-turns=5. Two blind cohorts on the same
four MVP tasks (`t1_add_three`, `t2_length`, `t3_main_prints`,
`t4_count_zeros`); each cohort sees only its own form's
mini-spec. Run directory:
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
| metric | JSON cohort | AILX cohort |
|---|---|---|
| reached green | 1/4 | 2/4 |
| first-attempt green | 0/4 | 1/4 |
| mean turns-to-green (green only) | 2.0 | 2.0 |
| total prompt tokens | 191,768 | 95,417 |
| total completion tokens | 15,376 | 2,587 |
| top error class | check (×4) | parse (×3) |
The AILX cohort reached green on more tasks at roughly half the
prompt-token cost and one-sixth the completion-token cost. The
only first-attempt success in the entire run was AILX on
`t3_main_prints` (~5 lines of tagged s-expr; the JSON-cohort
counterpart was ~23 lines of structured JSON and required a
correction turn). JSON-cohort failures clustered in the
typecheck-stage; AILX-cohort failures clustered in the parse stage,
which is the symmetric front-of-pipeline check for that form.
**Scope of this addendum:** single subject, single deterministic
run. This is a data point, not a verdict. The universal claim of
this Decision (".ailx is the AI authoring projection") remains
pending a multi-subject expansion — see the roadmap entry
"Multi-subject expansion (cross-model authoring-form follow-up)"
for the next-step queue.
## Decision 8: explicit, verified tail calls
For an LLM author, recursion is the natural iteration form