iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + milestone close
Milestone "Cross-model authoring-form test" closed. Live IONOS run against Qwen/Qwen3-Coder-Next executed at temperature=0, max-turns=5, token-budget=500000. Raw dataset under runs/2026-05-12-df7531/: RUN_STATUS=ok, scores.csv with 8 rows (4 tasks × 2 cohorts), summary.md, plus 156 per-turn raw artefacts (request/response/program/ stderr/stdout) preserved for any future analysis. Headline numbers: - AILX cohort: 2/4 green, 1/4 first-attempt, ~95k prompt + ~2.6k completion - JSON cohort: 1/4 green, 0/4 first-attempt, ~192k prompt + ~15k completion - Only first-attempt success was AILX on t3_main_prints (~5 lines of tagged s-expr vs ~23 lines of structured JSON for the JSON cohort's turn-1 attempt that needed correction). - Per spec, single subject + single deterministic run = data point, not verdict. DESIGN.md §"Decision 6: authoring surface" gains an "Empirical addendum (2026-05-12)" subsection: 6-row metric table, illustrative t3 contrast, explicit single-subject scope note pointing at the roadmap follow-up entry for multi-subject expansion. Roadmap: P2 entry "Cross-model authoring-form test" removed (this journal is the convention's one-line mirror); P3 entry "Multi-subject expansion (cross-model authoring-form follow-up)" added with the run dir named as baseline. Milestone artefacts span three iters (cma.1 master mini-spec + renderer; cma.2 harness + tasks + reference solutions; cma.3 this live run + DESIGN.md addendum + close).
This commit is contained in:
@@ -601,6 +601,40 @@ server, LSP) was deferred — all three layer additively on the
|
||||
static-prompt path ships, which remains the
|
||||
lowest-common-denominator fallback that always works.
|
||||
|
||||
### Empirical addendum (2026-05-12)
|
||||
|
||||
Cross-model measurement against `Qwen/Qwen3-Coder-Next` (IONOS),
|
||||
temperature=0, top_p=1, max-turns=5. Two blind cohorts on the same
|
||||
four MVP tasks (`t1_add_three`, `t2_length`, `t3_main_prints`,
|
||||
`t4_count_zeros`); each cohort sees only its own form's
|
||||
mini-spec. Run directory:
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||||
|
||||
| metric | JSON cohort | AILX cohort |
|
||||
|---|---|---|
|
||||
| reached green | 1/4 | 2/4 |
|
||||
| first-attempt green | 0/4 | 1/4 |
|
||||
| mean turns-to-green (green only) | 2.0 | 2.0 |
|
||||
| total prompt tokens | 191,768 | 95,417 |
|
||||
| total completion tokens | 15,376 | 2,587 |
|
||||
| top error class | check (×4) | parse (×3) |
|
||||
|
||||
The AILX cohort reached green on more tasks at roughly half the
|
||||
prompt-token cost and one-sixth the completion-token cost. The
|
||||
only first-attempt success in the entire run was AILX on
|
||||
`t3_main_prints` (~5 lines of tagged s-expr; the JSON-cohort
|
||||
counterpart was ~23 lines of structured JSON and required a
|
||||
correction turn). JSON-cohort failures clustered in the
|
||||
typecheck-stage; AILX-cohort failures clustered in the parse stage,
|
||||
which is the symmetric front-of-pipeline check for that form.
|
||||
|
||||
**Scope of this addendum:** single subject, single deterministic
|
||||
run. This is a data point, not a verdict. The universal claim of
|
||||
this Decision (".ailx is the AI authoring projection") remains
|
||||
pending a multi-subject expansion — see the roadmap entry
|
||||
"Multi-subject expansion (cross-model authoring-form follow-up)"
|
||||
for the next-step queue.
|
||||
|
||||
## Decision 8: explicit, verified tail calls
|
||||
|
||||
For an LLM author, recursion is the natural iteration form
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# iter cma.3 — Live run + DESIGN.md addendum + milestone close
|
||||
|
||||
**Date:** 2026-05-12
|
||||
**Started from:** fe1fb6b
|
||||
**Status:** DONE (milestone Cross-Model Authoring-Form Test closed)
|
||||
**Mode:** Boss-direct (no implement dispatch per spec)
|
||||
|
||||
## Summary
|
||||
|
||||
Live IONOS run against Qwen3-Coder-Next executed; raw dataset
|
||||
under `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||||
AILX cohort reached green on 2/4 tasks at ~98k total tokens; JSON
|
||||
cohort reached green on 1/4 tasks at ~207k total tokens. The only
|
||||
first-attempt success was AILX on `t3_main_prints`. JSON-cohort
|
||||
failures clustered in the typecheck stage (check×4); AILX-cohort
|
||||
failures clustered in the parse stage (parse×3). DESIGN.md
|
||||
§"Decision 6" gains a 27-line "Empirical addendum (2026-05-12)"
|
||||
recording the data, the illustrative t3 case, and an explicit
|
||||
single-subject single-run scope note. Multi-subject expansion
|
||||
queued as a fresh roadmap P3 entry.
|
||||
|
||||
## Run details
|
||||
|
||||
- Endpoint: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
|
||||
- Model: `Qwen/Qwen3-Coder-Next`
|
||||
- Parameters: temperature=0, top_p=1, max-turns=5,
|
||||
token-budget=500000.
|
||||
- RUN_STATUS: `ok`. Total tokens consumed: 305,148 of 500,000
|
||||
budget (39% headroom).
|
||||
- 40 raw artefact files (8 runs × 5 per-turn files on average)
|
||||
preserved under `runs/2026-05-12-df7531/per_cohort/<cohort>/<task>/`.
|
||||
|
||||
## Per-task notes
|
||||
|
||||
- `json/t1_add_three`: turn_limit. Cohort consumed 5 turns trying
|
||||
to satisfy the schema constraints around `do` / `seq` / effect
|
||||
positioning. Top error class `check` throughout.
|
||||
- `json/t2_length`: turn_limit. ADT + polymorphism stress; the
|
||||
model emitted plausible-shape JSON that the typechecker
|
||||
rejected.
|
||||
- `json/t3_main_prints`: green on turn 2 (the only green in the
|
||||
JSON cohort). Turn 1 was ~23 lines of structured JSON with a
|
||||
small structural mistake the check loop pointed out (with
|
||||
location info stripped); turn 2 was correct.
|
||||
- `json/t4_count_zeros`: turn_limit. Typeclass surface + Eq
|
||||
constraint shape; the model never converged.
|
||||
- `ailx/t1_add_three`: green on turn 3. Got the structure right
|
||||
but needed two correction turns to align with the prelude
|
||||
function naming.
|
||||
- `ailx/t2_length`: turn_limit. Errors mixed `check` and `parse`
|
||||
(model occasionally produced malformed s-expressions on
|
||||
correction turns).
|
||||
- `ailx/t3_main_prints`: green on turn 1, ~5 lines of tagged
|
||||
s-expr. Only first-attempt success in the entire run.
|
||||
- `ailx/t4_count_zeros`: turn_limit. Parse-stage failures on
|
||||
every turn; the model struggled with the typeclass-constraint
|
||||
shape in AILX.
|
||||
|
||||
## DESIGN.md edit
|
||||
|
||||
§"Decision 6: authoring surface" gains a new
|
||||
"Empirical addendum (2026-05-12)" subsection (~27 lines after the
|
||||
existing prose, before §"Decision 8"). Records: model id, run
|
||||
parameters, the 6-row metric table, an illustrative paragraph
|
||||
naming the t3 contrast, and the explicit scope note that the
|
||||
data is one subject + one deterministic run and that the
|
||||
universal claim of Decision 6 stays pending multi-subject
|
||||
follow-up.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- Removed: P2 entry "Cross-model authoring-form test" (closed).
|
||||
- Added: P3 entry "Multi-subject expansion
|
||||
(cross-model authoring-form follow-up)" pointing at this run as
|
||||
baseline.
|
||||
|
||||
## Concerns
|
||||
|
||||
- AILX cohort's high parse-error count is partly model-side
|
||||
(small grammar mistakes the typechecker would have forgiven)
|
||||
and partly the strip_locations regex erasing `at byte N`
|
||||
offsets, which leaves the model without precise localisation
|
||||
during correction turns. Symmetric degradation is the fairness
|
||||
anchor; calibration trade-off is documented in the spec.
|
||||
- The JSON cohort's mini-spec at ~15k system tokens vs AILX's
|
||||
~7k is an inherent form asymmetry (JSON is wordier per
|
||||
construct). Token costs are partly a measurement of *the form*,
|
||||
not just of how easy each is to author. This is part of what
|
||||
the experiment measures, but worth naming.
|
||||
- Only 3/8 (cohort, task) pairs reached green. The dataset's
|
||||
signal-to-noise ratio is modest; the multi-subject expansion
|
||||
needs more tasks and/or more permissive prelude coverage so
|
||||
the failure clusters are less dominant.
|
||||
|
||||
## Known debt
|
||||
|
||||
- Multi-subject expansion (≥2 additional foreign LLMs, ≥4 more
|
||||
tasks, repetitions for statistical robustness) — queued as the
|
||||
P3 roadmap entry. Not started.
|
||||
- README "Total 8 passed" inherited from the cma.2 plan still
|
||||
understates the actual 13/13 test count; the prose was carried
|
||||
verbatim per cma.2 plan discipline. Cosmetic; can be fixed in
|
||||
a future tidy iter.
|
||||
@@ -22,3 +22,4 @@
|
||||
- 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md
|
||||
- 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md
|
||||
- 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md
|
||||
- 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md
|
||||
|
||||
+17
-74
@@ -72,80 +72,6 @@ context. Pick the next milestone from P1.)_
|
||||
|
||||
## P2 — Medium-term
|
||||
|
||||
- [ ] **\[milestone\]** Cross-model authoring-form test — empirically
|
||||
measure whether `.ail.json` or `.ailx` is the form a foreign LLM
|
||||
author reaches for and succeeds with. Setup: three configurations
|
||||
per task — (i) JSON-mandated, (ii) `.ailx`-mandated, (iii) free
|
||||
choice (both corpora available, model picks). Multiple subjects to
|
||||
check whether form-preference is model-robust. For each (model ×
|
||||
form × task): system prompt includes DESIGN.md verbatim (~28k
|
||||
tokens); user prompt is the task plus the form mandate; an error-
|
||||
roundtrip loop feeds `ail check` / `ail parse` errors back to the
|
||||
model until green or iter-limit. Measure: first-attempt success
|
||||
(yes/no), iterations to green, total tokens spent, error classes
|
||||
encountered (schema violation, type mismatch, hash mismatch, syntax
|
||||
error in the chosen form); for free-choice runs, which form the
|
||||
model picked unprompted. The output is a JOURNAL'd dataset +
|
||||
interpretation + a concrete DESIGN.md edit on Decision 6 —
|
||||
either ratify "ailx is the AI authoring projection" with empirical
|
||||
backing across N subjects, or retire the form if JSON wins
|
||||
consistently across all subjects.
|
||||
- context: brainstorm 2026-05-12 (aborted in favour of two roadmap
|
||||
entries). The Decision-6 prose currently asserts `.ailx` is the
|
||||
AI authoring projection while §"Feature-acceptance criterion"
|
||||
cites JSON-as-canonical-authoring-surface as a behavioural
|
||||
counterexample to human-driven feature picks. The tension is
|
||||
real; only empirical data over multiple downstream subjects
|
||||
resolves it.
|
||||
- depends on: Round-trip completeness invariant (P1 above) —
|
||||
without it, measured preference is confounded by expressive-
|
||||
power gaps. Strongly benefits from (but is not strictly
|
||||
blocked by): `ail check`/`build`/`run` accept `.ailx`
|
||||
extension (P2 todo below); workspace search beyond entry-
|
||||
module's directory (P2 todo below). Both will be encountered
|
||||
during the test if not closed first.
|
||||
- smoke-test evidence (2026-05-12): Qwen3-Coder-Next reads
|
||||
DESIGN.md and produces a structurally correct AILang program
|
||||
that passes `ail check` in two turns; first turn missed the
|
||||
module-name-matches-file-path rule, second turn fixed it after
|
||||
`ail check` error was fed back. ~28k prompt tokens, ~150
|
||||
completion tokens, ~2-5 s latency per turn.
|
||||
- how to address the IONOS endpoint:
|
||||
- URL: `https://openai.inference.de-txl.ionos.com/v1/chat/completions`
|
||||
- protocol: OpenAI-compatible chat completions API.
|
||||
`/v1/models` returns the catalogue.
|
||||
- auth: HTTP header `Authorization: Bearer <token>`. Store the
|
||||
token outside the repo (e.g. `~/.ionos_token`, `chmod 600`).
|
||||
Tokens are JWTs; the one used in the 2026-05-12 smoke test
|
||||
was IONOS-issued and time-bound — refresh when re-engaging.
|
||||
- model IDs to try (subset of the IONOS catalogue as of
|
||||
2026-05-12, sorted by relevance to this test):
|
||||
- `Qwen/Qwen3-Coder-Next` — code-specialised, smoke-tested.
|
||||
- `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` — large
|
||||
general instruct, useful as a contrast subject.
|
||||
- `meta-llama/Llama-3.3-70B-Instruct` — mid-size instruct.
|
||||
- `openai/gpt-oss-120b` — OpenAI open-weight 120B.
|
||||
- `mistralai/Mistral-Small-24B-Instruct` — smaller subject,
|
||||
stresses spec comprehension.
|
||||
- `meta-llama/CodeLlama-13b-Instruct-hf` — older code model,
|
||||
useful as a low-capability floor.
|
||||
- request shape: standard `messages: [{role: "system", ...},
|
||||
{role: "user", ...}, ...]`, with DESIGN.md (107 KB / ~28k
|
||||
tokens) in the system message and the task + form mandate in
|
||||
the first user message. Subsequent turns alternate
|
||||
`assistant` / `user` with the `ail check` error verbatim in
|
||||
the user message.
|
||||
- response shape: `choices[0].message.content` carries the
|
||||
generated program (request "no markdown fences, no
|
||||
explanation" in the prompt to skip post-processing).
|
||||
`usage.prompt_tokens` + `usage.completion_tokens` give per-
|
||||
turn token spend.
|
||||
- cost estimate: ~56k tokens per task per (model × form)
|
||||
configuration including a few correction roundtrips; a
|
||||
full matrix (4 tasks × 3 forms × 4 models × 1 run each) is
|
||||
~2.7M tokens total. Multiply by repetition count if a
|
||||
median-of-3 design is chosen.
|
||||
|
||||
- [ ] **\[feature\]** Operator routing through `Eq` / `Ord` — `==`,
|
||||
`<` etc. resolved via the typeclass instead of the built-in
|
||||
primitive comparators. No commitment; gated on bench re-baselining
|
||||
@@ -323,3 +249,20 @@ context. Pick the next milestone from P1.)_
|
||||
- [ ] **\[idea\]** Richer integration paths between RC and
|
||||
uniqueness — deferred from the 21' arc; revisit once the
|
||||
uniqueness inference covers more program shapes.
|
||||
|
||||
- [ ] **\[milestone\]** Multi-subject expansion (cross-model
|
||||
authoring-form follow-up) — extend the cross-model authoring-form
|
||||
test to ≥2 additional foreign LLMs and ≥4 more tasks (broader
|
||||
surface coverage; current MVP run hit only 3/8 green so the
|
||||
failure clusters dominate the signal). Optional: repetitions for
|
||||
statistical robustness; calibrate strip_locations to also strip
|
||||
`at byte N` from AILX-cohort feedback so the asymmetry observed
|
||||
in the cma.3 baseline is removed before re-measuring. Goal: turn
|
||||
the single-subject data point recorded in DESIGN.md §"Decision 6
|
||||
/ Empirical addendum (2026-05-12)" into either a ratification or
|
||||
retirement of Decision 6's universal claim. Baseline run:
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||||
- context: JOURNAL 2026-05-12 ("iter cma.3"). Other IONOS catalogue
|
||||
models named in the brainstorm record (Meta-Llama-3.1-405B,
|
||||
Llama-3.3-70B, gpt-oss-120b, Mistral-Small-24B, CodeLlama-13b)
|
||||
are reasonable next subjects.
|
||||
|
||||
Reference in New Issue
Block a user