AILang/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md

SMA authoring probe — Qwen3-Coder-Next via IONOS

Date: 2026-06-02 Model: Qwen/Qwen3-Coder-Next (IONOS, temperature 0.2) Task: write the Series SMA worked example (examples/series_sma.ail) in AILang Form-A, given only the Form-A mini-spec (rendered/ail.md) + the Series library API — not the reference solution. Multi-turn (≤6), each attempt run through ail check → ail run, the diagnostic fed back on failure. Driver: sma_probe.py. Raw per-turn logs: sma-probe-run1.md, sma-probe-run2.md, sma-probe-run3.md. Three runs, escalating the amount of reference material handed to the model.

Expected stdout: 3.0 / 5.33333 / 5.66667 / 5.33333 (window 3 over [1.0, 5.0, 3.0, 8.0, 6.0, 2.0]).

Outcome: did NOT reach green in any of the three runs. Runs 1→2 were a clean progression (each sharpening moved the model one hurdle forward); run 3 inverted it — more material made the model degenerate. See the Verdict at the bottom.

Run 1 — API doc said (new Series (con Float) N)

• 6 turns, byte-identical program every turn, all failing at ail check.
• Blocker: Qwen wrote (app new (con Float) 3) — it treated new as an ordinary function and passed the type (con Float) as a term argument → [surface-parse-error] unknown term head 'con'. new is a built-in term head, not an app callee.
• Structure: instead of recursion over a list (as the reference does), Qwen hand-unrolled the six pushes into a deeply nested (seq (seq (seq …))) tower.
• Tokens: prompt 72 783, completion 12 000.

Run 2 — new clarified as a term head, with a (let s (new Series …) …) snippet

• The new parse error vanished — the sharpening worked; Qwen now writes (new Series (con Float) 3) correctly.
• New blocker: [arity-mismatch] Series.push: expected 2 args, got 4 (5/6 turns; turn 1 a stray expected ) parse error).
• Notably, the ownership threading was idiomatically correct: nested (let s (app Series.push s v) …), re-binding s to each push's result — the right shape for Series.push's own-in/own-out signature. The failure was S-expression bracketing: a missing ) let the push application swallow following terms, so it saw 4 args. Minor un-idiom: (app int_to_float 1) instead of the float literal 1.0.
• Again non-convergent: the bracketing bug persisted across all 6 turns despite the explicit "expected 2 args, got 4" diagnostic each time.
• Tokens: prompt 47 041, completion 1 852.

Cross-cutting findings (n=1 model, single session)

1. Per-API-quirk hand-holding. Qwen clears a hurdle only when the exact surface rule is spelled out (the new-is-a-term-head fix). With the stock API doc it does not infer it.
2. No convergence from compiler feedback. In neither run did the ail check diagnostic move the model off its failure — run 1 reproduced the same bytes 6×, run 2 kept the same bracketing bug 6×. The retry-with-diagnostic loop bought nothing here.
3. Form-A's nesting is itself a hurdle. Both failure modes are S-expression-shape errors (term-head confusion; unbalanced parens), not semantic ones — and in run 2 the semantics (push threading) were right. This is a concrete data point on the Form-A authoring surface for a foreign model: it gets the program logic but stumbles on the bracket discipline.

Run 3 — fully equipped: complete construct grammar + mini-spec + real Series library source

The hypothesis was that the bracket failures of runs 1–2 would clear if Qwen had the formal grammar. The opposite happened. With the larger context (~80k prompt tokens: a distilled Form-A construct grammar with explicit arity rules + rendered/ail.md + the verbatim series/source.ail), Qwen degenerated: every one of the 6 turns produced an identical pathological (seq (seq (seq …))) tower — 321 open parens vs 10 close, identical bytes each turn — with whitespace indentation exploding to hundreds of spaces per line. It never closes the tower; it just runs out of tokens (unexpected end of input). Raw log: sma-probe-run3.md.

A confound check (re-run with max_tokens doubled to 4000) did not yield a complete program — it produced HTTP 413 (request too large): with more budget Qwen emits an even larger degenerate output, so the accumulated message history blows past the endpoint's size limit. That confirms degeneration rather than truncation.

So the extra equipment made the model worse, not better. The best run was the middle one (run 2): just enough guidance to clear the new quirk, without the context overload that tipped run 3 into a repetition loop.

Verdict (n=1 model, single session)

• Qwen3-Coder-Next did not write the SMA in any configuration. Its only semantically-correct attempt (run 2) still failed on S-expression bracketing.
• More reference material did not help — it hurt. The fully-equipped run degenerated into a non-terminating seq tower; the lean run got the algorithm right. This is the sharpest finding: for this model on Form-A, context volume past a point is actively harmful.
• No convergence from compiler feedback in any run. Every run repeated its failure byte-for-byte (or shape-for-shape) across all 6 turns; the retry-with-diagnostic loop never moved the model.
• All failures are Form-A shape errors (term-head confusion, paren imbalance, repetition), never semantics. The bracket discipline of the fully-parenthesised authoring surface is the real wall for this model.

Natural next steps (deliberate, not automatic): a different model class via the same harness; or testing whether the JSON authoring form — which removes the human-style paren-counting burden — changes the bracket-failure picture for the same model.