AILang/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md

SMA authoring probe — Qwen3-Coder-Next via IONOS

Date: 2026-06-02 Model: Qwen/Qwen3-Coder-Next (IONOS, temperature 0.2) Task: write the Series SMA worked example (examples/series_sma.ail) in AILang Form-A, given only the Form-A mini-spec (rendered/ail.md) + the Series library API — not the reference solution. Multi-turn (≤6), each attempt run through ail check → ail run, the diagnostic fed back on failure. Driver: sma_probe.py. Raw per-turn logs: sma-probe-run1.md, sma-probe-run2.md.

Expected stdout: 3.0 / 5.33333 / 5.66667 / 5.33333 (window 3 over [1.0, 5.0, 3.0, 8.0, 6.0, 2.0]).

Outcome: did NOT reach green in either run. But the two runs are a clean progression — each sharpening of the API doc moved the model exactly one hurdle forward.

Run 1 — API doc said (new Series (con Float) N)

• 6 turns, byte-identical program every turn, all failing at ail check.
• Blocker: Qwen wrote (app new (con Float) 3) — it treated new as an ordinary function and passed the type (con Float) as a term argument → [surface-parse-error] unknown term head 'con'. new is a built-in term head, not an app callee.
• Structure: instead of recursion over a list (as the reference does), Qwen hand-unrolled the six pushes into a deeply nested (seq (seq (seq …))) tower.
• Tokens: prompt 72 783, completion 12 000.

Run 2 — new clarified as a term head, with a (let s (new Series …) …) snippet

• The new parse error vanished — the sharpening worked; Qwen now writes (new Series (con Float) 3) correctly.
• New blocker: [arity-mismatch] Series.push: expected 2 args, got 4 (5/6 turns; turn 1 a stray expected ) parse error).
• Notably, the ownership threading was idiomatically correct: nested (let s (app Series.push s v) …), re-binding s to each push's result — the right shape for Series.push's own-in/own-out signature. The failure was S-expression bracketing: a missing ) let the push application swallow following terms, so it saw 4 args. Minor un-idiom: (app int_to_float 1) instead of the float literal 1.0.
• Again non-convergent: the bracketing bug persisted across all 6 turns despite the explicit "expected 2 args, got 4" diagnostic each time.
• Tokens: prompt 47 041, completion 1 852.

Cross-cutting findings (n=1 model, single session)

1. Per-API-quirk hand-holding. Qwen clears a hurdle only when the exact surface rule is spelled out (the new-is-a-term-head fix). With the stock API doc it does not infer it.
2. No convergence from compiler feedback. In neither run did the ail check diagnostic move the model off its failure — run 1 reproduced the same bytes 6×, run 2 kept the same bracketing bug 6×. The retry-with-diagnostic loop bought nothing here.
3. Form-A's nesting is itself a hurdle. Both failure modes are S-expression-shape errors (term-head confusion; unbalanced parens), not semantic ones — and in run 2 the semantics (push threading) were right. This is a concrete data point on the Form-A authoring surface for a foreign model: it gets the program logic but stumbles on the bracket discipline.

Did not pursue

A run 3 with the bracketing/push-arity spelled out would likely clear the next hurdle too — but at that point the prompt is dictating the solution shape rather than measuring what the model produces. Stopped here; the finding (clears hurdles only with per-quirk guidance, never converges from diagnostics) is robust across the two runs. A stronger-guidance run, or a different model, is a deliberate next step, not an automatic one.