AILang/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md

# SMA authoring probe — Qwen3-Coder-Next via IONOS

**Date:** 2026-06-02
**Model:** Qwen/Qwen3-Coder-Next (IONOS, temperature 0.2)
**Task:** write the Series SMA worked example (`examples/series_sma.ail`) in
AILang Form-A, given only the Form-A mini-spec (`rendered/ail.md`) + the
Series library API — *not* the reference solution. Multi-turn (≤6), each
attempt run through `ail check` → `ail run`, the diagnostic fed back on
failure. Driver: `sma_probe.py`. Raw per-turn logs: `sma-probe-run1.md`,
`sma-probe-run2.md`, `sma-probe-run3.md`. Three runs, escalating the amount
of reference material handed to the model.

**Expected stdout:** `3.0 / 5.33333 / 5.66667 / 5.33333` (window 3 over
`[1.0, 5.0, 3.0, 8.0, 6.0, 2.0]`).

**Outcome: did NOT reach green in any of the three runs.** Runs 1→2 were a
clean progression (each sharpening moved the model one hurdle forward); run 3
inverted it — more material made the model degenerate. See the Verdict at the
bottom.

## Run 1 — API doc said `(new Series (con Float) N)`

- 6 turns, **byte-identical program every turn**, all failing at `ail check`.
- Blocker: Qwen wrote `(app new (con Float) 3)` — it treated `new` as an
  ordinary function and passed the *type* `(con Float)` as a term argument →
  `[surface-parse-error] unknown term head 'con'`. `new` is a built-in term
  head, not an `app` callee.
- Structure: instead of recursion over a list (as the reference does), Qwen
  hand-unrolled the six pushes into a deeply nested `(seq (seq (seq …)))`
  tower.
- Tokens: prompt 72 783, completion 12 000.

## Run 2 — `new` clarified as a term head, with a `(let s (new Series …) …)` snippet

- The `new` parse error vanished — the sharpening worked; Qwen now writes
  `(new Series (con Float) 3)` correctly.
- New blocker: `[arity-mismatch] Series.push: expected 2 args, got 4`
  (5/6 turns; turn 1 a stray `expected )` parse error).
- Notably, the *ownership threading was idiomatically correct*: nested
  `(let s (app Series.push s v) …)`, re-binding `s` to each push's result —
  the right shape for `Series.push`'s own-in/own-out signature. The failure
  was S-expression bracketing: a missing `)` let the `push` application
  swallow following terms, so it saw 4 args. Minor un-idiom: `(app
  int_to_float 1)` instead of the float literal `1.0`.
- Again **non-convergent**: the bracketing bug persisted across all 6 turns
  despite the explicit "expected 2 args, got 4" diagnostic each time.
- Tokens: prompt 47 041, completion 1 852.

## Cross-cutting findings (n=1 model, single session)

1. **Per-API-quirk hand-holding.** Qwen clears a hurdle only when the exact
   surface rule is spelled out (the `new`-is-a-term-head fix). With the
   stock API doc it does not infer it.
2. **No convergence from compiler feedback.** In neither run did the
   `ail check` diagnostic move the model off its failure — run 1 reproduced
   the same bytes 6×, run 2 kept the same bracketing bug 6×. The
   retry-with-diagnostic loop bought nothing here.
3. **Form-A's nesting is itself a hurdle.** Both failure modes are
   S-expression-shape errors (term-head confusion; unbalanced parens), not
   semantic ones — and in run 2 the *semantics* (push threading) were right.
   This is a concrete data point on the Form-A authoring surface for a
   foreign model: it gets the program logic but stumbles on the bracket
   discipline.

## Run 3 — fully equipped: complete construct grammar + mini-spec + real Series library source

The hypothesis was that the bracket failures of runs 1–2 would clear if Qwen
had the formal grammar. The opposite happened. With the larger context
(~80k prompt tokens: a distilled Form-A construct grammar with explicit
arity rules + `rendered/ail.md` + the verbatim `series/source.ail`), Qwen
**degenerated**: every one of the 6 turns produced an identical pathological
`(seq (seq (seq …)))` tower — **321 open parens vs 10 close, identical bytes
each turn** — with whitespace indentation exploding to hundreds of spaces per
line. It never closes the tower; it just runs out of tokens (`unexpected end
of input`). Raw log: `sma-probe-run3.md`.

A confound check (re-run with `max_tokens` doubled to 4000) did **not** yield
a complete program — it produced HTTP 413 (request too large): with more
budget Qwen emits an even larger degenerate output, so the accumulated
message history blows past the endpoint's size limit. That confirms
degeneration rather than truncation.

So the extra equipment made the model **worse**, not better. The best run was
the middle one (run 2): just enough guidance to clear the `new` quirk,
without the context overload that tipped run 3 into a repetition loop.

## Verdict (n=1 model, single session)

- **Qwen3-Coder-Next did not write the SMA in any configuration.** Its only
  semantically-correct attempt (run 2) still failed on S-expression
  bracketing.
- **More reference material did not help — it hurt.** The fully-equipped run
  degenerated into a non-terminating `seq` tower; the lean run got the
  algorithm right. This is the sharpest finding: for this model on Form-A,
  context volume past a point is actively harmful.
- **No convergence from compiler feedback in any run.** Every run repeated
  its failure byte-for-byte (or shape-for-shape) across all 6 turns; the
  retry-with-diagnostic loop never moved the model.
- **All failures are Form-A *shape* errors** (term-head confusion, paren
  imbalance, repetition), never semantics. The bracket discipline of the
  fully-parenthesised authoring surface is the real wall for this model.

Natural next steps (deliberate, not automatic): a different model class via
the same harness; or testing whether the JSON authoring form — which removes
the human-style paren-counting burden — changes the bracket-failure picture
for the same model.