Files

T

Brummel da99ebaba8 experiment(cma): ablation finds WHY Qwen fails at Form-A — two distinct walls (refs #68 )

Try-and-error ablation, minimal few-shot context, single shot per level, a
ladder L0 (fn returns 42) to L7 (full SMA). Two ladders: base few-shot that
only destructures, then one that also constructs (term-ctor).

Finding: Qwen is NOT generally incapable. With minimal correct context it
writes 6/8 tasks green, including ADTs, match, recursive fns, and Series use.
The failures are two specific walls:

- Wall 1 (cheap): unfamiliar constructs. It built lists with (app Cons ...) —
  calling the ctor like a function — because it had only seen match, never
  term-ctor. Adding ONE term-ctor example flipped L3 and L5 to green and
  removed the Cons error. The model generalises a construct from one example.

- Wall 2 (hard): paren/nesting discipline at depth. The two that stay red
  (L4, L7) fail purely on bracketing, not knowledge. In L4 the recursive
  function is flawless; main closes a 4x-nested term-ctor value one paren
  early so the let body slides inside it — total paren count is BALANCED
  (43/43), the parens are just misplaced. Green/red tracks nesting depth, not
  feature. Examples do not fix it. This is the same weakness the full-SMA
  probes hit (run 2 had correct logic, died on brackets; run 3's long context
  tipped it into a seq-repetition loop).

Takeaway: the bottleneck is the fully-parenthesised surface at depth, not the
semantics. Natural next test: the .ail.json authoring form, which removes
human paren-counting. Driver qwen_ablation.py; ladders in qwen-ablation-min.md
and qwen-ablation-ctor.md; synthesis in qwen-ablation.md.

2026-06-02 18:09:31 +02:00

3.8 KiB

Raw Permalink Blame History

Why does Qwen3-Coder-Next fail at AILang Form-A? — ablation finding

Date: 2026-06-02 Model: Qwen/Qwen3-Coder-Next (IONOS, temp 0.2) Method: try-and-error from a minimum upward. A ladder L0 (trivial) → L7 (full SMA), each level run independently with minimal few-shot context, single shot. Two ladders: ladder 1 with a base few-shot that only destructures (match); ladder 2 after adding one example that constructs (term-ctor). Raw logs: qwen-ablation-min.md (ladder 1), qwen-ablation-ctor.md (ladder 2). Driver: qwen_ablation.py.

The two ladders

Level	task	ladder 1 (min)	ladder 2 (+term-ctor example)
L0	fn returns 42	✅	✅
L1	fn ×3	✅	✅
L2	main prints 7	✅	✅
L3	ADT + match	❌ check	✅
L4	build list + recursive length	❌ `Cons` unbound	❌ paren misplace
L5	Series, print len	❌ missing newline	✅
L6	Series.at	✅	✅
L7	full SMA	❌ check	❌ paren imbalance

One example of the missing construct moved Qwen from 4/8 to 6/8.

The answer: it is NOT general incapacity — there are two distinct walls

Qwen is not "stupid" at Form-A. With minimal, correct few-shot context it writes 6 of 8 tasks green — including ADTs, match, recursive functions, and Series library use (new Series, push, at, len). The two failures are specific and different in kind:

Wall 1 — unfamiliar constructs (SOLVABLE by one example)

Qwen does not know constructs it hasn't seen, and guesses plausibly wrong:

L4 ladder 1: it built a list with (app Cons 1 (app Cons 2 …)) and bare Nil — calling the constructor like a function → unbound-var: Cons. It had only seen match/pat-ctor (destructuring), never term-ctor (construction).
Adding ONE few-shot that uses term-ctor flipped L3 and L5 to green, and removed the Cons-as-function error in L4. The model generalises from a single example of a construct.

This is the cheap wall: feed the construct once and it clears.

Wall 2 — paren/nesting discipline at depth (NOT solved by examples)

The two that stayed red (L4, L7) both fail on S-expression bracketing, not on any missing knowledge:

L4 ladder 2: the length function is flawless. main builds (let xs (term-ctor … (term-ctor … (term-ctor … (term-ctor IntList Nil))) — closing the nested term-ctor value one paren too early, so the let body slides inside the value. Total paren count is balanced (43 open / 43 close) — the parens are merely misplaced. The model is not under-closing globally; it loses track of the nesting structure at depth.
L7 (SMA): 49 open / 47 close — under-closed.

The grin/red split tracks nesting depth, not feature: every green task is shallow (one or two levels), every red one is deeply nested (4× nested term-ctor in a let; the SMA's stream-loop composition). This is a mechanical limit on tracking deep paren nesting, and examples do not fix it — consistent with the earlier full-SMA probes (run 2: correct ownership threading, died on brackets; run 3: ~80k-token context tipped the same weakness into a non-terminating (seq (seq …)) repetition loop).

Takeaway

The bottleneck for this model on Form-A is the fully-parenthesised surface itself at depth, not the language's semantics or its vocabulary. Two levers follow, both deliberate next steps, not run here:

Few-shot every non-obvious construct (cheap, clears Wall 1).
For Wall 2, the surface is the problem — the natural test is the .ail.json authoring form, which removes human-style paren-counting (structure is carried by JSON nesting the model is far more trained on). The cma harness already renders both cohorts; this is the obvious A/B.

3.8 KiB Raw Permalink Blame History Unescape Escape