Files

T

Brummel da99ebaba8 experiment(cma): ablation finds WHY Qwen fails at Form-A — two distinct walls (refs #68 )

Try-and-error ablation, minimal few-shot context, single shot per level, a
ladder L0 (fn returns 42) to L7 (full SMA). Two ladders: base few-shot that
only destructures, then one that also constructs (term-ctor).

Finding: Qwen is NOT generally incapable. With minimal correct context it
writes 6/8 tasks green, including ADTs, match, recursive fns, and Series use.
The failures are two specific walls:

- Wall 1 (cheap): unfamiliar constructs. It built lists with (app Cons ...) —
  calling the ctor like a function — because it had only seen match, never
  term-ctor. Adding ONE term-ctor example flipped L3 and L5 to green and
  removed the Cons error. The model generalises a construct from one example.

- Wall 2 (hard): paren/nesting discipline at depth. The two that stay red
  (L4, L7) fail purely on bracketing, not knowledge. In L4 the recursive
  function is flawless; main closes a 4x-nested term-ctor value one paren
  early so the let body slides inside it — total paren count is BALANCED
  (43/43), the parens are just misplaced. Green/red tracks nesting depth, not
  feature. Examples do not fix it. This is the same weakness the full-SMA
  probes hit (run 2 had correct logic, died on brackets; run 3's long context
  tipped it into a seq-repetition loop).

Takeaway: the bottleneck is the fully-parenthesised surface at depth, not the
semantics. Natural next test: the .ail.json authoring form, which removes
human paren-counting. Driver qwen_ablation.py; ladders in qwen-ablation-min.md
and qwen-ablation-ctor.md; synthesis in qwen-ablation.md.

2026-06-02 18:09:31 +02:00

4.4 KiB

Raw Permalink Blame History

Qwen3-Coder-Next ablation — where does Form-A authoring break?

Model: Qwen/Qwen3-Coder-Next; single-shot per level; minimal few-shot context.

Each level is independent (fresh context, one call, no retry).

L0_const — ✅ green

ctx=BASE tokens: prompt=375 completion=37

(module m
  (fn answer
    (type (fn-type (params) (ret (own (con Int)))))
    (params)
    (body 42)))

L1_arith — ✅ green

ctx=BASE tokens: prompt=377 completion=50

(module m
  (fn triple
    (type (fn-type (params (own (con Int))) (ret (own (con Int)))))
    (params x)
    (body (app + x (app + x x)))))

L2_print — ✅ green

ctx=BASE tokens: prompt=372 completion=54

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body (seq (app print 7) (do io/print_str "\n")))))

L3_adt — ✅ green

ctx=BASE tokens: prompt=391 completion=140

(module m
  (data Light (ctor Red) (ctor Green))
  (fn code
    (type (fn-type (params (own (con Light))) (ret (own (con Int)))))
    (params x)
    (body
      (match x
        (case (pat-ctor Red) 0)
        (case (pat-ctor Green) 1))))
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (seq (app print (app code (term-ctor Light Red))) (do io/print_str "\n")))))

L4_rec — ❌ check

ctx=BASE tokens: prompt=403 completion=194

(module m
  (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
  (fn length
    (type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
    (params xs)
    (body
      (match xs
        (case (pat-ctor Nil) 0)
        (case (pat-ctor Cons _ t) (app + 1 (app length t))))))
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Cons 3 (term-ctor IntList Nil)))
        (seq (app print (app length xs)) (do io/print_str "\n"))))))

ail check:

error: [surface-parse-error] parse error: expected term, got `)` at byte 579

L5_series_len — ✅ green

ctx=SERIES tokens: prompt=613 completion=122

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s1 (app Series.push s 1.0)
          (let s2 (app Series.push s1 2.0)
            (let s3 (app Series.push s2 3.0)
              (seq (app print (app Series.len s3)) (do io/print_str "\n")))))))))

L6_series_at — ✅ green

ctx=SERIES tokens: prompt=621 completion=124

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s1 (app Series.push s 1.0)
          (let s2 (app Series.push s1 2.0)
            (let s3 (app Series.push s2 3.0)
              (seq (app print (app Series.at s3 0)) (do io/print_str "\n")))))))))

L7_sma — ❌ check

ctx=SERIES tokens: prompt=703 completion=285

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s1 (app Series.push s 1.0)
          (let s2 (app Series.push s1 5.0)
            (let s3 (app Series.push s2 3.0)
              (let s4 (app Series.push s3 8.0)
                (let s5 (app Series.push s4 6.0)
                  (let s6 (app Series.push s5 2.0)
                    (seq
                      (seq
                        (seq
                          (seq
                            (seq
                              (app print (app / (app Series.total_count s3) 3.0))
                              (do io/print_str "\n"))
                            (app print (app / (app Series.total_count s4) 3.0)))
                          (do io/print_str "\n"))
                        (app print (app / (app Series.total_count s5) 3.0)))
                      (do io/print_str "\n"))
                    (app print (app / (app Series.total_count s6) 3.0)))
                  (do io/print_str "\n"))))))))