Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

experiment(cma): ablation finds WHY Qwen fails at Form-A — two distinct walls (refs #68)

Browse Source 
Try-and-error ablation, minimal few-shot context, single shot per level, a
ladder L0 (fn returns 42) to L7 (full SMA). Two ladders: base few-shot that
only destructures, then one that also constructs (term-ctor).

Finding: Qwen is NOT generally incapable. With minimal correct context it
writes 6/8 tasks green, including ADTs, match, recursive fns, and Series use.
The failures are two specific walls:

- Wall 1 (cheap): unfamiliar constructs. It built lists with (app Cons ...) —
  calling the ctor like a function — because it had only seen match, never
  term-ctor. Adding ONE term-ctor example flipped L3 and L5 to green and
  removed the Cons error. The model generalises a construct from one example.

- Wall 2 (hard): paren/nesting discipline at depth. The two that stay red
  (L4, L7) fail purely on bracketing, not knowledge. In L4 the recursive
  function is flawless; main closes a 4x-nested term-ctor value one paren
  early so the let body slides inside it — total paren count is BALANCED
  (43/43), the parens are just misplaced. Green/red tracks nesting depth, not
  feature. Examples do not fix it. This is the same weakness the full-SMA
  probes hit (run 2 had correct logic, died on brackets; run 3's long context
  tipped it into a seq-repetition loop).

Takeaway: the bottleneck is the fully-parenthesised surface at depth, not the
semantics. Natural next test: the .ail.json authoring form, which removes
human paren-counting. Driver qwen_ablation.py; ladders in qwen-ablation-min.md
and qwen-ablation-ctor.md; synthesis in qwen-ablation.md.
This commit is contained in:
Brummel
2026-06-02 18:09:31 +02:00
parent 8b7e683cc9
commit da99ebaba8
4 changed files with 575 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experiments/2026-05-12-cross-model-authoring/qwen-ablation-ctor.md
+181
View File
@@ -0,0 +1,181 @@
# Qwen3-Coder-Next ablation — where does Form-A authoring break?
**Model:** Qwen/Qwen3-Coder-Next; single-shot per level; minimal few-shot context.
Each level is independent (fresh context, one call, no retry).
---
## L0_const — ✅ green
ctx=BASE tokens: prompt=375 completion=37
```
(module m
(fn answer
(type (fn-type (params) (ret (own (con Int)))))
(params)
(body 42)))
```
## L1_arith — ✅ green
ctx=BASE tokens: prompt=377 completion=50
```
(module m
(fn triple
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
(params x)
(body (app + x (app + x x)))))
```
## L2_print — ✅ green
ctx=BASE tokens: prompt=372 completion=54
```
(module m
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (seq (app print 7) (do io/print_str "\n")))))
```
## L3_adt — ✅ green
ctx=BASE tokens: prompt=391 completion=140
```
(module m
(data Light (ctor Red) (ctor Green))
(fn code
(type (fn-type (params (own (con Light))) (ret (own (con Int)))))
(params x)
(body
(match x
(case (pat-ctor Red) 0)
(case (pat-ctor Green) 1))))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(seq (app print (app code (term-ctor Light Red))) (do io/print_str "\n")))))
```
## L4_rec — ❌ check
ctx=BASE tokens: prompt=403 completion=194
```
(module m
(data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
(fn length
(type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
(params xs)
(body
(match xs
(case (pat-ctor Nil) 0)
(case (pat-ctor Cons _ t) (app + 1 (app length t))))))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Cons 3 (term-ctor IntList Nil)))
(seq (app print (app length xs)) (do io/print_str "\n"))))))
```
**`ail check`:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 579
```
## L5_series_len — ✅ green
ctx=SERIES tokens: prompt=613 completion=122
```
(module m
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let s (new Series (con Float) 3)
(let s1 (app Series.push s 1.0)
(let s2 (app Series.push s1 2.0)
(let s3 (app Series.push s2 3.0)
(seq (app print (app Series.len s3)) (do io/print_str "\n")))))))))
```
## L6_series_at — ✅ green
ctx=SERIES tokens: prompt=621 completion=124
```
(module m
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let s (new Series (con Float) 3)
(let s1 (app Series.push s 1.0)
(let s2 (app Series.push s1 2.0)
(let s3 (app Series.push s2 3.0)
(seq (app print (app Series.at s3 0)) (do io/print_str "\n")))))))))
```
## L7_sma — ❌ check
ctx=SERIES tokens: prompt=703 completion=285
```
(module m
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let s (new Series (con Float) 3)
(let s1 (app Series.push s 1.0)
(let s2 (app Series.push s1 5.0)
(let s3 (app Series.push s2 3.0)
(let s4 (app Series.push s3 8.0)
(let s5 (app Series.push s4 6.0)
(let s6 (app Series.push s5 2.0)
(seq
(seq
(seq
(seq
(seq
(app print (app / (app Series.total_count s3) 3.0))
(do io/print_str "\n"))
(app print (app / (app Series.total_count s4) 3.0)))
(do io/print_str "\n"))
(app print (app / (app Series.total_count s5) 3.0)))
(do io/print_str "\n"))
(app print (app / (app Series.total_count s6) 3.0)))
(do io/print_str "\n"))))))))
```
**`ail check`:**
```
error: [surface-parse-error] parse error: expected `)` (end of let-term), got `(` at byte 980
```
---
## Ladder
- L0_const: ✅ green
- L1_arith: ✅ green
- L2_print: ✅ green
- L3_adt: ✅ green
- L4_rec: ❌ check
- L5_series_len: ✅ green
- L6_series_at: ✅ green
- L7_sma: ❌ check
experiments/2026-05-12-cross-model-authoring/qwen-ablation-min.md
+178
View File
File diff suppressed because one or more lines are too long
experiments/2026-05-12-cross-model-authoring/qwen-ablation.md
+75
View File
@@ -0,0 +1,75 @@
# Why does Qwen3-Coder-Next fail at AILang Form-A? — ablation finding
**Date:** 2026-06-02 **Model:** Qwen/Qwen3-Coder-Next (IONOS, temp 0.2)
**Method:** try-and-error from a minimum upward. A ladder L0 (trivial) → L7
(full SMA), each level run independently with minimal few-shot context, single
shot. Two ladders: ladder 1 with a base few-shot that only *destructures*
(`match`); ladder 2 after adding one example that *constructs* (`term-ctor`).
Raw logs: `qwen-ablation-min.md` (ladder 1), `qwen-ablation-ctor.md` (ladder 2).
Driver: `qwen_ablation.py`.
## The two ladders
| Level | task | ladder 1 (min) | ladder 2 (+term-ctor example) |
|---|---|---|---|
| L0 | fn returns 42 | ✅ | ✅ |
| L1 | fn ×3 | ✅ | ✅ |
| L2 | main prints 7 | ✅ | ✅ |
| L3 | ADT + match | ❌ check | ✅ |
| L4 | build list + recursive length | ❌ `Cons` unbound | ❌ paren misplace |
| L5 | Series, print len | ❌ missing newline | ✅ |
| L6 | Series.at | ✅ | ✅ |
| L7 | full SMA | ❌ check | ❌ paren imbalance |
One example of the missing construct moved Qwen from 4/8 to **6/8**.
## The answer: it is NOT general incapacity — there are two distinct walls
**Qwen is not "stupid" at Form-A.** With minimal, *correct* few-shot context
it writes 6 of 8 tasks green — including ADTs, `match`, recursive functions,
and Series library use (`new Series`, `push`, `at`, `len`). The two failures
are specific and different in kind:
### Wall 1 — unfamiliar constructs (SOLVABLE by one example)
Qwen does not know constructs it hasn't seen, and guesses plausibly wrong:
- L4 ladder 1: it built a list with `(app Cons 1 (app Cons 2 …))` and bare
`Nil` — calling the constructor like a function → `unbound-var: Cons`. It
had only seen `match`/`pat-ctor` (destructuring), never `term-ctor`
(construction).
- Adding ONE few-shot that uses `term-ctor` flipped L3 and L5 to green, and
removed the `Cons`-as-function error in L4. The model generalises from a
single example of a construct.
This is the cheap wall: feed the construct once and it clears.
### Wall 2 — paren/nesting discipline at depth (NOT solved by examples)
The two that stayed red (L4, L7) both fail on **S-expression bracketing**,
not on any missing knowledge:
- L4 ladder 2: the `length` function is flawless. `main` builds
`(let xs (term-ctor … (term-ctor … (term-ctor … (term-ctor IntList Nil)))`
— closing the nested `term-ctor` value one paren too early, so the `let`
body slides inside the value. **Total paren count is balanced (43 open / 43
close)** — the parens are merely *misplaced*. The model is not under-closing
globally; it loses track of the nesting structure at depth.
- L7 (SMA): 49 open / 47 close — under-closed.
The grin/red split tracks **nesting depth**, not feature: every green task is
shallow (one or two levels), every red one is deeply nested (4× nested
`term-ctor` in a `let`; the SMA's stream-loop composition). This is a
mechanical limit on tracking deep paren nesting, and examples do not fix it —
consistent with the earlier full-SMA probes (run 2: correct ownership
threading, died on brackets; run 3: ~80k-token context tipped the same
weakness into a non-terminating `(seq (seq …))` repetition loop).
## Takeaway
The bottleneck for this model on Form-A is the **fully-parenthesised surface
itself** at depth, not the language's semantics or its vocabulary. Two
levers follow, both deliberate next steps, not run here:
1. Few-shot every non-obvious construct (cheap, clears Wall 1).
2. For Wall 2, the surface is the problem — the natural test is the
**`.ail.json` authoring form**, which removes human-style paren-counting
(structure is carried by JSON nesting the model is far more trained on).
The cma harness already renders both cohorts; this is the obvious A/B.
experiments/2026-05-12-cross-model-authoring/qwen_ablation.py
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Ablation probe: WHY does Qwen3-Coder-Next fail at AILang Form-A?
Try-and-error from a minimum upward. A ladder of tasks of increasing
complexity (L0 trivial → L7 full SMA), each run INDEPENDENTLY: fresh,
minimal few-shot context, single shot (no retry, no dialog accumulation —
prior probes showed retries don't converge and long context degenerates).
The first level that breaks localises the wall.
Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent.
"""
from __future__ import annotations
import json, os, re, subprocess, urllib.request
from pathlib import Path
ENDPOINT = "https://openai.inference.de-txl.ionos.com/v1/chat/completions"
MODEL = "Qwen/Qwen3-Coder-Next"
REPO = Path(__file__).resolve().parents[2]
AIL = os.environ.get("AIL_BIN", str(REPO / "target/debug/ail"))
DOC = REPO / "experiments/2026-05-12-cross-model-authoring/qwen-ablation-ctor.md"
DEMO_A = """(module demo_a
(fn double
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
(params x)
(body (app + x x)))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (seq (app print (app double 21)) (do io/print_str "\\n")))))"""
DEMO_B = """(module demo_b
(data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
(fn sum
(type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
(params xs)
(body
(match xs
(case (pat-ctor Nil) 0)
(case (pat-ctor Cons h t) (app + h (app sum t))))))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Nil)))
(seq (app print (app sum xs)) (do io/print_str "\\n"))))))"""
DEMO_S = """(module demo_s
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body
(let s (new Series (con Float) 3)
(let s2 (app Series.push s 1.0)
(let s3 (app Series.push s2 2.0)
(seq (app print (app Series.len s3)) (do io/print_str "\\n"))))))))"""
INTRO = ("AILang is a fully-parenthesised S-expression language (\"Form-A\"). "
"Study these complete, valid example modules, then write the requested "
"module. Return ONLY the AILang source for the module — no prose, no "
"markdown fences.\n\n")
BASE = INTRO + "Example 1:\n" + DEMO_A + "\n\nExample 2:\n" + DEMO_B + "\n"
SERIES = (BASE + "\nExample 3 (uses the Series ring-buffer library — "
"`(new Series (con Float) N)` allocates one of capacity N; "
"`Series.push : (own (Series a)) (own a) -> (own (Series a))` appends "
"and returns the new series; `Series.at : (borrow (Series a)) (own Int) "
"-> (own a)` reads index i back from newest (0 = newest); "
"`Series.len`/`Series.total_count : (borrow (Series a)) -> (own Int)`):\n"
+ DEMO_S + "\n")
SMA_OUT = "3.0\n5.33333\n5.66667\n5.33333\n"
SMA_TASK = ("Write a module `m` computing a simple moving average, window 3, over "
"the float stream 1.0, 5.0, 3.0, 8.0, 6.0, 2.0 using the Series library: "
"create a Float Series lookback 3, push each value in turn, and after each "
"push once at least 3 values have been pushed, print the average of the "
"most recent 3 (their sum / 3.0) on its own line. Expected:\n" + SMA_OUT)
# (id, context, task, expected_stdout_or_None[=check-only])
STAGES = [
("L0_const", BASE, "Write a module `m` with a function `answer` taking no parameters that returns the Int 42.", None),
("L1_arith", BASE, "Write a module `m` with a function `triple` taking one Int parameter, returning it multiplied by 3.", None),
("L2_print", BASE, "Write a module `m` whose `main` prints the Int 7 on its own line.", "7\n"),
("L3_adt", BASE, "Write a module `m` with an ADT `Light` (ctors Red, Green) and a function `code` returning Int 0 for Red and 1 for Green via match.", None),
("L4_rec", BASE, "Write a module `m` with an IntList ADT (Nil/Cons), a function `length` counting elements by recursion, and a `main` that prints the length of the list [1,2,3] on its own line.", "3\n"),
("L5_series_len", SERIES, "Write a module `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print its Series.len on its own line.", "3\n"),
("L6_series_at", SERIES, "Write a module `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print (Series.at on index 0, the newest) on its own line.", "3.0\n"),
("L7_sma", SERIES, SMA_TASK, SMA_OUT),
]
def call(system, task):
body = json.dumps({"model": MODEL, "temperature": 0.2, "max_tokens": 1500,
"messages": [{"role":"system","content":system},{"role":"user","content":task}]}).encode()
req = urllib.request.Request(ENDPOINT, data=body, headers={
"Authorization": f"Bearer {os.environ['IONOS_API_TOKEN']}", "Content-Type":"application/json"})
with urllib.request.urlopen(req, timeout=120) as r:
d = json.loads(r.read())
return d["choices"][0]["message"]["content"], d.get("usage", {})
def extract(resp):
m = re.search(r"```(?:[a-zA-Z]*)?\n(.*?)```", resp, re.S)
code = m.group(1) if m else resp
i = code.find("(module")
return code[i:].strip() if i >= 0 else code.strip()
def evaluate(code, expected):
m = re.search(r"\(module\s+([A-Za-z_][A-Za-z0-9_]*)", code)
name = m.group(1) if m else "m"
f = Path(f"/tmp/abl/{name}.ail"); f.parent.mkdir(exist_ok=True); f.write_text(code+"\n")
chk = subprocess.run([AIL,"check",str(f)], capture_output=True, text=True)
if chk.returncode != 0: return "check", (chk.stderr or chk.stdout).strip()
if expected is None: return "green", "(check-only)"
run = subprocess.run([AIL,"run",str(f)], capture_output=True, text=True)
if run.returncode != 0: return "run", (run.stderr or run.stdout).strip()
if run.stdout != expected: return "output", f"got {run.stdout!r}, expected {expected!r}"
return "green", run.stdout
def main():
log = ["# Qwen3-Coder-Next ablation — where does Form-A authoring break?\n",
f"**Model:** {MODEL}; single-shot per level; minimal few-shot context.\n",
"Each level is independent (fresh context, one call, no retry).\n", "---\n"]
summary = []
for sid, ctx, task, exp in STAGES:
resp, usage = call(ctx, task)
code = extract(resp)
stage, fb = evaluate(code, exp)
ok = stage == "green"
mark = "✅ green" if ok else f"❌ {stage}"
summary.append((sid, mark))
print(f"{sid}: {mark}")
log += [f"## {sid} — {mark}",
f"\nctx={'BASE' if ctx is BASE else 'SERIES'} tokens: "
f"prompt={usage.get('prompt_tokens','?')} completion={usage.get('completion_tokens','?')}\n",
"```\n"+code+"\n```\n",
("" if ok else f"**`ail {stage}`:**\n```\n{fb[:800]}\n```\n")]
log += ["\n---\n## Ladder\n"] + [f"- {s}: {m}" for s,m in summary]
DOC.write_text("\n".join(log))
print("doc →", DOC)
if __name__ == "__main__":
main()