Brummel
ded28e2f75
Ran the two core formats (plain Form-A, annotated parens) on Llama-3.1-405B (405B dense, ~135x Qwen3-Coder-Next's 3B active) to separate format-effect from small-model-effect. Scripts parametrized by PROBE_MODEL/PROBE_TAG (Qwen default preserved). Watch-out caught: Llama-3.1 on IONOS leaks its chat template (repeats "assistant", generates endless variants to max_tokens — an IONOS-documented quirk). First run read 0/8; that was the instrument. Fixed with a stop sequence and a balanced-first-module extractor. Result reverses the effect: Qwen3-Coder (3B): plain 6/8, annotated 7/8 Llama-405B (405B): plain 7/8, annotated 6/8 Llama solves L4 (Qwen's bracket wall) with plain parens, so that wall was a small-model effect. Annotated parens help the small model and hurt the large one (at L1 Llama invented a spurious (lit 3) wrapper under the annotated dialect). Both still fail L7 (SMA) — a genuine complexity ceiling. Synthesis: redundant explicit structure is a crutch for weak models. A frontier model tracks plain brackets fine and the unfamiliar scaffolding only adds noise. The right surface is model-relative. Synthesis in format-findings.md; raw ladders llama405-*.md.
169 lines
4.3 KiB
Markdown
169 lines
4.3 KiB
Markdown
# Qwen3-Coder-Next ablation — where does Form-A authoring break?
|
|
|
|
**Model:** meta-llama/Meta-Llama-3.1-405B-Instruct-FP8; single-shot per level; minimal few-shot context.
|
|
|
|
Each level is independent (fresh context, one call, no retry).
|
|
|
|
---
|
|
|
|
## L0_const — ✅ green
|
|
|
|
ctx=BASE tokens: prompt=395 completion=38
|
|
|
|
```
|
|
(module m
|
|
(fn answer
|
|
(type (fn-type (params) (ret (own (con Int)))))
|
|
(params)
|
|
(body 42)))
|
|
```
|
|
|
|
|
|
## L1_arith — ✅ green
|
|
|
|
ctx=BASE tokens: prompt=398 completion=48
|
|
|
|
```
|
|
(module m
|
|
(fn triple
|
|
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
|
|
(params x)
|
|
(body (app * x 3))))
|
|
```
|
|
|
|
|
|
## L2_print — ✅ green
|
|
|
|
ctx=BASE tokens: prompt=393 completion=56
|
|
|
|
```
|
|
(module m
|
|
(fn main
|
|
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
|
|
(params)
|
|
(body (seq (app print 7) (do io/print_str "\n")))))
|
|
```
|
|
|
|
|
|
## L3_adt — ✅ green
|
|
|
|
ctx=BASE tokens: prompt=412 completion=84
|
|
|
|
```
|
|
(module m
|
|
(data Light (ctor Red) (ctor Green))
|
|
(fn code
|
|
(type (fn-type (params (own (con Light))) (ret (own (con Int)))))
|
|
(params x)
|
|
(body
|
|
(match x
|
|
(case (pat-ctor Red) 0)
|
|
(case (pat-ctor Green) 1)))))
|
|
```
|
|
|
|
|
|
## L4_rec — ✅ green
|
|
|
|
ctx=BASE tokens: prompt=424 completion=196
|
|
|
|
```
|
|
(module m
|
|
(data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
|
|
(fn length
|
|
(type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
|
|
(params xs)
|
|
(body
|
|
(match xs
|
|
(case (pat-ctor Nil) 0)
|
|
(case (pat-ctor Cons _ t) (app + 1 (app length t))))))
|
|
(fn main
|
|
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
|
|
(params)
|
|
(body
|
|
(let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Cons 3 (term-ctor IntList Nil))))
|
|
(seq (app print (app length xs)) (do io/print_str "\n"))))))
|
|
```
|
|
|
|
|
|
## L5_series_len — ✅ green
|
|
|
|
ctx=SERIES tokens: prompt=634 completion=124
|
|
|
|
```
|
|
(module m
|
|
(fn main
|
|
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
|
|
(params)
|
|
(body
|
|
(let s (new Series (con Float) 3)
|
|
(let s2 (app Series.push s 1.0)
|
|
(let s3 (app Series.push s2 2.0)
|
|
(let s4 (app Series.push s3 3.0)
|
|
(seq (app print (app Series.len s4)) (do io/print_str "\n")))))))))
|
|
```
|
|
|
|
|
|
## L6_series_at — ✅ green
|
|
|
|
ctx=SERIES tokens: prompt=642 completion=126
|
|
|
|
```
|
|
(module m
|
|
(fn main
|
|
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
|
|
(params)
|
|
(body
|
|
(let s (new Series (con Float) 3)
|
|
(let s2 (app Series.push s 1.0)
|
|
(let s3 (app Series.push s2 2.0)
|
|
(let s4 (app Series.push s3 3.0)
|
|
(seq (app print (app Series.at s4 0)) (do io/print_str "\n")))))))))
|
|
```
|
|
|
|
|
|
## L7_sma — ❌ check
|
|
|
|
ctx=SERIES tokens: prompt=714 completion=363
|
|
|
|
```
|
|
(module m
|
|
(fn main
|
|
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
|
|
(params)
|
|
(body
|
|
(let s (new Series (con Float) 3)
|
|
(let s2 (app Series.push s 1.0)
|
|
(let s3 (app Series.push s2 5.0)
|
|
(let s4 (app Series.push s3 3.0)
|
|
(seq
|
|
(app print (/ (app + (app Series.at s4 0) (app + (app Series.at s4 1) (app Series.at s4 2))) 3.0))
|
|
(do io/print_str "\n")
|
|
(let s5 (app Series.push s4 8.0)
|
|
(seq
|
|
(app print (/ (app + (app Series.at s5 0) (app + (app Series.at s5 1) (app Series.at s5 2))) 3.0))
|
|
(do io/print_str "\n")
|
|
(let s6 (app Series.push s5 6.0)
|
|
(seq
|
|
(app print (/ (app + (app Series.at s6 0) (app + (app Series.at s6 1) (app Series.at s6 2))) 3.0))
|
|
(do io/print_str "\n")
|
|
(let s7 (app Series.push s6 2.0)
|
|
(app print (/ (app + (app Series.at s7 0) (app + (app Series.at s7 1) (app Series.at s7 2))) 3.0))))))))))))))
|
|
```
|
|
|
|
**`ail check`:**
|
|
```
|
|
error: [surface-parse-error] parse error in term: unknown term head `/`; expected one of `app`, `tail-app`, `lam`, `let`, `let-rec`, `if`, `match`, `do`, `tail-do`, `seq`, `term-ctor`, `clone`, `reuse-as`, `loop`, `recur`, `new`, `lit-unit` at byte 324
|
|
```
|
|
|
|
|
|
---
|
|
## Ladder
|
|
|
|
- L0_const: ✅ green
|
|
- L1_arith: ✅ green
|
|
- L2_print: ✅ green
|
|
- L3_adt: ✅ green
|
|
- L4_rec: ✅ green
|
|
- L5_series_len: ✅ green
|
|
- L6_series_at: ✅ green
|
|
- L7_sma: ❌ check |