Files

T

Brummel ded28e2f75 experiment(cma): large-model control on Llama-3.1-405B — the format effect REVERSES (refs #68 )

Ran the two core formats (plain Form-A, annotated parens) on Llama-3.1-405B
(405B dense, ~135x Qwen3-Coder-Next's 3B active) to separate format-effect from
small-model-effect. Scripts parametrized by PROBE_MODEL/PROBE_TAG (Qwen default
preserved).

Watch-out caught: Llama-3.1 on IONOS leaks its chat template (repeats
"assistant", generates endless variants to max_tokens — an IONOS-documented
quirk). First run read 0/8; that was the instrument. Fixed with a stop sequence
and a balanced-first-module extractor.

Result reverses the effect:
  Qwen3-Coder (3B):  plain 6/8, annotated 7/8
  Llama-405B (405B): plain 7/8, annotated 6/8
Llama solves L4 (Qwen's bracket wall) with plain parens, so that wall was a
small-model effect. Annotated parens help the small model and hurt the large
one (at L1 Llama invented a spurious (lit 3) wrapper under the annotated
dialect). Both still fail L7 (SMA) — a genuine complexity ceiling.

Synthesis: redundant explicit structure is a crutch for weak models. A
frontier model tracks plain brackets fine and the unfamiliar scaffolding only
adds noise. The right surface is model-relative. Synthesis in
format-findings.md; raw ladders llama405-*.md.

2026-06-02 19:59:17 +02:00

4.3 KiB

Raw Blame History

Qwen3-Coder-Next ablation — where does Form-A authoring break?

Model: meta-llama/Meta-Llama-3.1-405B-Instruct-FP8; single-shot per level; minimal few-shot context.

Each level is independent (fresh context, one call, no retry).

L0_const — ✅ green

ctx=BASE tokens: prompt=395 completion=38

(module m
  (fn answer
    (type (fn-type (params) (ret (own (con Int)))))
    (params)
    (body 42)))

L1_arith — ✅ green

ctx=BASE tokens: prompt=398 completion=48

(module m
  (fn triple
    (type (fn-type (params (own (con Int))) (ret (own (con Int)))))
    (params x)
    (body (app * x 3))))

L2_print — ✅ green

ctx=BASE tokens: prompt=393 completion=56

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body (seq (app print 7) (do io/print_str "\n")))))

L3_adt — ✅ green

ctx=BASE tokens: prompt=412 completion=84

(module m
  (data Light (ctor Red) (ctor Green))
  (fn code
    (type (fn-type (params (own (con Light))) (ret (own (con Int)))))
    (params x)
    (body
      (match x
        (case (pat-ctor Red) 0)
        (case (pat-ctor Green) 1)))))

L4_rec — ✅ green

ctx=BASE tokens: prompt=424 completion=196

(module m
  (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
  (fn length
    (type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
    (params xs)
    (body
      (match xs
        (case (pat-ctor Nil) 0)
        (case (pat-ctor Cons _ t) (app + 1 (app length t))))))
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Cons 3 (term-ctor IntList Nil))))
        (seq (app print (app length xs)) (do io/print_str "\n"))))))

L5_series_len — ✅ green

ctx=SERIES tokens: prompt=634 completion=124

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s2 (app Series.push s 1.0)
          (let s3 (app Series.push s2 2.0)
            (let s4 (app Series.push s3 3.0)
              (seq (app print (app Series.len s4)) (do io/print_str "\n")))))))))

L6_series_at — ✅ green

ctx=SERIES tokens: prompt=642 completion=126

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s2 (app Series.push s 1.0)
          (let s3 (app Series.push s2 2.0)
            (let s4 (app Series.push s3 3.0)
              (seq (app print (app Series.at s4 0)) (do io/print_str "\n")))))))))

L7_sma — ❌ check

ctx=SERIES tokens: prompt=714 completion=363

(module m
  (fn main
    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
    (params)
    (body
      (let s (new Series (con Float) 3)
        (let s2 (app Series.push s 1.0)
          (let s3 (app Series.push s2 5.0)
            (let s4 (app Series.push s3 3.0)
              (seq
                (app print (/ (app + (app Series.at s4 0) (app + (app Series.at s4 1) (app Series.at s4 2))) 3.0))
                (do io/print_str "\n")
                (let s5 (app Series.push s4 8.0)
                  (seq
                    (app print (/ (app + (app Series.at s5 0) (app + (app Series.at s5 1) (app Series.at s5 2))) 3.0))
                    (do io/print_str "\n")
                    (let s6 (app Series.push s5 6.0)
                      (seq
                        (app print (/ (app + (app Series.at s6 0) (app + (app Series.at s6 1) (app Series.at s6 2))) 3.0))
                        (do io/print_str "\n")
                        (let s7 (app Series.push s6 2.0)
                          (app print (/ (app + (app Series.at s7 0) (app + (app Series.at s7 1) (app Series.at s7 2))) 3.0))))))))))))))

ail check:

error: [surface-parse-error] parse error in term: unknown term head `/`; expected one of `app`, `tail-app`, `lam`, `let`, `let-rec`, `if`, `match`, `do`, `tail-do`, `seq`, `term-ctor`, `clone`, `reuse-as`, `loop`, `recur`, `new`, `lit-unit` at byte 324

Ladder

L0_const: ✅ green
L1_arith: ✅ green
L2_print: ✅ green
L3_adt: ✅ green
L4_rec: ✅ green
L5_series_len: ✅ green
L6_series_at: ✅ green
L7_sma: ❌ check

4.3 KiB Raw Blame History