experiment(cma): SMA probe run 3 — full grammar+spec+lib makes Qwen degenerate (refs #68)

Third run of the SMA authoring probe, this time with Qwen fully equipped: a
distilled complete Form-A construct grammar (with explicit arity/paren rules),
the mini-spec, and the verbatim Series library source. Hypothesis: the formal
grammar clears the bracket failures of runs 1-2. It did the opposite.

With ~80k prompt tokens the model degenerated into an identical pathological
(seq (seq (seq ...))) tower every turn (321 open parens vs 10 close,
byte-identical across all 6 turns, indentation exploding). Doubling max_tokens
to confirm it was not mere truncation produced HTTP 413 (request too large) —
more budget yields an even larger degenerate output, confirming degeneration.

Verdict (n=1 model): Qwen3-Coder-Next wrote no working SMA in any of three
configurations; more reference material made it WORSE, not better (the lean
run 2 got the algorithm right, the fully-equipped run 3 looped); it never
converged from ail-check feedback in any run; all failures are Form-A
bracket-shape errors, never semantics. Driver + raw logs updated.

experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md

@@ -7,14 +7,16 @@ AILang Form-A, given only the Form-A mini-spec (`rendered/ail.md`) + the
 Series library API — *not* the reference solution. Multi-turn (≤6), each
 attempt run through `ail check` → `ail run`, the diagnostic fed back on
 failure. Driver: `sma_probe.py`. Raw per-turn logs: `sma-probe-run1.md`,
-`sma-probe-run2.md`.
+`sma-probe-run2.md`, `sma-probe-run3.md`. Three runs, escalating the amount
+of reference material handed to the model.
 
 **Expected stdout:** `3.0 / 5.33333 / 5.66667 / 5.33333` (window 3 over
 `[1.0, 5.0, 3.0, 8.0, 6.0, 2.0]`).
 
-**Outcome: did NOT reach green in either run.** But the two runs are a clean
+**Outcome: did NOT reach green in any of the three runs.** Runs 1→2 were a
-progression — each sharpening of the API doc moved the model exactly one
+clean progression (each sharpening moved the model one hurdle forward); run 3
-hurdle forward.
+inverted it — more material made the model degenerate. See the Verdict at the
+bottom.
 
 ## Run 1 — API doc said `(new Series (con Float) N)`
 
@@ -60,11 +62,45 @@ hurdle forward.
    foreign model: it gets the program logic but stumbles on the bracket
    discipline.
 
-## Did not pursue
+## Run 3 — fully equipped: complete construct grammar + mini-spec + real Series library source
 
-A run 3 with the bracketing/`push`-arity spelled out would likely clear the
+The hypothesis was that the bracket failures of runs 1–2 would clear if Qwen
-next hurdle too — but at that point the prompt is dictating the solution
+had the formal grammar. The opposite happened. With the larger context
-shape rather than measuring what the model produces. Stopped here; the
+(~80k prompt tokens: a distilled Form-A construct grammar with explicit
-finding (clears hurdles only with per-quirk guidance, never converges from
+arity rules + `rendered/ail.md` + the verbatim `series/source.ail`), Qwen
-diagnostics) is robust across the two runs. A stronger-guidance run, or a
+**degenerated**: every one of the 6 turns produced an identical pathological
-different model, is a deliberate next step, not an automatic one.
+`(seq (seq (seq …)))` tower — **321 open parens vs 10 close, identical bytes
+each turn** — with whitespace indentation exploding to hundreds of spaces per
+line. It never closes the tower; it just runs out of tokens (`unexpected end
+of input`). Raw log: `sma-probe-run3.md`.
+
+A confound check (re-run with `max_tokens` doubled to 4000) did **not** yield
+a complete program — it produced HTTP 413 (request too large): with more
+budget Qwen emits an even larger degenerate output, so the accumulated
+message history blows past the endpoint's size limit. That confirms
+degeneration rather than truncation.
+
+So the extra equipment made the model **worse**, not better. The best run was
+the middle one (run 2): just enough guidance to clear the `new` quirk,
+without the context overload that tipped run 3 into a repetition loop.
+
+## Verdict (n=1 model, single session)
+
+- **Qwen3-Coder-Next did not write the SMA in any configuration.** Its only
+  semantically-correct attempt (run 2) still failed on S-expression
+  bracketing.
+- **More reference material did not help — it hurt.** The fully-equipped run
+  degenerated into a non-terminating `seq` tower; the lean run got the
+  algorithm right. This is the sharpest finding: for this model on Form-A,
+  context volume past a point is actively harmful.
+- **No convergence from compiler feedback in any run.** Every run repeated
+  its failure byte-for-byte (or shape-for-shape) across all 6 turns; the
+  retry-with-diagnostic loop never moved the model.
+- **All failures are Form-A *shape* errors** (term-head confusion, paren
+  imbalance, repetition), never semantics. The bracket discipline of the
+  fully-parenthesised authoring surface is the real wall for this model.
+
+Natural next steps (deliberate, not automatic): a different model class via
+the same harness; or testing whether the JSON authoring form — which removes
+the human-style paren-counting burden — changes the bracket-failure picture
+for the same model.

experiments/2026-05-12-cross-model-authoring/sma_probe.py

@@ -21,32 +21,64 @@ REPO = Path(__file__).resolve().parents[2]
 AIL  = os.environ.get("AIL_BIN", str(REPO / "target/debug/ail"))
 SPEC = (REPO / "experiments/2026-05-12-cross-model-authoring/rendered/ail.md").read_text()
 EXPECTED = "3.0\n5.33333\n5.66667\n5.33333\n"
-DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md"
+DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md"
 
-SERIES_API = """\
+GRAMMAR = """\
-## Series library API
+## AILang Form-A grammar (complete construct grammar)
 
-`Series a` is a bounded ring buffer (financial indexing: index 0 = newest).
+Every program is fully-parenthesised S-expressions. ARITY IS FIXED PER HEAD —
-The element type `a` is `Int` or `Float`. Available operations:
+count your parens. `seq` takes EXACTLY 2 subterms; `if` EXACTLY 3; `let`
+EXACTLY 3.
 
-- `new` is a built-in TERM HEAD (like `let`, `if`, `match`, `app`) — NOT an
+Module:  (module NAME DECL*)
-  ordinary function. Do NOT write `(app new ...)`. To allocate an empty Float
+Decl:
-  Series with capacity (lookback) `N`, write the term `(new Series (con Float) N)`
+  (data NAME (vars V*)? (doc S)? (ctor CNAME FIELDTYPE*)*)
-  directly. It evaluates to `(own (con Series (con Float)))`. For example, to
+  (fn NAME (doc S)? (type TYPE) (params P*) (body TERM))
-  bind one: `(let s (new Series (con Float) 3) <body-using-s>)`.
+Type:
-- `Series.push : (own (con Series a)) (own a) -> (own (con Series a))` —
+  (con NAME TYPEARG*)             ; type constructor: (con Int), (con Series (con Float))
-  append a value, evicting the oldest element when full.
+  V                               ; a bare identifier is a type variable, e.g. a
-- `Series.at : (borrow (con Series a)) (own (con Int)) -> (own a)` — read the
+  (fn-type (params PT*) (ret PT) (effects E*)?)
-  element `i` positions back from newest (`i=0` is the most recent push).
+  (forall (vars V*) TYPE)
-- `Series.len : (borrow (con Series a)) -> (own (con Int))` — number of live
+  (own TYPE) | (borrow TYPE)      ; parameter / return MODE wrappers (mandatory)
-  elements (saturates at the lookback).
+Term (head forms):
-- `Series.total_count : (borrow (con Series a)) -> (own (con Int))` — monotone
+  (app FN ARG*)                   ; call FN with N value args
-  count of all pushes ever (does NOT saturate).
+  (tail-app FN ARG*)              ; tail call (use for recursion)
+  (term-ctor TYPENAME CNAME ARG*) ; build an ADT value
+  (new TYPENAME TYPEARG* VALARG*) ; BUILT-IN allocator term (NOT a function, never (app new ...)):
+                                  ;   (new Series (con Float) 3)
+  (if COND THEN ELSE)             ; exactly 3 subterms
+  (let NAME VALUE BODY)           ; exactly 3: bind NAME=VALUE, then BODY
+  (let-rec NAME (type T) (params P*) (body B) IN)   ; local recursive fn, then IN
+  (match SCRUT (case PAT ARM)*)   ; one or more arms
+  (seq LHS RHS)                   ; EXACTLY 2; for >2 effects, NEST: (seq A (seq B C))
+  (do OP ARG*)                    ; effect op: (do io/print_str "\\n")
+  (loop (NAME TYPE INIT)* BODY)   ; typed loop binders, then body
+  (recur ARG*)                    ; re-enter the enclosing loop
+  literals: 42 (Int)  1.0 (Float)  "txt" (Str)  true | false (Bool)
+  NAME                            ; a bare identifier is a variable reference
+Pattern:
+  (pat-ctor CNAME FIELDPAT*)      ; FIELDPAT = bare name (binds) or pat-wild
+  (pat-lit LIT) | pat-wild
+Params:  (params a b c)           ; bare parameter names
 
-Useful prelude functions: `+ - * /` (arithmetic, `forall a. (a,a)->a`),
+Prelude functions you will need: `+ - * /` (`forall a. (a,a)->a`),
-`eq`/`ge`/`gt`/`lt`/`le` (comparisons returning Bool), `int_to_float :
+`eq ge gt lt le` (comparisons -> Bool), `int_to_float : (Int)->Float`,
-(Int)->Float`, `print : forall a. (a) -> Unit !IO` (no trailing newline),
+`print : forall a. (a) -> Unit !IO` (NO trailing newline; emit one with
-`io/print_str : (Str) -> Unit !IO` (effect op, e.g. `(do io/print_str "\\n")`).
+`(do io/print_str "\\n")`).
+"""
+
+SERIES_LIB = """\
+## Series library reference (this module already exists — CALL these functions, do not redefine them)
+
+`Series a` is a bounded ring buffer; element type `a` is `Int` or `Float`;
+financial indexing (index 0 = newest). The library source (its public fns and
+their exact types/behaviour):
+
+""" + (REPO / "crates/ailang-kernel/src/series/source.ail").read_text() + """
+
+Call its functions as `Series.new`, `Series.push`, `Series.at`, `Series.len`,
+`Series.total_count`. Allocate a fresh one with the `new` term:
+`(new Series (con Float) N)`.
 """
 
 TASK = """\
@@ -80,7 +112,7 @@ def extract_code(resp: str) -> str:
 
 def call(messages) -> tuple[str, dict]:
     body = json.dumps({"model": MODEL, "messages": messages,
-                       "temperature": 0.2, "max_tokens": 2000}).encode()
+                       "temperature": 0.2, "max_tokens": 4000}).encode()
     req = urllib.request.Request(ENDPOINT, data=body, headers={
         "Authorization": f"Bearer {os.environ['IONOS_API_TOKEN']}",
         "Content-Type": "application/json"})
@@ -104,7 +136,7 @@ def evaluate(code: str):
     return True, "green", run.stdout
 
 def main():
-    system = SPEC + "\n\n" + SERIES_API
+    system = GRAMMAR + "\n\n" + SPEC + "\n\n" + SERIES_LIB
     messages = [{"role": "system", "content": system},
                 {"role": "user", "content": TASK}]
     log = ["# SMA authoring probe — Qwen3-Coder-Next via IONOS\n",