experiment(cma): SMA probe run 3 — full grammar+spec+lib makes Qwen degenerate (refs #68)
Third run of the SMA authoring probe, this time with Qwen fully equipped: a distilled complete Form-A construct grammar (with explicit arity/paren rules), the mini-spec, and the verbatim Series library source. Hypothesis: the formal grammar clears the bracket failures of runs 1-2. It did the opposite. With ~80k prompt tokens the model degenerated into an identical pathological (seq (seq (seq ...))) tower every turn (321 open parens vs 10 close, byte-identical across all 6 turns, indentation exploding). Doubling max_tokens to confirm it was not mere truncation produced HTTP 413 (request too large) — more budget yields an even larger degenerate output, confirming degeneration. Verdict (n=1 model): Qwen3-Coder-Next wrote no working SMA in any of three configurations; more reference material made it WORSE, not better (the lean run 2 got the algorithm right, the fully-equipped run 3 looped); it never converged from ail-check feedback in any run; all failures are Form-A bracket-shape errors, never semantics. Driver + raw logs updated.
This commit is contained in:
@@ -7,14 +7,16 @@ AILang Form-A, given only the Form-A mini-spec (`rendered/ail.md`) + the
|
||||
Series library API — *not* the reference solution. Multi-turn (≤6), each
|
||||
attempt run through `ail check` → `ail run`, the diagnostic fed back on
|
||||
failure. Driver: `sma_probe.py`. Raw per-turn logs: `sma-probe-run1.md`,
|
||||
`sma-probe-run2.md`.
|
||||
`sma-probe-run2.md`, `sma-probe-run3.md`. Three runs, escalating the amount
|
||||
of reference material handed to the model.
|
||||
|
||||
**Expected stdout:** `3.0 / 5.33333 / 5.66667 / 5.33333` (window 3 over
|
||||
`[1.0, 5.0, 3.0, 8.0, 6.0, 2.0]`).
|
||||
|
||||
**Outcome: did NOT reach green in either run.** But the two runs are a clean
|
||||
progression — each sharpening of the API doc moved the model exactly one
|
||||
hurdle forward.
|
||||
**Outcome: did NOT reach green in any of the three runs.** Runs 1→2 were a
|
||||
clean progression (each sharpening moved the model one hurdle forward); run 3
|
||||
inverted it — more material made the model degenerate. See the Verdict at the
|
||||
bottom.
|
||||
|
||||
## Run 1 — API doc said `(new Series (con Float) N)`
|
||||
|
||||
@@ -60,11 +62,45 @@ hurdle forward.
|
||||
foreign model: it gets the program logic but stumbles on the bracket
|
||||
discipline.
|
||||
|
||||
## Did not pursue
|
||||
## Run 3 — fully equipped: complete construct grammar + mini-spec + real Series library source
|
||||
|
||||
A run 3 with the bracketing/`push`-arity spelled out would likely clear the
|
||||
next hurdle too — but at that point the prompt is dictating the solution
|
||||
shape rather than measuring what the model produces. Stopped here; the
|
||||
finding (clears hurdles only with per-quirk guidance, never converges from
|
||||
diagnostics) is robust across the two runs. A stronger-guidance run, or a
|
||||
different model, is a deliberate next step, not an automatic one.
|
||||
The hypothesis was that the bracket failures of runs 1–2 would clear if Qwen
|
||||
had the formal grammar. The opposite happened. With the larger context
|
||||
(~80k prompt tokens: a distilled Form-A construct grammar with explicit
|
||||
arity rules + `rendered/ail.md` + the verbatim `series/source.ail`), Qwen
|
||||
**degenerated**: every one of the 6 turns produced an identical pathological
|
||||
`(seq (seq (seq …)))` tower — **321 open parens vs 10 close, identical bytes
|
||||
each turn** — with whitespace indentation exploding to hundreds of spaces per
|
||||
line. It never closes the tower; it just runs out of tokens (`unexpected end
|
||||
of input`). Raw log: `sma-probe-run3.md`.
|
||||
|
||||
A confound check (re-run with `max_tokens` doubled to 4000) did **not** yield
|
||||
a complete program — it produced HTTP 413 (request too large): with more
|
||||
budget Qwen emits an even larger degenerate output, so the accumulated
|
||||
message history blows past the endpoint's size limit. That confirms
|
||||
degeneration rather than truncation.
|
||||
|
||||
So the extra equipment made the model **worse**, not better. The best run was
|
||||
the middle one (run 2): just enough guidance to clear the `new` quirk,
|
||||
without the context overload that tipped run 3 into a repetition loop.
|
||||
|
||||
## Verdict (n=1 model, single session)
|
||||
|
||||
- **Qwen3-Coder-Next did not write the SMA in any configuration.** Its only
|
||||
semantically-correct attempt (run 2) still failed on S-expression
|
||||
bracketing.
|
||||
- **More reference material did not help — it hurt.** The fully-equipped run
|
||||
degenerated into a non-terminating `seq` tower; the lean run got the
|
||||
algorithm right. This is the sharpest finding: for this model on Form-A,
|
||||
context volume past a point is actively harmful.
|
||||
- **No convergence from compiler feedback in any run.** Every run repeated
|
||||
its failure byte-for-byte (or shape-for-shape) across all 6 turns; the
|
||||
retry-with-diagnostic loop never moved the model.
|
||||
- **All failures are Form-A *shape* errors** (term-head confusion, paren
|
||||
imbalance, repetition), never semantics. The bracket discipline of the
|
||||
fully-parenthesised authoring surface is the real wall for this model.
|
||||
|
||||
Natural next steps (deliberate, not automatic): a different model class via
|
||||
the same harness; or testing whether the JSON authoring form — which removes
|
||||
the human-style paren-counting burden — changes the bracket-failure picture
|
||||
for the same model.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -21,32 +21,64 @@ REPO = Path(__file__).resolve().parents[2]
|
||||
AIL = os.environ.get("AIL_BIN", str(REPO / "target/debug/ail"))
|
||||
SPEC = (REPO / "experiments/2026-05-12-cross-model-authoring/rendered/ail.md").read_text()
|
||||
EXPECTED = "3.0\n5.33333\n5.66667\n5.33333\n"
|
||||
DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md"
|
||||
DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md"
|
||||
|
||||
SERIES_API = """\
|
||||
## Series library API
|
||||
GRAMMAR = """\
|
||||
## AILang Form-A grammar (complete construct grammar)
|
||||
|
||||
`Series a` is a bounded ring buffer (financial indexing: index 0 = newest).
|
||||
The element type `a` is `Int` or `Float`. Available operations:
|
||||
Every program is fully-parenthesised S-expressions. ARITY IS FIXED PER HEAD —
|
||||
count your parens. `seq` takes EXACTLY 2 subterms; `if` EXACTLY 3; `let`
|
||||
EXACTLY 3.
|
||||
|
||||
- `new` is a built-in TERM HEAD (like `let`, `if`, `match`, `app`) — NOT an
|
||||
ordinary function. Do NOT write `(app new ...)`. To allocate an empty Float
|
||||
Series with capacity (lookback) `N`, write the term `(new Series (con Float) N)`
|
||||
directly. It evaluates to `(own (con Series (con Float)))`. For example, to
|
||||
bind one: `(let s (new Series (con Float) 3) <body-using-s>)`.
|
||||
- `Series.push : (own (con Series a)) (own a) -> (own (con Series a))` —
|
||||
append a value, evicting the oldest element when full.
|
||||
- `Series.at : (borrow (con Series a)) (own (con Int)) -> (own a)` — read the
|
||||
element `i` positions back from newest (`i=0` is the most recent push).
|
||||
- `Series.len : (borrow (con Series a)) -> (own (con Int))` — number of live
|
||||
elements (saturates at the lookback).
|
||||
- `Series.total_count : (borrow (con Series a)) -> (own (con Int))` — monotone
|
||||
count of all pushes ever (does NOT saturate).
|
||||
Module: (module NAME DECL*)
|
||||
Decl:
|
||||
(data NAME (vars V*)? (doc S)? (ctor CNAME FIELDTYPE*)*)
|
||||
(fn NAME (doc S)? (type TYPE) (params P*) (body TERM))
|
||||
Type:
|
||||
(con NAME TYPEARG*) ; type constructor: (con Int), (con Series (con Float))
|
||||
V ; a bare identifier is a type variable, e.g. a
|
||||
(fn-type (params PT*) (ret PT) (effects E*)?)
|
||||
(forall (vars V*) TYPE)
|
||||
(own TYPE) | (borrow TYPE) ; parameter / return MODE wrappers (mandatory)
|
||||
Term (head forms):
|
||||
(app FN ARG*) ; call FN with N value args
|
||||
(tail-app FN ARG*) ; tail call (use for recursion)
|
||||
(term-ctor TYPENAME CNAME ARG*) ; build an ADT value
|
||||
(new TYPENAME TYPEARG* VALARG*) ; BUILT-IN allocator term (NOT a function, never (app new ...)):
|
||||
; (new Series (con Float) 3)
|
||||
(if COND THEN ELSE) ; exactly 3 subterms
|
||||
(let NAME VALUE BODY) ; exactly 3: bind NAME=VALUE, then BODY
|
||||
(let-rec NAME (type T) (params P*) (body B) IN) ; local recursive fn, then IN
|
||||
(match SCRUT (case PAT ARM)*) ; one or more arms
|
||||
(seq LHS RHS) ; EXACTLY 2; for >2 effects, NEST: (seq A (seq B C))
|
||||
(do OP ARG*) ; effect op: (do io/print_str "\\n")
|
||||
(loop (NAME TYPE INIT)* BODY) ; typed loop binders, then body
|
||||
(recur ARG*) ; re-enter the enclosing loop
|
||||
literals: 42 (Int) 1.0 (Float) "txt" (Str) true | false (Bool)
|
||||
NAME ; a bare identifier is a variable reference
|
||||
Pattern:
|
||||
(pat-ctor CNAME FIELDPAT*) ; FIELDPAT = bare name (binds) or pat-wild
|
||||
(pat-lit LIT) | pat-wild
|
||||
Params: (params a b c) ; bare parameter names
|
||||
|
||||
Useful prelude functions: `+ - * /` (arithmetic, `forall a. (a,a)->a`),
|
||||
`eq`/`ge`/`gt`/`lt`/`le` (comparisons returning Bool), `int_to_float :
|
||||
(Int)->Float`, `print : forall a. (a) -> Unit !IO` (no trailing newline),
|
||||
`io/print_str : (Str) -> Unit !IO` (effect op, e.g. `(do io/print_str "\\n")`).
|
||||
Prelude functions you will need: `+ - * /` (`forall a. (a,a)->a`),
|
||||
`eq ge gt lt le` (comparisons -> Bool), `int_to_float : (Int)->Float`,
|
||||
`print : forall a. (a) -> Unit !IO` (NO trailing newline; emit one with
|
||||
`(do io/print_str "\\n")`).
|
||||
"""
|
||||
|
||||
SERIES_LIB = """\
|
||||
## Series library reference (this module already exists — CALL these functions, do not redefine them)
|
||||
|
||||
`Series a` is a bounded ring buffer; element type `a` is `Int` or `Float`;
|
||||
financial indexing (index 0 = newest). The library source (its public fns and
|
||||
their exact types/behaviour):
|
||||
|
||||
""" + (REPO / "crates/ailang-kernel/src/series/source.ail").read_text() + """
|
||||
|
||||
Call its functions as `Series.new`, `Series.push`, `Series.at`, `Series.len`,
|
||||
`Series.total_count`. Allocate a fresh one with the `new` term:
|
||||
`(new Series (con Float) N)`.
|
||||
"""
|
||||
|
||||
TASK = """\
|
||||
@@ -80,7 +112,7 @@ def extract_code(resp: str) -> str:
|
||||
|
||||
def call(messages) -> tuple[str, dict]:
|
||||
body = json.dumps({"model": MODEL, "messages": messages,
|
||||
"temperature": 0.2, "max_tokens": 2000}).encode()
|
||||
"temperature": 0.2, "max_tokens": 4000}).encode()
|
||||
req = urllib.request.Request(ENDPOINT, data=body, headers={
|
||||
"Authorization": f"Bearer {os.environ['IONOS_API_TOKEN']}",
|
||||
"Content-Type": "application/json"})
|
||||
@@ -104,7 +136,7 @@ def evaluate(code: str):
|
||||
return True, "green", run.stdout
|
||||
|
||||
def main():
|
||||
system = SPEC + "\n\n" + SERIES_API
|
||||
system = GRAMMAR + "\n\n" + SPEC + "\n\n" + SERIES_LIB
|
||||
messages = [{"role": "system", "content": system},
|
||||
{"role": "user", "content": TASK}]
|
||||
log = ["# SMA authoring probe — Qwen3-Coder-Next via IONOS\n",
|
||||
|
||||
Reference in New Issue
Block a user