From 8b7e683cc9b0b23e10b38f19f990ef639b9ef9ef Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 2 Jun 2026 17:49:23 +0200 Subject: [PATCH] =?UTF-8?q?experiment(cma):=20SMA=20probe=20run=203=20?= =?UTF-8?q?=E2=80=94=20full=20grammar+spec+lib=20makes=20Qwen=20degenerate?= =?UTF-8?q?=20(refs=20#68)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third run of the SMA authoring probe, this time with Qwen fully equipped: a distilled complete Form-A construct grammar (with explicit arity/paren rules), the mini-spec, and the verbatim Series library source. Hypothesis: the formal grammar clears the bracket failures of runs 1-2. It did the opposite. With ~80k prompt tokens the model degenerated into an identical pathological (seq (seq (seq ...))) tower every turn (321 open parens vs 10 close, byte-identical across all 6 turns, indentation exploding). Doubling max_tokens to confirm it was not mere truncation produced HTTP 413 (request too large) — more budget yields an even larger degenerate output, confirming degeneration. Verdict (n=1 model): Qwen3-Coder-Next wrote no working SMA in any of three configurations; more reference material made it WORSE, not better (the lean run 2 got the algorithm right, the fully-equipped run 3 looped); it never converged from ail-check feedback in any run; all failures are Form-A bracket-shape errors, never semantics. Driver + raw logs updated. --- .../sma-probe-progress.md | 58 +- .../sma-probe-run3.md | 1971 +++++++++++++++++ .../sma_probe.py | 80 +- 3 files changed, 2074 insertions(+), 35 deletions(-) create mode 100644 experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md diff --git a/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md b/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md index b2e7d5e..e6b3f83 100644 --- a/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md +++ b/experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md @@ -7,14 +7,16 @@ AILang Form-A, given only the Form-A mini-spec (`rendered/ail.md`) + the Series library API — *not* the reference solution. Multi-turn (≤6), each attempt run through `ail check` → `ail run`, the diagnostic fed back on failure. Driver: `sma_probe.py`. Raw per-turn logs: `sma-probe-run1.md`, -`sma-probe-run2.md`. +`sma-probe-run2.md`, `sma-probe-run3.md`. Three runs, escalating the amount +of reference material handed to the model. **Expected stdout:** `3.0 / 5.33333 / 5.66667 / 5.33333` (window 3 over `[1.0, 5.0, 3.0, 8.0, 6.0, 2.0]`). -**Outcome: did NOT reach green in either run.** But the two runs are a clean -progression — each sharpening of the API doc moved the model exactly one -hurdle forward. +**Outcome: did NOT reach green in any of the three runs.** Runs 1→2 were a +clean progression (each sharpening moved the model one hurdle forward); run 3 +inverted it — more material made the model degenerate. See the Verdict at the +bottom. ## Run 1 — API doc said `(new Series (con Float) N)` @@ -60,11 +62,45 @@ hurdle forward. foreign model: it gets the program logic but stumbles on the bracket discipline. -## Did not pursue +## Run 3 — fully equipped: complete construct grammar + mini-spec + real Series library source -A run 3 with the bracketing/`push`-arity spelled out would likely clear the -next hurdle too — but at that point the prompt is dictating the solution -shape rather than measuring what the model produces. Stopped here; the -finding (clears hurdles only with per-quirk guidance, never converges from -diagnostics) is robust across the two runs. A stronger-guidance run, or a -different model, is a deliberate next step, not an automatic one. +The hypothesis was that the bracket failures of runs 1–2 would clear if Qwen +had the formal grammar. The opposite happened. With the larger context +(~80k prompt tokens: a distilled Form-A construct grammar with explicit +arity rules + `rendered/ail.md` + the verbatim `series/source.ail`), Qwen +**degenerated**: every one of the 6 turns produced an identical pathological +`(seq (seq (seq …)))` tower — **321 open parens vs 10 close, identical bytes +each turn** — with whitespace indentation exploding to hundreds of spaces per +line. It never closes the tower; it just runs out of tokens (`unexpected end +of input`). Raw log: `sma-probe-run3.md`. + +A confound check (re-run with `max_tokens` doubled to 4000) did **not** yield +a complete program — it produced HTTP 413 (request too large): with more +budget Qwen emits an even larger degenerate output, so the accumulated +message history blows past the endpoint's size limit. That confirms +degeneration rather than truncation. + +So the extra equipment made the model **worse**, not better. The best run was +the middle one (run 2): just enough guidance to clear the `new` quirk, +without the context overload that tipped run 3 into a repetition loop. + +## Verdict (n=1 model, single session) + +- **Qwen3-Coder-Next did not write the SMA in any configuration.** Its only + semantically-correct attempt (run 2) still failed on S-expression + bracketing. +- **More reference material did not help — it hurt.** The fully-equipped run + degenerated into a non-terminating `seq` tower; the lean run got the + algorithm right. This is the sharpest finding: for this model on Form-A, + context volume past a point is actively harmful. +- **No convergence from compiler feedback in any run.** Every run repeated + its failure byte-for-byte (or shape-for-shape) across all 6 turns; the + retry-with-diagnostic loop never moved the model. +- **All failures are Form-A *shape* errors** (term-head confusion, paren + imbalance, repetition), never semantics. The bracket discipline of the + fully-parenthesised authoring surface is the real wall for this model. + +Natural next steps (deliberate, not automatic): a different model class via +the same harness; or testing whether the JSON authoring form — which removes +the human-style paren-counting burden — changes the bracket-failure picture +for the same model. diff --git a/experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md b/experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md new file mode 100644 index 0000000..88e59d7 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md @@ -0,0 +1,1971 @@ +# SMA authoring probe — Qwen3-Coder-Next via IONOS + +**Model:** Qwen/Qwen3-Coder-Next +**Max turns:** 6 +**Task:** write the Series SMA worked example in AILang Form-A, given the Form-A mini-spec + Series API only (not the reference). + +**Expected stdout:** `'3.0\n5.33333\n5.66667\n5.33333\n'` + +--- + +## Turn 1 — `check` ❌ + +Tokens this turn: prompt=8060, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + +## Turn 2 — `check` ❌ + +Tokens this turn: prompt=10111, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + +## Turn 3 — `check` ❌ + +Tokens this turn: prompt=12162, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + +## Turn 4 — `check` ❌ + +Tokens this turn: prompt=14213, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + +## Turn 5 — `check` ❌ + +Tokens this turn: prompt=16264, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + +## Turn 6 — `check` ❌ + +Tokens this turn: prompt=18315, completion=2000 + +``` +(module sma + (fn main + (doc "Compute a 3-point simple moving average over a stream of 6 floats, printing the average after each push once the window is full.") + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let s (app Series/new 3) + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq + (seq +``` + +**`ail check` feedback:** +``` +error: [surface-parse-error] parse error: unexpected end of input, expected term +``` + + +--- + +**Outcome:** did NOT reach green after 6 turn(s). +**Total tokens:** prompt=79125, completion=12000. diff --git a/experiments/2026-05-12-cross-model-authoring/sma_probe.py b/experiments/2026-05-12-cross-model-authoring/sma_probe.py index 0ad9761..0b5b3f0 100644 --- a/experiments/2026-05-12-cross-model-authoring/sma_probe.py +++ b/experiments/2026-05-12-cross-model-authoring/sma_probe.py @@ -21,32 +21,64 @@ REPO = Path(__file__).resolve().parents[2] AIL = os.environ.get("AIL_BIN", str(REPO / "target/debug/ail")) SPEC = (REPO / "experiments/2026-05-12-cross-model-authoring/rendered/ail.md").read_text() EXPECTED = "3.0\n5.33333\n5.66667\n5.33333\n" -DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-progress.md" +DOC = REPO / "experiments/2026-05-12-cross-model-authoring/sma-probe-run3.md" -SERIES_API = """\ -## Series library API +GRAMMAR = """\ +## AILang Form-A grammar (complete construct grammar) -`Series a` is a bounded ring buffer (financial indexing: index 0 = newest). -The element type `a` is `Int` or `Float`. Available operations: +Every program is fully-parenthesised S-expressions. ARITY IS FIXED PER HEAD — +count your parens. `seq` takes EXACTLY 2 subterms; `if` EXACTLY 3; `let` +EXACTLY 3. -- `new` is a built-in TERM HEAD (like `let`, `if`, `match`, `app`) — NOT an - ordinary function. Do NOT write `(app new ...)`. To allocate an empty Float - Series with capacity (lookback) `N`, write the term `(new Series (con Float) N)` - directly. It evaluates to `(own (con Series (con Float)))`. For example, to - bind one: `(let s (new Series (con Float) 3) )`. -- `Series.push : (own (con Series a)) (own a) -> (own (con Series a))` — - append a value, evicting the oldest element when full. -- `Series.at : (borrow (con Series a)) (own (con Int)) -> (own a)` — read the - element `i` positions back from newest (`i=0` is the most recent push). -- `Series.len : (borrow (con Series a)) -> (own (con Int))` — number of live - elements (saturates at the lookback). -- `Series.total_count : (borrow (con Series a)) -> (own (con Int))` — monotone - count of all pushes ever (does NOT saturate). +Module: (module NAME DECL*) +Decl: + (data NAME (vars V*)? (doc S)? (ctor CNAME FIELDTYPE*)*) + (fn NAME (doc S)? (type TYPE) (params P*) (body TERM)) +Type: + (con NAME TYPEARG*) ; type constructor: (con Int), (con Series (con Float)) + V ; a bare identifier is a type variable, e.g. a + (fn-type (params PT*) (ret PT) (effects E*)?) + (forall (vars V*) TYPE) + (own TYPE) | (borrow TYPE) ; parameter / return MODE wrappers (mandatory) +Term (head forms): + (app FN ARG*) ; call FN with N value args + (tail-app FN ARG*) ; tail call (use for recursion) + (term-ctor TYPENAME CNAME ARG*) ; build an ADT value + (new TYPENAME TYPEARG* VALARG*) ; BUILT-IN allocator term (NOT a function, never (app new ...)): + ; (new Series (con Float) 3) + (if COND THEN ELSE) ; exactly 3 subterms + (let NAME VALUE BODY) ; exactly 3: bind NAME=VALUE, then BODY + (let-rec NAME (type T) (params P*) (body B) IN) ; local recursive fn, then IN + (match SCRUT (case PAT ARM)*) ; one or more arms + (seq LHS RHS) ; EXACTLY 2; for >2 effects, NEST: (seq A (seq B C)) + (do OP ARG*) ; effect op: (do io/print_str "\\n") + (loop (NAME TYPE INIT)* BODY) ; typed loop binders, then body + (recur ARG*) ; re-enter the enclosing loop + literals: 42 (Int) 1.0 (Float) "txt" (Str) true | false (Bool) + NAME ; a bare identifier is a variable reference +Pattern: + (pat-ctor CNAME FIELDPAT*) ; FIELDPAT = bare name (binds) or pat-wild + (pat-lit LIT) | pat-wild +Params: (params a b c) ; bare parameter names -Useful prelude functions: `+ - * /` (arithmetic, `forall a. (a,a)->a`), -`eq`/`ge`/`gt`/`lt`/`le` (comparisons returning Bool), `int_to_float : -(Int)->Float`, `print : forall a. (a) -> Unit !IO` (no trailing newline), -`io/print_str : (Str) -> Unit !IO` (effect op, e.g. `(do io/print_str "\\n")`). +Prelude functions you will need: `+ - * /` (`forall a. (a,a)->a`), +`eq ge gt lt le` (comparisons -> Bool), `int_to_float : (Int)->Float`, +`print : forall a. (a) -> Unit !IO` (NO trailing newline; emit one with +`(do io/print_str "\\n")`). +""" + +SERIES_LIB = """\ +## Series library reference (this module already exists — CALL these functions, do not redefine them) + +`Series a` is a bounded ring buffer; element type `a` is `Int` or `Float`; +financial indexing (index 0 = newest). The library source (its public fns and +their exact types/behaviour): + +""" + (REPO / "crates/ailang-kernel/src/series/source.ail").read_text() + """ + +Call its functions as `Series.new`, `Series.push`, `Series.at`, `Series.len`, +`Series.total_count`. Allocate a fresh one with the `new` term: +`(new Series (con Float) N)`. """ TASK = """\ @@ -80,7 +112,7 @@ def extract_code(resp: str) -> str: def call(messages) -> tuple[str, dict]: body = json.dumps({"model": MODEL, "messages": messages, - "temperature": 0.2, "max_tokens": 2000}).encode() + "temperature": 0.2, "max_tokens": 4000}).encode() req = urllib.request.Request(ENDPOINT, data=body, headers={ "Authorization": f"Bearer {os.environ['IONOS_API_TOKEN']}", "Content-Type": "application/json"}) @@ -104,7 +136,7 @@ def evaluate(code: str): return True, "green", run.stdout def main(): - system = SPEC + "\n\n" + SERIES_API + system = GRAMMAR + "\n\n" + SPEC + "\n\n" + SERIES_LIB messages = [{"role": "system", "content": system}, {"role": "user", "content": TASK}] log = ["# SMA authoring probe — Qwen3-Coder-Next via IONOS\n",