diff --git a/experiments/2026-05-12-cross-model-authoring/format-findings.md b/experiments/2026-05-12-cross-model-authoring/format-findings.md index 1a2cb12..44366cf 100644 --- a/experiments/2026-05-12-cross-model-authoring/format-findings.md +++ b/experiments/2026-05-12-cross-model-authoring/format-findings.md @@ -87,24 +87,51 @@ tested, same ladder, each with a round-trip-verified converter: (L3 ADT, L4 data-def) the indentation structure breaks (e.g. a `con` field type lands as a `data` attribute — the model lost the indent level). -## Overall finding across all four formats +## M-expressions tested: middle of the pack + +McCarthy's `head[arg; arg]` notation — explicit structure (brackets + +semicolons) in the familiar function-call shape. Hypothesis: explicit + a +shape LLMs know cold should win. Result: 4–5/8 (model variance between two +runs), BELOW plain parens, ABOVE the bracket-free formats. Two watch-out +catches kept the measurement honest: +- A trailing `;` before `]` (Qwen wrote `let[s; v; ]`, a normal trailing + separator) — my parser was too strict and rejected it. Fixed to tolerate it + (round-trip preserved). Without this, M-expr would have looked unfairly bad. +- L4 then failed with **43 `[` vs 42 `]`** — a genuine Qwen bracket imbalance + (not truncation: 301 completion tokens). The same depth-tracking failure as + plain parens, just with `[` instead of `(`. + +Why M-expr is worse than plain parens despite being "explicit": swapping `()` +for `[]` does nothing for the balance burden (square is no easier than round), +and the semicolons *add* a second consistency requirement Qwen does not +reliably meet (the trailing-`;` slips). It is explicit, but with extra load, +not redundant cue. + +## Overall finding across all five formats | format | green | structure marking | |---|---|---| | **format 4 — annotated parens** | **7/8** | maximal-explicit (every paren + its depth) | | plain Form-A — parens | 6/8 | explicit (parens) | +| M-expressions `head[a; b]` | 4–5/8 | explicit, but `[]`+`;` adds a separator burden | | yamlish / indent (own parser) | 3/8 | implicit (indentation) | | YAML (library) | 2/8 | implicit + special-char collision | -The result is monotone and the opposite of the starting hypothesis: -**more explicit structure marking helps this model; less hurts.** Qwen's -weakness is structure-tracking at depth — and indentation makes it track the -nesting itself, which is just as error-prone as matching parens but *without* -the redundant cue. Parens are a help, not a burden; making them **more** -explicit (depth-annotated, format 4) is the win. The closing-token repetition -tail at L7 is a separate single-shot ceiling, not something a bracket-free -surface fixes — indeed bracket-free surfaces fail earlier, before that ceiling -is even reached. +The result is monotone and the opposite of the starting hypothesis. The +discriminator is sharper than "explicit vs implicit": it is **redundant cue vs +extra burden.** Qwen's weakness is structure-tracking at depth. What helps is a +marking that makes that tracking *easier without adding work* — depth-annotated +parens carry the nesting depth redundantly, so the model cannot lose its place +(format 4, best). What hurts is either (a) moving the tracking onto an implicit +channel it must maintain itself — indentation (yamlish/YAML, worst) — or (b) +adding a second thing to keep consistent on top of bracket balance — the +M-expr semicolons (middle). Plain parens sit in between: one explicit channel, +no redundancy, no extra burden. -**For an LLM-authored language, the lever is therefore redundant explicit -structure (e.g. depth-annotated brackets), not a lighter surface.** +The L7 closing-token repetition tail is a separate single-shot ceiling, not +something any surface here fixes — the bracket-free surfaces fail earlier, +before that ceiling is even reached. + +**For an LLM-authored language the lever is redundant, self-checking structure +(depth-annotated brackets), not a lighter surface and not a more familiar one +that carries extra bookkeeping.** diff --git a/experiments/2026-05-12-cross-model-authoring/qwen-mexpr.md b/experiments/2026-05-12-cross-model-authoring/qwen-mexpr.md new file mode 100644 index 0000000..f3b6ccb --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/qwen-mexpr.md @@ -0,0 +1,510 @@ +# Qwen format probe — M-expressions `head[a; b]` + +**Model:** Qwen/Qwen3-Coder-Next; single-shot; own literal parser (atoms verbatim). + +--- + +## L0_const — ✅ green + +tokens: prompt=616 completion=56 + +model output: +``` +module[ + m; + fn[ + answer; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Int] + ] + ] + ] + ]; + params[]; + body[ + 42 + ] + ] +] +``` + +converted Form-A: +``` +(module m (fn answer (type (fn-type (params) (ret (own (con Int))))) (params) (body 42))) +``` + + +## L1_arith — ✅ green + +tokens: prompt=618 completion=72 + +model output: +``` +module[ + m; + fn[ + triple; + type[ + fn-type[ + params[ + own[ + con[Int] + ] + ]; + ret[ + own[ + con[Int] + ] + ] + ] + ]; + params[x]; + body[ + app[*; x; 3] + ] + ] +] +``` + +converted Form-A: +``` +(module m (fn triple (type (fn-type (params (own (con Int))) (ret (own (con Int))))) (params x) (body (app * x 3)))) +``` + + +## L2_print — ✅ green + +tokens: prompt=613 completion=79 + +model output: +``` +module[ + m; + fn[ + main; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Unit] + ] + ]; + effects[IO] + ] + ]; + params[]; + body[ + seq[ + app[print; 7]; + do[io/print_str; "\n"] + ] + ] + ] +] +``` + +converted Form-A: +``` +(module m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (seq (app print 7) (do io/print_str "\n"))))) +``` + + +## L3_adt — ✅ green + +tokens: prompt=632 completion=122 + +model output: +``` +module[ + m; + data[ + Light; + ctor[Red]; + ctor[Green] + ]; + fn[ + code; + type[ + fn-type[ + params[ + own[ + con[Light] + ] + ]; + ret[ + own[ + con[Int] + ] + ] + ] + ]; + params[x]; + body[ + match[ + x; + case[ + pat-ctor[Red]; + 0 + ]; + case[ + pat-ctor[Green]; + 1 + ] + ] + ] + ] +] +``` + +converted Form-A: +``` +(module m (data Light (ctor Red) (ctor Green)) (fn code (type (fn-type (params (own (con Light))) (ret (own (con Int))))) (params x) (body (match x (case (pat-ctor Red) 0) (case (pat-ctor Green) 1))))) +``` + + +## L4_rec — ❌ convert + +tokens: prompt=644 completion=301 + +model output: +``` +module[ + m; + data[ + IntList; + ctor[Nil]; + ctor[ + Cons; + con[Int]; + con[IntList] + ] + ]; + fn[ + length; + type[ + fn-type[ + params[ + own[ + con[IntList] + ] + ]; + ret[ + own[ + con[Int] + ] + ] + ] + ]; + params[xs]; + body[ + match[ + xs; + case[ + pat-ctor[Nil]; + 0 + ]; + case[ + pat-ctor[Cons; h; t]; + app[ + +; + 1; + app[length; t] + ] + ] + ] + ] + ]; + fn[ + main; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Unit] + ] + ]; + effects[IO] + ] + ]; + params[]; + body[ + let[ + xs; + term-ctor[ + IntList; + Cons; + 1; + term-ctor[ + IntList; + Cons; + 2; + term-ctor[ + IntList; + Cons; + 3; + term-ctor[IntList; Nil] + ] + ]; + seq[ + app[ + print; + app[length; xs] + ]; + do[io/print_str; "\n"] + ] + ] + ] + ] +] +``` + + +**convert:** +``` +own parser rejected output: unterminated [ +``` + +## L5_series_len — ❌ check + +tokens: prompt=875 completion=182 + +model output: +``` +module[ + demo_m; + fn[ + main; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Unit] + ] + ]; + effects[IO] + ] + ]; + params[]; + body[ + let[ + s; + new[ + Series; + con[Float]; + 3 + ]; + ]; + let[ + s1; + app[Series.push; s; 1.0]; + ]; + let[ + s2; + app[Series.push; s1; 2.0]; + ]; + let[ + s3; + app[Series.push; s2; 3.0]; + ]; + seq[ + app[ + print; + app[Series.len; s3] + ]; + do[io/print_str; "\n"] + ] + ] + ] +] +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s1 (app Series.push s 1.0)) (let s2 (app Series.push s1 2.0)) (let s3 (app Series.push s2 3.0)) (seq (app print (app Series.len s3)) (do io/print_str "\n"))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + +## L6_series_at — ❌ check + +tokens: prompt=883 completion=190 + +model output: +``` +module[ + demo_m; + fn[ + main; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Unit] + ] + ]; + effects[IO] + ] + ]; + params[]; + body[ + let[ + s; + new[ + Series; + con[Float]; + 3 + ]; + ]; + let[ + s1; + app[Series.push; s; 1.0]; + ]; + let[ + s2; + app[Series.push; s1; 2.0]; + ]; + let[ + s3; + app[Series.push; s2; 3.0]; + ]; + seq[ + app[ + print; + app[ + Series.at; + s3; + 0 + ] + ]; + do[io/print_str; "\n"] + ] + ] + ] +] +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s1 (app Series.push s 1.0)) (let s2 (app Series.push s1 2.0)) (let s3 (app Series.push s2 3.0)) (seq (app print (app Series.at s3 0)) (do io/print_str "\n"))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + +## L7_sma — ❌ check + +tokens: prompt=966 completion=327 + +model output: +``` +module[ + demo_m; + fn[ + main; + type[ + fn-type[ + params[]; + ret[ + own[ + con[Unit] + ] + ]; + effects[IO] + ] + ]; + params[]; + body[ + let[ + s; + new[ + Series; + con[Float]; + 3 + ]; + let[ + s1; + app[Series.push; s; 1.0]; + ]; + let[ + s2; + app[Series.push; s1; 5.0]; + ]; + let[ + s3; + app[Series.push; s2; 3.0]; + ]; + seq[ + app[ + print; + app[ + /; + app[Series.sum; s3]; + 3.0 + ] + ]; + let[ + s4; + app[Series.push; s3; 8.0]; + seq[ + app[ + print; + app[ + /; + app[Series.sum; s4]; + 3.0 + ] + ]; + let[ + s5; + app[Series.push; s4; 6.0]; + seq[ + app[ + print; + app[ + /; + app[Series.sum; s5]; + 3.0 + ] + ]; + let[ + s6; + app[Series.push; s5; 2.0]; + ] + +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3) (let s1 (app Series.push s 1.0)) (let s2 (app Series.push s1 5.0)) (let s3 (app Series.push s2 3.0)) (seq (app print (app / (app Series.sum s3) 3.0)) (let s4 (app Series.push s3 8.0) (seq (app print (app / (app Series.sum s4) 3.0)) (let s5 (app Series.push s4 6.0) (seq (app print (app / (app Series.sum s5) 3.0)) (let s6 (app Series.push s5 2.0))))))))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 166 +``` + + +--- +## Ladder + +- L0_const: ✅ green +- L1_arith: ✅ green +- L2_print: ✅ green +- L3_adt: ✅ green +- L4_rec: ❌ convert +- L5_series_len: ❌ check +- L6_series_at: ❌ check +- L7_sma: ❌ check \ No newline at end of file diff --git a/experiments/2026-05-12-cross-model-authoring/qwen_mexpr.py b/experiments/2026-05-12-cross-model-authoring/qwen_mexpr.py new file mode 100644 index 0000000..5070f51 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/qwen_mexpr.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Format probe — M-expressions. McCarthy's original Lisp notation: +`head[arg1; arg2; ...]` — explicit structure (brackets + semicolons) but in +the function-call shape `f[x; y]` that LLMs know from C/Python/JS. Tests the +"more explicit structure helps" finding in a familiar guise. + +Own literal parser (atoms verbatim, whitespace-agnostic), round-trip-verified +before judging output. Same ablation ladder. + +Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent. +""" +from __future__ import annotations +import json, os, re, subprocess, urllib.request +from pathlib import Path + +ENDPOINT="https://openai.inference.de-txl.ionos.com/v1/chat/completions" +MODEL="Qwen/Qwen3-Coder-Next" +REPO=Path(__file__).resolve().parents[2] +AIL=os.environ.get("AIL_BIN", str(REPO/"target/debug/ail")) +DOC=REPO/"experiments/2026-05-12-cross-model-authoring/qwen-mexpr.md" + +def tokenize(s): + t=[];i=0 + while i=0 else body.strip() + +def evaluate(mx,expected): + try: plain=from_mexpr(mx) + except Exception as e: return "convert",f"own parser rejected output: {e}","" + m=re.search(r"\(module\s+([A-Za-z_][A-Za-z0-9_]*)",plain) + name=m.group(1) if m else "m" + f=Path(f"/tmp/abl/{name}.ail"); f.parent.mkdir(exist_ok=True); f.write_text(plain+"\n") + chk=subprocess.run([AIL,"check",str(f)],capture_output=True,text=True) + if chk.returncode!=0: return "check",(chk.stderr or chk.stdout).strip(),plain + if expected is None: return "green","(check-only)",plain + run=subprocess.run([AIL,"run",str(f)],capture_output=True,text=True) + if run.returncode!=0: return "run",(run.stderr or run.stdout).strip(),plain + if run.stdout!=expected: return "output",f"got {run.stdout!r}, expected {expected!r}",plain + return "green",run.stdout,plain + +def main(): + log=["# Qwen format probe — M-expressions `head[a; b]`\n", + f"**Model:** {MODEL}; single-shot; own literal parser (atoms verbatim).\n","---\n"] + summary=[] + for sid,ctx,task,exp in STAGES: + resp,usage=call(ctx,task); mx=extract(resp) + stage,fb,plain=evaluate(mx,exp) + ok=stage=="green"; mark="✅ green" if ok else f"❌ {stage}" + summary.append((sid,mark)); print(f"{sid}: {mark}") + log+=[f"## {sid} — {mark}", + f"\ntokens: prompt={usage.get('prompt_tokens','?')} completion={usage.get('completion_tokens','?')}\n", + "model output:\n```\n"+mx[:1400]+"\n```\n", + ("converted Form-A:\n```\n"+plain[:1100]+"\n```\n" if plain else ""), + ("" if ok else f"**{stage}:**\n```\n{fb[:600]}\n```\n")] + log+=["\n---\n## Ladder\n"]+[f"- {s}: {m}" for s,m in summary] + DOC.write_text("\n".join(log)); print("doc →",DOC) + +if __name__=="__main__": + main()