diff --git a/experiments/2026-05-12-cross-model-authoring/format-findings.md b/experiments/2026-05-12-cross-model-authoring/format-findings.md index 1c5e6cc..1a2cb12 100644 --- a/experiments/2026-05-12-cross-model-authoring/format-findings.md +++ b/experiments/2026-05-12-cross-model-authoring/format-findings.md @@ -66,7 +66,45 @@ still failed — now on two model-behaviour limits, not surface encoding: making `seq` n-ary) and (b) a single-shot complexity ceiling where the model simplifies and falls into a closing-token repetition loop. - The repetition tail is paren-driven (`#0)` repeated). The natural next test - is a **bracket-free** format (indent, or keyword-delimited blocks) where - there is no closing token to loop on — that directly targets the one limit - format 4 does not remove. It needs a new round-trip-verified converter - (an offside / keyword parser), so it is a deliberate next step, not run here. + was a **bracket-free** format — run below. + +## Bracket-free formats tested: indent / YAML — they made it WORSE + +The hypothesis was that removing the closing token would remove the repetition +tail. It did the opposite — bracket-free formats are clearly worse. Two were +tested, same ladder, each with a round-trip-verified converter: + +- **YAML via PyYAML** (`qwen-yaml.md`): 2/8. First lesson was about the + *instrument*, not the model: a real YAML library auto-quotes and gives `*`, + `-`, `true` special meaning — and the AILang operators collide. Qwen wrote + `- *` (multiply), valid as an atom but a YAML alias marker, so the library + crashed. **You cannot use an auto-quoting parser for an operator-rich + vocabulary.** Discarded. +- **yamlish, own literal parser** (`qwen-yamlish.md`): 3/8. Re-built with a + hand-written indent parser that takes atoms verbatim (no quoting, no alias) — + `- *` round-trips fine. With the instrument fixed, the model still only + reached 3/8: trivial programs (L0–L2) pass, but at the first real nesting + (L3 ADT, L4 data-def) the indentation structure breaks (e.g. a `con` field + type lands as a `data` attribute — the model lost the indent level). + +## Overall finding across all four formats + +| format | green | structure marking | +|---|---|---| +| **format 4 — annotated parens** | **7/8** | maximal-explicit (every paren + its depth) | +| plain Form-A — parens | 6/8 | explicit (parens) | +| yamlish / indent (own parser) | 3/8 | implicit (indentation) | +| YAML (library) | 2/8 | implicit + special-char collision | + +The result is monotone and the opposite of the starting hypothesis: +**more explicit structure marking helps this model; less hurts.** Qwen's +weakness is structure-tracking at depth — and indentation makes it track the +nesting itself, which is just as error-prone as matching parens but *without* +the redundant cue. Parens are a help, not a burden; making them **more** +explicit (depth-annotated, format 4) is the win. The closing-token repetition +tail at L7 is a separate single-shot ceiling, not something a bracket-free +surface fixes — indeed bracket-free surfaces fail earlier, before that ceiling +is even reached. + +**For an LLM-authored language, the lever is therefore redundant explicit +structure (e.g. depth-annotated brackets), not a lighter surface.** diff --git a/experiments/2026-05-12-cross-model-authoring/qwen-yaml.md b/experiments/2026-05-12-cross-model-authoring/qwen-yaml.md new file mode 100644 index 0000000..9e02014 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/qwen-yaml.md @@ -0,0 +1,542 @@ +# Qwen format probe — YAML surface + +**Model:** Qwen/Qwen3-Coder-Next; single-shot; YAML dialect; converted via verified PyYAML round-trip. + +--- + +## L0_const — ✅ green + +tokens: prompt=649 completion=58 + +model YAML: +```yaml +module: +- m +- fn: + - answer + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Int + - params: [] + - body: + - '42' +``` + +converted Form-A: +``` +(module m (fn answer (type (fn-type (params) (ret (own (con Int))))) (params) (body 42))) +``` + + +## L1_arith — ❌ yaml-convert + +tokens: prompt=651 completion=87 + +model YAML: +```yaml +module: +- demo_m +- fn: + - triple + - type: + - fn-type: + - params: + - own: + - con: + - Int + - ret: + - own: + - con: + - Int + - params: + - x + - body: + - app: + - app: + - * + - x + - '3' +``` + + +**yaml-convert:** +``` +converter rejected model YAML: while scanning an alias + in "", line 20, column 11: + - * + ^ +expected alphabetic or numeric character, but found '\n' + in "", line 20, column 12: + - * + ^ +``` + +## L2_print — ❌ yaml-convert + +tokens: prompt=646 completion=94 + +model YAML: +```yaml +module: +- demo_c +- fn: + - m + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: [] + - body: + - seq: + - app: + - print + - '7 + - do: + - io/print_str + - '"\n"' +``` + + +**yaml-convert:** +``` +converter rejected model YAML: while parsing a block mapping + in "", line 17, column 9: + - app: + ^ +expected , but found '' + in "", line 22, column 12: + - '"\n"' + ^ +``` + +## L3_adt — ✅ green + +tokens: prompt=665 completion=136 + +model YAML: +```yaml +module: +- demo_m +- data: + - Light + - ctor: + - Red + - ctor: + - Green +- fn: + - code + - type: + - fn-type: + - params: + - own: + - con: + - Light + - ret: + - own: + - con: + - Int + - params: + - l + - body: + - match: + - l + - case: + - pat-ctor: + - Red + - '0' + - case: + - pat-ctor: + - Green + - '1' +``` + +converted Form-A: +``` +(module demo_m (data Light (ctor Red) (ctor Green)) (fn code (type (fn-type (params (own (con Light))) (ret (own (con Int))))) (params l) (body (match l (case (pat-ctor Red) 0) (case (pat-ctor Green) 1))))) +``` + + +## L4_rec — ❌ yaml-convert + +tokens: prompt=677 completion=359 + +model YAML: +```yaml +module: +- demo_m +- data: + - IntList + - ctor: + - Nil + - ctor: + - Cons + - con: + - Int + - con: + - IntList +- fn: + - length + - type: + - fn-type: + - params: + - own: + - con: + - IntList + - ret: + - own: + - con: + - Int + - params: + - xs + - body: + - match: + - xs + - case: + - pat-ctor: + - Nil + - '0' + - case: + - pat-ctor: + - Cons + - h + - t + - app: + - + + - '1' + - app: + - length + - t +- fn: + - main + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: [] + - body: + - let: + - xs + - term-ctor: + - IntList + - Cons + - '1' + - term-ctor: + - IntList + - Cons + - '2' + - term-ctor: + - IntList + - Cons + - '3' + - term-ctor: + - IntList + - Nil + - seq: + - app: + - print + - app: + - length + - xs + - do: + - io/print_str + - '"\n" +``` + + +**yaml-convert:** +``` +converter rejected model YAML: while scanning a quoted scalar + in "", line 83, column 11: + - '"\n" + ^ +found unexpected end of stream + in "", line 83, column 16: + - '"\n" + ^ +``` + +## L5_series_len — ❌ check + +tokens: prompt=945 completion=215 + +model YAML: +```yaml +module: +- demo_m +- fn: + - main + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: [] + - body: + - let: + - s + - new: + - Series + - con: + - Float + - '3' + - let: + - s + - app: + - Series.push + - s + - '1.0' + - let: + - s + - app: + - Series.push + - s + - '2.0' + - let: + - s + - app: + - Series.push + - s + - '3.0' + - seq: + - app: + - print + - app: + - Series.len + - s + - do: + - io/print_str + - '"\n"' +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 2.0)) (let s (app Series.push s 3.0)) (seq (app print) (app Series.len) s) (do io/print_str "\n")))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + +## L6_series_at — ❌ check + +tokens: prompt=953 completion=226 + +model YAML: +```yaml +module: +- demo_m +- fn: + - main + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: [] + - body: + - let: + - s + - new: + - Series + - con: + - Float + - '3' + - let: + - s1 + - app: + - Series.push + - s + - '1.0' + - let: + - s2 + - app: + - Series.push + - s1 + - '2.0' + - let: + - s3 + - app: + - Series.push + - s2 + - '3.0' + - seq: + - app: + - print + - app: + - Series.at + - s3 + - '0' + - do: + - io/print_str + - '"\n"' +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s1 (app Series.push s 1.0)) (let s2 (app Series.push s1 2.0)) (let s3 (app Series.push s2 3.0)) (seq (app print) (app Series.at) s3 0) (do io/print_str "\n")))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + +## L7_sma — ❌ check + +tokens: prompt=1036 completion=469 + +model YAML: +```yaml +module: +- demo_m +- fn: + - main + - type: + - fn-type: + - params: [] + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: [] + - body: + - let: + - s + - new: + - Series + - con: + - Float + - '3' + - let: + - s + - app: + - Series.push + - s + - '1.0' + - let: + - s + - app: + - Series.push + - s + - '5.0' + - seq: + - app: + - print + - app: + - / + - app: + - Series.sum + - s + - '3.0' + - let: + - s + - app: + - Series.push + - s + - '3.0' + - seq: + - app: + - print + - app: + - / + - app: + - Series.sum + - s + - '3.0' + - let: + - s + - app: + - Series.push + - s + - '8.0' + - seq: + - app: + - print + - app: + - / + - app: + - Series.sum + - s + - '3.0' + - let: + - s + - app: + - Series.push + - s + - '6.0' + - seq: + - app: + - print + - app: + - / + - app: + - Series.sum + - s + - '3.0' + - let: + - s + - app: + - Series.push + - s + - '2.0' + - seq: + - app: + - print + - app: + - / + - app: + - Series.sum + - s + - '3.0' + - do: + - io/print_str + - '"\n"' +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 5.0)) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 3.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 8.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 6.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 2.0) (seq (app print) (app /) (app Series.sum) s 3.0) (do io/print_str "\n")))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + + +--- +## Ladder + +- L0_const: ✅ green +- L1_arith: ❌ yaml-convert +- L2_print: ❌ yaml-convert +- L3_adt: ✅ green +- L4_rec: ❌ yaml-convert +- L5_series_len: ❌ check +- L6_series_at: ❌ check +- L7_sma: ❌ check \ No newline at end of file diff --git a/experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md b/experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md new file mode 100644 index 0000000..2cf4f74 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md @@ -0,0 +1,545 @@ +# Qwen format probe — yamlish (indented tree, own literal parser) + +**Model:** Qwen/Qwen3-Coder-Next; single-shot; atoms verbatim (no YAML lib, no quoting). + +--- + +## L0_const — ✅ green + +tokens: prompt=683 completion=127 + +model output: +``` +module: + - demo_c + - fn: + - answer + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Int + - params: + - body: + - 42 + - fn: + - m + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - app: + - print + - app: + - answer +``` + +converted Form-A: +``` +(module demo_c (fn answer (type (fn-type (params) (ret (own (con Int))))) (params) (body 42)) (fn m (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (app print (app answer))))) +``` + + +## L1_arith — ✅ green + +tokens: prompt=685 completion=84 + +model output: +``` +module: + - demo_m + - fn: + - triple + - type: + - fn-type: + - params: + - own: + - con: + - Int + - ret: + - own: + - con: + - Int + - params: + - x + - body: + - app: + - * + - x + - 3 +``` + +converted Form-A: +``` +(module demo_m (fn triple (type (fn-type (params (own (con Int))) (ret (own (con Int))))) (params x) (body (app * x 3)))) +``` + + +## L2_print — ✅ green + +tokens: prompt=680 completion=146 + +model output: +``` +module: + - demo_c + - fn: + - m + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Int + - params: + - body: + - 7 + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - seq: + - app: + - print + - app: + - m + - do: + - io/print_str + - "\n" +``` + +converted Form-A: +``` +(module demo_c (fn m (type (fn-type (params) (ret (own (con Int))))) (params) (body 7)) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (seq (app print (app m)) (do io/print_str "\n"))))) +``` + + +## L3_adt — ❌ check + +tokens: prompt=699 completion=241 + +model output: +``` +module: + - demo_c + - data: + - Light + - ctor: + - Red + - ctor: + - Green + - fn: + - code + - type: + - fn-type: + - params: + - own: + - con: + - Light + - ret: + - own: + - con: + - Int + - params: + - l + - body: + - match: + - l + - case: + - pat-ctor: + - Red + - 0 + - case: + - pat-ctor: + - Green + - 1 + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - seq: + - app: + - print + - app: + - code + - term-ctor: + - Light + - Red + - do: + - io/print_str + - "\n" +``` + +converted Form-A: +``` +(module demo_c (data Light (ctor Red) (ctor Green)) (fn code (type (fn-type (params (own (con Light))) (ret (own (con Int))))) (params l) (body (match l (case (pat-ctor Red) 0) (case (pat-ctor Green) 1)))) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (seq (app print (app code) (term-ctor Light Red)) (do io/print_str "\n"))))) +``` + +**check:** +``` +error: [arity-mismatch] main: arity mismatch for `print`: expected 1 args, got 2 +``` + +## L4_rec — ❌ check + +tokens: prompt=711 completion=360 + +model output: +``` +module: + - demo_m + - data: + - IntList + - ctor: + - Nil + - ctor: + - Cons + - con: + - Int + - con: + - IntList + - fn: + - length + - type: + - fn-type: + - params: + - own: + - con: + - IntList + - ret: + - own: + - con: + - Int + - params: + - xs + - body: + - match: + - xs + - case: + - pat-ctor: + - Nil + - 0 + - case: + - pat-ctor: + - Cons + - h + - t + - app: + - + + - 1 + - app: + - length + - t + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - let: + - xs + - term-ctor: + - IntList + - Cons + - 1 + - term-ctor: + - IntList + - Cons + - 2 + - term-ctor: + - IntList + - Cons + - 3 + - term-ctor: + - IntList + - Nil + - seq: + - app: + - print + - app: + - length + - xs + - do: + - io/print_str + - "\n" +``` + +converted Form-A: +``` +(module demo_m (data IntList (ctor Nil) (ctor Cons) (con Int) (con IntList)) (fn length (type (fn-type (params (own (con IntList))) (ret (own (con Int))))) (params xs) (body (match xs (case (pat-ctor Nil) 0) (case (pat-ctor Cons h t) (app + 1 (app length t)))))) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons) 2 (term-ctor IntList Cons) 3 (term-ctor IntList) Nil) (seq (app print) (app length) xs) (do io/print_str "\n"))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error in data-def: unknown data attribute `con`; expected `doc`, `ctor`, `drop-iterative`, or `param-in` at byte 52 +``` + +## L5_series_len — ❌ check + +tokens: prompt=975 completion=214 + +model output: +``` +module: + - demo_m + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - let: + - s + - new: + - Series + - con: + - Float + - 3 + - let: + - s + - app: + - Series.push + - s + - 1.0 + - let: + - s + - app: + - Series.push + - s + - 2.0 + - let: + - s + - app: + - Series.push + - s + - 3.0 + - seq: + - app: + - print + - app: + - Series.len + - s + - do: + - io/print_str + - "\n" +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 2.0)) (let s (app Series.push s 3.0)) (seq (app print (app Series.len) s)) (do io/print_str "\n")))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 134 +``` + +## L6_series_at — ❌ check + +tokens: prompt=983 completion=225 + +model output: +``` +module: + - demo_m + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - let: + - s + - new: + - Series + - con: + - Float + - 3 + - let: + - s1 + - app: + - Series.push + - s + - 1.0 + - let: + - s2 + - app: + - Series.push + - s1 + - 2.0 + - let: + - s3 + - app: + - Series.push + - s2 + - 3.0 + - seq: + - app: + - print + - app: + - Series.at + - s3 + - 0 + - do: + - io/print_str + - "\n" +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3) (let) s1 (app Series.push s 1.0) (let) s2 (app Series.push s1 2.0) (let) s3 (app Series.push s2 3.0) (seq (app print (app Series.at s3 0)) (do io/print_str "\n")))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected ident (let-bound name), got `)` at byte 139 +``` + +## L7_sma — ❌ check + +tokens: prompt=1066 completion=811 + +model output: +``` +module: + - demo_m + - fn: + - main + - type: + - fn-type: + - params: + - ret: + - own: + - con: + - Unit + - effects: + - IO + - params: + - body: + - let: + - s + - new: + - Series + - con: + - Float + - 3 + - seq: + - let: + - s + - app: + - Series.push + - s + - 1.0 + - seq: + - let: + - s + - app: + - Series.push + - s + - 5.0 + - seq: + - app: + - print + - app: + - / + - app: + - + + - app: + - Series.at + - s + - 0 + - app: + - + + - app: + - Series.at + - s + - 1 + - app: + - Series.at + - s + - 2 + - 3.0 + - do: + - io/print_str + - "\n" + - let: + - s + - app: + - Series.push + - s + - 3.0 + - seq: + - +``` + +converted Form-A: +``` +(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3) (seq (let s (app Series.push s 1.0) (seq (let s (app Series.push s 5.0)) (seq (app print (app /) (app +) (app Series.at s 0) (app +) (app Series.at s 1) (app Series.at s 2) 3.0)) (do io/print_str "\n"))) (let s (app Series.push s 3.0)) (seq (app print) (app /) (app +) (app Series.at) s 0 (app +) (app Series.at) s 1 (app Series.at) s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 8.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 6.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 2.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n")))))) +``` + +**check:** +``` +error: [surface-parse-error] parse error: expected term, got `)` at byte 206 +``` + + +--- +## Ladder + +- L0_const: ✅ green +- L1_arith: ✅ green +- L2_print: ✅ green +- L3_adt: ❌ check +- L4_rec: ❌ check +- L5_series_len: ❌ check +- L6_series_at: ❌ check +- L7_sma: ❌ check \ No newline at end of file diff --git a/experiments/2026-05-12-cross-model-authoring/qwen_yaml.py b/experiments/2026-05-12-cross-model-authoring/qwen_yaml.py new file mode 100644 index 0000000..029d715 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/qwen_yaml.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Format probe — YAML surface. Does encoding the tree as YAML (no parens, +native nesting, massively in LLM training) help Qwen, esp. at depth where +format 4 hit a closing-token repetition tail? + +Node `(head a b ...)` <-> YAML mapping `head: [a, b, ...]`; each arg is a +scalar atom or another mapping. Converter uses the STANDARD PyYAML parser +(round-trip-verified incl. the -,*,/,true,false special-char atoms) before +judging any model output. Same ablation ladder as the other probes. + +Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent. +""" +from __future__ import annotations +import json, os, re, subprocess, urllib.request, yaml +from pathlib import Path + +ENDPOINT="https://openai.inference.de-txl.ionos.com/v1/chat/completions" +MODEL="Qwen/Qwen3-Coder-Next" +REPO=Path(__file__).resolve().parents[2] +AIL=os.environ.get("AIL_BIN", str(REPO/"target/debug/ail")) +DOC=REPO/"experiments/2026-05-12-cross-model-authoring/qwen-yaml.md" + +# ---- S-expr <-> tree <-> YAML (verified converter) ---- +def tokenize(s): + toks=[]; i=0 + while i `head:` line with args as `- ` items indented 2 deeper; +an arg is an atom (literal) or another `head:` node. Same ablation ladder. + +Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent. +""" +from __future__ import annotations +import json, os, re, subprocess, urllib.request +from pathlib import Path + +ENDPOINT="https://openai.inference.de-txl.ionos.com/v1/chat/completions" +MODEL="Qwen/Qwen3-Coder-Next" +REPO=Path(__file__).resolve().parents[2] +AIL=os.environ.get("AIL_BIN", str(REPO/"target/debug/ail")) +DOC=REPO/"experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md" + +def tokenize(s): + t=[];i=0 + while i