Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

experiment(cma): bracket-free formats tested — they make Qwen WORSE, not better (refs #68)

Browse Source 
Tested the user's bracket-free ideas (indent, YAML) against the annotated-paren
result. Hypothesis was that removing the closing token removes the repetition
tail. The opposite held.

Instrument note (the watch-out paid off twice): a real YAML library auto-quotes
and gives *,-,true special meaning, which collides with the AILang operators —
Qwen wrote `- *` (multiply), a YAML alias marker, and the library crashed. That
is a measurement bug, not a model failure. Rebuilt with a hand-written literal
indent parser (atoms verbatim, no quoting); `- *` then round-trips fine.

With the instrument fixed, the result is monotone across four formats:
  format 4 annotated parens 7/8 > plain Form-A parens 6/8 > yamlish indent 3/8
  > YAML library 2/8.
More explicit structure marking helps this model; less hurts. Qwen's weakness
is structure-tracking at depth, and indentation makes it track the nesting
itself — as error-prone as matching parens but without the redundant cue. The
bracket-free surfaces fail EARLIER (L3/L4), before the L7 repetition ceiling is
even reached.

Lever for an LLM-authored language: redundant explicit structure (depth-
annotated brackets), not a lighter surface. Synthesis in format-findings.md;
raw ladders qwen-yaml.md (PyYAML, discarded) and qwen-yamlish.md (own parser).
This commit is contained in:
Brummel
2026-06-02 18:49:35 +02:00
parent 682b935ba3
commit 525112e32e
5 changed files with 1501 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experiments/2026-05-12-cross-model-authoring/format-findings.md
+42 -4
View File
@@ -66,7 +66,45 @@ still failed — now on two model-behaviour limits, not surface encoding:
making `seq` n-ary) and (b) a single-shot complexity ceiling where the model
simplifies and falls into a closing-token repetition loop.
- The repetition tail is paren-driven (`#0)` repeated). The natural next test
is a **bracket-free** format (indent, or keyword-delimited blocks) where
there is no closing token to loop on — that directly targets the one limit
format 4 does not remove. It needs a new round-trip-verified converter
(an offside / keyword parser), so it is a deliberate next step, not run here.
was a **bracket-free** format — run below.
## Bracket-free formats tested: indent / YAML — they made it WORSE
The hypothesis was that removing the closing token would remove the repetition
tail. It did the opposite — bracket-free formats are clearly worse. Two were
tested, same ladder, each with a round-trip-verified converter:
- **YAML via PyYAML** (`qwen-yaml.md`): 2/8. First lesson was about the
*instrument*, not the model: a real YAML library auto-quotes and gives `*`,
`-`, `true` special meaning — and the AILang operators collide. Qwen wrote
`- *` (multiply), valid as an atom but a YAML alias marker, so the library
crashed. **You cannot use an auto-quoting parser for an operator-rich
vocabulary.** Discarded.
- **yamlish, own literal parser** (`qwen-yamlish.md`): 3/8. Re-built with a
hand-written indent parser that takes atoms verbatim (no quoting, no alias) —
`- *` round-trips fine. With the instrument fixed, the model still only
reached 3/8: trivial programs (L0–L2) pass, but at the first real nesting
(L3 ADT, L4 data-def) the indentation structure breaks (e.g. a `con` field
type lands as a `data` attribute — the model lost the indent level).
## Overall finding across all four formats
| format | green | structure marking |
|---|---|---|
| **format 4 — annotated parens** | **7/8** | maximal-explicit (every paren + its depth) |
| plain Form-A — parens | 6/8 | explicit (parens) |
| yamlish / indent (own parser) | 3/8 | implicit (indentation) |
| YAML (library) | 2/8 | implicit + special-char collision |
The result is monotone and the opposite of the starting hypothesis:
**more explicit structure marking helps this model; less hurts.** Qwen's
weakness is structure-tracking at depth — and indentation makes it track the
nesting itself, which is just as error-prone as matching parens but *without*
the redundant cue. Parens are a help, not a burden; making them **more**
explicit (depth-annotated, format 4) is the win. The closing-token repetition
tail at L7 is a separate single-shot ceiling, not something a bracket-free
surface fixes — indeed bracket-free surfaces fail earlier, before that ceiling
is even reached.
**For an LLM-authored language, the lever is therefore redundant explicit
structure (e.g. depth-annotated brackets), not a lighter surface.**
experiments/2026-05-12-cross-model-authoring/qwen-yaml.md
+542
View File
@@ -0,0 +1,542 @@
# Qwen format probe — YAML surface
**Model:** Qwen/Qwen3-Coder-Next; single-shot; YAML dialect; converted via verified PyYAML round-trip.
---
## L0_const — ✅ green
tokens: prompt=649 completion=58
model YAML:
```yaml
module:
- m
- fn:
- answer
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Int
- params: []
- body:
- '42'
```
converted Form-A:
```
(module m (fn answer (type (fn-type (params) (ret (own (con Int))))) (params) (body 42)))
```
## L1_arith — ❌ yaml-convert
tokens: prompt=651 completion=87
model YAML:
```yaml
module:
- demo_m
- fn:
- triple
- type:
- fn-type:
- params:
- own:
- con:
- Int
- ret:
- own:
- con:
- Int
- params:
- x
- body:
- app:
- app:
- *
- x
- '3'
```
**yaml-convert:**
```
converter rejected model YAML: while scanning an alias
in "<unicode string>", line 20, column 11:
- *
^
expected alphabetic or numeric character, but found '\n'
in "<unicode string>", line 20, column 12:
- *
^
```
## L2_print — ❌ yaml-convert
tokens: prompt=646 completion=94
model YAML:
```yaml
module:
- demo_c
- fn:
- m
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params: []
- body:
- seq:
- app:
- print
- '7
- do:
- io/print_str
- '"\n"'
```
**yaml-convert:**
```
converter rejected model YAML: while parsing a block mapping
in "<unicode string>", line 17, column 9:
- app:
^
expected <block end>, but found '<scalar>'
in "<unicode string>", line 22, column 12:
- '"\n"'
^
```
## L3_adt — ✅ green
tokens: prompt=665 completion=136
model YAML:
```yaml
module:
- demo_m
- data:
- Light
- ctor:
- Red
- ctor:
- Green
- fn:
- code
- type:
- fn-type:
- params:
- own:
- con:
- Light
- ret:
- own:
- con:
- Int
- params:
- l
- body:
- match:
- l
- case:
- pat-ctor:
- Red
- '0'
- case:
- pat-ctor:
- Green
- '1'
```
converted Form-A:
```
(module demo_m (data Light (ctor Red) (ctor Green)) (fn code (type (fn-type (params (own (con Light))) (ret (own (con Int))))) (params l) (body (match l (case (pat-ctor Red) 0) (case (pat-ctor Green) 1)))))
```
## L4_rec — ❌ yaml-convert
tokens: prompt=677 completion=359
model YAML:
```yaml
module:
- demo_m
- data:
- IntList
- ctor:
- Nil
- ctor:
- Cons
- con:
- Int
- con:
- IntList
- fn:
- length
- type:
- fn-type:
- params:
- own:
- con:
- IntList
- ret:
- own:
- con:
- Int
- params:
- xs
- body:
- match:
- xs
- case:
- pat-ctor:
- Nil
- '0'
- case:
- pat-ctor:
- Cons
- h
- t
- app:
- +
- '1'
- app:
- length
- t
- fn:
- main
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params: []
- body:
- let:
- xs
- term-ctor:
- IntList
- Cons
- '1'
- term-ctor:
- IntList
- Cons
- '2'
- term-ctor:
- IntList
- Cons
- '3'
- term-ctor:
- IntList
- Nil
- seq:
- app:
- print
- app:
- length
- xs
- do:
- io/print_str
- '"\n"
```
**yaml-convert:**
```
converter rejected model YAML: while scanning a quoted scalar
in "<unicode string>", line 83, column 11:
- '"\n"
^
found unexpected end of stream
in "<unicode string>", line 83, column 16:
- '"\n"
^
```
## L5_series_len — ❌ check
tokens: prompt=945 completion=215
model YAML:
```yaml
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params: []
- body:
- let:
- s
- new:
- Series
- con:
- Float
- '3'
- let:
- s
- app:
- Series.push
- s
- '1.0'
- let:
- s
- app:
- Series.push
- s
- '2.0'
- let:
- s
- app:
- Series.push
- s
- '3.0'
- seq:
- app:
- print
- app:
- Series.len
- s
- do:
- io/print_str
- '"\n"'
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 2.0)) (let s (app Series.push s 3.0)) (seq (app print) (app Series.len) s) (do io/print_str "\n"))))
```
**check:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 134
```
## L6_series_at — ❌ check
tokens: prompt=953 completion=226
model YAML:
```yaml
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params: []
- body:
- let:
- s
- new:
- Series
- con:
- Float
- '3'
- let:
- s1
- app:
- Series.push
- s
- '1.0'
- let:
- s2
- app:
- Series.push
- s1
- '2.0'
- let:
- s3
- app:
- Series.push
- s2
- '3.0'
- seq:
- app:
- print
- app:
- Series.at
- s3
- '0'
- do:
- io/print_str
- '"\n"'
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s1 (app Series.push s 1.0)) (let s2 (app Series.push s1 2.0)) (let s3 (app Series.push s2 3.0)) (seq (app print) (app Series.at) s3 0) (do io/print_str "\n"))))
```
**check:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 134
```
## L7_sma — ❌ check
tokens: prompt=1036 completion=469
model YAML:
```yaml
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params: []
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params: []
- body:
- let:
- s
- new:
- Series
- con:
- Float
- '3'
- let:
- s
- app:
- Series.push
- s
- '1.0'
- let:
- s
- app:
- Series.push
- s
- '5.0'
- seq:
- app:
- print
- app:
- /
- app:
- Series.sum
- s
- '3.0'
- let:
- s
- app:
- Series.push
- s
- '3.0'
- seq:
- app:
- print
- app:
- /
- app:
- Series.sum
- s
- '3.0'
- let:
- s
- app:
- Series.push
- s
- '8.0'
- seq:
- app:
- print
- app:
- /
- app:
- Series.sum
- s
- '3.0'
- let:
- s
- app:
- Series.push
- s
- '6.0'
- seq:
- app:
- print
- app:
- /
- app:
- Series.sum
- s
- '3.0'
- let:
- s
- app:
- Series.push
- s
- '2.0'
- seq:
- app:
- print
- app:
- /
- app:
- Series.sum
- s
- '3.0'
- do:
- io/print_str
- '"\n"'
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 5.0)) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 3.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 8.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 6.0) (seq (app print) (app /) (app Series.sum) s 3.0) (let s (app Series.push) s 2.0) (seq (app print) (app /) (app Series.sum) s 3.0) (do io/print_str "\n"))))
```
**check:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 134
```
---
## Ladder
- L0_const: ✅ green
- L1_arith: ❌ yaml-convert
- L2_print: ❌ yaml-convert
- L3_adt: ✅ green
- L4_rec: ❌ yaml-convert
- L5_series_len: ❌ check
- L6_series_at: ❌ check
- L7_sma: ❌ check
experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md
+545
View File
@@ -0,0 +1,545 @@
# Qwen format probe — yamlish (indented tree, own literal parser)
**Model:** Qwen/Qwen3-Coder-Next; single-shot; atoms verbatim (no YAML lib, no quoting).
---
## L0_const — ✅ green
tokens: prompt=683 completion=127
model output:
```
module:
- demo_c
- fn:
- answer
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Int
- params:
- body:
- 42
- fn:
- m
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- app:
- print
- app:
- answer
```
converted Form-A:
```
(module demo_c (fn answer (type (fn-type (params) (ret (own (con Int))))) (params) (body 42)) (fn m (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (app print (app answer)))))
```
## L1_arith — ✅ green
tokens: prompt=685 completion=84
model output:
```
module:
- demo_m
- fn:
- triple
- type:
- fn-type:
- params:
- own:
- con:
- Int
- ret:
- own:
- con:
- Int
- params:
- x
- body:
- app:
- *
- x
- 3
```
converted Form-A:
```
(module demo_m (fn triple (type (fn-type (params (own (con Int))) (ret (own (con Int))))) (params x) (body (app * x 3))))
```
## L2_print — ✅ green
tokens: prompt=680 completion=146
model output:
```
module:
- demo_c
- fn:
- m
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Int
- params:
- body:
- 7
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- seq:
- app:
- print
- app:
- m
- do:
- io/print_str
- "\n"
```
converted Form-A:
```
(module demo_c (fn m (type (fn-type (params) (ret (own (con Int))))) (params) (body 7)) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (seq (app print (app m)) (do io/print_str "\n")))))
```
## L3_adt — ❌ check
tokens: prompt=699 completion=241
model output:
```
module:
- demo_c
- data:
- Light
- ctor:
- Red
- ctor:
- Green
- fn:
- code
- type:
- fn-type:
- params:
- own:
- con:
- Light
- ret:
- own:
- con:
- Int
- params:
- l
- body:
- match:
- l
- case:
- pat-ctor:
- Red
- 0
- case:
- pat-ctor:
- Green
- 1
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- seq:
- app:
- print
- app:
- code
- term-ctor:
- Light
- Red
- do:
- io/print_str
- "\n"
```
converted Form-A:
```
(module demo_c (data Light (ctor Red) (ctor Green)) (fn code (type (fn-type (params (own (con Light))) (ret (own (con Int))))) (params l) (body (match l (case (pat-ctor Red) 0) (case (pat-ctor Green) 1)))) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (seq (app print (app code) (term-ctor Light Red)) (do io/print_str "\n")))))
```
**check:**
```
error: [arity-mismatch] main: arity mismatch for `print`: expected 1 args, got 2
```
## L4_rec — ❌ check
tokens: prompt=711 completion=360
model output:
```
module:
- demo_m
- data:
- IntList
- ctor:
- Nil
- ctor:
- Cons
- con:
- Int
- con:
- IntList
- fn:
- length
- type:
- fn-type:
- params:
- own:
- con:
- IntList
- ret:
- own:
- con:
- Int
- params:
- xs
- body:
- match:
- xs
- case:
- pat-ctor:
- Nil
- 0
- case:
- pat-ctor:
- Cons
- h
- t
- app:
- +
- 1
- app:
- length
- t
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- let:
- xs
- term-ctor:
- IntList
- Cons
- 1
- term-ctor:
- IntList
- Cons
- 2
- term-ctor:
- IntList
- Cons
- 3
- term-ctor:
- IntList
- Nil
- seq:
- app:
- print
- app:
- length
- xs
- do:
- io/print_str
- "\n"
```
converted Form-A:
```
(module demo_m (data IntList (ctor Nil) (ctor Cons) (con Int) (con IntList)) (fn length (type (fn-type (params (own (con IntList))) (ret (own (con Int))))) (params xs) (body (match xs (case (pat-ctor Nil) 0) (case (pat-ctor Cons h t) (app + 1 (app length t)))))) (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons) 2 (term-ctor IntList Cons) 3 (term-ctor IntList) Nil) (seq (app print) (app length) xs) (do io/print_str "\n")))))
```
**check:**
```
error: [surface-parse-error] parse error in data-def: unknown data attribute `con`; expected `doc`, `ctor`, `drop-iterative`, or `param-in` at byte 52
```
## L5_series_len — ❌ check
tokens: prompt=975 completion=214
model output:
```
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- let:
- s
- new:
- Series
- con:
- Float
- 3
- let:
- s
- app:
- Series.push
- s
- 1.0
- let:
- s
- app:
- Series.push
- s
- 2.0
- let:
- s
- app:
- Series.push
- s
- 3.0
- seq:
- app:
- print
- app:
- Series.len
- s
- do:
- io/print_str
- "\n"
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3)) (let s (app Series.push s 1.0)) (let s (app Series.push s 2.0)) (let s (app Series.push s 3.0)) (seq (app print (app Series.len) s)) (do io/print_str "\n"))))
```
**check:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 134
```
## L6_series_at — ❌ check
tokens: prompt=983 completion=225
model output:
```
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- let:
- s
- new:
- Series
- con:
- Float
- 3
- let:
- s1
- app:
- Series.push
- s
- 1.0
- let:
- s2
- app:
- Series.push
- s1
- 2.0
- let:
- s3
- app:
- Series.push
- s2
- 3.0
- seq:
- app:
- print
- app:
- Series.at
- s3
- 0
- do:
- io/print_str
- "\n"
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3) (let) s1 (app Series.push s 1.0) (let) s2 (app Series.push s1 2.0) (let) s3 (app Series.push s2 3.0) (seq (app print (app Series.at s3 0)) (do io/print_str "\n"))))))
```
**check:**
```
error: [surface-parse-error] parse error: expected ident (let-bound name), got `)` at byte 139
```
## L7_sma — ❌ check
tokens: prompt=1066 completion=811
model output:
```
module:
- demo_m
- fn:
- main
- type:
- fn-type:
- params:
- ret:
- own:
- con:
- Unit
- effects:
- IO
- params:
- body:
- let:
- s
- new:
- Series
- con:
- Float
- 3
- seq:
- let:
- s
- app:
- Series.push
- s
- 1.0
- seq:
- let:
- s
- app:
- Series.push
- s
- 5.0
- seq:
- app:
- print
- app:
- /
- app:
- +
- app:
- Series.at
- s
- 0
- app:
- +
- app:
- Series.at
- s
- 1
- app:
- Series.at
- s
- 2
- 3.0
- do:
- io/print_str
- "\n"
- let:
- s
- app:
- Series.push
- s
- 3.0
- seq:
-
```
converted Form-A:
```
(module demo_m (fn main (type (fn-type (params) (ret (own (con Unit))) (effects IO))) (params) (body (let s (new Series (con Float) 3) (seq (let s (app Series.push s 1.0) (seq (let s (app Series.push s 5.0)) (seq (app print (app /) (app +) (app Series.at s 0) (app +) (app Series.at s 1) (app Series.at s 2) 3.0)) (do io/print_str "\n"))) (let s (app Series.push s 3.0)) (seq (app print) (app /) (app +) (app Series.at) s 0 (app +) (app Series.at) s 1 (app Series.at) s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 8.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 6.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n") (let s (app) Series.push s 2.0) (seq (app) print (app) / (app) + (app) Series.at s 0 (app) + (app) Series.at s 1 (app) Series.at s 2 3.0) (do io/print_str "\n"))))))
```
**check:**
```
error: [surface-parse-error] parse error: expected term, got `)` at byte 206
```
---
## Ladder
- L0_const: ✅ green
- L1_arith: ✅ green
- L2_print: ✅ green
- L3_adt: ❌ check
- L4_rec: ❌ check
- L5_series_len: ❌ check
- L6_series_at: ❌ check
- L7_sma: ❌ check
experiments/2026-05-12-cross-model-authoring/qwen_yaml.py
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Format probe — YAML surface. Does encoding the tree as YAML (no parens,
native nesting, massively in LLM training) help Qwen, esp. at depth where
format 4 hit a closing-token repetition tail?
Node `(head a b ...)` <-> YAML mapping `head: [a, b, ...]`; each arg is a
scalar atom or another mapping. Converter uses the STANDARD PyYAML parser
(round-trip-verified incl. the -,*,/,true,false special-char atoms) before
judging any model output. Same ablation ladder as the other probes.
Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent.
"""
from __future__ import annotations
import json, os, re, subprocess, urllib.request, yaml
from pathlib import Path
ENDPOINT="https://openai.inference.de-txl.ionos.com/v1/chat/completions"
MODEL="Qwen/Qwen3-Coder-Next"
REPO=Path(__file__).resolve().parents[2]
AIL=os.environ.get("AIL_BIN", str(REPO/"target/debug/ail"))
DOC=REPO/"experiments/2026-05-12-cross-model-authoring/qwen-yaml.md"
# ---- S-expr <-> tree <-> YAML (verified converter) ----
def tokenize(s):
toks=[]; i=0
while i<len(s):
c=s[i]
if c.isspace(): i+=1; continue
if c in '()': toks.append(c); i+=1; continue
if c=='"':
j=i+1
while j<len(s):
if s[j]=='\\': j+=2; continue
if s[j]=='"': break
j+=1
toks.append(s[i:j+1]); i=j+1; continue
j=i
while j<len(s) and not s[j].isspace() and s[j] not in '()"': j+=1
toks.append(s[i:j]); i=j
return toks
def parse(toks):
t=toks.pop(0)
if t=='(':
node=[]
while toks and toks[0]!=')': node.append(parse(toks))
if not toks: raise ValueError("unbalanced (")
toks.pop(0); return node
return t
def to_tree(src):
toks=tokenize(src); t=parse(toks)
return t
def emit(n): return '('+' '.join(emit(x) for x in n)+')' if isinstance(n,list) else n
def to_yamlobj(n):
if isinstance(n,list):
if not n or not isinstance(n[0],str): raise ValueError("node head must be an atom")
return {n[0]:[to_yamlobj(a) for a in n[1:]]}
return n
def from_yamlobj(o):
if isinstance(o,dict):
if len(o)!=1: raise ValueError(f"each node must be a single-key mapping, got keys {list(o)}")
(k,v),=o.items()
if v is None: v=[]
if not isinstance(v,list): raise ValueError(f"value of `{k}` must be a list, got {type(v).__name__}")
return [str(k)]+[from_yamlobj(x) for x in v]
return str(o)
def to_yaml(src): return yaml.dump(to_yamlobj(to_tree(src)),default_flow_style=False,sort_keys=False)
def from_yaml(y):
o=yaml.safe_load(y)
return emit(from_yamlobj(o))
A="""(module demo_a
(fn double
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
(params x)
(body (app + x x)))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (seq (app print (app double 21)) (do io/print_str "\\n")))))"""
B="""(module demo_b
(data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
(fn sum
(type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
(params xs)
(body (match xs (case (pat-ctor Nil) 0) (case (pat-ctor Cons h t) (app + h (app sum t))))))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Nil)))
(seq (app print (app sum xs)) (do io/print_str "\\n"))))))"""
S="""(module demo_s
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (let s (new Series (con Float) 3)
(let s2 (app Series.push s 1.0)
(let s3 (app Series.push s2 2.0)
(seq (app print (app Series.len s3)) (do io/print_str "\\n"))))))))"""
INTRO=("This programming language is written as YAML. Every node is a single-key "
"mapping `head:` whose value is a LIST of its arguments; each argument is "
"either a scalar atom (a number, name, or operator like `+`) or another "
"single-key mapping. An empty argument list is `[]`. Study these complete, "
"valid examples, then write the requested program in the SAME YAML "
"structure. Return ONLY the YAML, no prose, no fences.\n\n")
BASE=INTRO+"Example 1:\n"+to_yaml(A)+"\nExample 2:\n"+to_yaml(B)+"\n"
SERIES=(BASE+"\nExample 3 (Series ring-buffer library: `new` with head `new`, "
"args `Series`, a `con` Float type, and capacity N; `Series.push` takes "
"an owned series and an element, returns the new series; `Series.at` "
"takes a series and an Int index (0=newest); `Series.len` / "
"`Series.total_count` take a series, return Int):\n"+to_yaml(S)+"\n")
SMA_OUT="3.0\n5.33333\n5.66667\n5.33333\n"
SMA_TASK=("Write a program `m` computing a simple moving average, window 3, over the "
"float stream 1.0, 5.0, 3.0, 8.0, 6.0, 2.0 using the Series library: create "
"a Float Series lookback 3, push each value in turn, and after each push once "
"at least 3 values have been pushed, print the average of the most recent 3 "
"(their sum divided by 3.0) on its own line. Expected:\n"+SMA_OUT)
STAGES=[
("L0_const",BASE,"Write a program `m` with a function `answer` taking no parameters that returns the Int 42.",None),
("L1_arith",BASE,"Write a program `m` with a function `triple` taking one Int parameter, returning it multiplied by 3.",None),
("L2_print",BASE,"Write a program `m` whose `main` prints the Int 7 on its own line.","7\n"),
("L3_adt",BASE,"Write a program `m` with an ADT `Light` (ctors Red, Green) and a function `code` returning Int 0 for Red and 1 for Green via match.",None),
("L4_rec",BASE,"Write a program `m` with an IntList ADT (Nil/Cons), a function `length` counting elements by recursion, and a `main` that prints the length of the list [1,2,3] on its own line.","3\n"),
("L5_series_len",SERIES,"Write a program `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print its Series.len on its own line.","3\n"),
("L6_series_at",SERIES,"Write a program `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print (Series.at on index 0, the newest) on its own line.","3.0\n"),
("L7_sma",SERIES,SMA_TASK,SMA_OUT),
]
def call(system,task):
body=json.dumps({"model":MODEL,"temperature":0.2,"max_tokens":2000,
"messages":[{"role":"system","content":system},{"role":"user","content":task}]}).encode()
req=urllib.request.Request(ENDPOINT,data=body,headers={
"Authorization":f"Bearer {os.environ['IONOS_API_TOKEN']}","Content-Type":"application/json"})
with urllib.request.urlopen(req,timeout=120) as r: d=json.loads(r.read())
return d["choices"][0]["message"]["content"], d.get("usage",{})
def extract_yaml(resp):
m=re.search(r"```(?:ya?ml)?\n(.*?)```",resp,re.S)
return (m.group(1) if m else resp).strip()
def evaluate(yaml_src,expected):
try:
plain=from_yaml(yaml_src)
except Exception as e:
return "yaml-convert",f"converter rejected model YAML: {e}",""
m=re.search(r"\(module\s+([A-Za-z_][A-Za-z0-9_]*)",plain)
name=m.group(1) if m else "m"
f=Path(f"/tmp/abl/{name}.ail"); f.parent.mkdir(exist_ok=True); f.write_text(plain+"\n")
chk=subprocess.run([AIL,"check",str(f)],capture_output=True,text=True)
if chk.returncode!=0: return "check",(chk.stderr or chk.stdout).strip(),plain
if expected is None: return "green","(check-only)",plain
run=subprocess.run([AIL,"run",str(f)],capture_output=True,text=True)
if run.returncode!=0: return "run",(run.stderr or run.stdout).strip(),plain
if run.stdout!=expected: return "output",f"got {run.stdout!r}, expected {expected!r}",plain
return "green",run.stdout,plain
def main():
log=["# Qwen format probe — YAML surface\n",
f"**Model:** {MODEL}; single-shot; YAML dialect; converted via verified PyYAML round-trip.\n","---\n"]
summary=[]
for sid,ctx,task,exp in STAGES:
resp,usage=call(ctx,task)
y=extract_yaml(resp)
stage,fb,plain=evaluate(y,exp)
ok=stage=="green"; mark="✅ green" if ok else f"❌ {stage}"
summary.append((sid,mark)); print(f"{sid}: {mark}")
log+=[f"## {sid} — {mark}",
f"\ntokens: prompt={usage.get('prompt_tokens','?')} completion={usage.get('completion_tokens','?')}\n",
"model YAML:\n```yaml\n"+y[:1500]+"\n```\n",
("converted Form-A:\n```\n"+plain[:1200]+"\n```\n" if plain else ""),
("" if ok else f"**{stage}:**\n```\n{fb[:600]}\n```\n")]
log+=["\n---\n## Ladder\n"]+[f"- {s}: {m}" for s,m in summary]
DOC.write_text("\n".join(log)); print("doc →",DOC)
if __name__=="__main__":
main()
experiments/2026-05-12-cross-model-authoring/qwen_yamlish.py
+194
View File
@@ -0,0 +1,194 @@
#!/usr/bin/env python3
"""Format probe — yamlish (indented tree, YAML-LOOKING but parsed by our OWN
literal parser). No real YAML library: atoms are taken verbatim as bytes, so
the AILang operators (+ - * /), bools, and string literals carry no special
meaning (the PyYAML run failed precisely because `*` is a YAML alias marker and
the lib auto-quotes). Converter is round-trip-verified before judging output.
Node `(head a b)` <-> `head:` line with args as `- ` items indented 2 deeper;
an arg is an atom (literal) or another `head:` node. Same ablation ladder.
Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent.
"""
from __future__ import annotations
import json, os, re, subprocess, urllib.request
from pathlib import Path
ENDPOINT="https://openai.inference.de-txl.ionos.com/v1/chat/completions"
MODEL="Qwen/Qwen3-Coder-Next"
REPO=Path(__file__).resolve().parents[2]
AIL=os.environ.get("AIL_BIN", str(REPO/"target/debug/ail"))
DOC=REPO/"experiments/2026-05-12-cross-model-authoring/qwen-yamlish.md"
def tokenize(s):
t=[];i=0
while i<len(s):
c=s[i]
if c.isspace(): i+=1; continue
if c in '()': t.append(c); i+=1; continue
if c=='"':
j=i+1
while j<len(s):
if s[j]=='\\': j+=2; continue
if s[j]=='"': break
j+=1
t.append(s[i:j+1]); i=j+1; continue
j=i
while j<len(s) and not s[j].isspace() and s[j] not in '()"': j+=1
t.append(s[i:j]); i=j
return t
def parse(toks):
t=toks.pop(0)
if t=='(':
n=[]
while toks and toks[0]!=')': n.append(parse(toks))
if not toks: raise ValueError("unbalanced")
toks.pop(0); return n
return t
def to_tree(s): return parse(tokenize(s))
def emit_sexpr(n): return '('+' '.join(emit_sexpr(x) for x in n)+')' if isinstance(n,list) else n
def emit_yamlish(root):
lines=[]
def args(n,col):
sp=" "*col
for a in n[1:]:
if isinstance(a,list): lines.append(f"{sp}- {a[0]}:"); args(a,col+2)
else: lines.append(f"{sp}- {a}")
lines.append(f"{root[0]}:"); args(root,2)
return "\n".join(lines)
def parse_yamlish(text):
lines=[l for l in text.split("\n") if l.strip()]
if not lines: raise ValueError("empty")
def col(l): return len(l)-len(l.lstrip(" "))
root=lines[0].rstrip()
if not root.endswith(":"): raise ValueError(f"root must be a `head:` line, got {root!r}")
node=[root[:-1].strip()]; i=[1]
def parse_args(node,ec):
while i[0]<len(lines):
line=lines[i[0]].rstrip(); c=col(line)
if c<ec: return
content=line.strip()
if not content.startswith("- "): raise ValueError(f"expected `- ` item, got {line!r}")
rest=content[2:]
if rest.endswith(":"):
child=[rest[:-1].strip()]; node.append(child); i[0]+=1; parse_args(child,ec+2)
else:
node.append(rest); i[0]+=1
parse_args(node,2)
return node
A="""(module demo_a
(fn double
(type (fn-type (params (own (con Int))) (ret (own (con Int)))))
(params x)
(body (app + x x)))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (seq (app print (app double 21)) (do io/print_str "\\n")))))"""
B="""(module demo_b
(data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
(fn sum
(type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
(params xs)
(body (match xs (case (pat-ctor Nil) 0) (case (pat-ctor Cons h t) (app + h (app sum t))))))
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (let xs (term-ctor IntList Cons 1 (term-ctor IntList Cons 2 (term-ctor IntList Nil)))
(seq (app print (app sum xs)) (do io/print_str "\\n"))))))"""
S="""(module demo_s
(fn main
(type (fn-type (params) (ret (own (con Unit))) (effects IO)))
(params)
(body (let s (new Series (con Float) 3)
(let s2 (app Series.push s 1.0)
(let s3 (app Series.push s2 2.0)
(seq (app print (app Series.len s3)) (do io/print_str "\\n"))))))))"""
INTRO=("This program is written as an indented tree. Every node is a line `head:` "
"whose arguments follow on the next lines as `- ` items, indented two spaces "
"deeper than the head. An argument is either an ATOM written literally (a "
"number, a name, an operator like + or * or /, or a quoted string like "
"\"\\n\") or another `head:` node. A node with no arguments is just `head:` "
"with nothing under it. Study these complete, valid examples and write the "
"requested program in the SAME indented style. Return ONLY the tree — no "
"prose, no code fences.\n\n")
BASE=INTRO+"Example 1:\n"+emit_yamlish(to_tree(A))+"\n\nExample 2:\n"+emit_yamlish(to_tree(B))+"\n"
SERIES=(BASE+"\nExample 3 (Series ring-buffer library: `new` builds a series with "
"args `Series`, a `con` Float, and capacity N; `Series.push` takes an owned "
"series and an element, returns the new series; `Series.at` takes a series "
"and an Int index (0=newest); `Series.len`/`Series.total_count` take a "
"series, return Int):\n"+emit_yamlish(to_tree(S))+"\n")
SMA_OUT="3.0\n5.33333\n5.66667\n5.33333\n"
SMA_TASK=("Write a program `m` computing a simple moving average, window 3, over the "
"float stream 1.0, 5.0, 3.0, 8.0, 6.0, 2.0 using the Series library: create "
"a Float Series lookback 3, push each value in turn, and after each push once "
"at least 3 values have been pushed, print the average of the most recent 3 "
"(their sum divided by 3.0) on its own line. Expected:\n"+SMA_OUT)
STAGES=[
("L0_const",BASE,"Write a program `m` with a function `answer` taking no parameters that returns the Int 42.",None),
("L1_arith",BASE,"Write a program `m` with a function `triple` taking one Int parameter, returning it multiplied by 3.",None),
("L2_print",BASE,"Write a program `m` whose `main` prints the Int 7 on its own line.","7\n"),
("L3_adt",BASE,"Write a program `m` with an ADT `Light` (ctors Red, Green) and a function `code` returning Int 0 for Red and 1 for Green via match.",None),
("L4_rec",BASE,"Write a program `m` with an IntList ADT (Nil/Cons), a function `length` counting elements by recursion, and a `main` that prints the length of the list [1,2,3] on its own line.","3\n"),
("L5_series_len",SERIES,"Write a program `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print its Series.len on its own line.","3\n"),
("L6_series_at",SERIES,"Write a program `m` using the Series library: create a Float Series lookback 3, push 1.0 then 2.0 then 3.0, then print (Series.at on index 0, the newest) on its own line.","3.0\n"),
("L7_sma",SERIES,SMA_TASK,SMA_OUT),
]
def call(system,task):
body=json.dumps({"model":MODEL,"temperature":0.2,"max_tokens":2000,
"messages":[{"role":"system","content":system},{"role":"user","content":task}]}).encode()
req=urllib.request.Request(ENDPOINT,data=body,headers={
"Authorization":f"Bearer {os.environ['IONOS_API_TOKEN']}","Content-Type":"application/json"})
with urllib.request.urlopen(req,timeout=120) as r: d=json.loads(r.read())
return d["choices"][0]["message"]["content"], d.get("usage",{})
def extract(resp):
m=re.search(r"```(?:[a-zA-Z]*)?\n(.*?)```",resp,re.S)
body=(m.group(1) if m else resp)
# keep from first `head:`-looking line
lines=body.split("\n")
for k,l in enumerate(lines):
if re.match(r'^\S+:\s*$',l.rstrip()):
return "\n".join(lines[k:]).strip()
return body.strip()
def evaluate(yl,expected):
try: plain=emit_sexpr(parse_yamlish(yl))
except Exception as e: return "convert",f"own parser rejected output: {e}",""
m=re.search(r"\(module\s+([A-Za-z_][A-Za-z0-9_]*)",plain)
name=m.group(1) if m else "m"
f=Path(f"/tmp/abl/{name}.ail"); f.parent.mkdir(exist_ok=True); f.write_text(plain+"\n")
chk=subprocess.run([AIL,"check",str(f)],capture_output=True,text=True)
if chk.returncode!=0: return "check",(chk.stderr or chk.stdout).strip(),plain
if expected is None: return "green","(check-only)",plain
run=subprocess.run([AIL,"run",str(f)],capture_output=True,text=True)
if run.returncode!=0: return "run",(run.stderr or run.stdout).strip(),plain
if run.stdout!=expected: return "output",f"got {run.stdout!r}, expected {expected!r}",plain
return "green",run.stdout,plain
def main():
log=["# Qwen format probe — yamlish (indented tree, own literal parser)\n",
f"**Model:** {MODEL}; single-shot; atoms verbatim (no YAML lib, no quoting).\n","---\n"]
summary=[]
for sid,ctx,task,exp in STAGES:
resp,usage=call(ctx,task); yl=extract(resp)
stage,fb,plain=evaluate(yl,exp)
ok=stage=="green"; mark="✅ green" if ok else f"❌ {stage}"
summary.append((sid,mark)); print(f"{sid}: {mark}")
log+=[f"## {sid} — {mark}",
f"\ntokens: prompt={usage.get('prompt_tokens','?')} completion={usage.get('completion_tokens','?')}\n",
"model output:\n```\n"+yl[:1400]+"\n```\n",
("converted Form-A:\n```\n"+plain[:1100]+"\n```\n" if plain else ""),
("" if ok else f"**{stage}:**\n```\n{fb[:600]}\n```\n")]
log+=["\n---\n## Ladder\n"]+[f"- {s}: {m}" for s,m in summary]
DOC.write_text("\n".join(log)); print("doc →",DOC)
if __name__=="__main__":
main()