Brummel
2b9a1083b5
User asked whether Qwen's tool-calling could help. Verified tool-calling is
supported on IONOS (a dummy add tool returns a correct tool_call). Then tested
the promising version: submit the program as canonical AST JSON via a forced
submit_program tool — no surface syntax, no converter risk (the tool argument
IS the .ail.json, written and checked directly).
Result: 2/8, tied for worst. Two failure modes, both the model's: valid JSON
but wrong AST (arity etc.), and — at L4/L7 — bad-json: the tool arguments were
not valid JSON at all, because completion hit the 2500-token cap. AST JSON is
so verbose (every leaf is {"t":"var","name":"x"}) that a non-trivial program
overruns the budget and truncates. IONOS does not constrain decoding to the
schema, so the full structure burden stays on the model, and it is the heaviest
of all surfaces (braces+brackets+keys+quotes+commas+"t": per node).
This confirms the rule across six surfaces (annotated-parens 7/8 > plain parens
6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8 = tool-AST 2/8): the discriminator is
redundant cue vs extra burden, not explicit vs implicit. Even tool-calling, the
model's home turf, loses because AST JSON maximises bookkeeping with no
redundancy. Lever for an LLM-authored language: redundant self-checking
structure (depth-annotated brackets). Synthesis in format-findings.md.
8.0 KiB
8.0 KiB
Qwen format probe — tool-calling / structured AST
Model: Qwen/Qwen3-Coder-Next; submit_program tool, forced; AST written directly to .ail.json.
L0_const — ✅ green
tokens: prompt=1078 completion=114
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "answer",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"lit": {
"kind": "int",
"value": 42
},
"t": "lit"
}
}
]
}
L1_arith — ❌ check
tokens: prompt=1079 completion=170
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "triple",
"params": [
"x"
],
"type": {
"k": "fn",
"param_modes": [
"own"
],
"params": [
{
"k": "con",
"name": "Int"
}
],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"t": "app",
"fn": {
"t": "var",
"name": "*"
},
"args": [
{
"t": "var",
"name": "x"
},
{
"t": "var",
"name": "x"
},
{
"t": "var",
"name": "x"
}
]
}
}
]
}
check:
error: [arity-mismatch] triple: arity mismatch for `*`: expected 2 args, got 3
L2_print — ✅ green
tokens: prompt=1075 completion=280
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "main",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit",
"effects": [
"IO"
]
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"lhs": {
"args": [
{
"lit": {
"kind": "int",
"value": 7
},
"t": "lit"
}
],
"fn": {
"name": "print",
"t": "var"
},
"t": "app"
},
"rhs": {
"args": [
{
"lit": {
"kind": "str",
"value": "\n"
},
"t": "lit"
}
],
"op": "io/print_str",
"t": "do"
},
"t": "seq"
}
}
]
}
L3_adt — ❌ check
tokens: prompt=1092 completion=263
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "type",
"name": "Light",
"ctors": [
{
"name": "Red",
"fields": []
},
{
"name": "Green",
"fields": []
}
]
},
{
"kind": "fn",
"name": "code",
"params": [
"light"
],
"type": {
"k": "fn",
"param_modes": [
"own"
],
"params": [
{
"k": "con",
"name": "Light"
}
],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"kind": "match",
"scrutinee": {
"name": "light",
"t": "var"
},
"arms": [
{
"pat": {
"p": "ctor",
"ctor": "Red",
"fields": []
},
"body": {
"kind": "lit",
"lit": {
"kind": "int",
"value": 0
}
}
},
{
"pat": {
"p": "ctor",
"ctor": "Green",
"fields": []
},
"body": {
"kind": "lit",
"lit": {
"kind": "int",
"value": 1
}
}
}
]
}
}
]
}
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `t` at line 82 column 3
Caused by:
0: json: missing field `t` at line 82 column 3
1: missing field `t` at line 82 column 3
L4_rec — ❌ bad-json
tokens: prompt=1104 completion=2500
submitted AST:
{
"module": {
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "type",
"name": "IntList",
"ctors": [
{
"name": "Nil",
"fields": []
},
{
"name": "Cons",
"fields": [
{
"k": "con",
"name": "Int"
bad-json:
tool arguments not valid JSON: Unterminated string starting at: line 487 column 143 (char 39567)
L5_series_len — ❌ check
tokens: prompt=1494 completion=763
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "s",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"args": [
{
"name": "s",
"t": "var"
},
{
"kind": "float",
"bits": "4000000000000000",
"t": "lit"
}
],
"fn": {
"name": "Series.push",
"t": "var"
},
"t": "app"
},
"t": "let",
"value": {
"args": [
{
"kind": "type",
"value": {
"k": "con",
"name": "Float"
}
},
{
"kind": "value",
"value": {
"lit": {
"kind": "int",
"value": 3
},
"t": "lit"
}
},
"t"
],
"t": "new",
"type": "Series"
}
}
]
}
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `lit` at line 67 column 3
Caused by:
0: json: missing field `lit` at line 67 column 3
1: missing field `lit` at line 67 column 3
L6_series_at — ❌ check
tokens: prompt=1502 completion=797
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "main",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"body": {
"body": {
"body": {
"lhs": {
"args": [
{
"name": "s3",
"t": "var"
},
{
"args": [
{
"lit": {
"kind": "int",
"value": 0
},
"t": "lit"
},
"fn",
"name",
"Series",
"at",
"t",
"var",
"app",
"app"
],
"fn": {
"name": "print",
"t": "var"
},
"t": "app"
},
"rhs",
"args",
"lit",
"str",
"value",
"\n",
"t",
"lit",
"op",
"io/print_str",
"t",
"do",
"seq"
],
"name": "s3",
"t": "let",
"value": {
"args": [
{
"name": "s2",
"t": "var"
},
{
"lit": {
"kind": "float",
"bits": "3ff000000000000"
},
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `t` at line 167 column 3
Caused by:
0: json: missing field `t` at line 167 column 3
1: missing field `t` at line 167 column 3
L7_sma — ❌ bad-json
tokens: prompt=1588 completion=2500
submitted AST:
{
"module": {
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "main",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
bad-json:
tool arguments not valid JSON: Expecting property name enclosed in double quotes: line 347 column 129 (char 110384)
Ladder
- L0_const: ✅ green
- L1_arith: ❌ check
- L2_print: ✅ green
- L3_adt: ❌ check
- L4_rec: ❌ bad-json
- L5_series_len: ❌ check
- L6_series_at: ❌ check
- L7_sma: ❌ bad-json