Brummel
8391209618
User noted the context window is ~128k and the bad-json failures looked like
truncation, so the max_tokens=2500 cap may have skewed the tool-calling test.
Fair point: the L4/L7 bad-json cases WERE truncation. Verified max_tokens=8000
is accepted, re-ran.
Result: 1/8, slightly worse, not better. The bad-json cases became check
failures (full AST now produced, but semantically wrong), confirming
truncation was a real but secondary factor. And the extra room exposed the
deeper problem: at L2 ("print 7", trivial) Qwen emitted 8000 tokens / ~20KB of
AST JSON and ran off the end — a repetition loop on a one-line program; at L7
it nested JSON deep enough to hit the parser's recursion limit. The verbose
AST-JSON surface is not budget-limited in a recoverable way — more room just
lets it loop or over-nest. Tool-calling stays at the bottom; the higher budget
makes it degenerate, even on trivial tasks. Updated format-findings.md.
10 KiB
10 KiB
Qwen format probe — tool-calling / structured AST
Model: Qwen/Qwen3-Coder-Next; submit_program tool, forced; AST written directly to .ail.json.
L0_const — ✅ green
tokens: prompt=1078 completion=162
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "answer",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Int",
"effects": []
},
"ret_mode": "own",
"effects": []
},
"body": {
"lit": {
"kind": "int",
"value": 42
},
"t": "lit"
}
}
]
}
L1_arith — ❌ check
tokens: prompt=1079 completion=170
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "triple",
"params": [
"x"
],
"type": {
"k": "fn",
"param_modes": [
"own"
],
"params": [
{
"k": "con",
"name": "Int"
}
],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"t": "app",
"fn": {
"t": "var",
"name": "*"
},
"args": [
{
"t": "var",
"name": "x"
},
{
"t": "var",
"name": "x"
},
{
"t": "var",
"name": "x"
}
]
}
}
]
}
check:
error: [arity-mismatch] triple: arity mismatch for `*`: expected 2 args, got 3
L2_print — ❌ bad-json
tokens: prompt=1075 completion=8000
submitted AST:
{"module": {"schema": "ailang/v0", "name": "m", "imports": [], "defs": [{"kind": "fn", "name": "main", "params": [], "type": {"k": "fn", "param_modes": [], "params": [], "ret": {"k": "con", "name": "Unit"}, "effects": ["IO"]}, "body": {"t": "seq", "lhs": {"t": "app", "fn": {"t": "var", "name": "print"}, "args": [{"t": "app", "fn": {"t": "var", "name": "print"}, "args": [{"t": "var", "name": "x"}]}
bad-json:
tool arguments not valid JSON: Unterminated string starting at: line 1 column 20034 (char 20033)
L3_adt — ❌ check
tokens: prompt=1092 completion=253
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "type",
"name": "Light",
"ctors": [
{
"name": "Red",
"fields": []
},
{
"name": "Green",
"fields": []
}
]
},
{
"kind": "fn",
"name": "code",
"params": [
"light"
],
"type": {
"k": "fn",
"param_modes": [
"own"
],
"params": [
{
"k": "con",
"name": "Light"
}
],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"kind": "match",
"scrutinee": {
"name": "light",
"t": "var"
},
"arms": [
{
"pat": {
"p": "ctor",
"ctor": "Red"
},
"body": {
"lit": {
"kind": "int",
"value": 0
},
"t": "lit"
}
},
{
"pat": {
"p": "ctor",
"ctor": "Green"
},
"body": {
"lit": {
"kind": "int",
"value": 1
},
"t": "lit"
}
}
]
}
}
]
}
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `t` at line 80 column 3
Caused by:
0: json: missing field `t` at line 80 column 3
1: missing field `t` at line 80 column 3
L4_rec — ❌ check
tokens: prompt=1104 completion=1031
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "type",
"name": "IntList",
"ctors": [
{
"name": "Nil",
"fields": []
},
{
"name": "Cons",
"fields": [
{
"k": "con",
"name": "Int"
},
{
"k": "con",
"name": "IntList"
}
]
}
]
},
{
"kind": "fn",
"name": "length",
"params": [
"xs"
],
"type": {
"k": "fn",
"param_modes": [
"own"
],
"params": [
{
"k": "con",
"name": "IntList"
}
],
"ret": {
"k": "con",
"name": "Int"
},
"ret_mode": "own",
"effects": []
},
"body": {
"t": "match",
"scrutinee": {
"t": "var",
"name": "xs"
},
"arms": [
{
"t": "arm",
"pat": {
"p": "ctor",
"ctor": "Nil",
"fields": []
},
"body": {
"t": "lit",
"lit": {
"kind": "int",
"value": 0
}
},
"arms": [
{
"t": "arm",
"pat": {
"p": "ctor",
"ctor": "Cons",
"fields": [
{
"p": "var",
"name": "h"
},
{
"p": "var",
"name": "t"
}
]
},
"body": {
"t": "app",
"fn": {
"t": "var",
"name": "
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `lit` at line 227 column 3
Caused by:
0: json: missing field `lit` at line 227 column 3
1: missing field `lit` at line 227 column 3
L5_series_len — ❌ check
tokens: prompt=1494 completion=651
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "s",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"body": {
"body": {
"body": {
"lhs": {
"args": [
{
"name": "s3",
"t": "var"
}
],
"fn": {
"name": "Series.len",
"t": "var"
},
"t": "app"
},
"fn": {
"name": "print",
"t": "var"
},
"t": "app"
},
"rhs": {
"args": [
{
"kind": "str",
"value": "\n",
"t": "lit"
}
],
"op": "io/print_str",
"t": "do"
},
"t": "seq"
},
"name": "s3",
"t": "let",
"value": {
"args": [
{
"name": "s2",
"t": "var"
},
{
"kind": "float",
"bits": "400000000000000",
"t": "lit"
}
],
"fn": {
"name": "Series.push",
"t": "var"
},
"t": "app"
}
},
"name": "s2",
"t": "let",
"value": {
"args": [
{
"name": "s",
"t": "var"
},
{
"kind": "float",
"bits": "3ff00
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `lit` at line 124 column 3
Caused by:
0: json: missing field `lit` at line 124 column 3
1: missing field `lit` at line 124 column 3
L6_series_at — ❌ check
tokens: prompt=1502 completion=740
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "main",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"body": {
"body": {
"body": {
"lhs": {
"args": [
{
"args": [
{
"name": "s3",
"t": "var"
},
{
"kind": "value",
"value": {
"lit": {
"kind": "int",
"value": 0
},
"t": "lit"
}
}
],
"fn": {
"name": "Series.at",
"t": "var"
},
"t": "app"
}
],
"fn": {
"name": "print",
"t": "var"
},
"t": "app"
},
"rhs": {
"args": [
{
"lit": {
"kind": "str",
"value": "\n"
},
"t": "lit"
},
{
"op": "io/print_str",
"t": "do"
}
],
"t": "seq"
},
"name": "s3",
"t": "let",
"value": {
"args": [
{
"name": "s2",
"t": "var"
},
{
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: missing field `t` at line 149 column 3
Caused by:
0: json: missing field `t` at line 149 column 3
1: missing field `t` at line 149 column 3
L7_sma — ❌ check
tokens: prompt=1588 completion=4382
submitted AST:
{
"schema": "ailang/v0",
"name": "m",
"imports": [],
"defs": [
{
"kind": "fn",
"name": "main",
"params": [],
"type": {
"k": "fn",
"param_modes": [],
"params": [],
"ret": {
"k": "con",
"name": "Unit"
},
"ret_mode": "own",
"effects": [
"IO"
]
},
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
"body": {
check:
Error: schema/parse error in /tmp/abl/m.ail.json: json: recursion limit exceeded at line 147 column 263
Caused by:
0: json: recursion limit exceeded at line 147 column 263
1: recursion limit exceeded at line 147 column 263
Ladder
- L0_const: ✅ green
- L1_arith: ❌ check
- L2_print: ❌ bad-json
- L3_adt: ❌ check
- L4_rec: ❌ check
- L5_series_len: ❌ check
- L6_series_at: ❌ check
- L7_sma: ❌ check