User noted the context window is ~128k and the bad-json failures looked like
truncation, so the max_tokens=2500 cap may have skewed the tool-calling test.
Fair point: the L4/L7 bad-json cases WERE truncation. Verified max_tokens=8000
is accepted, re-ran.
Result: 1/8, slightly worse, not better. The bad-json cases became check
failures (full AST now produced, but semantically wrong), confirming
truncation was a real but secondary factor. And the extra room exposed the
deeper problem: at L2 ("print 7", trivial) Qwen emitted 8000 tokens / ~20KB of
AST JSON and ran off the end — a repetition loop on a one-line program; at L7
it nested JSON deep enough to hit the parser's recursion limit. The verbose
AST-JSON surface is not budget-limited in a recoverable way — more room just
lets it loop or over-nest. Tool-calling stays at the bottom; the higher budget
makes it degenerate, even on trivial tasks. Updated format-findings.md.
User asked whether Qwen's tool-calling could help. Verified tool-calling is
supported on IONOS (a dummy add tool returns a correct tool_call). Then tested
the promising version: submit the program as canonical AST JSON via a forced
submit_program tool — no surface syntax, no converter risk (the tool argument
IS the .ail.json, written and checked directly).
Result: 2/8, tied for worst. Two failure modes, both the model's: valid JSON
but wrong AST (arity etc.), and — at L4/L7 — bad-json: the tool arguments were
not valid JSON at all, because completion hit the 2500-token cap. AST JSON is
so verbose (every leaf is {"t":"var","name":"x"}) that a non-trivial program
overruns the budget and truncates. IONOS does not constrain decoding to the
schema, so the full structure burden stays on the model, and it is the heaviest
of all surfaces (braces+brackets+keys+quotes+commas+"t": per node).
This confirms the rule across six surfaces (annotated-parens 7/8 > plain parens
6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8 = tool-AST 2/8): the discriminator is
redundant cue vs extra burden, not explicit vs implicit. Even tool-calling, the
model's home turf, loses because AST JSON maximises bookkeeping with no
redundancy. Lever for an LLM-authored language: redundant self-checking
structure (depth-annotated brackets). Synthesis in format-findings.md.
Brummel