Brummel
8391209618
User noted the context window is ~128k and the bad-json failures looked like
truncation, so the max_tokens=2500 cap may have skewed the tool-calling test.
Fair point: the L4/L7 bad-json cases WERE truncation. Verified max_tokens=8000
is accepted, re-ran.
Result: 1/8, slightly worse, not better. The bad-json cases became check
failures (full AST now produced, but semantically wrong), confirming
truncation was a real but secondary factor. And the extra room exposed the
deeper problem: at L2 ("print 7", trivial) Qwen emitted 8000 tokens / ~20KB of
AST JSON and ran off the end — a repetition loop on a one-line program; at L7
it nested JSON deep enough to hit the parser's recursion limit. The verbose
AST-JSON surface is not budget-limited in a recoverable way — more room just
lets it loop or over-nest. Tool-calling stays at the bottom; the higher budget
makes it degenerate, even on trivial tasks. Updated format-findings.md.
172 lines
9.5 KiB
Markdown
172 lines
9.5 KiB
Markdown
# Do alternative surface formats help Qwen? — format 4 (annotated parens)
|
||
|
||
**Date:** 2026-06-02 **Model:** Qwen/Qwen3-Coder-Next (IONOS)
|
||
**Question (user):** the fully-parenthesised surface is only one encoding of
|
||
the tree. If no human must read it, try formats that lift the paren-counting
|
||
burden — indent, annotated indent, keywords, annotated parens. Does any help?
|
||
|
||
**Discipline (the "watch out"):** these formats have NO `ail` parser, so
|
||
`ail check` cannot judge them directly. For each, a converter (format → Form-A)
|
||
is built and **round-trip-verified** (Form-A → format → back must be
|
||
parse-identical to the original, on the known-good demos) BEFORE any Qwen
|
||
output is judged. Format 4's converter passed: parse-identity + green check on
|
||
all three demos, and end-to-end on a fenced annotated program.
|
||
|
||
## Format 4 tested: annotated parens `(#N ... #N)`
|
||
|
||
Every paren carries its nesting depth; opener `(#N ` pairs with closer ` #N)`.
|
||
Qwen writes the annotated dialect; the verified stripper removes the
|
||
annotations back to plain Form-A, then `ail check`/`run`. Same ablation ladder
|
||
as the plain-Form-A baseline, so results are comparable. Raw: `qwen-fmt4.md`,
|
||
`qwen-fmt4-seqhint.md`.
|
||
|
||
### Result: it helps, measurably
|
||
|
||
| Level | plain Form-A | **format 4** |
|
||
|---|---|---|
|
||
| L0–L3 | ✅ | ✅ |
|
||
| **L4** build-list + recursion | ❌ paren misplaced | **✅** |
|
||
| L5, L6 Series | ✅ | ✅ |
|
||
| L7 SMA | ❌ | ❌ (but different — see below) |
|
||
|
||
Plain Form-A: 6/8. **Format 4: 7/8.** The annotation cracked the
|
||
bracket-balance wall that broke L4 — the per-paren depth keeps openers and
|
||
closers paired.
|
||
|
||
### What format 4 does to L7 (the hardest level)
|
||
|
||
In plain Form-A, L7 failed on bracket imbalance. In format 4, Qwen produced
|
||
**perfectly balanced brackets even at the deepest nesting** (the depth
|
||
annotations all matched through the whole SMA) AND the **correct SMA logic**
|
||
(`(/ (+ (at 0) (+ (at 1) (at 2))) 3.0)`). Proof the building blocks were
|
||
right: taking Qwen's exact logic and only fixing the one remaining issue makes
|
||
it run and emit the exact expected output (`3.0 / 5.33333 / 5.66667 / 5.33333`)
|
||
— see `/tmp/abl/qwen_seqfix.ail` reconstruction in session.
|
||
|
||
The one remaining issue was **not a format problem**: Qwen treats `seq` as
|
||
variadic (Lisp `progn`) — `(seq A B C)` — but AILang `seq` is strictly binary.
|
||
|
||
### Pushing one more level (format 4 + an explicit `seq`-is-binary hint)
|
||
|
||
Gave Qwen the binary-`seq` rule plus a nested-`seq` example, re-ran L7. It
|
||
**fixed the seq nesting** (correct `(seq (seq print nl) (let …))`), but L7
|
||
still failed — now on two model-behaviour limits, not surface encoding:
|
||
1. it **simplified the logic** (printed `at 0` instead of the average), and
|
||
2. a **termination-degeneration tail**: hundreds of repeated `#0)` — Qwen
|
||
*knows* via `#0` it is at the outermost level, yet keeps closing. A pure
|
||
repetition loop, the same failure mode the ~80k-token full-SMA run hit.
|
||
|
||
## Takeaway
|
||
|
||
- **Annotated parens (format 4) measurably help**: they crack the
|
||
bracket-balance wall (L4 green; L7 brackets perfect). For everything up to
|
||
medium-deep nesting, the format lifts the exact burden the ablation isolated.
|
||
- **The hardest level (L7) is no longer a bracketing problem** — it is (a) a
|
||
construct-arity quirk (`seq` binary vs variadic; fixable by an example or by
|
||
making `seq` n-ary) and (b) a single-shot complexity ceiling where the model
|
||
simplifies and falls into a closing-token repetition loop.
|
||
- The repetition tail is paren-driven (`#0)` repeated). The natural next test
|
||
was a **bracket-free** format — run below.
|
||
|
||
## Bracket-free formats tested: indent / YAML — they made it WORSE
|
||
|
||
The hypothesis was that removing the closing token would remove the repetition
|
||
tail. It did the opposite — bracket-free formats are clearly worse. Two were
|
||
tested, same ladder, each with a round-trip-verified converter:
|
||
|
||
- **YAML via PyYAML** (`qwen-yaml.md`): 2/8. First lesson was about the
|
||
*instrument*, not the model: a real YAML library auto-quotes and gives `*`,
|
||
`-`, `true` special meaning — and the AILang operators collide. Qwen wrote
|
||
`- *` (multiply), valid as an atom but a YAML alias marker, so the library
|
||
crashed. **You cannot use an auto-quoting parser for an operator-rich
|
||
vocabulary.** Discarded.
|
||
- **yamlish, own literal parser** (`qwen-yamlish.md`): 3/8. Re-built with a
|
||
hand-written indent parser that takes atoms verbatim (no quoting, no alias) —
|
||
`- *` round-trips fine. With the instrument fixed, the model still only
|
||
reached 3/8: trivial programs (L0–L2) pass, but at the first real nesting
|
||
(L3 ADT, L4 data-def) the indentation structure breaks (e.g. a `con` field
|
||
type lands as a `data` attribute — the model lost the indent level).
|
||
|
||
## M-expressions tested: middle of the pack
|
||
|
||
McCarthy's `head[arg; arg]` notation — explicit structure (brackets +
|
||
semicolons) in the familiar function-call shape. Hypothesis: explicit + a
|
||
shape LLMs know cold should win. Result: 4–5/8 (model variance between two
|
||
runs), BELOW plain parens, ABOVE the bracket-free formats. Two watch-out
|
||
catches kept the measurement honest:
|
||
- A trailing `;` before `]` (Qwen wrote `let[s; v; ]`, a normal trailing
|
||
separator) — my parser was too strict and rejected it. Fixed to tolerate it
|
||
(round-trip preserved). Without this, M-expr would have looked unfairly bad.
|
||
- L4 then failed with **43 `[` vs 42 `]`** — a genuine Qwen bracket imbalance
|
||
(not truncation: 301 completion tokens). The same depth-tracking failure as
|
||
plain parens, just with `[` instead of `(`.
|
||
|
||
Why M-expr is worse than plain parens despite being "explicit": swapping `()`
|
||
for `[]` does nothing for the balance burden (square is no easier than round),
|
||
and the semicolons *add* a second consistency requirement Qwen does not
|
||
reliably meet (the trailing-`;` slips). It is explicit, but with extra load,
|
||
not redundant cue.
|
||
|
||
## Tool-calling / structured AST tested: WORST, and it confirms the rule
|
||
|
||
The intuitive bet: Qwen3-Coder is trained hard on tool-calling, the API
|
||
validates the arguments, so submitting the program as canonical AST JSON via a
|
||
`submit_program` tool should be the most reliable channel. Tool-calling is
|
||
supported (verified: a dummy `add` tool returned a correct `tool_call`). But
|
||
the result is **2/8 — tied for worst.** Two failure modes, both the model's:
|
||
- `check` (L1/L3/L5/L6): valid JSON, wrong AST — e.g. `* ` given 3 args.
|
||
- **`bad-json` (L4/L7): the tool arguments were not valid JSON at all** —
|
||
completion hit the 2500-token cap. The AST JSON is so verbose (every leaf is
|
||
`{"t":"var","name":"x"}`) that a non-trivial program overruns the budget and
|
||
the JSON is truncated. IONOS does **not** constrain decoding to the schema,
|
||
so the structure burden is fully on the model — and it is the *heaviest*
|
||
burden of all: braces + brackets + keys + quotes + commas + `"t":`
|
||
discriminators, every node.
|
||
|
||
So the "modern" structured-output channel is the worst surface here, for
|
||
exactly the reason the rule predicts: AST JSON is maximal extra bookkeeping
|
||
with no redundant cue.
|
||
|
||
**Re-run with max_tokens raised 2500 → 8000** (the bad-json cases at 2500 were
|
||
truncation, so this is the fair correction): it did NOT help — 1/8, slightly
|
||
worse. The full budget just gives the model more room to degenerate. At L2
|
||
("print 7", a trivial program) Qwen emitted **8000 tokens — ~20 KB of AST JSON
|
||
— and ran off the end** (a repetition loop on a one-line program); at L7 it
|
||
produced JSON nested so deep the parser hit its recursion limit. So the AST-JSON
|
||
surface is not budget-limited in a recoverable way: given more room it loops or
|
||
nests without end. The verbose surface invites the same degeneration the
|
||
annotated-paren run hit at its hardest level — but here it strikes even the
|
||
trivial tasks.
|
||
|
||
## Overall finding across all six surfaces
|
||
|
||
| surface | green | structure marking |
|
||
|---|---|---|
|
||
| **format 4 — annotated parens** | **7/8** | maximal-explicit, redundant (every paren + its depth) |
|
||
| plain Form-A — parens | 6/8 | explicit, one channel |
|
||
| M-expressions `head[a; b]` | 4–5/8 | explicit, but `[]`+`;` adds a separator burden |
|
||
| yamlish / indent (own parser) | 3/8 | implicit (indentation) |
|
||
| YAML (library) | 2/8 | implicit + special-char collision |
|
||
| tool-calling / AST JSON | 2/8 | maximal bookkeeping, verbose, no redundancy |
|
||
|
||
The result is monotone and the opposite of the starting hypothesis. The
|
||
discriminator is sharper than "explicit vs implicit": it is **redundant cue vs
|
||
extra burden.** Qwen's weakness is structure-tracking at depth. What helps is a
|
||
marking that makes that tracking *easier without adding work* — depth-annotated
|
||
parens carry the nesting depth redundantly, so the model cannot lose its place
|
||
(format 4, best). What hurts is either (a) moving the tracking onto an implicit
|
||
channel it must maintain itself — indentation (yamlish/YAML, worst) — or (b)
|
||
adding a second thing to keep consistent on top of bracket balance — the
|
||
M-expr semicolons (middle), or (c) maximising the bookkeeping outright — AST
|
||
JSON over the tool channel, the heaviest surface and tied for worst, even
|
||
though tool-calling is the model's home turf. Plain parens sit in between: one
|
||
explicit channel, no redundancy, no extra burden.
|
||
|
||
The L7 closing-token repetition tail is a separate single-shot ceiling, not
|
||
something any surface here fixes — the bracket-free surfaces fail earlier,
|
||
before that ceiling is even reached.
|
||
|
||
**For an LLM-authored language the lever is redundant, self-checking structure
|
||
(depth-annotated brackets), not a lighter surface and not a more familiar one
|
||
that carries extra bookkeeping.**
|