Brummel
682b935ba3
Tests the user's hypothesis: if no human reads the surface, lift the paren- counting burden. Format 4 = annotated parens, every paren carries its nesting depth. Discipline: the format has no ail parser, so the converter is round-trip verified (parse-identity on known-good demos) before judging any model output. Result: it helps. Plain Form-A 6/8 -> format 4 7/8 on the same ablation ladder. The depth annotation cracks the bracket-balance wall that broke L4. At the hardest level (L7 SMA) Qwen produced perfectly balanced brackets through the deepest nesting AND correct SMA logic; the only remaining issue was a construct quirk (it treats seq as variadic, but AILang seq is binary). Proof the blocks were right: Qwen's exact logic with a manual binary-seq fix runs and emits the exact expected SMA output. Pushing further (format 4 + explicit seq-binary hint) fixed the seq nesting but L7 still failed on two MODEL-behaviour limits, not encoding: it simplified the logic, and fell into a closing-token repetition tail (hundreds of #0) ). The tail is paren-driven, so the natural next test is a bracket-free format (indent / keyword-delimited) with no closing token to loop on — a deliberate next step needing its own verified converter. Synthesis in format-findings.md.
3.8 KiB
Do alternative surface formats help Qwen? — format 4 (annotated parens)
Date: 2026-06-02 Model: Qwen/Qwen3-Coder-Next (IONOS) Question (user): the fully-parenthesised surface is only one encoding of the tree. If no human must read it, try formats that lift the paren-counting burden — indent, annotated indent, keywords, annotated parens. Does any help?
Discipline (the "watch out"): these formats have NO ail parser, so
ail check cannot judge them directly. For each, a converter (format → Form-A)
is built and round-trip-verified (Form-A → format → back must be
parse-identical to the original, on the known-good demos) BEFORE any Qwen
output is judged. Format 4's converter passed: parse-identity + green check on
all three demos, and end-to-end on a fenced annotated program.
Format 4 tested: annotated parens (#N ... #N)
Every paren carries its nesting depth; opener (#N pairs with closer #N).
Qwen writes the annotated dialect; the verified stripper removes the
annotations back to plain Form-A, then ail check/run. Same ablation ladder
as the plain-Form-A baseline, so results are comparable. Raw: qwen-fmt4.md,
qwen-fmt4-seqhint.md.
Result: it helps, measurably
| Level | plain Form-A | format 4 |
|---|---|---|
| L0–L3 | ✅ | ✅ |
| L4 build-list + recursion | ❌ paren misplaced | ✅ |
| L5, L6 Series | ✅ | ✅ |
| L7 SMA | ❌ | ❌ (but different — see below) |
Plain Form-A: 6/8. Format 4: 7/8. The annotation cracked the bracket-balance wall that broke L4 — the per-paren depth keeps openers and closers paired.
What format 4 does to L7 (the hardest level)
In plain Form-A, L7 failed on bracket imbalance. In format 4, Qwen produced
perfectly balanced brackets even at the deepest nesting (the depth
annotations all matched through the whole SMA) AND the correct SMA logic
((/ (+ (at 0) (+ (at 1) (at 2))) 3.0)). Proof the building blocks were
right: taking Qwen's exact logic and only fixing the one remaining issue makes
it run and emit the exact expected output (3.0 / 5.33333 / 5.66667 / 5.33333)
— see /tmp/abl/qwen_seqfix.ail reconstruction in session.
The one remaining issue was not a format problem: Qwen treats seq as
variadic (Lisp progn) — (seq A B C) — but AILang seq is strictly binary.
Pushing one more level (format 4 + an explicit seq-is-binary hint)
Gave Qwen the binary-seq rule plus a nested-seq example, re-ran L7. It
fixed the seq nesting (correct (seq (seq print nl) (let …))), but L7
still failed — now on two model-behaviour limits, not surface encoding:
- it simplified the logic (printed
at 0instead of the average), and - a termination-degeneration tail: hundreds of repeated
#0)— Qwen knows via#0it is at the outermost level, yet keeps closing. A pure repetition loop, the same failure mode the ~80k-token full-SMA run hit.
Takeaway
- Annotated parens (format 4) measurably help: they crack the bracket-balance wall (L4 green; L7 brackets perfect). For everything up to medium-deep nesting, the format lifts the exact burden the ablation isolated.
- The hardest level (L7) is no longer a bracketing problem — it is (a) a
construct-arity quirk (
seqbinary vs variadic; fixable by an example or by makingseqn-ary) and (b) a single-shot complexity ceiling where the model simplifies and falls into a closing-token repetition loop. - The repetition tail is paren-driven (
#0)repeated). The natural next test is a bracket-free format (indent, or keyword-delimited blocks) where there is no closing token to loop on — that directly targets the one limit format 4 does not remove. It needs a new round-trip-verified converter (an offside / keyword parser), so it is a deliberate next step, not run here.