Brummel
682b935ba3
Tests the user's hypothesis: if no human reads the surface, lift the paren- counting burden. Format 4 = annotated parens, every paren carries its nesting depth. Discipline: the format has no ail parser, so the converter is round-trip verified (parse-identity on known-good demos) before judging any model output. Result: it helps. Plain Form-A 6/8 -> format 4 7/8 on the same ablation ladder. The depth annotation cracks the bracket-balance wall that broke L4. At the hardest level (L7 SMA) Qwen produced perfectly balanced brackets through the deepest nesting AND correct SMA logic; the only remaining issue was a construct quirk (it treats seq as variadic, but AILang seq is binary). Proof the blocks were right: Qwen's exact logic with a manual binary-seq fix runs and emits the exact expected SMA output. Pushing further (format 4 + explicit seq-binary hint) fixed the seq nesting but L7 still failed on two MODEL-behaviour limits, not encoding: it simplified the logic, and fell into a closing-token repetition tail (hundreds of #0) ). The tail is paren-driven, so the natural next test is a bracket-free format (indent / keyword-delimited) with no closing token to loop on — a deliberate next step needing its own verified converter. Synthesis in format-findings.md.
73 lines
3.8 KiB
Markdown
73 lines
3.8 KiB
Markdown
# Do alternative surface formats help Qwen? — format 4 (annotated parens)
|
||
|
||
**Date:** 2026-06-02 **Model:** Qwen/Qwen3-Coder-Next (IONOS)
|
||
**Question (user):** the fully-parenthesised surface is only one encoding of
|
||
the tree. If no human must read it, try formats that lift the paren-counting
|
||
burden — indent, annotated indent, keywords, annotated parens. Does any help?
|
||
|
||
**Discipline (the "watch out"):** these formats have NO `ail` parser, so
|
||
`ail check` cannot judge them directly. For each, a converter (format → Form-A)
|
||
is built and **round-trip-verified** (Form-A → format → back must be
|
||
parse-identical to the original, on the known-good demos) BEFORE any Qwen
|
||
output is judged. Format 4's converter passed: parse-identity + green check on
|
||
all three demos, and end-to-end on a fenced annotated program.
|
||
|
||
## Format 4 tested: annotated parens `(#N ... #N)`
|
||
|
||
Every paren carries its nesting depth; opener `(#N ` pairs with closer ` #N)`.
|
||
Qwen writes the annotated dialect; the verified stripper removes the
|
||
annotations back to plain Form-A, then `ail check`/`run`. Same ablation ladder
|
||
as the plain-Form-A baseline, so results are comparable. Raw: `qwen-fmt4.md`,
|
||
`qwen-fmt4-seqhint.md`.
|
||
|
||
### Result: it helps, measurably
|
||
|
||
| Level | plain Form-A | **format 4** |
|
||
|---|---|---|
|
||
| L0–L3 | ✅ | ✅ |
|
||
| **L4** build-list + recursion | ❌ paren misplaced | **✅** |
|
||
| L5, L6 Series | ✅ | ✅ |
|
||
| L7 SMA | ❌ | ❌ (but different — see below) |
|
||
|
||
Plain Form-A: 6/8. **Format 4: 7/8.** The annotation cracked the
|
||
bracket-balance wall that broke L4 — the per-paren depth keeps openers and
|
||
closers paired.
|
||
|
||
### What format 4 does to L7 (the hardest level)
|
||
|
||
In plain Form-A, L7 failed on bracket imbalance. In format 4, Qwen produced
|
||
**perfectly balanced brackets even at the deepest nesting** (the depth
|
||
annotations all matched through the whole SMA) AND the **correct SMA logic**
|
||
(`(/ (+ (at 0) (+ (at 1) (at 2))) 3.0)`). Proof the building blocks were
|
||
right: taking Qwen's exact logic and only fixing the one remaining issue makes
|
||
it run and emit the exact expected output (`3.0 / 5.33333 / 5.66667 / 5.33333`)
|
||
— see `/tmp/abl/qwen_seqfix.ail` reconstruction in session.
|
||
|
||
The one remaining issue was **not a format problem**: Qwen treats `seq` as
|
||
variadic (Lisp `progn`) — `(seq A B C)` — but AILang `seq` is strictly binary.
|
||
|
||
### Pushing one more level (format 4 + an explicit `seq`-is-binary hint)
|
||
|
||
Gave Qwen the binary-`seq` rule plus a nested-`seq` example, re-ran L7. It
|
||
**fixed the seq nesting** (correct `(seq (seq print nl) (let …))`), but L7
|
||
still failed — now on two model-behaviour limits, not surface encoding:
|
||
1. it **simplified the logic** (printed `at 0` instead of the average), and
|
||
2. a **termination-degeneration tail**: hundreds of repeated `#0)` — Qwen
|
||
*knows* via `#0` it is at the outermost level, yet keeps closing. A pure
|
||
repetition loop, the same failure mode the ~80k-token full-SMA run hit.
|
||
|
||
## Takeaway
|
||
|
||
- **Annotated parens (format 4) measurably help**: they crack the
|
||
bracket-balance wall (L4 green; L7 brackets perfect). For everything up to
|
||
medium-deep nesting, the format lifts the exact burden the ablation isolated.
|
||
- **The hardest level (L7) is no longer a bracketing problem** — it is (a) a
|
||
construct-arity quirk (`seq` binary vs variadic; fixable by an example or by
|
||
making `seq` n-ary) and (b) a single-shot complexity ceiling where the model
|
||
simplifies and falls into a closing-token repetition loop.
|
||
- The repetition tail is paren-driven (`#0)` repeated). The natural next test
|
||
is a **bracket-free** format (indent, or keyword-delimited blocks) where
|
||
there is no closing token to loop on — that directly targets the one limit
|
||
format 4 does not remove. It needs a new round-trip-verified converter
|
||
(an offside / keyword parser), so it is a deliberate next step, not run here.
|