AILang/experiments/2026-05-12-cross-model-authoring/format-findings.md

# Do alternative surface formats help Qwen? — format 4 (annotated parens)

**Date:** 2026-06-02  **Model:** Qwen/Qwen3-Coder-Next (IONOS)
**Question (user):** the fully-parenthesised surface is only one encoding of
the tree. If no human must read it, try formats that lift the paren-counting
burden — indent, annotated indent, keywords, annotated parens. Does any help?

**Discipline (the "watch out"):** these formats have NO `ail` parser, so
`ail check` cannot judge them directly. For each, a converter (format → Form-A)
is built and **round-trip-verified** (Form-A → format → back must be
parse-identical to the original, on the known-good demos) BEFORE any Qwen
output is judged. Format 4's converter passed: parse-identity + green check on
all three demos, and end-to-end on a fenced annotated program.

## Format 4 tested: annotated parens `(#N ... #N)`

Every paren carries its nesting depth; opener `(#N ` pairs with closer ` #N)`.
Qwen writes the annotated dialect; the verified stripper removes the
annotations back to plain Form-A, then `ail check`/`run`. Same ablation ladder
as the plain-Form-A baseline, so results are comparable. Raw: `qwen-fmt4.md`,
`qwen-fmt4-seqhint.md`.

### Result: it helps, measurably

| Level | plain Form-A | **format 4** |
|---|---|---|
| L0–L3 | ✅ | ✅ |
| **L4** build-list + recursion | ❌ paren misplaced | **✅** |
| L5, L6 Series | ✅ | ✅ |
| L7 SMA | ❌ | ❌ (but different — see below) |

Plain Form-A: 6/8. **Format 4: 7/8.** The annotation cracked the
bracket-balance wall that broke L4 — the per-paren depth keeps openers and
closers paired.

### What format 4 does to L7 (the hardest level)

In plain Form-A, L7 failed on bracket imbalance. In format 4, Qwen produced
**perfectly balanced brackets even at the deepest nesting** (the depth
annotations all matched through the whole SMA) AND the **correct SMA logic**
(`(/ (+ (at 0) (+ (at 1) (at 2))) 3.0)`). Proof the building blocks were
right: taking Qwen's exact logic and only fixing the one remaining issue makes
it run and emit the exact expected output (`3.0 / 5.33333 / 5.66667 / 5.33333`)
— see `/tmp/abl/qwen_seqfix.ail` reconstruction in session.

The one remaining issue was **not a format problem**: Qwen treats `seq` as
variadic (Lisp `progn`) — `(seq A B C)` — but AILang `seq` is strictly binary.

### Pushing one more level (format 4 + an explicit `seq`-is-binary hint)

Gave Qwen the binary-`seq` rule plus a nested-`seq` example, re-ran L7. It
**fixed the seq nesting** (correct `(seq (seq print nl) (let …))`), but L7
still failed — now on two model-behaviour limits, not surface encoding:
1. it **simplified the logic** (printed `at 0` instead of the average), and
2. a **termination-degeneration tail**: hundreds of repeated `#0)` — Qwen
   *knows* via `#0` it is at the outermost level, yet keeps closing. A pure
   repetition loop, the same failure mode the ~80k-token full-SMA run hit.

## Takeaway

- **Annotated parens (format 4) measurably help**: they crack the
  bracket-balance wall (L4 green; L7 brackets perfect). For everything up to
  medium-deep nesting, the format lifts the exact burden the ablation isolated.
- **The hardest level (L7) is no longer a bracketing problem** — it is (a) a
  construct-arity quirk (`seq` binary vs variadic; fixable by an example or by
  making `seq` n-ary) and (b) a single-shot complexity ceiling where the model
  simplifies and falls into a closing-token repetition loop.
- The repetition tail is paren-driven (`#0)` repeated). The natural next test
  was a **bracket-free** format — run below.

## Bracket-free formats tested: indent / YAML — they made it WORSE

The hypothesis was that removing the closing token would remove the repetition
tail. It did the opposite — bracket-free formats are clearly worse. Two were
tested, same ladder, each with a round-trip-verified converter:

- **YAML via PyYAML** (`qwen-yaml.md`): 2/8. First lesson was about the
  *instrument*, not the model: a real YAML library auto-quotes and gives `*`,
  `-`, `true` special meaning — and the AILang operators collide. Qwen wrote
  `- *` (multiply), valid as an atom but a YAML alias marker, so the library
  crashed. **You cannot use an auto-quoting parser for an operator-rich
  vocabulary.** Discarded.
- **yamlish, own literal parser** (`qwen-yamlish.md`): 3/8. Re-built with a
  hand-written indent parser that takes atoms verbatim (no quoting, no alias) —
  `- *` round-trips fine. With the instrument fixed, the model still only
  reached 3/8: trivial programs (L0–L2) pass, but at the first real nesting
  (L3 ADT, L4 data-def) the indentation structure breaks (e.g. a `con` field
  type lands as a `data` attribute — the model lost the indent level).

## M-expressions tested: middle of the pack

McCarthy's `head[arg; arg]` notation — explicit structure (brackets +
semicolons) in the familiar function-call shape. Hypothesis: explicit + a
shape LLMs know cold should win. Result: 4–5/8 (model variance between two
runs), BELOW plain parens, ABOVE the bracket-free formats. Two watch-out
catches kept the measurement honest:
- A trailing `;` before `]` (Qwen wrote `let[s; v; ]`, a normal trailing
  separator) — my parser was too strict and rejected it. Fixed to tolerate it
  (round-trip preserved). Without this, M-expr would have looked unfairly bad.
- L4 then failed with **43 `[` vs 42 `]`** — a genuine Qwen bracket imbalance
  (not truncation: 301 completion tokens). The same depth-tracking failure as
  plain parens, just with `[` instead of `(`.

Why M-expr is worse than plain parens despite being "explicit": swapping `()`
for `[]` does nothing for the balance burden (square is no easier than round),
and the semicolons *add* a second consistency requirement Qwen does not
reliably meet (the trailing-`;` slips). It is explicit, but with extra load,
not redundant cue.

## Tool-calling / structured AST tested: WORST, and it confirms the rule

The intuitive bet: Qwen3-Coder is trained hard on tool-calling, the API
validates the arguments, so submitting the program as canonical AST JSON via a
`submit_program` tool should be the most reliable channel. Tool-calling is
supported (verified: a dummy `add` tool returned a correct `tool_call`). But
the result is **2/8 — tied for worst.** Two failure modes, both the model's:
- `check` (L1/L3/L5/L6): valid JSON, wrong AST — e.g. `* ` given 3 args.
- **`bad-json` (L4/L7): the tool arguments were not valid JSON at all** —
  completion hit the 2500-token cap. The AST JSON is so verbose (every leaf is
  `{"t":"var","name":"x"}`) that a non-trivial program overruns the budget and
  the JSON is truncated. IONOS does **not** constrain decoding to the schema,
  so the structure burden is fully on the model — and it is the *heaviest*
  burden of all: braces + brackets + keys + quotes + commas + `"t":`
  discriminators, every node.

So the "modern" structured-output channel is the worst surface here, for
exactly the reason the rule predicts: AST JSON is maximal extra bookkeeping
with no redundant cue.

## Overall finding across all six surfaces

| surface | green | structure marking |
|---|---|---|
| **format 4 — annotated parens** | **7/8** | maximal-explicit, redundant (every paren + its depth) |
| plain Form-A — parens | 6/8 | explicit, one channel |
| M-expressions `head[a; b]` | 4–5/8 | explicit, but `[]`+`;` adds a separator burden |
| yamlish / indent (own parser) | 3/8 | implicit (indentation) |
| YAML (library) | 2/8 | implicit + special-char collision |
| tool-calling / AST JSON | 2/8 | maximal bookkeeping, verbose, no redundancy |

The result is monotone and the opposite of the starting hypothesis. The
discriminator is sharper than "explicit vs implicit": it is **redundant cue vs
extra burden.** Qwen's weakness is structure-tracking at depth. What helps is a
marking that makes that tracking *easier without adding work* — depth-annotated
parens carry the nesting depth redundantly, so the model cannot lose its place
(format 4, best). What hurts is either (a) moving the tracking onto an implicit
channel it must maintain itself — indentation (yamlish/YAML, worst) — or (b)
adding a second thing to keep consistent on top of bracket balance — the
M-expr semicolons (middle), or (c) maximising the bookkeeping outright — AST
JSON over the tool channel, the heaviest surface and tied for worst, even
though tool-calling is the model's home turf. Plain parens sit in between: one
explicit channel, no redundancy, no extra burden.

The L7 closing-token repetition tail is a separate single-shot ceiling, not
something any surface here fixes — the bracket-free surfaces fail earlier,
before that ceiling is even reached.

**For an LLM-authored language the lever is redundant, self-checking structure
(depth-annotated brackets), not a lighter surface and not a more familiar one
that carries extra bookkeeping.**