Follows up the SMA controls with the sharper question: does AILang cost a
frontier model MORE than a language it knows from training (Rust)? Claude
passing SMA is binary and cannot tell "effortless" from "barely". Measured with
pass@1 over 6 samples of the same task in AILang (two-example few-shot) vs Rust,
single-shot, fresh clueless agents, no tools, oracled afterward.
Task 1 (Expr-evaluator — ADT + match + recursion, all in the few-shot):
AILang 6/6 = Rust 6/6, no gap; all six AILang answers byte-identical. When every
construct is in the few-shot, AILang is not harder.
Task 2 (count-greater-than — needs a comparison, OUTSIDE the few-shot): AILang
0/6 vs Rust 6/6. But the logic was correct in all six; they fell on a guessed
name (`>`, which AILang lacks). A vocabulary gap, not a reasoning gap — and the
single-shot setup is unfair: in Rust Claude implicitly has its training plus the
obvious ability to scan an unknown crate, which AILang was denied.
The fair test (scan + compiler-loop, how Claude actually works): one Claude
context, count-greater-than, given the prelude as a scanned library, then
iterated on raw compiler output. Converges in two feedback rounds —
gt+match-on-Bool -> (case true ..) -> match (compare h N) .. GT -> green.
Decisive contrast: no-scan + terse errors -> Claude DIVERGES (invents gtPos,
sub, non-existent pat-var); scan + diagnostic errors -> Claude CONVERGES. Same
model; the difference is the discovery affordances, not the model. The 0/6 was
the artefact of an unfair test, not an intrinsic AILang weakness — with the
scan-plus-iterate workflow, the "language is harder" gap dissolves.
Two genuine AILang defects the thread exposed, filed as issues:
- #69: `ail builtins` is an incomplete API scan — it lists no comparison
operator; gt/lt/le/ge/compare live in the prelude, which builtins does not
surface. A model scanning the obvious discovery tool never finds half the
comparison stdlib.
- #70: `match` on Bool passes `ail check` but codegen rejects it
(`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`,
branching on a Bool has no working surface — one must route through
compare->Ordering. A check/codegen inconsistency under an "internal:" prefix.
Answer, decomposed: cognitively AILang is not harder (the algorithm transfers);
vocabulary-without-scan is harder but that is any unfamiliar language; with scan
+ iteration Claude drives AILang like a foreign crate; where it genuinely is
harder is the two fixable tooling/compiler defects above. Caveat: n=6, two small
tasks, one model, low sampling variance — exploratory, not a study. Evidence
under experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/.
Every surface run in format-findings.md hit the same L7 (SMA) wall, read so far
as a "genuine single-shot complexity ceiling". Three controls on the plain
Form-A baseline (identical L7 prompt + ail oracle) pin down what that ceiling
actually is.
Control 1 — blind frontier model. Ran the full L0–L7 ladder through eight
fresh, clueless Claude Opus 4.8 agents, each given verbatim the same prompt the
IONOS models got, with no repo access, no compiler, no oracle (all returned
tool_uses: 0). Oracled afterward exactly as the harness oracles Qwen/Llama:
8/8 green, including L7. So L7 is not a language ceiling — Form-A is authorable,
SMA included, by a strong enough model. The eight modules are kept under
clueless-agents/ (one m.ail per dir to satisfy module-name=stem) and re-check
green; L7 also runs to the expected output. Claude generalised operators never
shown in the few-shot (float +, a / absent from every example, float print).
Control 2 — agentic Qwen (qwen_agentic.py). The fairer question: does the
compiler-as-a-tool close the gap? Runs the real Claude-Code loop — model
proposes a module, harness runs ail, the raw unedited compiler output goes back
into the dialogue, up to 8 turns. Result: never solved. From turn 1 Qwen
collapses into a repetition loop (~274× "(seq" until max_tokens, completion=1500
every turn); the module is truncated mid-cascade, the oracle returns the correct
parse error every turn, and Qwen repeats. Tool access is not an equaliser — it
lifts a model only when its reasoning already operates in the right
neighbourhood. Also settles the context question: turn 1 collapses at
prompt=729 (no accumulated history), so the failure is "can't", not "drowns in
context". Raw doc qwen-agentic-sma.md (repetition cascades collapsed to keep the
repo small; the collapse marker records the original run length).
Control 3 — same SMA in Python (qwen_python_sma.py). Is the AILang failure
about coding ability or the unfamiliar surface? Given the same task zero-shot in
Python, Qwen returns a clean, idiomatic, correct sliding-window in 113 tokens
(verified by inspection + hand-trace; the harness deliberately does not
auto-execute LLM code). 113 tokens of correct Python vs a 1500-token collapse in
AILang, same model and temperature.
Synthesis: the L7 wall is neither an algorithm ceiling nor a "model is weak"
ceiling. Qwen owns the algorithm (Python) and a frontier model owns the surface
(Claude, blind, first try). Qwen's AILang collapse is specifically the
unfamiliar, fully-parenthesised surface it never saw in training, and a
compiler-in-the-loop does not rescue it. For an LLM-authored language: making
Form-A compiler-driven does not lower the model bar — AILang either targets
frontier-class authors, or its surface must drop the structure-tracking load
enough for a small model to keep its place.
Live IONOS limited to the consented Qwen agentic + Python runs.
Ran the two core formats (plain Form-A, annotated parens) on Llama-3.1-405B
(405B dense, ~135x Qwen3-Coder-Next's 3B active) to separate format-effect from
small-model-effect. Scripts parametrized by PROBE_MODEL/PROBE_TAG (Qwen default
preserved).
Watch-out caught: Llama-3.1 on IONOS leaks its chat template (repeats
"assistant", generates endless variants to max_tokens — an IONOS-documented
quirk). First run read 0/8; that was the instrument. Fixed with a stop sequence
and a balanced-first-module extractor.
Result reverses the effect:
Qwen3-Coder (3B): plain 6/8, annotated 7/8
Llama-405B (405B): plain 7/8, annotated 6/8
Llama solves L4 (Qwen's bracket wall) with plain parens, so that wall was a
small-model effect. Annotated parens help the small model and hurt the large
one (at L1 Llama invented a spurious (lit 3) wrapper under the annotated
dialect). Both still fail L7 (SMA) — a genuine complexity ceiling.
Synthesis: redundant explicit structure is a crutch for weak models. A
frontier model tracks plain brackets fine and the unfamiliar scaffolding only
adds noise. The right surface is model-relative. Synthesis in
format-findings.md; raw ladders llama405-*.md.
Researched the model under test. Qwen3-Coder-Next is a 3B-activated MoE (80B
total, Feb 2026) trained for AGENTIC coding (codebase + tools + RL on
environment feedback), not one-shot generation of unfamiliar syntax — exactly
our task, so we tested its weakest discipline. Its tool-calling strength is
agentic environment interaction, not emitting deep structured AST JSON,
consistent with the tool probe being worst. The format direction is likely
general but the magnitude is model-specific; bigger active models on the same
IONOS endpoint (Llama-3.1-405B, Llama-3.3-70B) were never tried and are the
obvious next step to separate format-effect from small-model-effect.
User noted the context window is ~128k and the bad-json failures looked like
truncation, so the max_tokens=2500 cap may have skewed the tool-calling test.
Fair point: the L4/L7 bad-json cases WERE truncation. Verified max_tokens=8000
is accepted, re-ran.
Result: 1/8, slightly worse, not better. The bad-json cases became check
failures (full AST now produced, but semantically wrong), confirming
truncation was a real but secondary factor. And the extra room exposed the
deeper problem: at L2 ("print 7", trivial) Qwen emitted 8000 tokens / ~20KB of
AST JSON and ran off the end — a repetition loop on a one-line program; at L7
it nested JSON deep enough to hit the parser's recursion limit. The verbose
AST-JSON surface is not budget-limited in a recoverable way — more room just
lets it loop or over-nest. Tool-calling stays at the bottom; the higher budget
makes it degenerate, even on trivial tasks. Updated format-findings.md.
User asked whether Qwen's tool-calling could help. Verified tool-calling is
supported on IONOS (a dummy add tool returns a correct tool_call). Then tested
the promising version: submit the program as canonical AST JSON via a forced
submit_program tool — no surface syntax, no converter risk (the tool argument
IS the .ail.json, written and checked directly).
Result: 2/8, tied for worst. Two failure modes, both the model's: valid JSON
but wrong AST (arity etc.), and — at L4/L7 — bad-json: the tool arguments were
not valid JSON at all, because completion hit the 2500-token cap. AST JSON is
so verbose (every leaf is {"t":"var","name":"x"}) that a non-trivial program
overruns the budget and truncates. IONOS does not constrain decoding to the
schema, so the full structure burden stays on the model, and it is the heaviest
of all surfaces (braces+brackets+keys+quotes+commas+"t": per node).
This confirms the rule across six surfaces (annotated-parens 7/8 > plain parens
6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8 = tool-AST 2/8): the discriminator is
redundant cue vs extra burden, not explicit vs implicit. Even tool-calling, the
model's home turf, loses because AST JSON maximises bookkeeping with no
redundancy. Lever for an LLM-authored language: redundant self-checking
structure (depth-annotated brackets). Synthesis in format-findings.md.
Tested McCarthy's head[a; b] notation (user request). Explicit structure
(brackets + semicolons) in the function-call shape LLMs know cold. Result
4-5/8 (model variance over two runs): below plain parens, above bracket-free.
Two watch-out catches kept it honest: (1) a trailing ';' before ']' — a normal
trailing separator Qwen wrote — was rejected by my too-strict parser; fixed to
tolerate it, round-trip preserved (without this M-expr looked unfairly bad).
(2) L4 then failed with 43 '[' vs 42 ']', a genuine Qwen bracket imbalance (not
truncation), the same depth-tracking failure as plain parens.
Why worse than plain parens despite being explicit: swapping () for [] does
nothing for the balance burden, and the semicolons ADD a second consistency
requirement Qwen does not reliably meet.
This sharpens the overall finding across five formats (annotated-parens 7/8 >
plain parens 6/8 > M-expr 4-5/8 > indent 3/8 > YAML 2/8): the discriminator is
not "explicit vs implicit" but REDUNDANT CUE vs EXTRA BURDEN. Depth-annotated
parens carry nesting redundantly so the model can't lose its place (best);
indentation moves tracking to an implicit channel (worst); M-expr semicolons
add bookkeeping on top of bracket balance (middle). Lever for an LLM-authored
language: redundant self-checking structure, not a lighter or more familiar
surface that carries extra bookkeeping. Synthesis in format-findings.md.
Tested the user's bracket-free ideas (indent, YAML) against the annotated-paren
result. Hypothesis was that removing the closing token removes the repetition
tail. The opposite held.
Instrument note (the watch-out paid off twice): a real YAML library auto-quotes
and gives *,-,true special meaning, which collides with the AILang operators —
Qwen wrote `- *` (multiply), a YAML alias marker, and the library crashed. That
is a measurement bug, not a model failure. Rebuilt with a hand-written literal
indent parser (atoms verbatim, no quoting); `- *` then round-trips fine.
With the instrument fixed, the result is monotone across four formats:
format 4 annotated parens 7/8 > plain Form-A parens 6/8 > yamlish indent 3/8
> YAML library 2/8.
More explicit structure marking helps this model; less hurts. Qwen's weakness
is structure-tracking at depth, and indentation makes it track the nesting
itself — as error-prone as matching parens but without the redundant cue. The
bracket-free surfaces fail EARLIER (L3/L4), before the L7 repetition ceiling is
even reached.
Lever for an LLM-authored language: redundant explicit structure (depth-
annotated brackets), not a lighter surface. Synthesis in format-findings.md;
raw ladders qwen-yaml.md (PyYAML, discarded) and qwen-yamlish.md (own parser).
Tests the user's hypothesis: if no human reads the surface, lift the paren-
counting burden. Format 4 = annotated parens, every paren carries its nesting
depth. Discipline: the format has no ail parser, so the converter is round-trip
verified (parse-identity on known-good demos) before judging any model output.
Result: it helps. Plain Form-A 6/8 -> format 4 7/8 on the same ablation ladder.
The depth annotation cracks the bracket-balance wall that broke L4. At the
hardest level (L7 SMA) Qwen produced perfectly balanced brackets through the
deepest nesting AND correct SMA logic; the only remaining issue was a construct
quirk (it treats seq as variadic, but AILang seq is binary). Proof the blocks
were right: Qwen's exact logic with a manual binary-seq fix runs and emits the
exact expected SMA output.
Pushing further (format 4 + explicit seq-binary hint) fixed the seq nesting but
L7 still failed on two MODEL-behaviour limits, not encoding: it simplified the
logic, and fell into a closing-token repetition tail (hundreds of #0) ). The
tail is paren-driven, so the natural next test is a bracket-free format
(indent / keyword-delimited) with no closing token to loop on — a deliberate
next step needing its own verified converter. Synthesis in format-findings.md.
Brummel