Brummel
fdff6cd613
Follows up the SMA controls with the sharper question: does AILang cost a frontier model MORE than a language it knows from training (Rust)? Claude passing SMA is binary and cannot tell "effortless" from "barely". Measured with pass@1 over 6 samples of the same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless agents, no tools, oracled afterward. Task 1 (Expr-evaluator — ADT + match + recursion, all in the few-shot): AILang 6/6 = Rust 6/6, no gap; all six AILang answers byte-identical. When every construct is in the few-shot, AILang is not harder. Task 2 (count-greater-than — needs a comparison, OUTSIDE the few-shot): AILang 0/6 vs Rust 6/6. But the logic was correct in all six; they fell on a guessed name (`>`, which AILang lacks). A vocabulary gap, not a reasoning gap — and the single-shot setup is unfair: in Rust Claude implicitly has its training plus the obvious ability to scan an unknown crate, which AILang was denied. The fair test (scan + compiler-loop, how Claude actually works): one Claude context, count-greater-than, given the prelude as a scanned library, then iterated on raw compiler output. Converges in two feedback rounds — gt+match-on-Bool -> (case true ..) -> match (compare h N) .. GT -> green. Decisive contrast: no-scan + terse errors -> Claude DIVERGES (invents gtPos, sub, non-existent pat-var); scan + diagnostic errors -> Claude CONVERGES. Same model; the difference is the discovery affordances, not the model. The 0/6 was the artefact of an unfair test, not an intrinsic AILang weakness — with the scan-plus-iterate workflow, the "language is harder" gap dissolves. Two genuine AILang defects the thread exposed, filed as issues: - #69: `ail builtins` is an incomplete API scan — it lists no comparison operator; gt/lt/le/ge/compare live in the prelude, which builtins does not surface. A model scanning the obvious discovery tool never finds half the comparison stdlib. - #70: `match` on Bool passes `ail check` but codegen rejects it (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`, branching on a Bool has no working surface — one must route through compare->Ordering. A check/codegen inconsistency under an "internal:" prefix. Answer, decomposed: cognitively AILang is not harder (the algorithm transfers); vocabulary-without-scan is harder but that is any unfamiliar language; with scan + iteration Claude drives AILang like a foreign crate; where it genuinely is harder is the two fixable tooling/compiler defects above. Caveat: n=6, two small tasks, one model, low sampling variance — exploratory, not a study. Evidence under experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/.
2.4 KiB
Is AILang harder for a frontier model than a familiar language?
Evidence corpus for the "Is AILang harder for a frontier model…" section of
../format-findings.md. The question: does AILang cost Claude more than a
language it knows from training (Rust)? Method: pass@1 over 6 samples of the
same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless
agents, no tools, oracled afterward (ail check/run resp. rustc + stdout).
task1-eval — ADT + match + recursion (all in the few-shot)
Evaluate (A + B) * C over a small Expr ADT. AILang 6/6 = Rust 6/6, no
gap — and all six AILang answers were byte-identical. When every construct is in
the few-shot, AILang is not harder for Claude. ail/m.ail runs to 20.
task2-countgt — count elements > N (needs a comparison, OUTSIDE the few-shot)
-
ail-noscan/m.ail— the single-shot answer. The logic is correct (recurse, test, accumulate); it fails only on a guessed name:>is not an AILang identifier. Representative of 0/6 in this condition. The Rust cell (rust.rs, idiomaticfilter().count()) is 6/6. So the single-shot gap is a vocabulary gap, not a reasoning gap — and the single-shot setup is unfair: in Rust Claude has its training plus the obvious ability to scan a crate, which AILang was denied. -
ail-scan-loop/m.ail— the fair test: same task, but Claude is given the prelude as a scanned library and then iterates on raw compiler output (how it actually works on an unfamiliar crate). It converges in two feedback rounds:gt+(pat-ctor True)→(case true …)→ finallymatch (compare h N) … GT. Runs to3. The idiomatic path matches on theOrderingADT thatcomparereturns — never on aBool.
The finding
No scan + terse errors → Claude diverges. Scan + diagnostic errors → Claude converges. Same model; the difference is the discovery affordances. The "language is harder" gap dissolves under the workflow Claude really uses.
Two genuine AILang defects the thread exposed (filed as Gitea issues):
ail builtinslists no comparison operator — they live in the prelude, whichbuiltinsdoesn't surface. The discovery tool is an incomplete API scan.matchonBoolpassesail checkbut is rejected by codegen (match on non-ADT scrutinee (i1); MVP supports only ADTs); with noif, branching on a Bool is a dead end. A check/codegen inconsistency.