experiment(cma): familiar-vs-unfamiliar — AILang is not harder for a frontier model given a scan (refs #68 #69 #70)

Follows up the SMA controls with the sharper question: does AILang cost a
frontier model MORE than a language it knows from training (Rust)? Claude
passing SMA is binary and cannot tell "effortless" from "barely". Measured with
pass@1 over 6 samples of the same task in AILang (two-example few-shot) vs Rust,
single-shot, fresh clueless agents, no tools, oracled afterward.

Task 1 (Expr-evaluator — ADT + match + recursion, all in the few-shot):
AILang 6/6 = Rust 6/6, no gap; all six AILang answers byte-identical. When every
construct is in the few-shot, AILang is not harder.

Task 2 (count-greater-than — needs a comparison, OUTSIDE the few-shot): AILang
0/6 vs Rust 6/6. But the logic was correct in all six; they fell on a guessed
name (`>`, which AILang lacks). A vocabulary gap, not a reasoning gap — and the
single-shot setup is unfair: in Rust Claude implicitly has its training plus the
obvious ability to scan an unknown crate, which AILang was denied.

The fair test (scan + compiler-loop, how Claude actually works): one Claude
context, count-greater-than, given the prelude as a scanned library, then
iterated on raw compiler output. Converges in two feedback rounds —
gt+match-on-Bool -> (case true ..) -> match (compare h N) .. GT -> green.
Decisive contrast: no-scan + terse errors -> Claude DIVERGES (invents gtPos,
sub, non-existent pat-var); scan + diagnostic errors -> Claude CONVERGES. Same
model; the difference is the discovery affordances, not the model. The 0/6 was
the artefact of an unfair test, not an intrinsic AILang weakness — with the
scan-plus-iterate workflow, the "language is harder" gap dissolves.

Two genuine AILang defects the thread exposed, filed as issues:
- #69: `ail builtins` is an incomplete API scan — it lists no comparison
  operator; gt/lt/le/ge/compare live in the prelude, which builtins does not
  surface. A model scanning the obvious discovery tool never finds half the
  comparison stdlib.
- #70: `match` on Bool passes `ail check` but codegen rejects it
  (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`,
  branching on a Bool has no working surface — one must route through
  compare->Ordering. A check/codegen inconsistency under an "internal:" prefix.

Answer, decomposed: cognitively AILang is not harder (the algorithm transfers);
vocabulary-without-scan is harder but that is any unfamiliar language; with scan
+ iteration Claude drives AILang like a foreign crate; where it genuinely is
harder is the two fixable tooling/compiler defects above. Caveat: n=6, two small
tasks, one model, low sampling variance — exploratory, not a study. Evidence
under experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/.

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/README.md

@@ -0,0 +1,44 @@
+# Is AILang harder for a frontier model than a familiar language?
+
+Evidence corpus for the "Is AILang harder for a frontier model…" section of
+`../format-findings.md`. The question: does AILang cost Claude *more* than a
+language it knows from training (Rust)? Method: pass@1 over 6 samples of the
+same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless
+agents, no tools, oracled afterward (`ail check`/`run` resp. `rustc` + stdout).
+
+## task1-eval — ADT + match + recursion (all in the few-shot)
+
+Evaluate `(A + B) * C` over a small `Expr` ADT. **AILang 6/6 = Rust 6/6**, no
+gap — and all six AILang answers were byte-identical. When every construct is in
+the few-shot, AILang is not harder for Claude. `ail/m.ail` runs to `20`.
+
+## task2-countgt — count elements `> N` (needs a comparison, OUTSIDE the few-shot)
+
+- `ail-noscan/m.ail` — the single-shot answer. The *logic is correct* (recurse,
+  test, accumulate); it fails only on a guessed name: `> ` is not an AILang
+  identifier. Representative of **0/6** in this condition. The Rust cell
+  (`rust.rs`, idiomatic `filter().count()`) is **6/6**. So the single-shot gap
+  is a *vocabulary* gap, not a reasoning gap — and the single-shot setup is
+  unfair: in Rust Claude has its training plus the obvious ability to scan a
+  crate, which AILang was denied.
+
+- `ail-scan-loop/m.ail` — the **fair test**: same task, but Claude is given the
+  prelude as a scanned library and then iterates on raw compiler output (how it
+  actually works on an unfamiliar crate). It converges in two feedback rounds:
+  `gt`+`(pat-ctor True)` → `(case true …)` → finally `match (compare h N) … GT`.
+  Runs to `3`. The idiomatic path matches on the `Ordering` ADT that `compare`
+  returns — never on a `Bool`.
+
+## The finding
+
+*No scan + terse errors* → Claude diverges. *Scan + diagnostic errors* → Claude
+converges. Same model; the difference is the discovery affordances. The
+"language is harder" gap dissolves under the workflow Claude really uses.
+
+Two genuine AILang defects the thread exposed (filed as Gitea issues):
+
+1. `ail builtins` lists no comparison operator — they live in the prelude, which
+   `builtins` doesn't surface. The discovery tool is an incomplete API scan.
+2. `match` on `Bool` passes `ail check` but is rejected by codegen
+   (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`,
+   branching on a Bool is a dead end. A check/codegen inconsistency.

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/ail/m.ail

@@ -0,0 +1,16 @@
+(module m
+  (data Expr (ctor Lit (con Int)) (ctor Add (con Expr) (con Expr)) (ctor Mul (con Expr) (con Expr)))
+  (fn eval
+    (type (fn-type (params (own (con Expr))) (ret (own (con Int)))))
+    (params e)
+    (body
+      (match e
+        (case (pat-ctor Lit n) n)
+        (case (pat-ctor Add a b) (app + (app eval a) (app eval b)))
+        (case (pat-ctor Mul a b) (app * (app eval a) (app eval b))))))
+  (fn main
+    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
+    (params)
+    (body
+      (let e (term-ctor Expr Mul (term-ctor Expr Add (term-ctor Expr Lit 2) (term-ctor Expr Lit 3)) (term-ctor Expr Lit 4))
+        (seq (app print (app eval e)) (do io/print_str "\n"))))))

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/rust.rs

@@ -0,0 +1,15 @@
+enum Expr { Lit(i64), Add(Box<Expr>, Box<Expr>), Mul(Box<Expr>, Box<Expr>) }
+fn eval(e: &Expr) -> i64 {
+    match e {
+        Expr::Lit(n) => *n,
+        Expr::Add(a, b) => eval(a) + eval(b),
+        Expr::Mul(a, b) => eval(a) * eval(b),
+    }
+}
+fn main() {
+    let expr = Expr::Mul(
+        Box::new(Expr::Add(Box::new(Expr::Lit(2)), Box::new(Expr::Lit(3)))),
+        Box::new(Expr::Lit(4)),
+    );
+    println!("{}", eval(&expr));
+}

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-noscan/m.ail

@@ -0,0 +1,18 @@
+(module m
+  (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
+  (fn countGt
+    (type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
+    (params xs)
+    (body
+      (match xs
+        (case (pat-ctor Nil) 0)
+        (case (pat-ctor Cons h t)
+          (match (app > h 5)
+            (case (pat-ctor True) (app + 1 (app countGt t)))
+            (case (pat-ctor False) (app countGt t)))))))
+  (fn main
+    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
+    (params)
+    (body
+      (let xs (term-ctor IntList Cons 3 (term-ctor IntList Cons 7 (term-ctor IntList Cons 2 (term-ctor IntList Cons 9 (term-ctor IntList Cons 5 (term-ctor IntList Cons 8 (term-ctor IntList Nil)))))))
+        (seq (app print (app countGt xs)) (do io/print_str "\n"))))))

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-scan-loop/m.ail

@@ -0,0 +1,19 @@
+(module m
+  (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList)))
+  (fn countGt
+    (type (fn-type (params (own (con IntList))) (ret (own (con Int)))))
+    (params xs)
+    (body
+      (match xs
+        (case (pat-ctor Nil) 0)
+        (case (pat-ctor Cons h t)
+          (let rest (app countGt t)
+            (match (app compare h 5)
+              (case (pat-ctor GT) (app + 1 rest))
+              (case _ rest)))))))
+  (fn main
+    (type (fn-type (params) (ret (own (con Unit))) (effects IO)))
+    (params)
+    (body
+      (let xs (term-ctor IntList Cons 3 (term-ctor IntList Cons 7 (term-ctor IntList Cons 2 (term-ctor IntList Cons 9 (term-ctor IntList Cons 5 (term-ctor IntList Cons 8 (term-ctor IntList Nil)))))))
+        (seq (app print (app countGt xs)) (do io/print_str "\n"))))))

experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/rust.rs

@@ -0,0 +1,5 @@
+fn main() {
+    let numbers = [3, 7, 2, 9, 5, 8];
+    let count = numbers.iter().filter(|&&n| n > 5).count();
+    println!("{}", count);
+}

experiments/2026-05-12-cross-model-authoring/format-findings.md

@@ -318,3 +318,85 @@ bar; it rewards the model that would already be close blind. AILang either
 targets frontier-class authors, or its surface must drop the structure-tracking
 load enough for a small model to keep its place (the direction the annotated-
 paren result points, which still wasn't enough for L7).
+
+## Is AILang harder for a frontier model than a familiar language? (2026-06-02)
+
+Claude solving SMA is binary — it cannot tell "effortless" from "barely". The
+sharper question: does AILang cost a frontier model *more* than a language it
+knows (Rust)? Measured with pass@1 over 6 samples of the same task in AILang
+(few-shot) vs Rust (from training), single-shot, no tools, oracled afterward.
+
+### Two single-shot tasks
+
+| task | AILang | Rust |
+|---|---|---|
+| **Expr-evaluator** — ADT + match + recursion, all in the few-shot | **6/6** | 6/6 |
+| **count-greater-than** — needs a comparison, OUTSIDE the few-shot | **0/6** | 6/6 |
+
+Task 1: no gap. All six AILang answers came back byte-identical — Claude
+transfers the recursive eval with zero hesitation. When every construct is in
+the few-shot, AILang is not harder.
+
+Task 2: maximal gap — but **not at the logic**. All six AILang attempts had the
+correct structure (recurse, test each element, accumulate); they fell on a
+*name*: Claude guessed `>`, which AILang doesn't have. A vocabulary gap, not a
+reasoning gap. In Rust `>` is in Claude's head; in AILang it must be guessed.
+
+### The fair test: scan + compiler-loop (how Claude actually works)
+
+The single-shot setup is unfair: in Rust, Claude implicitly has its whole
+training plus the obvious ability to *scan* an unknown crate (`cargo doc`, the
+source) before writing. For AILang it had two examples and no scan. The fair
+comparison gives AILang the same affordance. One Claude context, count-greater-
+than, given the prelude as a scanned library, then iterated on raw compiler
+output:
+
+| round | what Claude did | oracle |
+|---|---|---|
+| 0, no scan | guesses `>` | `unknown identifier: >` |
+| 0, with prelude scan | uses `gt` correctly, guesses `(pat-ctor True)` | `True is not a ctor of Bool` |
+| 1, after feedback | corrects to `(case true …)` | `check` green, **codegen**: `match on i1` |
+| 2, after feedback | switches to `match (compare h N) … GT` | ✅ **green** |
+
+**The decisive contrast:** *no scan + terse errors* → Claude **diverges** (0/3,
+invents `gtPos`, `sub`, the non-existent `pat-var`, breaks the logic). *Scan +
+diagnostic errors* → Claude **converges** to the idiomatic path (match on the
+`Ordering` ADT that `compare` returns) in two rounds. Same model both times. The
+difference is the **discovery affordances**, not the model. The 0/6 single-shot
+number was the artefact of an unfair test (no scan, terse error), not an
+intrinsic AILang weakness — with the scan-plus-iterate workflow Claude uses on
+any unfamiliar crate, the "language is harder" gap dissolves.
+
+### …except two real AILang defects the thread exposed
+
+Independent of Claude, the probe surfaced two genuine tooling/compiler gaps:
+
+1. **`ail builtins` is an incomplete API scan.** The obvious discovery tool (the
+   `cargo doc` analogue) lists `+ - * / %` but **no comparison operator at all** —
+   `gt`/`lt`/`compare` live in the prelude, which `builtins` does not surface. A
+   model that dutifully scans the built-in discovery tool never finds half the
+   stdlib. (→ Gitea issue)
+2. **`match` on `Bool`: `check` accepts, codegen rejects.** `ail check` passes
+   `(match bool_expr (case true …))` (typecheck green), but codegen aborts with
+   `internal: match on non-ADT scrutinee (i1); MVP supports only ADTs`. A
+   check/codegen inconsistency that only bites at build time, under an "internal"
+   prefix. Combined with the absence of an `if`, branching on a Bool condition is
+   a dead end — one must route through `compare`→`Ordering`, which is far from
+   obvious. (→ Gitea issue)
+
+### Answer, decomposed
+
+- **Cognitively:** no — Task 1, and the correct logic everywhere, show the
+  algorithm transfers fully.
+- **Vocabulary, no scan:** yes, strongly — but that is *any* unfamiliar
+  language without docs, not AILang-specific.
+- **Vocabulary, with scan + iteration:** no — Claude drives AILang like a
+  foreign crate, *provided the affordances hold*.
+- **Where AILang genuinely is harder:** the two defects above — an incomplete
+  discovery surface and a check/codegen inconsistency with no `if`. Fixable
+  compiler/tooling gaps, not a property of "an LLM writing S-expressions".
+
+Caveat: n=6 per cell, two small tasks, one model; low sampling variance
+(byte-identical answers on Task 1) makes this exploratory, not a study. The
+signal is consistent across the layers. Evidence corpus under
+`familiar-vs-unfamiliar/`.