diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/README.md b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/README.md new file mode 100644 index 0000000..6bbe6f2 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/README.md @@ -0,0 +1,44 @@ +# Is AILang harder for a frontier model than a familiar language? + +Evidence corpus for the "Is AILang harder for a frontier model…" section of +`../format-findings.md`. The question: does AILang cost Claude *more* than a +language it knows from training (Rust)? Method: pass@1 over 6 samples of the +same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless +agents, no tools, oracled afterward (`ail check`/`run` resp. `rustc` + stdout). + +## task1-eval — ADT + match + recursion (all in the few-shot) + +Evaluate `(A + B) * C` over a small `Expr` ADT. **AILang 6/6 = Rust 6/6**, no +gap — and all six AILang answers were byte-identical. When every construct is in +the few-shot, AILang is not harder for Claude. `ail/m.ail` runs to `20`. + +## task2-countgt — count elements `> N` (needs a comparison, OUTSIDE the few-shot) + +- `ail-noscan/m.ail` — the single-shot answer. The *logic is correct* (recurse, + test, accumulate); it fails only on a guessed name: `> ` is not an AILang + identifier. Representative of **0/6** in this condition. The Rust cell + (`rust.rs`, idiomatic `filter().count()`) is **6/6**. So the single-shot gap + is a *vocabulary* gap, not a reasoning gap — and the single-shot setup is + unfair: in Rust Claude has its training plus the obvious ability to scan a + crate, which AILang was denied. + +- `ail-scan-loop/m.ail` — the **fair test**: same task, but Claude is given the + prelude as a scanned library and then iterates on raw compiler output (how it + actually works on an unfamiliar crate). It converges in two feedback rounds: + `gt`+`(pat-ctor True)` → `(case true …)` → finally `match (compare h N) … GT`. + Runs to `3`. The idiomatic path matches on the `Ordering` ADT that `compare` + returns — never on a `Bool`. + +## The finding + +*No scan + terse errors* → Claude diverges. *Scan + diagnostic errors* → Claude +converges. Same model; the difference is the discovery affordances. The +"language is harder" gap dissolves under the workflow Claude really uses. + +Two genuine AILang defects the thread exposed (filed as Gitea issues): + +1. `ail builtins` lists no comparison operator — they live in the prelude, which + `builtins` doesn't surface. The discovery tool is an incomplete API scan. +2. `match` on `Bool` passes `ail check` but is rejected by codegen + (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`, + branching on a Bool is a dead end. A check/codegen inconsistency. diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/ail/m.ail b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/ail/m.ail new file mode 100644 index 0000000..2872683 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/ail/m.ail @@ -0,0 +1,16 @@ +(module m + (data Expr (ctor Lit (con Int)) (ctor Add (con Expr) (con Expr)) (ctor Mul (con Expr) (con Expr))) + (fn eval + (type (fn-type (params (own (con Expr))) (ret (own (con Int))))) + (params e) + (body + (match e + (case (pat-ctor Lit n) n) + (case (pat-ctor Add a b) (app + (app eval a) (app eval b))) + (case (pat-ctor Mul a b) (app * (app eval a) (app eval b)))))) + (fn main + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let e (term-ctor Expr Mul (term-ctor Expr Add (term-ctor Expr Lit 2) (term-ctor Expr Lit 3)) (term-ctor Expr Lit 4)) + (seq (app print (app eval e)) (do io/print_str "\n")))))) diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/rust.rs b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/rust.rs new file mode 100644 index 0000000..cecd7c5 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task1-eval/rust.rs @@ -0,0 +1,15 @@ +enum Expr { Lit(i64), Add(Box, Box), Mul(Box, Box) } +fn eval(e: &Expr) -> i64 { + match e { + Expr::Lit(n) => *n, + Expr::Add(a, b) => eval(a) + eval(b), + Expr::Mul(a, b) => eval(a) * eval(b), + } +} +fn main() { + let expr = Expr::Mul( + Box::new(Expr::Add(Box::new(Expr::Lit(2)), Box::new(Expr::Lit(3)))), + Box::new(Expr::Lit(4)), + ); + println!("{}", eval(&expr)); +} diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-noscan/m.ail b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-noscan/m.ail new file mode 100644 index 0000000..ac6a7e8 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-noscan/m.ail @@ -0,0 +1,18 @@ +(module m + (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList))) + (fn countGt + (type (fn-type (params (own (con IntList))) (ret (own (con Int))))) + (params xs) + (body + (match xs + (case (pat-ctor Nil) 0) + (case (pat-ctor Cons h t) + (match (app > h 5) + (case (pat-ctor True) (app + 1 (app countGt t))) + (case (pat-ctor False) (app countGt t))))))) + (fn main + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let xs (term-ctor IntList Cons 3 (term-ctor IntList Cons 7 (term-ctor IntList Cons 2 (term-ctor IntList Cons 9 (term-ctor IntList Cons 5 (term-ctor IntList Cons 8 (term-ctor IntList Nil))))))) + (seq (app print (app countGt xs)) (do io/print_str "\n")))))) diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-scan-loop/m.ail b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-scan-loop/m.ail new file mode 100644 index 0000000..71d9f14 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/ail-scan-loop/m.ail @@ -0,0 +1,19 @@ +(module m + (data IntList (ctor Nil) (ctor Cons (con Int) (con IntList))) + (fn countGt + (type (fn-type (params (own (con IntList))) (ret (own (con Int))))) + (params xs) + (body + (match xs + (case (pat-ctor Nil) 0) + (case (pat-ctor Cons h t) + (let rest (app countGt t) + (match (app compare h 5) + (case (pat-ctor GT) (app + 1 rest)) + (case _ rest))))))) + (fn main + (type (fn-type (params) (ret (own (con Unit))) (effects IO))) + (params) + (body + (let xs (term-ctor IntList Cons 3 (term-ctor IntList Cons 7 (term-ctor IntList Cons 2 (term-ctor IntList Cons 9 (term-ctor IntList Cons 5 (term-ctor IntList Cons 8 (term-ctor IntList Nil))))))) + (seq (app print (app countGt xs)) (do io/print_str "\n")))))) diff --git a/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/rust.rs b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/rust.rs new file mode 100644 index 0000000..4c91838 --- /dev/null +++ b/experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/task2-countgt/rust.rs @@ -0,0 +1,5 @@ +fn main() { + let numbers = [3, 7, 2, 9, 5, 8]; + let count = numbers.iter().filter(|&&n| n > 5).count(); + println!("{}", count); +} diff --git a/experiments/2026-05-12-cross-model-authoring/format-findings.md b/experiments/2026-05-12-cross-model-authoring/format-findings.md index ee38959..9c6a710 100644 --- a/experiments/2026-05-12-cross-model-authoring/format-findings.md +++ b/experiments/2026-05-12-cross-model-authoring/format-findings.md @@ -318,3 +318,85 @@ bar; it rewards the model that would already be close blind. AILang either targets frontier-class authors, or its surface must drop the structure-tracking load enough for a small model to keep its place (the direction the annotated- paren result points, which still wasn't enough for L7). + +## Is AILang harder for a frontier model than a familiar language? (2026-06-02) + +Claude solving SMA is binary — it cannot tell "effortless" from "barely". The +sharper question: does AILang cost a frontier model *more* than a language it +knows (Rust)? Measured with pass@1 over 6 samples of the same task in AILang +(few-shot) vs Rust (from training), single-shot, no tools, oracled afterward. + +### Two single-shot tasks + +| task | AILang | Rust | +|---|---|---| +| **Expr-evaluator** — ADT + match + recursion, all in the few-shot | **6/6** | 6/6 | +| **count-greater-than** — needs a comparison, OUTSIDE the few-shot | **0/6** | 6/6 | + +Task 1: no gap. All six AILang answers came back byte-identical — Claude +transfers the recursive eval with zero hesitation. When every construct is in +the few-shot, AILang is not harder. + +Task 2: maximal gap — but **not at the logic**. All six AILang attempts had the +correct structure (recurse, test each element, accumulate); they fell on a +*name*: Claude guessed `>`, which AILang doesn't have. A vocabulary gap, not a +reasoning gap. In Rust `>` is in Claude's head; in AILang it must be guessed. + +### The fair test: scan + compiler-loop (how Claude actually works) + +The single-shot setup is unfair: in Rust, Claude implicitly has its whole +training plus the obvious ability to *scan* an unknown crate (`cargo doc`, the +source) before writing. For AILang it had two examples and no scan. The fair +comparison gives AILang the same affordance. One Claude context, count-greater- +than, given the prelude as a scanned library, then iterated on raw compiler +output: + +| round | what Claude did | oracle | +|---|---|---| +| 0, no scan | guesses `>` | `unknown identifier: >` | +| 0, with prelude scan | uses `gt` correctly, guesses `(pat-ctor True)` | `True is not a ctor of Bool` | +| 1, after feedback | corrects to `(case true …)` | `check` green, **codegen**: `match on i1` | +| 2, after feedback | switches to `match (compare h N) … GT` | ✅ **green** | + +**The decisive contrast:** *no scan + terse errors* → Claude **diverges** (0/3, +invents `gtPos`, `sub`, the non-existent `pat-var`, breaks the logic). *Scan + +diagnostic errors* → Claude **converges** to the idiomatic path (match on the +`Ordering` ADT that `compare` returns) in two rounds. Same model both times. The +difference is the **discovery affordances**, not the model. The 0/6 single-shot +number was the artefact of an unfair test (no scan, terse error), not an +intrinsic AILang weakness — with the scan-plus-iterate workflow Claude uses on +any unfamiliar crate, the "language is harder" gap dissolves. + +### …except two real AILang defects the thread exposed + +Independent of Claude, the probe surfaced two genuine tooling/compiler gaps: + +1. **`ail builtins` is an incomplete API scan.** The obvious discovery tool (the + `cargo doc` analogue) lists `+ - * / %` but **no comparison operator at all** — + `gt`/`lt`/`compare` live in the prelude, which `builtins` does not surface. A + model that dutifully scans the built-in discovery tool never finds half the + stdlib. (→ Gitea issue) +2. **`match` on `Bool`: `check` accepts, codegen rejects.** `ail check` passes + `(match bool_expr (case true …))` (typecheck green), but codegen aborts with + `internal: match on non-ADT scrutinee (i1); MVP supports only ADTs`. A + check/codegen inconsistency that only bites at build time, under an "internal" + prefix. Combined with the absence of an `if`, branching on a Bool condition is + a dead end — one must route through `compare`→`Ordering`, which is far from + obvious. (→ Gitea issue) + +### Answer, decomposed + +- **Cognitively:** no — Task 1, and the correct logic everywhere, show the + algorithm transfers fully. +- **Vocabulary, no scan:** yes, strongly — but that is *any* unfamiliar + language without docs, not AILang-specific. +- **Vocabulary, with scan + iteration:** no — Claude drives AILang like a + foreign crate, *provided the affordances hold*. +- **Where AILang genuinely is harder:** the two defects above — an incomplete + discovery surface and a check/codegen inconsistency with no `if`. Fixable + compiler/tooling gaps, not a property of "an LLM writing S-expressions". + +Caveat: n=6 per cell, two small tasks, one model; low sampling variance +(byte-identical answers on Task 1) makes this exploratory, not a study. The +signal is consistent across the layers. Evidence corpus under +`familiar-vs-unfamiliar/`.