Brummel
fdff6cd613
Follows up the SMA controls with the sharper question: does AILang cost a frontier model MORE than a language it knows from training (Rust)? Claude passing SMA is binary and cannot tell "effortless" from "barely". Measured with pass@1 over 6 samples of the same task in AILang (two-example few-shot) vs Rust, single-shot, fresh clueless agents, no tools, oracled afterward. Task 1 (Expr-evaluator — ADT + match + recursion, all in the few-shot): AILang 6/6 = Rust 6/6, no gap; all six AILang answers byte-identical. When every construct is in the few-shot, AILang is not harder. Task 2 (count-greater-than — needs a comparison, OUTSIDE the few-shot): AILang 0/6 vs Rust 6/6. But the logic was correct in all six; they fell on a guessed name (`>`, which AILang lacks). A vocabulary gap, not a reasoning gap — and the single-shot setup is unfair: in Rust Claude implicitly has its training plus the obvious ability to scan an unknown crate, which AILang was denied. The fair test (scan + compiler-loop, how Claude actually works): one Claude context, count-greater-than, given the prelude as a scanned library, then iterated on raw compiler output. Converges in two feedback rounds — gt+match-on-Bool -> (case true ..) -> match (compare h N) .. GT -> green. Decisive contrast: no-scan + terse errors -> Claude DIVERGES (invents gtPos, sub, non-existent pat-var); scan + diagnostic errors -> Claude CONVERGES. Same model; the difference is the discovery affordances, not the model. The 0/6 was the artefact of an unfair test, not an intrinsic AILang weakness — with the scan-plus-iterate workflow, the "language is harder" gap dissolves. Two genuine AILang defects the thread exposed, filed as issues: - #69: `ail builtins` is an incomplete API scan — it lists no comparison operator; gt/lt/le/ge/compare live in the prelude, which builtins does not surface. A model scanning the obvious discovery tool never finds half the comparison stdlib. - #70: `match` on Bool passes `ail check` but codegen rejects it (`match on non-ADT scrutinee (i1); MVP supports only ADTs`); with no `if`, branching on a Bool has no working surface — one must route through compare->Ordering. A check/codegen inconsistency under an "internal:" prefix. Answer, decomposed: cognitively AILang is not harder (the algorithm transfers); vocabulary-without-scan is harder but that is any unfamiliar language; with scan + iteration Claude drives AILang like a foreign crate; where it genuinely is harder is the two fixable tooling/compiler defects above. Caveat: n=6, two small tasks, one model, low sampling variance — exploratory, not a study. Evidence under experiments/2026-05-12-cross-model-authoring/familiar-vs-unfamiliar/.
Cross-model authoring-form test
Empirical measurement of whether .ail.json or .ail is the form a
foreign LLM author reaches for and succeeds with. Single subject for
v1: Qwen3-Coder-Next via IONOS. Two blind cohorts; same four tasks.
Parent spec: docs/specs/0017-cross-model-authoring-form-test.md.
Layout
master/spec.md— canonical mini-spec source (form-agnostic prose +{form-only: X}blocks +{example: id}markers).master/examples/*.ail.json— AST source-of-truth; each example prints to either form via the existing roundtrip machinery.master/tasks/*.task.json— task definitions consumed by the harness. Authored in cma.2, not cma.1.render/— standalone Cargo crate, outside the root workspace, builds the renderer binary.rendered/json.md,rendered/ail.md— projected mini-specs, checked into the repo for review.harness/— Authored in cma.2.runs/<date>-<hash>/— populated byharnessduring a live run.
Running the renderer
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml -- \
--master experiments/2026-05-12-cross-model-authoring/master \
--rendered experiments/2026-05-12-cross-model-authoring/rendered
Running the tests
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml
Three integration tests: example_roundtrip (each example loads,
prints to AIL, reparses to the same canonical bytes), spec_completeness
(every AST variant in ailang_core::ast is exercised by at least
one example), token_balance (form-only blocks balanced within ±5%
across the two rendered files).
Running the harness
Live mode (one full eight-run sweep, ~480k tokens budget by default):
export IONOS_API_TOKEN="<token>" # see roadmap entry for token provenance
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out experiments/2026-05-12-cross-model-authoring/runs \
--model Qwen/Qwen3-Coder-Next
The harness pre-flights ail --version and clang --version before
the first API call. Set AIL_BIN if ail is not on PATH.
Mock mode (offline; CI-friendly; bypasses the IONOS API):
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out /tmp/mock-runs \
--model mock \
--mock experiments/2026-05-12-cross-model-authoring/harness/tests/fixtures/mock_full_run.json
Tests:
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
Five suites: inline --lib unit tests for strip_locations (5),
plus integration tests strip_locations against captured stderr
fixtures (5), verify_references (1, drives every reference through
the real ail+clang pipeline), mock_full_run (1, full eight-row
mock E2E), budget_abort (1). Total 13 passed.