Brummel
9c833f65e4
Every surface run in format-findings.md hit the same L7 (SMA) wall, read so far as a "genuine single-shot complexity ceiling". Three controls on the plain Form-A baseline (identical L7 prompt + ail oracle) pin down what that ceiling actually is. Control 1 — blind frontier model. Ran the full L0–L7 ladder through eight fresh, clueless Claude Opus 4.8 agents, each given verbatim the same prompt the IONOS models got, with no repo access, no compiler, no oracle (all returned tool_uses: 0). Oracled afterward exactly as the harness oracles Qwen/Llama: 8/8 green, including L7. So L7 is not a language ceiling — Form-A is authorable, SMA included, by a strong enough model. The eight modules are kept under clueless-agents/ (one m.ail per dir to satisfy module-name=stem) and re-check green; L7 also runs to the expected output. Claude generalised operators never shown in the few-shot (float +, a / absent from every example, float print). Control 2 — agentic Qwen (qwen_agentic.py). The fairer question: does the compiler-as-a-tool close the gap? Runs the real Claude-Code loop — model proposes a module, harness runs ail, the raw unedited compiler output goes back into the dialogue, up to 8 turns. Result: never solved. From turn 1 Qwen collapses into a repetition loop (~274× "(seq" until max_tokens, completion=1500 every turn); the module is truncated mid-cascade, the oracle returns the correct parse error every turn, and Qwen repeats. Tool access is not an equaliser — it lifts a model only when its reasoning already operates in the right neighbourhood. Also settles the context question: turn 1 collapses at prompt=729 (no accumulated history), so the failure is "can't", not "drowns in context". Raw doc qwen-agentic-sma.md (repetition cascades collapsed to keep the repo small; the collapse marker records the original run length). Control 3 — same SMA in Python (qwen_python_sma.py). Is the AILang failure about coding ability or the unfamiliar surface? Given the same task zero-shot in Python, Qwen returns a clean, idiomatic, correct sliding-window in 113 tokens (verified by inspection + hand-trace; the harness deliberately does not auto-execute LLM code). 113 tokens of correct Python vs a 1500-token collapse in AILang, same model and temperature. Synthesis: the L7 wall is neither an algorithm ceiling nor a "model is weak" ceiling. Qwen owns the algorithm (Python) and a frontier model owns the surface (Claude, blind, first try). Qwen's AILang collapse is specifically the unfamiliar, fully-parenthesised surface it never saw in training, and a compiler-in-the-loop does not rescue it. For an LLM-authored language: making Form-A compiler-driven does not lower the model bar — AILang either targets frontier-class authors, or its surface must drop the structure-tracking load enough for a small model to keep its place. Live IONOS limited to the consented Qwen agentic + Python runs.
Cross-model authoring-form test
Empirical measurement of whether .ail.json or .ail is the form a
foreign LLM author reaches for and succeeds with. Single subject for
v1: Qwen3-Coder-Next via IONOS. Two blind cohorts; same four tasks.
Parent spec: docs/specs/0017-cross-model-authoring-form-test.md.
Layout
master/spec.md— canonical mini-spec source (form-agnostic prose +{form-only: X}blocks +{example: id}markers).master/examples/*.ail.json— AST source-of-truth; each example prints to either form via the existing roundtrip machinery.master/tasks/*.task.json— task definitions consumed by the harness. Authored in cma.2, not cma.1.render/— standalone Cargo crate, outside the root workspace, builds the renderer binary.rendered/json.md,rendered/ail.md— projected mini-specs, checked into the repo for review.harness/— Authored in cma.2.runs/<date>-<hash>/— populated byharnessduring a live run.
Running the renderer
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml -- \
--master experiments/2026-05-12-cross-model-authoring/master \
--rendered experiments/2026-05-12-cross-model-authoring/rendered
Running the tests
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/render/Cargo.toml
Three integration tests: example_roundtrip (each example loads,
prints to AIL, reparses to the same canonical bytes), spec_completeness
(every AST variant in ailang_core::ast is exercised by at least
one example), token_balance (form-only blocks balanced within ±5%
across the two rendered files).
Running the harness
Live mode (one full eight-run sweep, ~480k tokens budget by default):
export IONOS_API_TOKEN="<token>" # see roadmap entry for token provenance
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out experiments/2026-05-12-cross-model-authoring/runs \
--model Qwen/Qwen3-Coder-Next
The harness pre-flights ail --version and clang --version before
the first API call. Set AIL_BIN if ail is not on PATH.
Mock mode (offline; CI-friendly; bypasses the IONOS API):
cargo run --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml -- \
--rendered experiments/2026-05-12-cross-model-authoring/rendered \
--tasks experiments/2026-05-12-cross-model-authoring/master/tasks \
--out /tmp/mock-runs \
--model mock \
--mock experiments/2026-05-12-cross-model-authoring/harness/tests/fixtures/mock_full_run.json
Tests:
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
Five suites: inline --lib unit tests for strip_locations (5),
plus integration tests strip_locations against captured stderr
fixtures (5), verify_references (1, drives every reference through
the real ail+clang pipeline), mock_full_run (1, full eight-row
mock E2E), budget_abort (1). Total 13 passed.