Brummel
9c833f65e4
Every surface run in format-findings.md hit the same L7 (SMA) wall, read so far as a "genuine single-shot complexity ceiling". Three controls on the plain Form-A baseline (identical L7 prompt + ail oracle) pin down what that ceiling actually is. Control 1 — blind frontier model. Ran the full L0–L7 ladder through eight fresh, clueless Claude Opus 4.8 agents, each given verbatim the same prompt the IONOS models got, with no repo access, no compiler, no oracle (all returned tool_uses: 0). Oracled afterward exactly as the harness oracles Qwen/Llama: 8/8 green, including L7. So L7 is not a language ceiling — Form-A is authorable, SMA included, by a strong enough model. The eight modules are kept under clueless-agents/ (one m.ail per dir to satisfy module-name=stem) and re-check green; L7 also runs to the expected output. Claude generalised operators never shown in the few-shot (float +, a / absent from every example, float print). Control 2 — agentic Qwen (qwen_agentic.py). The fairer question: does the compiler-as-a-tool close the gap? Runs the real Claude-Code loop — model proposes a module, harness runs ail, the raw unedited compiler output goes back into the dialogue, up to 8 turns. Result: never solved. From turn 1 Qwen collapses into a repetition loop (~274× "(seq" until max_tokens, completion=1500 every turn); the module is truncated mid-cascade, the oracle returns the correct parse error every turn, and Qwen repeats. Tool access is not an equaliser — it lifts a model only when its reasoning already operates in the right neighbourhood. Also settles the context question: turn 1 collapses at prompt=729 (no accumulated history), so the failure is "can't", not "drowns in context". Raw doc qwen-agentic-sma.md (repetition cascades collapsed to keep the repo small; the collapse marker records the original run length). Control 3 — same SMA in Python (qwen_python_sma.py). Is the AILang failure about coding ability or the unfamiliar surface? Given the same task zero-shot in Python, Qwen returns a clean, idiomatic, correct sliding-window in 113 tokens (verified by inspection + hand-trace; the harness deliberately does not auto-execute LLM code). 113 tokens of correct Python vs a 1500-token collapse in AILang, same model and temperature. Synthesis: the L7 wall is neither an algorithm ceiling nor a "model is weak" ceiling. Qwen owns the algorithm (Python) and a frontier model owns the surface (Claude, blind, first try). Qwen's AILang collapse is specifically the unfamiliar, fully-parenthesised surface it never saw in training, and a compiler-in-the-loop does not rescue it. For an LLM-authored language: making Form-A compiler-driven does not lower the model bar — AILang either targets frontier-class authors, or its surface must drop the structure-tracking load enough for a small model to keep its place. Live IONOS limited to the consented Qwen agentic + Python runs.
35 lines
1.5 KiB
Markdown
35 lines
1.5 KiB
Markdown
# Blind frontier-model control — Claude Opus 4.8, single-shot, no tools
|
|
|
|
These eight modules are the raw output of Control 1 in `../format-findings.md`
|
|
("Authoring-capability controls"). Each was produced by a fresh, clueless
|
|
**Claude Opus 4.8** agent given verbatim the same plain-Form-A prompt the IONOS
|
|
models received (same SERIES few-shot, same per-level task) — with **no repo
|
|
access, no compiler, no oracle** (all eight returned with `tool_uses: 0`). The
|
|
output was oracled afterward, exactly as the harness oracles Qwen/Llama.
|
|
|
|
Result: **8/8 green, including L7 (SMA)** — the level no IONOS model solved on
|
|
any surface. So L7 is a *model* ceiling, not a language ceiling: Form-A is
|
|
authorable, SMA included, by a strong enough model.
|
|
|
|
Each module is named `m` (the task says "module `m`"), so it lives in its own
|
|
directory as `m.ail` to satisfy the module-name-equals-filename-stem rule.
|
|
|
|
Re-check all eight:
|
|
|
|
```sh
|
|
for d in */; do target/debug/ail check "$d/m.ail"; done # run from repo root with full path
|
|
# or from here:
|
|
for d in */; do "$REPO/target/debug/ail" check "$d/m.ail"; done
|
|
```
|
|
|
|
L7 (`L7_sma/m.ail`) also runs to the expected output:
|
|
|
|
```sh
|
|
ail run L7_sma/m.ail # => 3.0 / 5.33333 / 5.66667 / 5.33333
|
|
```
|
|
|
|
Note on L7: Claude solved it by hand-unrolling the six pushes into four windowed
|
|
prints (the task asks for the output, not a loop), and generalised operators
|
|
never shown in the few-shot — float `+`, a `/` operator absent from every
|
|
example, and float formatting via `print`. All correct.
|