Brummel
9c833f65e4
Every surface run in format-findings.md hit the same L7 (SMA) wall, read so far as a "genuine single-shot complexity ceiling". Three controls on the plain Form-A baseline (identical L7 prompt + ail oracle) pin down what that ceiling actually is. Control 1 — blind frontier model. Ran the full L0–L7 ladder through eight fresh, clueless Claude Opus 4.8 agents, each given verbatim the same prompt the IONOS models got, with no repo access, no compiler, no oracle (all returned tool_uses: 0). Oracled afterward exactly as the harness oracles Qwen/Llama: 8/8 green, including L7. So L7 is not a language ceiling — Form-A is authorable, SMA included, by a strong enough model. The eight modules are kept under clueless-agents/ (one m.ail per dir to satisfy module-name=stem) and re-check green; L7 also runs to the expected output. Claude generalised operators never shown in the few-shot (float +, a / absent from every example, float print). Control 2 — agentic Qwen (qwen_agentic.py). The fairer question: does the compiler-as-a-tool close the gap? Runs the real Claude-Code loop — model proposes a module, harness runs ail, the raw unedited compiler output goes back into the dialogue, up to 8 turns. Result: never solved. From turn 1 Qwen collapses into a repetition loop (~274× "(seq" until max_tokens, completion=1500 every turn); the module is truncated mid-cascade, the oracle returns the correct parse error every turn, and Qwen repeats. Tool access is not an equaliser — it lifts a model only when its reasoning already operates in the right neighbourhood. Also settles the context question: turn 1 collapses at prompt=729 (no accumulated history), so the failure is "can't", not "drowns in context". Raw doc qwen-agentic-sma.md (repetition cascades collapsed to keep the repo small; the collapse marker records the original run length). Control 3 — same SMA in Python (qwen_python_sma.py). Is the AILang failure about coding ability or the unfamiliar surface? Given the same task zero-shot in Python, Qwen returns a clean, idiomatic, correct sliding-window in 113 tokens (verified by inspection + hand-trace; the harness deliberately does not auto-execute LLM code). 113 tokens of correct Python vs a 1500-token collapse in AILang, same model and temperature. Synthesis: the L7 wall is neither an algorithm ceiling nor a "model is weak" ceiling. Qwen owns the algorithm (Python) and a frontier model owns the surface (Claude, blind, first try). Qwen's AILang collapse is specifically the unfamiliar, fully-parenthesised surface it never saw in training, and a compiler-in-the-loop does not rescue it. For an LLM-authored language: making Form-A compiler-driven does not lower the model bar — AILang either targets frontier-class authors, or its surface must drop the structure-tracking load enough for a small model to keep its place. Live IONOS limited to the consented Qwen agentic + Python runs.
62 lines
3.0 KiB
Python
62 lines
3.0 KiB
Python
#!/usr/bin/env python3
|
|
"""Control probe: the SAME SMA task, but in Python instead of AILang Form-A.
|
|
|
|
If the model nails this zero-shot, the AILang failures are about the unfamiliar
|
|
surface (structure-tracking in a never-seen language), NOT about coding ability.
|
|
If it fails here too, the model is just weak. Single-shot, temperature 0.2, same
|
|
as the AILang ablation — but no few-shot, because Python is the model's home turf.
|
|
|
|
This harness ONLY fetches the model's code and writes it to /tmp + a doc. It does
|
|
NOT execute it (LLM-generated code is run deliberately by the operator, after a
|
|
read). Reads $IONOS_API_TOKEN (never printed). Live IONOS — run with consent.
|
|
"""
|
|
from __future__ import annotations
|
|
import json, os, re, urllib.request
|
|
from pathlib import Path
|
|
|
|
ENDPOINT = "https://openai.inference.de-txl.ionos.com/v1/chat/completions"
|
|
MODEL = os.environ.get("PROBE_MODEL", "Qwen/Qwen3-Coder-Next")
|
|
TAG = os.environ.get("PROBE_TAG", "qwen")
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
DOC = REPO / f"experiments/2026-05-12-cross-model-authoring/{TAG}-python-sma.md"
|
|
OUT = Path("/tmp/qwen_sma.py")
|
|
|
|
TASK = ("Write a Python 3 program that computes a simple moving average with "
|
|
"window 3 over the float stream 1.0, 5.0, 3.0, 8.0, 6.0, 2.0: feed the "
|
|
"values in one at a time, and after each value — once at least 3 values "
|
|
"have been seen — print the average of the most recent 3 (their sum / 3) "
|
|
"on its own line. The four printed lines should be the averages "
|
|
"3.0, 5.33333..., 5.66667..., 5.33333... (exact float formatting is up to "
|
|
"you). Return the program in a single ```python code block.")
|
|
|
|
def call(task):
|
|
body = json.dumps({"model": MODEL, "temperature": 0.2, "max_tokens": 1500,
|
|
"messages": [
|
|
{"role": "system", "content": "You are an expert Python programmer."},
|
|
{"role": "user", "content": task}]}).encode()
|
|
req = urllib.request.Request(ENDPOINT, data=body, headers={
|
|
"Authorization": f"Bearer {os.environ['IONOS_API_TOKEN']}", "Content-Type": "application/json"})
|
|
with urllib.request.urlopen(req, timeout=300) as r:
|
|
d = json.loads(r.read())
|
|
return d["choices"][0]["message"]["content"], d.get("usage", {})
|
|
|
|
def extract(resp):
|
|
m = re.search(r"```(?:python)?\n(.*?)```", resp, re.S)
|
|
return (m.group(1) if m else resp).strip()
|
|
|
|
def main():
|
|
resp, usage = call(TASK)
|
|
code = extract(resp)
|
|
OUT.write_text(code + "\n")
|
|
log = [f"# Python control — same SMA task, in Python ({MODEL})\n",
|
|
f"tokens: prompt={usage.get('prompt_tokens','?')} completion={usage.get('completion_tokens','?')}\n",
|
|
"---\n", "extracted Python program:\n```python\n" + code + "\n```\n",
|
|
f"\nwritten to: {OUT} (NOT executed by this harness)\n"]
|
|
DOC.write_text("\n".join(log))
|
|
print(f"completion_tokens={usage.get('completion_tokens','?')}")
|
|
print(f"code → {OUT}")
|
|
print(f"doc → {DOC}")
|
|
|
|
if __name__ == "__main__":
|
|
main()
|