diff --git a/experiments/2026-05-12-cross-model-authoring/format-findings.md b/experiments/2026-05-12-cross-model-authoring/format-findings.md index 44cbaee..cb96639 100644 --- a/experiments/2026-05-12-cross-model-authoring/format-findings.md +++ b/experiments/2026-05-12-cross-model-authoring/format-findings.md @@ -169,3 +169,30 @@ before that ceiling is even reached. **For an LLM-authored language the lever is redundant, self-checking structure (depth-annotated brackets), not a lighter surface and not a more familiar one that carries extra bookkeeping.** + +## Model context (researched 2026-06-02) + +All of the above is Qwen3-Coder-Next, which the research reframes: it is a +Mixture-of-Experts model with only **3B activated parameters** (80B total), +released Feb 2026, and **trained specifically for *agentic* coding** — reading +a codebase, calling tools in an executable environment, writing fixes, and +recovering from failures via RL on environment feedback ([qwen.ai blog](https://qwen.ai/blog?id=qwen3-coder-next), +[technical report](https://arxiv.org/abs/2603.00729)). It is NOT optimised for +one-shot generation of an unfamiliar, non-mainstream syntax from a spec — which +is exactly our task. Two consequences: + +- The "excels at tool calling" claim is about *agentic environment + interaction* (simple calls in a loop), not emitting a deep structured AST as + one tool argument — consistent with why our submit_program/AST-JSON probe was + the worst. +- A 3B-active model is unusually sensitive to structure-tracking load, so the + format ranking (redundant cue > plain > burden) may be sharper for it than + for a larger model. The *direction* is likely general; the *magnitude* is + probably model-specific. + +**Bigger active models are available on the same IONOS endpoint** and were +never tried here: `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` (405B dense — +~135× the active params), `meta-llama/Llama-3.3-70B-Instruct` (70B), +`mistralai/Mistral-Small-24B-Instruct`. The obvious next step to separate +"format effect" from "small-model effect" is to re-run the annotated-paren vs +plain-paren comparison on Llama-3.1-405B.