docs(cma): add Qwen3-Coder-Next model context to format findings (refs #68)
Researched the model under test. Qwen3-Coder-Next is a 3B-activated MoE (80B total, Feb 2026) trained for AGENTIC coding (codebase + tools + RL on environment feedback), not one-shot generation of unfamiliar syntax — exactly our task, so we tested its weakest discipline. Its tool-calling strength is agentic environment interaction, not emitting deep structured AST JSON, consistent with the tool probe being worst. The format direction is likely general but the magnitude is model-specific; bigger active models on the same IONOS endpoint (Llama-3.1-405B, Llama-3.3-70B) were never tried and are the obvious next step to separate format-effect from small-model-effect.
This commit is contained in:
@@ -169,3 +169,30 @@ before that ceiling is even reached.
|
||||
**For an LLM-authored language the lever is redundant, self-checking structure
|
||||
(depth-annotated brackets), not a lighter surface and not a more familiar one
|
||||
that carries extra bookkeeping.**
|
||||
|
||||
## Model context (researched 2026-06-02)
|
||||
|
||||
All of the above is Qwen3-Coder-Next, which the research reframes: it is a
|
||||
Mixture-of-Experts model with only **3B activated parameters** (80B total),
|
||||
released Feb 2026, and **trained specifically for *agentic* coding** — reading
|
||||
a codebase, calling tools in an executable environment, writing fixes, and
|
||||
recovering from failures via RL on environment feedback ([qwen.ai blog](https://qwen.ai/blog?id=qwen3-coder-next),
|
||||
[technical report](https://arxiv.org/abs/2603.00729)). It is NOT optimised for
|
||||
one-shot generation of an unfamiliar, non-mainstream syntax from a spec — which
|
||||
is exactly our task. Two consequences:
|
||||
|
||||
- The "excels at tool calling" claim is about *agentic environment
|
||||
interaction* (simple calls in a loop), not emitting a deep structured AST as
|
||||
one tool argument — consistent with why our submit_program/AST-JSON probe was
|
||||
the worst.
|
||||
- A 3B-active model is unusually sensitive to structure-tracking load, so the
|
||||
format ranking (redundant cue > plain > burden) may be sharper for it than
|
||||
for a larger model. The *direction* is likely general; the *magnitude* is
|
||||
probably model-specific.
|
||||
|
||||
**Bigger active models are available on the same IONOS endpoint** and were
|
||||
never tried here: `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` (405B dense —
|
||||
~135× the active params), `meta-llama/Llama-3.3-70B-Instruct` (70B),
|
||||
`mistralai/Mistral-Small-24B-Instruct`. The obvious next step to separate
|
||||
"format effect" from "small-model effect" is to re-run the annotated-paren vs
|
||||
plain-paren comparison on Llama-3.1-405B.
|
||||
|
||||
Reference in New Issue
Block a user