docs(cma): add Qwen3-Coder-Next model context to format findings (refs #68)
Researched the model under test. Qwen3-Coder-Next is a 3B-activated MoE (80B total, Feb 2026) trained for AGENTIC coding (codebase + tools + RL on environment feedback), not one-shot generation of unfamiliar syntax — exactly our task, so we tested its weakest discipline. Its tool-calling strength is agentic environment interaction, not emitting deep structured AST JSON, consistent with the tool probe being worst. The format direction is likely general but the magnitude is model-specific; bigger active models on the same IONOS endpoint (Llama-3.1-405B, Llama-3.3-70B) were never tried and are the obvious next step to separate format-effect from small-model-effect.
This commit is contained in:
@@ -169,3 +169,30 @@ before that ceiling is even reached.
|
|||||||
**For an LLM-authored language the lever is redundant, self-checking structure
|
**For an LLM-authored language the lever is redundant, self-checking structure
|
||||||
(depth-annotated brackets), not a lighter surface and not a more familiar one
|
(depth-annotated brackets), not a lighter surface and not a more familiar one
|
||||||
that carries extra bookkeeping.**
|
that carries extra bookkeeping.**
|
||||||
|
|
||||||
|
## Model context (researched 2026-06-02)
|
||||||
|
|
||||||
|
All of the above is Qwen3-Coder-Next, which the research reframes: it is a
|
||||||
|
Mixture-of-Experts model with only **3B activated parameters** (80B total),
|
||||||
|
released Feb 2026, and **trained specifically for *agentic* coding** — reading
|
||||||
|
a codebase, calling tools in an executable environment, writing fixes, and
|
||||||
|
recovering from failures via RL on environment feedback ([qwen.ai blog](https://qwen.ai/blog?id=qwen3-coder-next),
|
||||||
|
[technical report](https://arxiv.org/abs/2603.00729)). It is NOT optimised for
|
||||||
|
one-shot generation of an unfamiliar, non-mainstream syntax from a spec — which
|
||||||
|
is exactly our task. Two consequences:
|
||||||
|
|
||||||
|
- The "excels at tool calling" claim is about *agentic environment
|
||||||
|
interaction* (simple calls in a loop), not emitting a deep structured AST as
|
||||||
|
one tool argument — consistent with why our submit_program/AST-JSON probe was
|
||||||
|
the worst.
|
||||||
|
- A 3B-active model is unusually sensitive to structure-tracking load, so the
|
||||||
|
format ranking (redundant cue > plain > burden) may be sharper for it than
|
||||||
|
for a larger model. The *direction* is likely general; the *magnitude* is
|
||||||
|
probably model-specific.
|
||||||
|
|
||||||
|
**Bigger active models are available on the same IONOS endpoint** and were
|
||||||
|
never tried here: `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8` (405B dense —
|
||||||
|
~135× the active params), `meta-llama/Llama-3.3-70B-Instruct` (70B),
|
||||||
|
`mistralai/Mistral-Small-24B-Instruct`. The obvious next step to separate
|
||||||
|
"format effect" from "small-model effect" is to re-run the annotated-paren vs
|
||||||
|
plain-paren comparison on Llama-3.1-405B.
|
||||||
|
|||||||
Reference in New Issue
Block a user