WhatsNew: cross-model authoring test — second subject added

2026-05-12 13:29:34 +02:00
parent 9dc6263b6e
commit 17b370bbb3
1 changed files with 4 additions and 0 deletions
@@ -76,3 +76,7 @@ Bench all-green. Roadmap is back to user-direction; the Show+print rewire is gat
 ## 2026-05-12 — Cross-model authoring test: first data point

 Cross-model authoring test landed. First run (Qwen3-Coder-Next, 4 tasks, two blind cohorts): textual-form cohort 2/4 green at ~half the token cost; structured-form 1/4. Only first-attempt success was textual-form on a print-two-integers task — five lines vs twenty-three. Recorded as a data point in the language reference; single-model scope explicit. Multi-subject expansion queued as next experimental milestone. Milestone closed; queue back to user-direction.
+
+## 2026-05-12 — Cross-model authoring test: second subject added
+
+Cross-model authoring test now has a second subject. Ran CodeLlama alongside Qwen on the same four tasks, fresh runs both. Both models point the same way: textual form is cheaper than structured form on tokens (61–80% of structured form's cost across the two models) and reaches working code at least as often (Qwen ties on this run; CodeLlama strictly dominates 2/4 vs 0/4). Three of four first-attempt successes across the two models are textual. A feedback-asymmetry bug surfaced during planning — the structured form was silently losing its underlying parse error on the way back to the model — and got fixed in the same milestone, so both subjects ran with the fix in place. Not a universal verdict yet (would need three or more models with repetitions), but the second data point points the same way as the first. Milestone closed; queue back to user-direction.