From 17b370bbb37b8aa1ced7f787c5e06eb91558e7b7 Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 12 May 2026 13:29:34 +0200 Subject: [PATCH] =?UTF-8?q?WhatsNew:=20cross-model=20authoring=20test=20?= =?UTF-8?q?=E2=80=94=20second=20subject=20added?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/WhatsNew.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/WhatsNew.md b/docs/WhatsNew.md index 3f019f0..03fdfe9 100644 --- a/docs/WhatsNew.md +++ b/docs/WhatsNew.md @@ -76,3 +76,7 @@ Bench all-green. Roadmap is back to user-direction; the Show+print rewire is gat ## 2026-05-12 — Cross-model authoring test: first data point Cross-model authoring test landed. First run (Qwen3-Coder-Next, 4 tasks, two blind cohorts): textual-form cohort 2/4 green at ~half the token cost; structured-form 1/4. Only first-attempt success was textual-form on a print-two-integers task — five lines vs twenty-three. Recorded as a data point in the language reference; single-model scope explicit. Multi-subject expansion queued as next experimental milestone. Milestone closed; queue back to user-direction. + +## 2026-05-12 — Cross-model authoring test: second subject added + +Cross-model authoring test now has a second subject. Ran CodeLlama alongside Qwen on the same four tasks, fresh runs both. Both models point the same way: textual form is cheaper than structured form on tokens (61–80% of structured form's cost across the two models) and reaches working code at least as often (Qwen ties on this run; CodeLlama strictly dominates 2/4 vs 0/4). Three of four first-attempt successes across the two models are textual. A feedback-asymmetry bug surfaced during planning — the structured form was silently losing its underlying parse error on the way back to the model — and got fixed in the same milestone, so both subjects ran with the fix in place. Not a universal verdict yet (would need three or more models with repetitions), but the second data point points the same way as the first. Milestone closed; queue back to user-direction.