iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run + DESIGN.md §Decision-6 addendum extended to two subjects
Two live IONOS harness invocations with the ms.1 fix in place:
- Qwen/Qwen3-Coder-Next → runs/2026-05-12-080864/ (RUN_STATUS=ok)
- meta-llama/CodeLlama-13b-Instruct-hf
→ runs/2026-05-12-9197fd/ (RUN_STATUS=ok;
2/8 JSON cells terminated as api_failure
on turn 1 zero-token — transient IONOS
terminal error, disclosed in addendum)
Per-cell results (reached/4, prompt tokens, completion tokens):
Qwen JSON 1/4 182,378 14,972
Qwen AILX 1/4 110,575 3,474
CodeLlama JSON 0/4 116,015 2,711 [2 cells api_failure]
CodeLlama AILX 2/4 93,017 2,234
Both subjects agree on direction: AILX cohort cheaper on prompt
tokens (Qwen 61%, CodeLlama 80% of JSON-cohort spend) and on
completion tokens (Qwen 23%, CodeLlama 82%); AILX reached-green
≥ JSON reached-green for each subject. Three of four
first-attempt-green cells across the two subjects are AILX.
Failure classes split symmetric to form: JSON cohorts fail at
typecheck, AILX cohorts at parse — each form's
front-of-pipeline check.
DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)"
replaced: 2-column single-subject view → 4-column two-subject
view, framing paragraph reports direction agreement and the
api_failure / Qwen-rerun-noise anomalies, scope paragraph
updated to 'two subjects, n=1 each, deterministic; not a
verdict — a universal claim would need ≥3 subjects with
statistical robustness'.
Roadmap P3 'Multi-subject expansion' entry removed; scoped
goal met for two subjects.
Milestone token spend: prompt 501,985 + completion 23,391
= 525,376.
This commit is contained in:
+48
-26
@@ -603,37 +603,59 @@ lowest-common-denominator fallback that always works.
|
||||
|
||||
### Empirical addendum (2026-05-12)
|
||||
|
||||
Cross-model measurement against `Qwen/Qwen3-Coder-Next` (IONOS),
|
||||
Cross-model measurement against two foreign LLMs via IONOS,
|
||||
temperature=0, top_p=1, max-turns=5. Two blind cohorts on the same
|
||||
four MVP tasks (`t1_add_three`, `t2_length`, `t3_main_prints`,
|
||||
`t4_count_zeros`); each cohort sees only its own form's
|
||||
mini-spec. Run directory:
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-df7531/`.
|
||||
`t4_count_zeros`); each cohort sees only its own form's mini-spec.
|
||||
Subjects (both runs use the pipeline-error-formatting fix from ms.1
|
||||
so JSON-cohort feedback carries the full anyhow cause chain):
|
||||
|
||||
| metric | JSON cohort | AILX cohort |
|
||||
|---|---|---|
|
||||
| reached green | 1/4 | 2/4 |
|
||||
| first-attempt green | 0/4 | 1/4 |
|
||||
| mean turns-to-green (green only) | 2.0 | 2.0 |
|
||||
| total prompt tokens | 191,768 | 95,417 |
|
||||
| total completion tokens | 15,376 | 2,587 |
|
||||
| top error class | check (×4) | parse (×3) |
|
||||
- `Qwen/Qwen3-Coder-Next` —
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/`
|
||||
- `meta-llama/CodeLlama-13b-Instruct-hf` —
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/`
|
||||
|
||||
The AILX cohort reached green on more tasks at roughly half the
|
||||
prompt-token cost and one-sixth the completion-token cost. The
|
||||
only first-attempt success in the entire run was AILX on
|
||||
`t3_main_prints` (~5 lines of tagged s-expr; the JSON-cohort
|
||||
counterpart was ~23 lines of structured JSON and required a
|
||||
correction turn). JSON-cohort failures clustered in the
|
||||
typecheck-stage; AILX-cohort failures clustered in the parse stage,
|
||||
which is the symmetric front-of-pipeline check for that form.
|
||||
| metric | Qwen / JSON | Qwen / AILX | CodeLlama / JSON | CodeLlama / AILX |
|
||||
|---|---|---|---|---|
|
||||
| reached green | 1/4 | 1/4 | 0/4 | 2/4 |
|
||||
| first-attempt green | 1/4 | 1/4 | 0/4 | 2/4 |
|
||||
| mean turns-to-green (green only) | 1.0 | 1.0 | — | 1.0 |
|
||||
| total prompt tokens | 182,378 | 110,575 | 116,015 | 93,017 |
|
||||
| total completion tokens | 14,972 | 3,474 | 2,711 | 2,234 |
|
||||
| top error class | check (×3) | parse (×3) | check (×2) | parse (×2) |
|
||||
|
||||
**Scope of this addendum:** single subject, single deterministic
|
||||
run. This is a data point, not a verdict. The universal claim of
|
||||
this Decision (".ailx is the AI authoring projection") remains
|
||||
pending a multi-subject expansion — see the roadmap entry
|
||||
"Multi-subject expansion (cross-model authoring-form follow-up)"
|
||||
for the next-step queue.
|
||||
Both subjects show the same direction on every metric the table
|
||||
tracks. AILX is cheaper than JSON on prompt tokens (Qwen 61%,
|
||||
CodeLlama 80% of the JSON cohort's spend) and cheaper on completion
|
||||
tokens (Qwen 23%, CodeLlama 82%). AILX reached-green is greater than
|
||||
or equal to JSON reached-green for each subject (Qwen ties at 1/4;
|
||||
CodeLlama strictly dominates, 2/4 vs 0/4). The failure-class
|
||||
symmetry observed on the original Qwen baseline holds for both
|
||||
subjects: JSON cohorts fail at the typecheck stage, AILX cohorts at
|
||||
the parse stage — each form's front-of-pipeline check. Three of the
|
||||
four first-attempt-green cells observed across the two subjects are
|
||||
AILX (Qwen-AILX t3, CodeLlama-AILX t1, CodeLlama-AILX t3); the
|
||||
fourth is Qwen-JSON t3, the same task the original baseline already
|
||||
flagged as the JSON-form's only first-attempt success. Two CodeLlama
|
||||
JSON-cohort cells (`t2_length`, `t4_count_zeros`) terminated as
|
||||
`api_failure` on turn 1 with zero tokens consumed — a transient
|
||||
IONOS-side terminal error, not a model-output failure; CodeLlama
|
||||
JSON-cohort numbers therefore average over 2 informative cells, not
|
||||
4. The Qwen re-run also shifted by one cell against the cma.3
|
||||
baseline (AILX 2/4 → 1/4, JSON 1/4 → 1/4) despite identical
|
||||
temperature=0 inputs; this is IONOS-side state-of-day noise on a
|
||||
single deterministic re-run, not an effect of the ms.1
|
||||
feedback-formatting fix (which only changes JSON-cohort prompt
|
||||
content).
|
||||
|
||||
**Scope of this addendum:** two subjects, n=1 each, deterministic.
|
||||
This is a second data point pointing in the same direction as the
|
||||
first, not a verdict. The universal claim of this Decision (".ailx
|
||||
is the AI authoring projection") would need ≥3 subjects with
|
||||
statistical robustness to ratify; both current points point the
|
||||
same way (AILX cohort cheaper and at-least-as-green) but neither
|
||||
the sample size nor the cross-call noise observed in the Qwen
|
||||
re-run is enough to close out the Decision.
|
||||
|
||||
## Decision 8: explicit, verified tail calls
|
||||
|
||||
|
||||
Reference in New Issue
Block a user