iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run + DESIGN.md §Decision-6 addendum extended to two subjects
Two live IONOS harness invocations with the ms.1 fix in place:
- Qwen/Qwen3-Coder-Next → runs/2026-05-12-080864/ (RUN_STATUS=ok)
- meta-llama/CodeLlama-13b-Instruct-hf
→ runs/2026-05-12-9197fd/ (RUN_STATUS=ok;
2/8 JSON cells terminated as api_failure
on turn 1 zero-token — transient IONOS
terminal error, disclosed in addendum)
Per-cell results (reached/4, prompt tokens, completion tokens):
Qwen JSON 1/4 182,378 14,972
Qwen AILX 1/4 110,575 3,474
CodeLlama JSON 0/4 116,015 2,711 [2 cells api_failure]
CodeLlama AILX 2/4 93,017 2,234
Both subjects agree on direction: AILX cohort cheaper on prompt
tokens (Qwen 61%, CodeLlama 80% of JSON-cohort spend) and on
completion tokens (Qwen 23%, CodeLlama 82%); AILX reached-green
≥ JSON reached-green for each subject. Three of four
first-attempt-green cells across the two subjects are AILX.
Failure classes split symmetric to form: JSON cohorts fail at
typecheck, AILX cohorts at parse — each form's
front-of-pipeline check.
DESIGN.md §"Decision 6 / Empirical addendum (2026-05-12)"
replaced: 2-column single-subject view → 4-column two-subject
view, framing paragraph reports direction agreement and the
api_failure / Qwen-rerun-noise anomalies, scope paragraph
updated to 'two subjects, n=1 each, deterministic; not a
verdict — a universal claim would need ≥3 subjects with
statistical robustness'.
Roadmap P3 'Multi-subject expansion' entry removed; scoped
goal met for two subjects.
Milestone token spend: prompt 501,985 + completion 23,391
= 525,376.
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
# iter ms.2 — Multi-subject run + DESIGN.md addendum extension
|
||||
|
||||
**Date:** 2026-05-12
|
||||
**Started from:** 8689f7a2386e0d7993609c374d91984bc28d215c
|
||||
**Status:** DONE
|
||||
**Tasks completed:** 6 of 6
|
||||
|
||||
## Summary
|
||||
|
||||
Two live IONOS harness invocations executed with the ms.1 fix in
|
||||
place: a Qwen3-Coder-Next retroactive re-run
|
||||
(`runs/2026-05-12-080864/`, RUN_STATUS=ok) and a first-ever
|
||||
CodeLlama-13b-Instruct run (`runs/2026-05-12-9197fd/`, RUN_STATUS=ok
|
||||
at the run level; 2 of 8 cells terminated as api_failure on turn 1
|
||||
with zero tokens — transient IONOS-side terminal error, disclosed
|
||||
in the addendum and below). DESIGN.md §"Decision 6 / Empirical
|
||||
addendum (2026-05-12)" replaced with a four-column per-cell table
|
||||
plus a framing paragraph; roadmap P3 "Multi-subject expansion"
|
||||
entry removed.
|
||||
|
||||
Both subjects point the same way on every measured axis: AILX
|
||||
cohort is cheaper than JSON cohort on prompt tokens (Qwen 61%,
|
||||
CodeLlama 80%) and on completion tokens (Qwen 23%, CodeLlama 82%),
|
||||
and AILX reached-green ≥ JSON reached-green for each subject (Qwen
|
||||
ties at 1/4; CodeLlama strictly dominates 2/4 vs 0/4). Three of the
|
||||
four first-attempt-green cells across the two subjects are AILX.
|
||||
Decision 6 is not ratified — n=1 per subject across two subjects is
|
||||
not statistical robustness — but the second data point points
|
||||
consistently. The roadmap entry was removed because the brainstorm
|
||||
spec was scoped to two subjects, not because the universal claim is
|
||||
closed; a future ratification iteration would re-open it.
|
||||
|
||||
Anomaly disclosure: the Qwen re-run shifted by one cell vs the cma.3
|
||||
baseline (AILX 2/4 → 1/4, JSON 1/4 → 1/4) at identical
|
||||
temperature=0; this is IONOS-side state-of-day noise, not an effect
|
||||
of the ms.1 feedback fix, which only changes JSON-cohort prompt
|
||||
content. The two CodeLlama JSON api_failure cells (`t2_length`,
|
||||
`t4_count_zeros`) were not re-run — the carrier constraint forbids
|
||||
exploratory token spend, and the addendum discloses the partial
|
||||
sample directly.
|
||||
|
||||
Roadmap P3 "Multi-subject expansion" entry removed in this iter.
|
||||
|
||||
## Per-task notes
|
||||
|
||||
- iter ms.2.1: Qwen retroactive re-run via harness — new run dir
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/`,
|
||||
RUN_STATUS=ok, 8/8 cells populated. AIL_BIN set to
|
||||
`target/release/ail` (no `~/.local/bin/ail`); harness picks it up
|
||||
via env-var resolution at `pipeline.rs:42`.
|
||||
- iter ms.2.2: CodeLlama-13b-Instruct first run via harness — new
|
||||
run dir
|
||||
`experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/`,
|
||||
model id `meta-llama/CodeLlama-13b-Instruct-hf` (Boss-pre-flighted
|
||||
IONOS catalogue id), RUN_STATUS=ok. 6/8 cells informative; 2/8
|
||||
json cells (`t2_length`, `t4_count_zeros`) `api_failure` on turn 1
|
||||
zero tokens.
|
||||
- iter ms.2.3: Per-cell metrics computed via direct scores.csv
|
||||
read — see addendum table. Total milestone token spend: prompt
|
||||
501,985 + completion 23,391 = 525,376.
|
||||
- iter ms.2.4: DESIGN.md lines 604-636 replaced with new
|
||||
four-column addendum block (~50 lines including the framing
|
||||
paragraph). Subjects named with canonical IONOS model ids and
|
||||
run-dir pointers; framing paragraph reports direction agreement,
|
||||
failure-class symmetry, first-attempt tally, api_failure anomaly,
|
||||
and Qwen-rerun noise. Scope paragraph extended to "two subjects,
|
||||
n=1 each".
|
||||
- iter ms.2.5: roadmap.md lines 253-268 removed (P3 multi-subject
|
||||
entry + context note); journal file written. INDEX.md is NOT
|
||||
edited by this agent per the Iron Law; the Boss appends the
|
||||
ms.2 line at commit time. Suggested INDEX line (per plan Task 5
|
||||
Step 3):
|
||||
`- 2026-05-12 — iter ms.2: Qwen retroactive re-run + first CodeLlama-13b-Instruct run; DESIGN.md §Decision-6 addendum extended to four cells; roadmap P3 removed → 2026-05-12-iter-ms.2.md`
|
||||
- iter ms.2.6: Spec acceptance criteria sweep (1-7 all green); 14
|
||||
harness tests still pass.
|
||||
|
||||
## Spec acceptance sweep
|
||||
|
||||
1. ms.1 `pipeline.rs:118` reads `format!("check: {e:#}")` + RED→GREEN unit test → green (ms.1 working tree).
|
||||
2. ms.1 existing 14 harness tests still pass → green (verified Task 6 Step 2).
|
||||
3. CodeLlama-13b-Instruct-hf runs to completion against IONOS endpoint, RUN_STATUS=ok at run level → green; model id recorded above.
|
||||
4. Qwen3-Coder-Next re-run completes, RUN_STATUS=ok, 8 cells; distinct dir from cma.3 baseline → green.
|
||||
5. DESIGN.md §"Decision 6 / Empirical addendum" extended with follow-up paragraph + four-cell-shaped table → green.
|
||||
6. Roadmap P3 entry removed; one-line mirror above → green.
|
||||
7. Per-iter journal (this file) written → green. INDEX.md ms.2 entry is Boss-side per the Iron Law; suggested line is in the end-report.
|
||||
|
||||
## Concerns
|
||||
|
||||
- 2 of 4 CodeLlama JSON cells are `api_failure` (terminal IONOS
|
||||
error on turn 1). Per the Plan's branch (c) and the carrier's
|
||||
"no exploratory re-runs" constraint, not re-attempted. The
|
||||
addendum and journal disclose this directly; the
|
||||
CodeLlama-JSON column's prompt-token average is therefore over 2
|
||||
informative cells, not 4. Effect on the AILX-cheaper conclusion:
|
||||
none in direction; the prompt-token gap is preserved over the
|
||||
two informative JSON cells. If a future iteration re-runs
|
||||
CodeLlama at lower IONOS load, this should resolve.
|
||||
- Qwen-re-run vs cma.3-baseline drift (1 cell) at identical
|
||||
temperature=0 is IONOS noise on a single deterministic call. A
|
||||
future ratification iteration should add ≥3 repetitions per
|
||||
subject to bound this noise, not rely on a single
|
||||
temperature-zero call.
|
||||
|
||||
## Known debt
|
||||
|
||||
- The pre-existing 3 warnings in `harness/src/ionos.rs` lines 86,
|
||||
125, 137 (`last_err` assigned but never read) survived ms.1
|
||||
and ms.2. Not in scope here; cma.2 inheritance.
|
||||
- Decision 6 universal-claim status is still "second data point,
|
||||
not verdict". A future ratification iteration would need ≥3
|
||||
subjects with statistical robustness. Out of scope for ms.
|
||||
|
||||
## Files touched
|
||||
|
||||
- `docs/DESIGN.md` (addendum extended)
|
||||
- `docs/roadmap.md` (P3 entry removed)
|
||||
- `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-080864/` (created by Qwen run)
|
||||
- `experiments/2026-05-12-cross-model-authoring/runs/2026-05-12-9197fd/` (created by CodeLlama run)
|
||||
- `docs/journals/2026-05-12-iter-ms.2.md` (this file)
|
||||
- `bench/orchestrator-stats/2026-05-12-iter-ms.2.json` (stats)
|
||||
|
||||
## Stats
|
||||
|
||||
bench/orchestrator-stats/2026-05-12-iter-ms.2.json
|
||||
@@ -26,3 +26,4 @@
|
||||
- 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md
|
||||
- 2026-05-12 — audit-cma: milestone close (Cross-model authoring-form test) — architect drift fixed in 2 record-keeping items (INDEX backfill for brainstorm Step 7.5, README test-count correction); bench all-green (5 unexplained improvements on latency.explicit_at_rc not coupled to milestone, baseline left pristine) → 2026-05-12-audit-cma.md
|
||||
- 2026-05-12 — iter ms.1: pipeline anyhow-chain preservation — `{e}` → `{e:#}` at `pipeline.rs:118` restores `serde_json` parse-error leaf in JSON-cohort feedback; pinning unit test added (14/14 green) → 2026-05-12-iter-ms.1.md
|
||||
- 2026-05-12 — iter ms.2: Qwen3-Coder-Next retroactive re-run + first CodeLlama-13b-Instruct run via IONOS; DESIGN.md §Decision-6 addendum extended from 2-col single-subject to 4-col two-subject; both subjects agree on direction (AILX cheaper + ≥ green); roadmap P3 multi-subject entry removed → 2026-05-12-iter-ms.2.md
|
||||
|
||||
Reference in New Issue
Block a user