audit-cma: close milestone Cross-model authoring-form test

Architect drift_found, both [low]:
- INDEX backfill for commit 90512d5 (brainstorm Step 7.5 SKILL.md
  edit + roadmap P1 todo removed without a paired INDEX mirror).
- README "Total 8 passed" understated the actual 13/13 sweep.

Both fixed inline (mechanical, ≤30 LOC) per CLAUDE.md trivial-edits
carve-out; no implementer dispatch.

Bench: all three scripts exit 0, 0 regressed, 5 unexplained
latency.explicit_at_rc improvements not coupled to milestone work
(no in-workspace crate touched). Baseline left pristine — future
audits will reveal whether the improvements hold.

All 9 spec acceptance criteria green. Milestone closed clean.
This commit is contained in:
2026-05-12 12:20:30 +02:00
parent e91e31aadb
commit 94d6963995
3 changed files with 129 additions and 2 deletions
+122
View File
@@ -0,0 +1,122 @@
# audit-cma — Milestone close: Cross-model authoring-form test
**Date:** 2026-05-12
**Milestone:** Cross-model authoring-form test (cma.1 + cma.2 + cma.3)
**Status:** Closed, clean (after two `[low]` tidy fixes)
## Architect drift review
`ailang-architect` reported `drift_found` with two items over
`44c6e56..HEAD`:
- `[low]` `docs/journals/INDEX.md` — commit `90512d5` ("brainstorm
Step 7.5 re-dispatch on any post-PASS spec edit") removed the
corresponding P1 todo from roadmap.md but did not add an INDEX
mirror. CLAUDE.md's roadmap convention requires a one-line mirror
in the journals when a roadmap entry is removed.
- `[low]` `experiments/.../README.md` — the "Total 8 passed" sentence
inherited from the cma.2 plan understates the actual harness
test sweep (13 passed across `--lib` unit + 4 integration suites).
Surface cosmetic; flagged in cma.3 known-debt.
Both items confined to record-keeping, neither blocks milestone
close. Architect recommendation: carry on as planned.
## Tidy fixes (boss-direct, inline)
1. **INDEX backfill for `90512d5`.** Added a single-line entry in
`docs/journals/INDEX.md` pointing at the commit and noting that
no per-iter journal accompanies it (a small skill-discipline
tweak rather than an iter). The format mirrors existing entries
so future agents can navigate from INDEX to the commit history
without reconstructing.
2. **README test-count correction.** The five-suite layout under
`experiments/2026-05-12-cross-model-authoring/README.md`'s
"Running the harness" section now lists all five suites
explicitly (inline `--lib` unit tests for strip_locations, plus
the four integration files) and reports the correct total
(13 passed).
Both edits are ≤30 LOC and require no design judgement (CLAUDE.md
"trivial mechanical edits" carve-out); no planner+implement
dispatch needed.
## Bench-regression check
`bench/check.py && bench/compile_check.py && bench/cross_lang.py`:
- `check.py`: exit 0; **63 metrics; 0 regressed, 5 improved beyond
tolerance, 58 stable**.
- `compile_check.py`: exit 0; 24 metrics; 0 regressed, 0 improved
beyond tolerance, 24 stable.
- `cross_lang.py`: exit 0; 25 metrics; 0 regressed, 0 improved
beyond tolerance, 25 stable.
All three exit-code gates green. The five improvements on
`check.py` cluster on `latency.explicit_at_rc` (p99 -36.5%, p99_9
-41.8%, max -41.7%, p99_over_median -38.0%) plus
`throughput.bench_list_sum.gc_over_bump` -8.7%.
**Decision: no baseline update (carry-on).** Rationale:
- The cma milestone touched no in-workspace crate. Every line of
code change lives under `experiments/2026-05-12-cross-model-authoring/`
which is out of the root Cargo workspace by design. No mechanism
in the milestone could plausibly improve runtime latency or
GC-over-bump throughput.
- The improvements therefore reflect either (a) runtime variance
/ scheduler / clang version drift between the prior audit-rt
baseline measurement and now, or (b) genuine system-level
improvement uncorrelated with the milestone.
- Ratifying an unexplained improvement would lock it into the
baseline and lose the ability to detect a future regression
back to the prior level. Better to leave the baseline pristine;
if the improvement holds across the next two or three audits,
the next audit can ratify with confidence.
- This is the audit-rt pattern carried forward: don't ratify what
the milestone didn't intentionally move.
## Spec acceptance criteria — final check
Reviewing the parent spec `docs/specs/2026-05-12-cross-model-authoring-form-test.md`
§"Acceptance criteria" against landed work:
1. ✓ Directory `experiments/2026-05-12-cross-model-authoring/`
exists with all four sub-components (master/, render/, harness/,
runs/) plus top-level README. (cma.1 + cma.2)
2.`master/spec.md` is vollumfänglich; `spec_completeness.rs`
gate green; 34/34 AST variants covered by the curated subset.
(cma.1)
3. ✓ Both `rendered/json.md` and `rendered/ailx.md` are checked
in; all `render/` tests green. (cma.1)
4.`harness/` builds, passes 13/13 tests including the
mock_full_run end-to-end. (cma.2)
5. ✓ Four task definitions + reference solutions exist; the
`verify_references` integration test confirms each reaches
green through the real ail+clang pipeline locally. (cma.2)
6. ✓ Live run executed against IONOS (Qwen3-Coder-Next,
temperature=0, top_p=1, max-turns=5) and recorded under
`runs/2026-05-12-df7531/` with RUN_STATUS=ok, full per-turn raw
artefacts for all eight (cohort × task) combinations,
scores.csv, and summary.md. (cma.3)
7. ✓ DESIGN.md §"Decision 6" gained an "Empirical addendum
(2026-05-12)" subsection — 27 lines, model id + run date +
per-cohort means table + illustrative t3 contrast + explicit
scope note deferring the universal claim to a multi-subject
expansion. (cma.3)
8. ✓ Journal entry `2026-05-12-iter-cma.3.md` linked from
INDEX.md summarises the data point. (cma.3)
9. ✓ Roadmap P2 entry removed; P3 entry "Multi-subject expansion
(cross-model authoring-form follow-up)" added pointing at the
run as baseline. (cma.3)
All nine criteria green.
## Outcome
Milestone closed clean after the two record-keeping tidies above.
The cross-model authoring-form test ships its harness, four MVP
tasks with verified reference solutions, a single-subject baseline
dataset, and a calibrated DESIGN.md addendum. The next step
(P3 roadmap: multi-subject expansion) is queued but not started.
+2
View File
@@ -20,6 +20,8 @@
- 2026-05-12 — iter rt.2: DESIGN.md anchor for Roundtrip Invariant (top-level section + Decision 6 Constraint 2 upward cross-reference) → 2026-05-12-iter-rt.2.md - 2026-05-12 — iter rt.2: DESIGN.md anchor for Roundtrip Invariant (top-level section + Decision 6 Constraint 2 upward cross-reference) → 2026-05-12-iter-rt.2.md
- 2026-05-12 — audit-rt: milestone close (Roundtrip Invariant) — architect drift fixed in rt.tidy (3 items: Direction-2 5th test, roadmap P1 removed, wording sync); bench all-green → 2026-05-12-audit-rt.md - 2026-05-12 — audit-rt: milestone close (Roundtrip Invariant) — architect drift fixed in rt.tidy (3 items: Direction-2 5th test, roadmap P1 removed, wording sync); bench all-green → 2026-05-12-audit-rt.md
- 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md - 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md
- 2026-05-12 — brainstorm Step 7.5: re-dispatch grounding-check on any post-PASS spec edit (skill SKILL.md edit + roadmap P1 todo removed) → see commit `90512d5` (no per-iter journal — small skill-discipline tweak)
- 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md - 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md
- 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md - 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md
- 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md - 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md
- 2026-05-12 — audit-cma: milestone close (Cross-model authoring-form test) — architect drift fixed in 2 record-keeping items (INDEX backfill for brainstorm Step 7.5, README test-count correction); bench all-green (5 unexplained improvements on latency.explicit_at_rc not coupled to milestone, baseline left pristine) → 2026-05-12-audit-cma.md
@@ -74,5 +74,8 @@ Tests:
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
``` ```
Four test suites: `strip_locations` (5), `verify_references` (1), Five suites: inline `--lib` unit tests for `strip_locations` (5),
`mock_full_run` (1), `budget_abort` (1). Total 8 passed. plus integration tests `strip_locations` against captured stderr
fixtures (5), `verify_references` (1, drives every reference through
the real ail+clang pipeline), `mock_full_run` (1, full eight-row
mock E2E), `budget_abort` (1). Total 13 passed.