From 94d6963995b060b22da2ef1e45885b8f8f95678c Mon Sep 17 00:00:00 2001 From: Brummel Date: Tue, 12 May 2026 12:20:30 +0200 Subject: [PATCH] audit-cma: close milestone Cross-model authoring-form test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architect drift_found, both [low]: - INDEX backfill for commit 90512d5 (brainstorm Step 7.5 SKILL.md edit + roadmap P1 todo removed without a paired INDEX mirror). - README "Total 8 passed" understated the actual 13/13 sweep. Both fixed inline (mechanical, ≤30 LOC) per CLAUDE.md trivial-edits carve-out; no implementer dispatch. Bench: all three scripts exit 0, 0 regressed, 5 unexplained latency.explicit_at_rc improvements not coupled to milestone work (no in-workspace crate touched). Baseline left pristine — future audits will reveal whether the improvements hold. All 9 spec acceptance criteria green. Milestone closed clean. --- docs/journals/2026-05-12-audit-cma.md | 122 ++++++++++++++++++ docs/journals/INDEX.md | 2 + .../README.md | 7 +- 3 files changed, 129 insertions(+), 2 deletions(-) create mode 100644 docs/journals/2026-05-12-audit-cma.md diff --git a/docs/journals/2026-05-12-audit-cma.md b/docs/journals/2026-05-12-audit-cma.md new file mode 100644 index 0000000..0e9eb32 --- /dev/null +++ b/docs/journals/2026-05-12-audit-cma.md @@ -0,0 +1,122 @@ +# audit-cma — Milestone close: Cross-model authoring-form test + +**Date:** 2026-05-12 +**Milestone:** Cross-model authoring-form test (cma.1 + cma.2 + cma.3) +**Status:** Closed, clean (after two `[low]` tidy fixes) + +## Architect drift review + +`ailang-architect` reported `drift_found` with two items over +`44c6e56..HEAD`: + +- `[low]` `docs/journals/INDEX.md` — commit `90512d5` ("brainstorm + Step 7.5 re-dispatch on any post-PASS spec edit") removed the + corresponding P1 todo from roadmap.md but did not add an INDEX + mirror. CLAUDE.md's roadmap convention requires a one-line mirror + in the journals when a roadmap entry is removed. +- `[low]` `experiments/.../README.md` — the "Total 8 passed" sentence + inherited from the cma.2 plan understates the actual harness + test sweep (13 passed across `--lib` unit + 4 integration suites). + Surface cosmetic; flagged in cma.3 known-debt. + +Both items confined to record-keeping, neither blocks milestone +close. Architect recommendation: carry on as planned. + +## Tidy fixes (boss-direct, inline) + +1. **INDEX backfill for `90512d5`.** Added a single-line entry in + `docs/journals/INDEX.md` pointing at the commit and noting that + no per-iter journal accompanies it (a small skill-discipline + tweak rather than an iter). The format mirrors existing entries + so future agents can navigate from INDEX to the commit history + without reconstructing. + +2. **README test-count correction.** The five-suite layout under + `experiments/2026-05-12-cross-model-authoring/README.md`'s + "Running the harness" section now lists all five suites + explicitly (inline `--lib` unit tests for strip_locations, plus + the four integration files) and reports the correct total + (13 passed). + +Both edits are ≤30 LOC and require no design judgement (CLAUDE.md +"trivial mechanical edits" carve-out); no planner+implement +dispatch needed. + +## Bench-regression check + +`bench/check.py && bench/compile_check.py && bench/cross_lang.py`: + +- `check.py`: exit 0; **63 metrics; 0 regressed, 5 improved beyond + tolerance, 58 stable**. +- `compile_check.py`: exit 0; 24 metrics; 0 regressed, 0 improved + beyond tolerance, 24 stable. +- `cross_lang.py`: exit 0; 25 metrics; 0 regressed, 0 improved + beyond tolerance, 25 stable. + +All three exit-code gates green. The five improvements on +`check.py` cluster on `latency.explicit_at_rc` (p99 -36.5%, p99_9 +-41.8%, max -41.7%, p99_over_median -38.0%) plus +`throughput.bench_list_sum.gc_over_bump` -8.7%. + +**Decision: no baseline update (carry-on).** Rationale: + +- The cma milestone touched no in-workspace crate. Every line of + code change lives under `experiments/2026-05-12-cross-model-authoring/` + which is out of the root Cargo workspace by design. No mechanism + in the milestone could plausibly improve runtime latency or + GC-over-bump throughput. +- The improvements therefore reflect either (a) runtime variance + / scheduler / clang version drift between the prior audit-rt + baseline measurement and now, or (b) genuine system-level + improvement uncorrelated with the milestone. +- Ratifying an unexplained improvement would lock it into the + baseline and lose the ability to detect a future regression + back to the prior level. Better to leave the baseline pristine; + if the improvement holds across the next two or three audits, + the next audit can ratify with confidence. +- This is the audit-rt pattern carried forward: don't ratify what + the milestone didn't intentionally move. + +## Spec acceptance criteria — final check + +Reviewing the parent spec `docs/specs/2026-05-12-cross-model-authoring-form-test.md` +§"Acceptance criteria" against landed work: + +1. ✓ Directory `experiments/2026-05-12-cross-model-authoring/` + exists with all four sub-components (master/, render/, harness/, + runs/) plus top-level README. (cma.1 + cma.2) +2. ✓ `master/spec.md` is vollumfänglich; `spec_completeness.rs` + gate green; 34/34 AST variants covered by the curated subset. + (cma.1) +3. ✓ Both `rendered/json.md` and `rendered/ailx.md` are checked + in; all `render/` tests green. (cma.1) +4. ✓ `harness/` builds, passes 13/13 tests including the + mock_full_run end-to-end. (cma.2) +5. ✓ Four task definitions + reference solutions exist; the + `verify_references` integration test confirms each reaches + green through the real ail+clang pipeline locally. (cma.2) +6. ✓ Live run executed against IONOS (Qwen3-Coder-Next, + temperature=0, top_p=1, max-turns=5) and recorded under + `runs/2026-05-12-df7531/` with RUN_STATUS=ok, full per-turn raw + artefacts for all eight (cohort × task) combinations, + scores.csv, and summary.md. (cma.3) +7. ✓ DESIGN.md §"Decision 6" gained an "Empirical addendum + (2026-05-12)" subsection — 27 lines, model id + run date + + per-cohort means table + illustrative t3 contrast + explicit + scope note deferring the universal claim to a multi-subject + expansion. (cma.3) +8. ✓ Journal entry `2026-05-12-iter-cma.3.md` linked from + INDEX.md summarises the data point. (cma.3) +9. ✓ Roadmap P2 entry removed; P3 entry "Multi-subject expansion + (cross-model authoring-form follow-up)" added pointing at the + run as baseline. (cma.3) + +All nine criteria green. + +## Outcome + +Milestone closed clean after the two record-keeping tidies above. +The cross-model authoring-form test ships its harness, four MVP +tasks with verified reference solutions, a single-subject baseline +dataset, and a calibrated DESIGN.md addendum. The next step +(P3 roadmap: multi-subject expansion) is queued but not started. diff --git a/docs/journals/INDEX.md b/docs/journals/INDEX.md index 0819158..8205d12 100644 --- a/docs/journals/INDEX.md +++ b/docs/journals/INDEX.md @@ -20,6 +20,8 @@ - 2026-05-12 — iter rt.2: DESIGN.md anchor for Roundtrip Invariant (top-level section + Decision 6 Constraint 2 upward cross-reference) → 2026-05-12-iter-rt.2.md - 2026-05-12 — audit-rt: milestone close (Roundtrip Invariant) — architect drift fixed in rt.tidy (3 items: Direction-2 5th test, roadmap P1 removed, wording sync); bench all-green → 2026-05-12-audit-rt.md - 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md +- 2026-05-12 — brainstorm Step 7.5: re-dispatch grounding-check on any post-PASS spec edit (skill SKILL.md edit + roadmap P1 todo removed) → see commit `90512d5` (no per-iter journal — small skill-discipline tweak) - 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md - 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md - 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md +- 2026-05-12 — audit-cma: milestone close (Cross-model authoring-form test) — architect drift fixed in 2 record-keeping items (INDEX backfill for brainstorm Step 7.5, README test-count correction); bench all-green (5 unexplained improvements on latency.explicit_at_rc not coupled to milestone, baseline left pristine) → 2026-05-12-audit-cma.md diff --git a/experiments/2026-05-12-cross-model-authoring/README.md b/experiments/2026-05-12-cross-model-authoring/README.md index 8c592ce..dce977a 100644 --- a/experiments/2026-05-12-cross-model-authoring/README.md +++ b/experiments/2026-05-12-cross-model-authoring/README.md @@ -74,5 +74,8 @@ Tests: cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml ``` -Four test suites: `strip_locations` (5), `verify_references` (1), -`mock_full_run` (1), `budget_abort` (1). Total 8 passed. +Five suites: inline `--lib` unit tests for `strip_locations` (5), +plus integration tests `strip_locations` against captured stderr +fixtures (5), `verify_references` (1, drives every reference through +the real ail+clang pipeline), `mock_full_run` (1, full eight-row +mock E2E), `budget_abort` (1). Total 13 passed.