audit-cma: close milestone Cross-model authoring-form test
Architect drift_found, both [low]:
- INDEX backfill for commit 90512d5 (brainstorm Step 7.5 SKILL.md
edit + roadmap P1 todo removed without a paired INDEX mirror).
- README "Total 8 passed" understated the actual 13/13 sweep.
Both fixed inline (mechanical, ≤30 LOC) per CLAUDE.md trivial-edits
carve-out; no implementer dispatch.
Bench: all three scripts exit 0, 0 regressed, 5 unexplained
latency.explicit_at_rc improvements not coupled to milestone work
(no in-workspace crate touched). Baseline left pristine — future
audits will reveal whether the improvements hold.
All 9 spec acceptance criteria green. Milestone closed clean.
This commit is contained in:
@@ -0,0 +1,122 @@
|
||||
# audit-cma — Milestone close: Cross-model authoring-form test
|
||||
|
||||
**Date:** 2026-05-12
|
||||
**Milestone:** Cross-model authoring-form test (cma.1 + cma.2 + cma.3)
|
||||
**Status:** Closed, clean (after two `[low]` tidy fixes)
|
||||
|
||||
## Architect drift review
|
||||
|
||||
`ailang-architect` reported `drift_found` with two items over
|
||||
`44c6e56..HEAD`:
|
||||
|
||||
- `[low]` `docs/journals/INDEX.md` — commit `90512d5` ("brainstorm
|
||||
Step 7.5 re-dispatch on any post-PASS spec edit") removed the
|
||||
corresponding P1 todo from roadmap.md but did not add an INDEX
|
||||
mirror. CLAUDE.md's roadmap convention requires a one-line mirror
|
||||
in the journals when a roadmap entry is removed.
|
||||
- `[low]` `experiments/.../README.md` — the "Total 8 passed" sentence
|
||||
inherited from the cma.2 plan understates the actual harness
|
||||
test sweep (13 passed across `--lib` unit + 4 integration suites).
|
||||
Surface cosmetic; flagged in cma.3 known-debt.
|
||||
|
||||
Both items confined to record-keeping, neither blocks milestone
|
||||
close. Architect recommendation: carry on as planned.
|
||||
|
||||
## Tidy fixes (boss-direct, inline)
|
||||
|
||||
1. **INDEX backfill for `90512d5`.** Added a single-line entry in
|
||||
`docs/journals/INDEX.md` pointing at the commit and noting that
|
||||
no per-iter journal accompanies it (a small skill-discipline
|
||||
tweak rather than an iter). The format mirrors existing entries
|
||||
so future agents can navigate from INDEX to the commit history
|
||||
without reconstructing.
|
||||
|
||||
2. **README test-count correction.** The five-suite layout under
|
||||
`experiments/2026-05-12-cross-model-authoring/README.md`'s
|
||||
"Running the harness" section now lists all five suites
|
||||
explicitly (inline `--lib` unit tests for strip_locations, plus
|
||||
the four integration files) and reports the correct total
|
||||
(13 passed).
|
||||
|
||||
Both edits are ≤30 LOC and require no design judgement (CLAUDE.md
|
||||
"trivial mechanical edits" carve-out); no planner+implement
|
||||
dispatch needed.
|
||||
|
||||
## Bench-regression check
|
||||
|
||||
`bench/check.py && bench/compile_check.py && bench/cross_lang.py`:
|
||||
|
||||
- `check.py`: exit 0; **63 metrics; 0 regressed, 5 improved beyond
|
||||
tolerance, 58 stable**.
|
||||
- `compile_check.py`: exit 0; 24 metrics; 0 regressed, 0 improved
|
||||
beyond tolerance, 24 stable.
|
||||
- `cross_lang.py`: exit 0; 25 metrics; 0 regressed, 0 improved
|
||||
beyond tolerance, 25 stable.
|
||||
|
||||
All three exit-code gates green. The five improvements on
|
||||
`check.py` cluster on `latency.explicit_at_rc` (p99 -36.5%, p99_9
|
||||
-41.8%, max -41.7%, p99_over_median -38.0%) plus
|
||||
`throughput.bench_list_sum.gc_over_bump` -8.7%.
|
||||
|
||||
**Decision: no baseline update (carry-on).** Rationale:
|
||||
|
||||
- The cma milestone touched no in-workspace crate. Every line of
|
||||
code change lives under `experiments/2026-05-12-cross-model-authoring/`
|
||||
which is out of the root Cargo workspace by design. No mechanism
|
||||
in the milestone could plausibly improve runtime latency or
|
||||
GC-over-bump throughput.
|
||||
- The improvements therefore reflect either (a) runtime variance
|
||||
/ scheduler / clang version drift between the prior audit-rt
|
||||
baseline measurement and now, or (b) genuine system-level
|
||||
improvement uncorrelated with the milestone.
|
||||
- Ratifying an unexplained improvement would lock it into the
|
||||
baseline and lose the ability to detect a future regression
|
||||
back to the prior level. Better to leave the baseline pristine;
|
||||
if the improvement holds across the next two or three audits,
|
||||
the next audit can ratify with confidence.
|
||||
- This is the audit-rt pattern carried forward: don't ratify what
|
||||
the milestone didn't intentionally move.
|
||||
|
||||
## Spec acceptance criteria — final check
|
||||
|
||||
Reviewing the parent spec `docs/specs/2026-05-12-cross-model-authoring-form-test.md`
|
||||
§"Acceptance criteria" against landed work:
|
||||
|
||||
1. ✓ Directory `experiments/2026-05-12-cross-model-authoring/`
|
||||
exists with all four sub-components (master/, render/, harness/,
|
||||
runs/) plus top-level README. (cma.1 + cma.2)
|
||||
2. ✓ `master/spec.md` is vollumfänglich; `spec_completeness.rs`
|
||||
gate green; 34/34 AST variants covered by the curated subset.
|
||||
(cma.1)
|
||||
3. ✓ Both `rendered/json.md` and `rendered/ailx.md` are checked
|
||||
in; all `render/` tests green. (cma.1)
|
||||
4. ✓ `harness/` builds, passes 13/13 tests including the
|
||||
mock_full_run end-to-end. (cma.2)
|
||||
5. ✓ Four task definitions + reference solutions exist; the
|
||||
`verify_references` integration test confirms each reaches
|
||||
green through the real ail+clang pipeline locally. (cma.2)
|
||||
6. ✓ Live run executed against IONOS (Qwen3-Coder-Next,
|
||||
temperature=0, top_p=1, max-turns=5) and recorded under
|
||||
`runs/2026-05-12-df7531/` with RUN_STATUS=ok, full per-turn raw
|
||||
artefacts for all eight (cohort × task) combinations,
|
||||
scores.csv, and summary.md. (cma.3)
|
||||
7. ✓ DESIGN.md §"Decision 6" gained an "Empirical addendum
|
||||
(2026-05-12)" subsection — 27 lines, model id + run date +
|
||||
per-cohort means table + illustrative t3 contrast + explicit
|
||||
scope note deferring the universal claim to a multi-subject
|
||||
expansion. (cma.3)
|
||||
8. ✓ Journal entry `2026-05-12-iter-cma.3.md` linked from
|
||||
INDEX.md summarises the data point. (cma.3)
|
||||
9. ✓ Roadmap P2 entry removed; P3 entry "Multi-subject expansion
|
||||
(cross-model authoring-form follow-up)" added pointing at the
|
||||
run as baseline. (cma.3)
|
||||
|
||||
All nine criteria green.
|
||||
|
||||
## Outcome
|
||||
|
||||
Milestone closed clean after the two record-keeping tidies above.
|
||||
The cross-model authoring-form test ships its harness, four MVP
|
||||
tasks with verified reference solutions, a single-subject baseline
|
||||
dataset, and a calibrated DESIGN.md addendum. The next step
|
||||
(P3 roadmap: multi-subject expansion) is queued but not started.
|
||||
@@ -20,6 +20,8 @@
|
||||
- 2026-05-12 — iter rt.2: DESIGN.md anchor for Roundtrip Invariant (top-level section + Decision 6 Constraint 2 upward cross-reference) → 2026-05-12-iter-rt.2.md
|
||||
- 2026-05-12 — audit-rt: milestone close (Roundtrip Invariant) — architect drift fixed in rt.tidy (3 items: Direction-2 5th test, roadmap P1 removed, wording sync); bench all-green → 2026-05-12-audit-rt.md
|
||||
- 2026-05-12 — iter boss: `/boss` skill — autonomous-mode discipline (Direction freedom, Notifications, WhatsNew procedure) extracted from CLAUDE.md into a user-invoked skill → 2026-05-12-iter-boss.md
|
||||
- 2026-05-12 — brainstorm Step 7.5: re-dispatch grounding-check on any post-PASS spec edit (skill SKILL.md edit + roadmap P1 todo removed) → see commit `90512d5` (no per-iter journal — small skill-discipline tweak)
|
||||
- 2026-05-12 — iter cma.1: master mini-spec + render binary for cross-model authoring experiment (13 fixtures cover 34/34 AST variants, 4 test gates green, rendered/{json,ailx}.md checked in; out-of-workspace nested crate) → 2026-05-12-iter-cma.1.md
|
||||
- 2026-05-12 — iter cma.2: harness binary + 4 tasks + reference solutions + 4 integration tests (six modules, real-pipeline preflight via verify_references, 13/13 tests green; pipeline.rs renames program file to module-name for ail check) → 2026-05-12-iter-cma.2.md
|
||||
- 2026-05-12 — iter cma.3: live Qwen3-Coder-Next run + DESIGN.md §Decision-6 empirical addendum + roadmap close (AILX 2/4 green vs JSON 1/4; ~98k vs ~207k tokens; single-subject scope explicit, multi-subject expansion queued as P3) → 2026-05-12-iter-cma.3.md
|
||||
- 2026-05-12 — audit-cma: milestone close (Cross-model authoring-form test) — architect drift fixed in 2 record-keeping items (INDEX backfill for brainstorm Step 7.5, README test-count correction); bench all-green (5 unexplained improvements on latency.explicit_at_rc not coupled to milestone, baseline left pristine) → 2026-05-12-audit-cma.md
|
||||
|
||||
@@ -74,5 +74,8 @@ Tests:
|
||||
cargo test --manifest-path experiments/2026-05-12-cross-model-authoring/harness/Cargo.toml
|
||||
```
|
||||
|
||||
Four test suites: `strip_locations` (5), `verify_references` (1),
|
||||
`mock_full_run` (1), `budget_abort` (1). Total 8 passed.
|
||||
Five suites: inline `--lib` unit tests for `strip_locations` (5),
|
||||
plus integration tests `strip_locations` against captured stderr
|
||||
fixtures (5), `verify_references` (1, drives every reference through
|
||||
the real ail+clang pipeline), `mock_full_run` (1, full eight-row
|
||||
mock E2E), `budget_abort` (1). Total 13 passed.
|
||||
|
||||
Reference in New Issue
Block a user