discipline: bench/check.py as co-equal tidy-iter gate

Adds a "Performance regressions" subsection to the iter-cycle
discipline. bench/check.py runs at every family close alongside
the architect drift report; its exit code is the gate. Skipping
it requires the same explicit JOURNAL entry as skipping the
drift review.

Codifies the discipline that 21'a's JOURNAL entry recommended
and explicitly punted to the orchestrator. The tooling shipped
yesterday; the rule that makes it load-bearing ships now.

Also clarifies two non-goals: per-metric tolerances are not the
language correctness bar (Decision-10 thresholds remain in
DESIGN.md, evaluated against absolute numbers); and improvements
do not auto-update the baseline — they surface for ratification.
This commit is contained in:
2026-05-09 00:49:51 +02:00
parent a1b0ad5723
commit 2e40699cb1
+40
View File
@@ -181,6 +181,46 @@ and the deferral compounds across families. Skipping a tidy-iter
requires an explicit JOURNAL entry naming the reason (e.g. a
sibling family is blocking, or the user has asked to defer).
### Performance regressions
`bench/check.py` is the second non-optional element of the
tidy-iter, alongside the architect drift report. Run it at every
family close. The exit code is the gate:
- **Exit 0 (green).** All metrics within their per-metric
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
- **Exit 1 (red).** At least one metric regressed beyond
tolerance. Treat the regression like an architect-drift item:
fix (revert the offending iter, refactor the hot path, diagnose
with `ailang-bencher` or `ailang-debugger`), or ratify
(`bench/check.py --update-baseline` together with a JOURNAL
entry naming the iter that intentionally moved the metric and
why).
- **Exit 2 (parser misalignment).** The bench-output format
changed under check.py's parsers. Update parsers or baseline
schema before re-running. Never claim a regression on exit-2.
Skipping the bench-check at a family boundary requires the same
explicit JOURNAL entry as skipping the architect drift review:
name the reason, name the sibling family that's blocking, name
when it'll be re-run.
A bug-fix iter that adds a new bench fixture (e.g. proving the
fix's invariant under load) bumps `bench/baseline.json` as part
of the same commit. The new fixture's numbers are baseline-zero
on first capture; they become regression-relevant from the next
tidy-iter onward.
Two non-goals to keep clear: (1) per-metric tolerances are tuned
for run-to-run noise on a quiet developer machine, NOT the
language's correctness bar — Decision-10 thresholds (rc/bump ≤
1.3× / p99/median ≤ 5×) live in DESIGN.md and are evaluated
against absolute numbers, separate from the regression check.
(2) Improvements (metric beat its baseline by more than tolerance)
do NOT auto-update the baseline; they are surfaced in the report
so the orchestrator can ratify them via `--update-baseline` with
a paired JOURNAL entry.
### Roles of JOURNAL.md and DESIGN.md
- **DESIGN.md** is the canonical specification. It describes what