discipline: bench/check.py as co-equal tidy-iter gate
Adds a "Performance regressions" subsection to the iter-cycle discipline. bench/check.py runs at every family close alongside the architect drift report; its exit code is the gate. Skipping it requires the same explicit JOURNAL entry as skipping the drift review. Codifies the discipline that 21'a's JOURNAL entry recommended and explicitly punted to the orchestrator. The tooling shipped yesterday; the rule that makes it load-bearing ships now. Also clarifies two non-goals: per-metric tolerances are not the language correctness bar (Decision-10 thresholds remain in DESIGN.md, evaluated against absolute numbers); and improvements do not auto-update the baseline — they surface for ratification.
This commit is contained in:
@@ -181,6 +181,46 @@ and the deferral compounds across families. Skipping a tidy-iter
|
||||
requires an explicit JOURNAL entry naming the reason (e.g. a
|
||||
sibling family is blocking, or the user has asked to defer).
|
||||
|
||||
### Performance regressions
|
||||
|
||||
`bench/check.py` is the second non-optional element of the
|
||||
tidy-iter, alongside the architect drift report. Run it at every
|
||||
family close. The exit code is the gate:
|
||||
|
||||
- **Exit 0 (green).** All metrics within their per-metric
|
||||
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
|
||||
- **Exit 1 (red).** At least one metric regressed beyond
|
||||
tolerance. Treat the regression like an architect-drift item:
|
||||
fix (revert the offending iter, refactor the hot path, diagnose
|
||||
with `ailang-bencher` or `ailang-debugger`), or ratify
|
||||
(`bench/check.py --update-baseline` together with a JOURNAL
|
||||
entry naming the iter that intentionally moved the metric and
|
||||
why).
|
||||
- **Exit 2 (parser misalignment).** The bench-output format
|
||||
changed under check.py's parsers. Update parsers or baseline
|
||||
schema before re-running. Never claim a regression on exit-2.
|
||||
|
||||
Skipping the bench-check at a family boundary requires the same
|
||||
explicit JOURNAL entry as skipping the architect drift review:
|
||||
name the reason, name the sibling family that's blocking, name
|
||||
when it'll be re-run.
|
||||
|
||||
A bug-fix iter that adds a new bench fixture (e.g. proving the
|
||||
fix's invariant under load) bumps `bench/baseline.json` as part
|
||||
of the same commit. The new fixture's numbers are baseline-zero
|
||||
on first capture; they become regression-relevant from the next
|
||||
tidy-iter onward.
|
||||
|
||||
Two non-goals to keep clear: (1) per-metric tolerances are tuned
|
||||
for run-to-run noise on a quiet developer machine, NOT the
|
||||
language's correctness bar — Decision-10 thresholds (rc/bump ≤
|
||||
1.3× / p99/median ≤ 5×) live in DESIGN.md and are evaluated
|
||||
against absolute numbers, separate from the regression check.
|
||||
(2) Improvements (metric beat its baseline by more than tolerance)
|
||||
do NOT auto-update the baseline; they are surfaced in the report
|
||||
so the orchestrator can ratify them via `--update-baseline` with
|
||||
a paired JOURNAL entry.
|
||||
|
||||
### Roles of JOURNAL.md and DESIGN.md
|
||||
|
||||
- **DESIGN.md** is the canonical specification. It describes what
|
||||
|
||||
Reference in New Issue
Block a user