bench: 21'c — compile-time regression bench (check.py + compile_check.py)
Closes the second axis the user named: every typechecker / codegen perf change was previously invisible to the tidy-iter gate. With Family 21 typeclasses (queued) and 21'b's poly-ADT additions both pushing on the typechecker, naive substitution loops would have landed silently and decayed the compile path. bench/compile_check.py is a separate script from bench/check.py because the methodology differs: sub-process spawn timing on small workloads (1ms scale for `ail check`, 65ms for `ail build`) vs. allocator-stress on large ones (multi-second). Tolerances differ by an order of magnitude (25% / 20% here vs. 5-15% there). Empirically: ail check is sub-ms across the corpus, dominated by subprocess spawn (~5-10ms on Linux); ail build is 63-69ms, dominated by clang's link step. The bench is a catastrophe detector (10x slowdowns visible) — finer regressions need a profiler. CLAUDE.md updated to list both scripts as co-equal tidy-iter gates alongside the architect drift report; exit 0/1/2 semantics are uniform across both. JOURNAL queue: 21'd (pure-compute fixtures) and 21'e (cross- language reference / hand-C ratio) remain to land the LLVM- linkable performance claim.
This commit is contained in:
@@ -183,9 +183,17 @@ sibling family is blocking, or the user has asked to defer).
|
||||
|
||||
### Performance regressions
|
||||
|
||||
`bench/check.py` is the second non-optional element of the
|
||||
tidy-iter, alongside the architect drift report. Run it at every
|
||||
family close. The exit code is the gate:
|
||||
Two scripts gate the tidy-iter alongside the architect drift
|
||||
report:
|
||||
|
||||
- **`bench/check.py`** — runtime regressions (gc/bump/rc throughput
|
||||
and PTY tail-latency over the bench fixture corpus, baselined in
|
||||
`bench/baseline.json`).
|
||||
- **`bench/compile_check.py`** — compile-time regressions
|
||||
(`ail check` and `ail build --opt=-O0` wall-time over a curated
|
||||
example corpus, baselined in `bench/baseline_compile.json`).
|
||||
|
||||
Run both at every family close. The exit code is the gate:
|
||||
|
||||
- **Exit 0 (green).** All metrics within their per-metric
|
||||
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
|
||||
@@ -193,12 +201,14 @@ family close. The exit code is the gate:
|
||||
tolerance. Treat the regression like an architect-drift item:
|
||||
fix (revert the offending iter, refactor the hot path, diagnose
|
||||
with `ailang-bencher` or `ailang-debugger`), or ratify
|
||||
(`bench/check.py --update-baseline` together with a JOURNAL
|
||||
entry naming the iter that intentionally moved the metric and
|
||||
why).
|
||||
- **Exit 2 (parser misalignment).** The bench-output format
|
||||
changed under check.py's parsers. Update parsers or baseline
|
||||
schema before re-running. Never claim a regression on exit-2.
|
||||
(`--update-baseline` on the firing script together with a
|
||||
JOURNAL entry naming the iter that intentionally moved the
|
||||
metric and why).
|
||||
- **Exit 2 (infrastructure failure).** Bench-output format
|
||||
changed (`check.py` parser misalignment), or a corpus fixture
|
||||
is missing / fails to spawn (`compile_check.py`). Fix the
|
||||
infrastructure before re-running. Never claim a regression on
|
||||
exit-2.
|
||||
|
||||
Skipping the bench-check at a family boundary requires the same
|
||||
explicit JOURNAL entry as skipping the architect drift review:
|
||||
|
||||
Reference in New Issue
Block a user