bench: 21'c — compile-time regression bench (check.py + compile_check.py)

Closes the second axis the user named: every typechecker / codegen
perf change was previously invisible to the tidy-iter gate. With
Family 21 typeclasses (queued) and 21'b's poly-ADT additions both
pushing on the typechecker, naive substitution loops would have
landed silently and decayed the compile path.

bench/compile_check.py is a separate script from bench/check.py
because the methodology differs: sub-process spawn timing on small
workloads (1ms scale for `ail check`, 65ms for `ail build`) vs.
allocator-stress on large ones (multi-second). Tolerances differ
by an order of magnitude (25% / 20% here vs. 5-15% there).

Empirically: ail check is sub-ms across the corpus, dominated by
subprocess spawn (~5-10ms on Linux); ail build is 63-69ms,
dominated by clang's link step. The bench is a catastrophe
detector (10x slowdowns visible) — finer regressions need a
profiler. CLAUDE.md updated to list both scripts as co-equal
tidy-iter gates alongside the architect drift report; exit 0/1/2
semantics are uniform across both.

JOURNAL queue: 21'd (pure-compute fixtures) and 21'e (cross-
language reference / hand-C ratio) remain to land the LLVM-
linkable performance claim.
This commit is contained in:
2026-05-09 00:59:00 +02:00
parent 07bff24527
commit 416d763b73
4 changed files with 450 additions and 9 deletions
+115
View File
@@ -10134,3 +10134,118 @@ fixture and baseline-file additions only.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork.
## 2026-05-09 — Iter 21'c: compile-time regression bench
Closes the second axis the user explicitly named — until this
iter, every typechecker / codegen perf change was invisible to the
tidy-iter gate. Family 21 typeclasses (queued) plus 21'b's poly-
ADT additions both push on the typechecker; without a tripwire,
naive substitution loops or O(n²) constraint resolution would
land silently and decay the whole compile path.
### What shipped
**`bench/compile_check.py`** — separate from `bench/check.py`
because the methodology is different (sub-process spawn timing
on small workloads vs. allocator-stress on large ones) and the
relevant tolerances differ by an order of magnitude. Two ops
per fixture: `ail check FILE` and `ail build --opt=-O0 FILE -o T`.
Same drop-slowest-of-N, median-of-rest convention as
`bench/run.sh`.
**`bench/baseline_compile.json`** — 18 metrics (9 fixtures × 2
ops). Curated corpus: 5 surface-coverage examples (`hello`,
`list_map_poly`, `local_rec_capture`, `borrow_own_demo`,
`nested_pat`) + 4 bench-throughput fixtures (correlation with
`bench/check.py`).
**Baselines on this machine**:
```
fixture | check(ms) | build(ms)
hello | 0.8 | 65.0
list_map_poly | 1.1 | 67.3
local_rec_capture | 0.9 | 65.3
borrow_own_demo | 1.0 | 64.3
nested_pat | 1.7 | 67.8
bench_list_sum | 0.9 | 63.6
bench_tree_walk | 0.9 | 65.8
bench_closure_chain | 0.9 | 69.0
bench_hof_pipeline | 1.0 | 66.6
```
### What the data tells us
`ail check` runs at **sub-millisecond per fixture** for everything
except `nested_pat` (1.7ms — its deeper pattern tree marginally
exceeds the noise floor). The typechecker is genuinely fast at
the current corpus scale; on this hardware the wall-clock is
dominated by subprocess spawn (~5-10ms on Linux), not by check
work. The bench detects catastrophes (10× slowdowns visible),
not subtler regressions — those want a profiler, not wall-clock.
`ail build --opt=-O0` runs at **63-69ms per fixture**, dominated
by clang's link step. The variance across fixtures is small —
~9% spread between fastest (`bench_list_sum` 63.6) and slowest
(`bench_closure_chain` 69.0). This is fine for catastrophe-
detection but not informative about codegen-quality differences.
For codegen-quality questions the runtime bench (rc/bump ratios)
remains the right tool.
### Tolerances
- **`check_ms`**: 25% per fixture. Justified empirically: a
re-run captured a +17.35% diff on `bench_hof_pipeline check`
with no code changes. Sub-millisecond timing is noisy.
- **`build_O0_ms`**: 20% per fixture. Build noise is materially
lower; observed re-run drift was ≤7% on every fixture.
These are catastrophe-detector tolerances. Tightening them
would mean false-positives on quiet-machine noise.
### CLAUDE.md update
The `Performance regressions` section now lists both
`bench/check.py` (runtime) and `bench/compile_check.py` (compile)
as co-equal tidy-iter gates alongside the architect drift report.
Exit 0 / 1 / 2 semantics are uniform across both scripts.
### What this iter does NOT do
- **No latency-harness methodology upgrade.** The wide
explicit_at_rc.p99 dispersion observed across today's three
captures (357.5 / 294.6 / 251.5) is a runtime-bench problem;
the compile bench is a different axis. Methodology upgrade
(n>=10 captures or tighter latency fixture) stays queued.
- **No O2 build bench.** `--opt=-O2` includes additional clang
passes that 2x-3x the build time. Useful for catching codegen
blowup-induced build slowdowns; not useful for detecting
AILang-side regressions, which are amply covered by the O0
pass. Future addition if/when warranted.
- **No incremental check bench.** Today every `ail check`
rebuilds the entire context. If incremental compilation is
added later (no current plan), re-baseline.
### Test state
288 / 0 / 3, unchanged. No Rust changes; the iter is bench-
infrastructure additions only.
### JOURNAL queue (updated)
- **21'd — pure-compute fixtures.** Mandelbrot / N-body / integer-
loop workloads. Heap-light, codegen-quality-heavy. Pairs
naturally with 21'e.
- **21'e — cross-language reference.** Hand-C variants of the
bench corpus, compiled with `clang -O2`. AILang/C ratio is the
honest answer to CLAUDE.md's "LLVM-linkable, performance is
extremely important" claim, which today is unbacked by data.
- **Latency methodology upgrade** — n=10+ captures or tighter
fixture for `explicit_at_rc.p99`. Could fold into 21'd or be
its own short iter.
- **`FnDef::synthetic(...)` factor-out** — unchanged.
- **Boehm full retirement** — unchanged.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork.