bench: 21'c — compile-time regression bench (check.py + compile_check.py)
Closes the second axis the user named: every typechecker / codegen perf change was previously invisible to the tidy-iter gate. With Family 21 typeclasses (queued) and 21'b's poly-ADT additions both pushing on the typechecker, naive substitution loops would have landed silently and decayed the compile path. bench/compile_check.py is a separate script from bench/check.py because the methodology differs: sub-process spawn timing on small workloads (1ms scale for `ail check`, 65ms for `ail build`) vs. allocator-stress on large ones (multi-second). Tolerances differ by an order of magnitude (25% / 20% here vs. 5-15% there). Empirically: ail check is sub-ms across the corpus, dominated by subprocess spawn (~5-10ms on Linux); ail build is 63-69ms, dominated by clang's link step. The bench is a catastrophe detector (10x slowdowns visible) — finer regressions need a profiler. CLAUDE.md updated to list both scripts as co-equal tidy-iter gates alongside the architect drift report; exit 0/1/2 semantics are uniform across both. JOURNAL queue: 21'd (pure-compute fixtures) and 21'e (cross- language reference / hand-C ratio) remain to land the LLVM- linkable performance claim.
This commit is contained in:
+115
@@ -10134,3 +10134,118 @@ fixture and baseline-file additions only.
|
||||
- **Deferred richer integration paths** (from 20f) — unchanged.
|
||||
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||
pattern-binding generalisation. Orchestrator-level fork.
|
||||
|
||||
## 2026-05-09 — Iter 21'c: compile-time regression bench
|
||||
|
||||
Closes the second axis the user explicitly named — until this
|
||||
iter, every typechecker / codegen perf change was invisible to the
|
||||
tidy-iter gate. Family 21 typeclasses (queued) plus 21'b's poly-
|
||||
ADT additions both push on the typechecker; without a tripwire,
|
||||
naive substitution loops or O(n²) constraint resolution would
|
||||
land silently and decay the whole compile path.
|
||||
|
||||
### What shipped
|
||||
|
||||
**`bench/compile_check.py`** — separate from `bench/check.py`
|
||||
because the methodology is different (sub-process spawn timing
|
||||
on small workloads vs. allocator-stress on large ones) and the
|
||||
relevant tolerances differ by an order of magnitude. Two ops
|
||||
per fixture: `ail check FILE` and `ail build --opt=-O0 FILE -o T`.
|
||||
Same drop-slowest-of-N, median-of-rest convention as
|
||||
`bench/run.sh`.
|
||||
|
||||
**`bench/baseline_compile.json`** — 18 metrics (9 fixtures × 2
|
||||
ops). Curated corpus: 5 surface-coverage examples (`hello`,
|
||||
`list_map_poly`, `local_rec_capture`, `borrow_own_demo`,
|
||||
`nested_pat`) + 4 bench-throughput fixtures (correlation with
|
||||
`bench/check.py`).
|
||||
|
||||
**Baselines on this machine**:
|
||||
|
||||
```
|
||||
fixture | check(ms) | build(ms)
|
||||
hello | 0.8 | 65.0
|
||||
list_map_poly | 1.1 | 67.3
|
||||
local_rec_capture | 0.9 | 65.3
|
||||
borrow_own_demo | 1.0 | 64.3
|
||||
nested_pat | 1.7 | 67.8
|
||||
bench_list_sum | 0.9 | 63.6
|
||||
bench_tree_walk | 0.9 | 65.8
|
||||
bench_closure_chain | 0.9 | 69.0
|
||||
bench_hof_pipeline | 1.0 | 66.6
|
||||
```
|
||||
|
||||
### What the data tells us
|
||||
|
||||
`ail check` runs at **sub-millisecond per fixture** for everything
|
||||
except `nested_pat` (1.7ms — its deeper pattern tree marginally
|
||||
exceeds the noise floor). The typechecker is genuinely fast at
|
||||
the current corpus scale; on this hardware the wall-clock is
|
||||
dominated by subprocess spawn (~5-10ms on Linux), not by check
|
||||
work. The bench detects catastrophes (10× slowdowns visible),
|
||||
not subtler regressions — those want a profiler, not wall-clock.
|
||||
|
||||
`ail build --opt=-O0` runs at **63-69ms per fixture**, dominated
|
||||
by clang's link step. The variance across fixtures is small —
|
||||
~9% spread between fastest (`bench_list_sum` 63.6) and slowest
|
||||
(`bench_closure_chain` 69.0). This is fine for catastrophe-
|
||||
detection but not informative about codegen-quality differences.
|
||||
For codegen-quality questions the runtime bench (rc/bump ratios)
|
||||
remains the right tool.
|
||||
|
||||
### Tolerances
|
||||
|
||||
- **`check_ms`**: 25% per fixture. Justified empirically: a
|
||||
re-run captured a +17.35% diff on `bench_hof_pipeline check`
|
||||
with no code changes. Sub-millisecond timing is noisy.
|
||||
- **`build_O0_ms`**: 20% per fixture. Build noise is materially
|
||||
lower; observed re-run drift was ≤7% on every fixture.
|
||||
|
||||
These are catastrophe-detector tolerances. Tightening them
|
||||
would mean false-positives on quiet-machine noise.
|
||||
|
||||
### CLAUDE.md update
|
||||
|
||||
The `Performance regressions` section now lists both
|
||||
`bench/check.py` (runtime) and `bench/compile_check.py` (compile)
|
||||
as co-equal tidy-iter gates alongside the architect drift report.
|
||||
Exit 0 / 1 / 2 semantics are uniform across both scripts.
|
||||
|
||||
### What this iter does NOT do
|
||||
|
||||
- **No latency-harness methodology upgrade.** The wide
|
||||
explicit_at_rc.p99 dispersion observed across today's three
|
||||
captures (357.5 / 294.6 / 251.5) is a runtime-bench problem;
|
||||
the compile bench is a different axis. Methodology upgrade
|
||||
(n>=10 captures or tighter latency fixture) stays queued.
|
||||
- **No O2 build bench.** `--opt=-O2` includes additional clang
|
||||
passes that 2x-3x the build time. Useful for catching codegen
|
||||
blowup-induced build slowdowns; not useful for detecting
|
||||
AILang-side regressions, which are amply covered by the O0
|
||||
pass. Future addition if/when warranted.
|
||||
- **No incremental check bench.** Today every `ail check`
|
||||
rebuilds the entire context. If incremental compilation is
|
||||
added later (no current plan), re-baseline.
|
||||
|
||||
### Test state
|
||||
|
||||
288 / 0 / 3, unchanged. No Rust changes; the iter is bench-
|
||||
infrastructure additions only.
|
||||
|
||||
### JOURNAL queue (updated)
|
||||
|
||||
- **21'd — pure-compute fixtures.** Mandelbrot / N-body / integer-
|
||||
loop workloads. Heap-light, codegen-quality-heavy. Pairs
|
||||
naturally with 21'e.
|
||||
- **21'e — cross-language reference.** Hand-C variants of the
|
||||
bench corpus, compiled with `clang -O2`. AILang/C ratio is the
|
||||
honest answer to CLAUDE.md's "LLVM-linkable, performance is
|
||||
extremely important" claim, which today is unbacked by data.
|
||||
- **Latency methodology upgrade** — n=10+ captures or tighter
|
||||
fixture for `explicit_at_rc.p99`. Could fold into 21'd or be
|
||||
its own short iter.
|
||||
- **`FnDef::synthetic(...)` factor-out** — unchanged.
|
||||
- **Boehm full retirement** — unchanged.
|
||||
- **Deferred richer integration paths** (from 20f) — unchanged.
|
||||
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||
pattern-binding generalisation. Orchestrator-level fork.
|
||||
|
||||
Reference in New Issue
Block a user