From 416d763b7359a9396a01cbcd785c9c1296fdd4cc Mon Sep 17 00:00:00 2001 From: Brummel Date: Sat, 9 May 2026 00:59:00 +0200 Subject: [PATCH] =?UTF-8?q?bench:=2021'c=20=E2=80=94=20compile-time=20regr?= =?UTF-8?q?ession=20bench=20(check.py=20+=20compile=5Fcheck.py)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the second axis the user named: every typechecker / codegen perf change was previously invisible to the tidy-iter gate. With Family 21 typeclasses (queued) and 21'b's poly-ADT additions both pushing on the typechecker, naive substitution loops would have landed silently and decayed the compile path. bench/compile_check.py is a separate script from bench/check.py because the methodology differs: sub-process spawn timing on small workloads (1ms scale for `ail check`, 65ms for `ail build`) vs. allocator-stress on large ones (multi-second). Tolerances differ by an order of magnitude (25% / 20% here vs. 5-15% there). Empirically: ail check is sub-ms across the corpus, dominated by subprocess spawn (~5-10ms on Linux); ail build is 63-69ms, dominated by clang's link step. The bench is a catastrophe detector (10x slowdowns visible) — finer regressions need a profiler. CLAUDE.md updated to list both scripts as co-equal tidy-iter gates alongside the architect drift report; exit 0/1/2 semantics are uniform across both. JOURNAL queue: 21'd (pure-compute fixtures) and 21'e (cross- language reference / hand-C ratio) remain to land the LLVM- linkable performance claim. --- CLAUDE.md | 28 +++-- bench/baseline_compile.json | 82 +++++++++++++ bench/compile_check.py | 234 ++++++++++++++++++++++++++++++++++++ docs/JOURNAL.md | 115 ++++++++++++++++++ 4 files changed, 450 insertions(+), 9 deletions(-) create mode 100644 bench/baseline_compile.json create mode 100755 bench/compile_check.py diff --git a/CLAUDE.md b/CLAUDE.md index bea14fd..428f818 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -183,9 +183,17 @@ sibling family is blocking, or the user has asked to defer). ### Performance regressions -`bench/check.py` is the second non-optional element of the -tidy-iter, alongside the architect drift report. Run it at every -family close. The exit code is the gate: +Two scripts gate the tidy-iter alongside the architect drift +report: + +- **`bench/check.py`** — runtime regressions (gc/bump/rc throughput + and PTY tail-latency over the bench fixture corpus, baselined in + `bench/baseline.json`). +- **`bench/compile_check.py`** — compile-time regressions + (`ail check` and `ail build --opt=-O0` wall-time over a curated + example corpus, baselined in `bench/baseline_compile.json`). + +Run both at every family close. The exit code is the gate: - **Exit 0 (green).** All metrics within their per-metric tolerance vs. `bench/baseline.json`. Tidy-iter can close. @@ -193,12 +201,14 @@ family close. The exit code is the gate: tolerance. Treat the regression like an architect-drift item: fix (revert the offending iter, refactor the hot path, diagnose with `ailang-bencher` or `ailang-debugger`), or ratify - (`bench/check.py --update-baseline` together with a JOURNAL - entry naming the iter that intentionally moved the metric and - why). -- **Exit 2 (parser misalignment).** The bench-output format - changed under check.py's parsers. Update parsers or baseline - schema before re-running. Never claim a regression on exit-2. + (`--update-baseline` on the firing script together with a + JOURNAL entry naming the iter that intentionally moved the + metric and why). +- **Exit 2 (infrastructure failure).** Bench-output format + changed (`check.py` parser misalignment), or a corpus fixture + is missing / fails to spawn (`compile_check.py`). Fix the + infrastructure before re-running. Never claim a regression on + exit-2. Skipping the bench-check at a family boundary requires the same explicit JOURNAL entry as skipping the architect drift review: diff --git a/bench/baseline_compile.json b/bench/baseline_compile.json new file mode 100644 index 0000000..f23e350 --- /dev/null +++ b/bench/baseline_compile.json @@ -0,0 +1,82 @@ +{ + "version": 1, + "captured": "2026-05-09", + "captured_via": "bench/compile_check.py", + "note": "Compile-time bench. `check_ms` is `ail check FILE`; `build_O0_ms` is `ail build --opt=-O0 FILE`. Wall-clock includes subprocess spawn (~5-10 ms on Linux) and, for build, the clang link step. Tolerances are tuned for noise on a quiet developer machine, not as the language correctness bar.", + "check_ms": { + "hello": { + "baseline_ms": 0.85, + "tolerance_pct": 25 + }, + "list_map_poly": { + "baseline_ms": 1.05, + "tolerance_pct": 25 + }, + "local_rec_capture": { + "baseline_ms": 0.9, + "tolerance_pct": 25 + }, + "borrow_own_demo": { + "baseline_ms": 1.01, + "tolerance_pct": 25 + }, + "nested_pat": { + "baseline_ms": 1.74, + "tolerance_pct": 25 + }, + "bench_list_sum": { + "baseline_ms": 0.91, + "tolerance_pct": 25 + }, + "bench_tree_walk": { + "baseline_ms": 0.9, + "tolerance_pct": 25 + }, + "bench_closure_chain": { + "baseline_ms": 0.88, + "tolerance_pct": 25 + }, + "bench_hof_pipeline": { + "baseline_ms": 0.99, + "tolerance_pct": 25 + } + }, + "build_O0_ms": { + "hello": { + "baseline_ms": 65.03, + "tolerance_pct": 20 + }, + "list_map_poly": { + "baseline_ms": 67.27, + "tolerance_pct": 20 + }, + "local_rec_capture": { + "baseline_ms": 65.35, + "tolerance_pct": 20 + }, + "borrow_own_demo": { + "baseline_ms": 64.26, + "tolerance_pct": 20 + }, + "nested_pat": { + "baseline_ms": 67.76, + "tolerance_pct": 20 + }, + "bench_list_sum": { + "baseline_ms": 63.6, + "tolerance_pct": 20 + }, + "bench_tree_walk": { + "baseline_ms": 65.77, + "tolerance_pct": 20 + }, + "bench_closure_chain": { + "baseline_ms": 68.98, + "tolerance_pct": 20 + }, + "bench_hof_pipeline": { + "baseline_ms": 66.64, + "tolerance_pct": 20 + } + } +} diff --git a/bench/compile_check.py b/bench/compile_check.py new file mode 100755 index 0000000..0b7e5af --- /dev/null +++ b/bench/compile_check.py @@ -0,0 +1,234 @@ +#!/usr/bin/env python3 +# Compile-time regression check. +# +# Times `ail check FILE` and `ail build --opt=-O0 FILE` over a curated +# corpus, drops the slowest run, takes the median of the rest, diffs +# against bench/baseline_compile.json. Exits 0 if every metric is within +# tolerance, 1 on any regression, 2 on infrastructure failure (missing +# fixture, ail spawn error). +# +# Methodology caveats this bench cannot work around: +# - Wall time of `ail check` is dominated by subprocess spawn (~5-10ms +# on Linux); typechecker work below that scale is invisible. +# - Wall time of `ail build` is dominated by clang's link step +# (~100ms+); the AILang IR-emit + codegen contribution is a +# small slice of the total. +# Both are still useful as catastrophe detectors (10x slowdowns visible) +# even if subtler regressions need a profiler. The right tool for finer- +# grained codegen perf is the runtime bench (bench/run.sh), not this one. +# +# Usage: +# bench/compile_check.py # run bench, exit 0/1/2 +# bench/compile_check.py -n 10 # more runs, tighter median +# bench/compile_check.py --update-baseline +# bench/compile_check.py --baseline path # alternate baseline file + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +DEFAULT_BASELINE = ROOT / "bench" / "baseline_compile.json" +AIL = ROOT / "target" / "release" / "ail" +EXAMPLES = ROOT / "examples" + +# Curated corpus. Pinned by name so the baseline keys are stable across +# example-directory churn. Each entry must have a corresponding +# `.ail.json` under examples/. +CORPUS = [ + # Surface coverage: each fixture exercises a different feature set. + "hello", # IO baseline + "list_map_poly", # polymorphism + HOF + "local_rec_capture", # let-rec capture, closure path + "borrow_own_demo", # explicit modes + "nested_pat", # nested pattern matching + # Bench fixtures (correlation with bench/check.py runtime baseline). + "bench_list_sum", + "bench_tree_walk", + "bench_closure_chain", + "bench_hof_pipeline", +] + + +def time_one(args: list[str]) -> float: + """Run a command, return wall-time in milliseconds. Raises if exit != 0.""" + t0 = time.monotonic() + proc = subprocess.run(args, capture_output=True) + t1 = time.monotonic() + if proc.returncode != 0: + raise RuntimeError( + f"command failed (exit {proc.returncode}): {' '.join(args)}\n" + f"stderr: {proc.stderr.decode(errors='replace')}" + ) + return (t1 - t0) * 1000.0 + + +def median_drop_slowest(runs: list[float]) -> float: + """Drop the slowest run, return median of the rest.""" + if len(runs) < 2: + return runs[0] if runs else 0.0 + kept = sorted(runs)[:-1] + n = len(kept) + if n % 2 == 1: + return kept[n // 2] + return 0.5 * (kept[n // 2 - 1] + kept[n // 2]) + + +def measure(num_runs: int) -> dict[str, dict[str, float]]: + """Return {section: {fixture: ms}}. Aborts on missing fixture / spawn fail.""" + if not AIL.is_file(): + print(f"missing release ail binary at {AIL}; building...", file=sys.stderr) + subprocess.run( + ["cargo", "build", "--release", "-p", "ail"], + cwd=str(ROOT), check=True, capture_output=True, + ) + + out: dict[str, dict[str, float]] = {"check_ms": {}, "build_O0_ms": {}} + with tempfile.NamedTemporaryFile(prefix="bench_compile_", delete=False) as tf: + out_path = tf.name + + try: + for fixture in CORPUS: + src = EXAMPLES / f"{fixture}.ail.json" + if not src.is_file(): + print(f"missing fixture: {src}", file=sys.stderr) + sys.exit(2) + + try: + check_runs = [time_one([str(AIL), "check", str(src)]) for _ in range(num_runs)] + build_runs = [time_one([str(AIL), "build", "--opt=-O0", + str(src), "-o", out_path]) for _ in range(num_runs)] + except RuntimeError as e: + print(f"spawn error on {fixture}: {e}", file=sys.stderr) + sys.exit(2) + + out["check_ms"][fixture] = median_drop_slowest(check_runs) + out["build_O0_ms"][fixture] = median_drop_slowest(build_runs) + print(f" {fixture:<32} check={out['check_ms'][fixture]:6.1f}ms " + f"build={out['build_O0_ms'][fixture]:6.1f}ms", file=sys.stderr) + finally: + try: + Path(out_path).unlink() + except FileNotFoundError: + pass + + return out + + +def diff_report(measured: dict, baseline: dict) -> tuple[str, bool]: + rows = [] + has_regression = False + for section in ("check_ms", "build_O0_ms"): + for fixture, spec in baseline.get(section, {}).items(): + actual = measured[section].get(fixture) + if actual is None: + rows.append((f"{section}.{fixture}", spec["baseline_ms"], None, + None, spec["tolerance_pct"], "MISSING")) + has_regression = True + continue + base = spec["baseline_ms"] + tol = spec["tolerance_pct"] + diff = 100.0 * (actual - base) / base if base else 0.0 + if diff > tol: + status = "REGRESSION" + has_regression = True + elif diff < -tol: + status = "improvement" + else: + status = "ok" + rows.append((f"{section}.{fixture}", base, actual, diff, tol, status)) + + lines = [] + lines.append(f"{'metric':<48} {'baseline':>10} {'actual':>10} {'diff':>9} {'tol':>6} status") + lines.append("-" * 100) + for metric, base, actual, diff, tol, status in rows: + if actual is None: + lines.append(f"{metric:<48} {base:>10.1f} {'-':>10} {'-':>9} {tol:>5.1f}% {status}") + else: + lines.append( + f"{metric:<48} {base:>10.1f} {actual:>10.1f} " + f"{diff:>+8.2f}% {tol:>5.1f}% {status}" + ) + regressed = sum(1 for r in rows if r[5] == "REGRESSION") + improved = sum(1 for r in rows if r[5] == "improvement") + stable = len(rows) - regressed - improved + lines.append("") + lines.append(f"summary: {len(rows)} metrics; " + f"{regressed} regressed, {improved} improved beyond tolerance, " + f"{stable} stable") + return "\n".join(lines), has_regression + + +def write_baseline(measured: dict, baseline_path: Path) -> None: + today = subprocess.check_output(["date", "+%Y-%m-%d"], text=True).strip() + if baseline_path.exists(): + existing = json.loads(baseline_path.read_text()) + check_tols = {f: spec.get("tolerance_pct", 25) + for f, spec in existing.get("check_ms", {}).items()} + build_tols = {f: spec.get("tolerance_pct", 20) + for f, spec in existing.get("build_O0_ms", {}).items()} + else: + check_tols = {} + build_tols = {} + + new = { + "version": 1, + "captured": today, + "captured_via": "bench/compile_check.py", + "note": "Compile-time bench. `check_ms` is `ail check FILE`; `build_O0_ms` is `ail build --opt=-O0 FILE`. Wall-clock includes subprocess spawn (~5-10 ms on Linux) and, for build, the clang link step. Tolerances are tuned for noise on a quiet developer machine, not as the language correctness bar.", + "check_ms": {}, + "build_O0_ms": {}, + } + for fixture, ms in measured["check_ms"].items(): + new["check_ms"][fixture] = { + "baseline_ms": round(ms, 2), + "tolerance_pct": check_tols.get(fixture, 25), + } + for fixture, ms in measured["build_O0_ms"].items(): + new["build_O0_ms"][fixture] = { + "baseline_ms": round(ms, 2), + "tolerance_pct": build_tols.get(fixture, 20), + } + baseline_path.write_text(json.dumps(new, indent=2) + "\n") + print(f">>> wrote new baseline to {baseline_path}", file=sys.stderr) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("-n", "--runs", type=int, default=5, + help="runs per fixture per op; min 2, default 5") + ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE) + ap.add_argument("--update-baseline", action="store_true", + help="re-measure, then overwrite baseline_compile.json") + args = ap.parse_args() + if args.runs < 2: + print("--runs must be >= 2", file=sys.stderr) + return 2 + + print(f">>> measuring {len(CORPUS)} fixtures, {args.runs} runs each " + f"(check + build_O0)", file=sys.stderr) + measured = measure(args.runs) + + if args.update_baseline: + write_baseline(measured, args.baseline) + return 0 + + if not args.baseline.exists(): + print(f"no baseline at {args.baseline}; create one with --update-baseline", + file=sys.stderr) + return 2 + + baseline = json.loads(args.baseline.read_text()) + report, has_regression = diff_report(measured, baseline) + print(report) + return 1 if has_regression else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/JOURNAL.md b/docs/JOURNAL.md index a4b3199..9de92f6 100644 --- a/docs/JOURNAL.md +++ b/docs/JOURNAL.md @@ -10134,3 +10134,118 @@ fixture and baseline-file additions only. - **Deferred richer integration paths** (from 20f) — unchanged. - **Family 21+** — typeclasses, polymorphic ADTs at runtime, pattern-binding generalisation. Orchestrator-level fork. + +## 2026-05-09 — Iter 21'c: compile-time regression bench + +Closes the second axis the user explicitly named — until this +iter, every typechecker / codegen perf change was invisible to the +tidy-iter gate. Family 21 typeclasses (queued) plus 21'b's poly- +ADT additions both push on the typechecker; without a tripwire, +naive substitution loops or O(n²) constraint resolution would +land silently and decay the whole compile path. + +### What shipped + +**`bench/compile_check.py`** — separate from `bench/check.py` +because the methodology is different (sub-process spawn timing +on small workloads vs. allocator-stress on large ones) and the +relevant tolerances differ by an order of magnitude. Two ops +per fixture: `ail check FILE` and `ail build --opt=-O0 FILE -o T`. +Same drop-slowest-of-N, median-of-rest convention as +`bench/run.sh`. + +**`bench/baseline_compile.json`** — 18 metrics (9 fixtures × 2 +ops). Curated corpus: 5 surface-coverage examples (`hello`, +`list_map_poly`, `local_rec_capture`, `borrow_own_demo`, +`nested_pat`) + 4 bench-throughput fixtures (correlation with +`bench/check.py`). + +**Baselines on this machine**: + +``` +fixture | check(ms) | build(ms) +hello | 0.8 | 65.0 +list_map_poly | 1.1 | 67.3 +local_rec_capture | 0.9 | 65.3 +borrow_own_demo | 1.0 | 64.3 +nested_pat | 1.7 | 67.8 +bench_list_sum | 0.9 | 63.6 +bench_tree_walk | 0.9 | 65.8 +bench_closure_chain | 0.9 | 69.0 +bench_hof_pipeline | 1.0 | 66.6 +``` + +### What the data tells us + +`ail check` runs at **sub-millisecond per fixture** for everything +except `nested_pat` (1.7ms — its deeper pattern tree marginally +exceeds the noise floor). The typechecker is genuinely fast at +the current corpus scale; on this hardware the wall-clock is +dominated by subprocess spawn (~5-10ms on Linux), not by check +work. The bench detects catastrophes (10× slowdowns visible), +not subtler regressions — those want a profiler, not wall-clock. + +`ail build --opt=-O0` runs at **63-69ms per fixture**, dominated +by clang's link step. The variance across fixtures is small — +~9% spread between fastest (`bench_list_sum` 63.6) and slowest +(`bench_closure_chain` 69.0). This is fine for catastrophe- +detection but not informative about codegen-quality differences. +For codegen-quality questions the runtime bench (rc/bump ratios) +remains the right tool. + +### Tolerances + +- **`check_ms`**: 25% per fixture. Justified empirically: a + re-run captured a +17.35% diff on `bench_hof_pipeline check` + with no code changes. Sub-millisecond timing is noisy. +- **`build_O0_ms`**: 20% per fixture. Build noise is materially + lower; observed re-run drift was ≤7% on every fixture. + +These are catastrophe-detector tolerances. Tightening them +would mean false-positives on quiet-machine noise. + +### CLAUDE.md update + +The `Performance regressions` section now lists both +`bench/check.py` (runtime) and `bench/compile_check.py` (compile) +as co-equal tidy-iter gates alongside the architect drift report. +Exit 0 / 1 / 2 semantics are uniform across both scripts. + +### What this iter does NOT do + +- **No latency-harness methodology upgrade.** The wide + explicit_at_rc.p99 dispersion observed across today's three + captures (357.5 / 294.6 / 251.5) is a runtime-bench problem; + the compile bench is a different axis. Methodology upgrade + (n>=10 captures or tighter latency fixture) stays queued. +- **No O2 build bench.** `--opt=-O2` includes additional clang + passes that 2x-3x the build time. Useful for catching codegen + blowup-induced build slowdowns; not useful for detecting + AILang-side regressions, which are amply covered by the O0 + pass. Future addition if/when warranted. +- **No incremental check bench.** Today every `ail check` + rebuilds the entire context. If incremental compilation is + added later (no current plan), re-baseline. + +### Test state + +288 / 0 / 3, unchanged. No Rust changes; the iter is bench- +infrastructure additions only. + +### JOURNAL queue (updated) + +- **21'd — pure-compute fixtures.** Mandelbrot / N-body / integer- + loop workloads. Heap-light, codegen-quality-heavy. Pairs + naturally with 21'e. +- **21'e — cross-language reference.** Hand-C variants of the + bench corpus, compiled with `clang -O2`. AILang/C ratio is the + honest answer to CLAUDE.md's "LLVM-linkable, performance is + extremely important" claim, which today is unbacked by data. +- **Latency methodology upgrade** — n=10+ captures or tighter + fixture for `explicit_at_rc.p99`. Could fold into 21'd or be + its own short iter. +- **`FnDef::synthetic(...)` factor-out** — unchanged. +- **Boehm full retirement** — unchanged. +- **Deferred richer integration paths** (from 20f) — unchanged. +- **Family 21+** — typeclasses, polymorphic ADTs at runtime, + pattern-binding generalisation. Orchestrator-level fork.