Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

bench: 21'c — compile-time regression bench (check.py + compile_check.py)

Browse Source 
Closes the second axis the user named: every typechecker / codegen
perf change was previously invisible to the tidy-iter gate. With
Family 21 typeclasses (queued) and 21'b's poly-ADT additions both
pushing on the typechecker, naive substitution loops would have
landed silently and decayed the compile path.

bench/compile_check.py is a separate script from bench/check.py
because the methodology differs: sub-process spawn timing on small
workloads (1ms scale for `ail check`, 65ms for `ail build`) vs.
allocator-stress on large ones (multi-second). Tolerances differ
by an order of magnitude (25% / 20% here vs. 5-15% there).

Empirically: ail check is sub-ms across the corpus, dominated by
subprocess spawn (~5-10ms on Linux); ail build is 63-69ms,
dominated by clang's link step. The bench is a catastrophe
detector (10x slowdowns visible) — finer regressions need a
profiler. CLAUDE.md updated to list both scripts as co-equal
tidy-iter gates alongside the architect drift report; exit 0/1/2
semantics are uniform across both.

JOURNAL queue: 21'd (pure-compute fixtures) and 21'e (cross-
language reference / hand-C ratio) remain to land the LLVM-
linkable performance claim.
This commit is contained in:
Brummel
2026-05-09 00:59:00 +02:00
parent 07bff24527
commit 416d763b73
4 changed files with 450 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+19 -9
View File
@@ -183,9 +183,17 @@ sibling family is blocking, or the user has asked to defer).
### Performance regressions
`bench/check.py` is the second non-optional element of the
tidy-iter, alongside the architect drift report. Run it at every
family close. The exit code is the gate:
Two scripts gate the tidy-iter alongside the architect drift
report:
- **`bench/check.py`** — runtime regressions (gc/bump/rc throughput
and PTY tail-latency over the bench fixture corpus, baselined in
`bench/baseline.json`).
- **`bench/compile_check.py`** — compile-time regressions
(`ail check` and `ail build --opt=-O0` wall-time over a curated
example corpus, baselined in `bench/baseline_compile.json`).
Run both at every family close. The exit code is the gate:
- **Exit 0 (green).** All metrics within their per-metric
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
@@ -193,12 +201,14 @@ family close. The exit code is the gate:
tolerance. Treat the regression like an architect-drift item:
fix (revert the offending iter, refactor the hot path, diagnose
with `ailang-bencher` or `ailang-debugger`), or ratify
(`bench/check.py --update-baseline` together with a JOURNAL
entry naming the iter that intentionally moved the metric and
why).
- **Exit 2 (parser misalignment).** The bench-output format
changed under check.py's parsers. Update parsers or baseline
schema before re-running. Never claim a regression on exit-2.
(`--update-baseline` on the firing script together with a
JOURNAL entry naming the iter that intentionally moved the
metric and why).
- **Exit 2 (infrastructure failure).** Bench-output format
changed (`check.py` parser misalignment), or a corpus fixture
is missing / fails to spawn (`compile_check.py`). Fix the
infrastructure before re-running. Never claim a regression on
exit-2.
Skipping the bench-check at a family boundary requires the same
explicit JOURNAL entry as skipping the architect drift review:
bench/baseline_compile.json
+82
View File
@@ -0,0 +1,82 @@
{
"version": 1,
"captured": "2026-05-09",
"captured_via": "bench/compile_check.py",
"note": "Compile-time bench. `check_ms` is `ail check FILE`; `build_O0_ms` is `ail build --opt=-O0 FILE`. Wall-clock includes subprocess spawn (~5-10 ms on Linux) and, for build, the clang link step. Tolerances are tuned for noise on a quiet developer machine, not as the language correctness bar.",
"check_ms": {
"hello": {
"baseline_ms": 0.85,
"tolerance_pct": 25
},
"list_map_poly": {
"baseline_ms": 1.05,
"tolerance_pct": 25
},
"local_rec_capture": {
"baseline_ms": 0.9,
"tolerance_pct": 25
},
"borrow_own_demo": {
"baseline_ms": 1.01,
"tolerance_pct": 25
},
"nested_pat": {
"baseline_ms": 1.74,
"tolerance_pct": 25
},
"bench_list_sum": {
"baseline_ms": 0.91,
"tolerance_pct": 25
},
"bench_tree_walk": {
"baseline_ms": 0.9,
"tolerance_pct": 25
},
"bench_closure_chain": {
"baseline_ms": 0.88,
"tolerance_pct": 25
},
"bench_hof_pipeline": {
"baseline_ms": 0.99,
"tolerance_pct": 25
}
},
"build_O0_ms": {
"hello": {
"baseline_ms": 65.03,
"tolerance_pct": 20
},
"list_map_poly": {
"baseline_ms": 67.27,
"tolerance_pct": 20
},
"local_rec_capture": {
"baseline_ms": 65.35,
"tolerance_pct": 20
},
"borrow_own_demo": {
"baseline_ms": 64.26,
"tolerance_pct": 20
},
"nested_pat": {
"baseline_ms": 67.76,
"tolerance_pct": 20
},
"bench_list_sum": {
"baseline_ms": 63.6,
"tolerance_pct": 20
},
"bench_tree_walk": {
"baseline_ms": 65.77,
"tolerance_pct": 20
},
"bench_closure_chain": {
"baseline_ms": 68.98,
"tolerance_pct": 20
},
"bench_hof_pipeline": {
"baseline_ms": 66.64,
"tolerance_pct": 20
}
}
}
bench/compile_check.py
Executable
+234
View File
@@ -0,0 +1,234 @@
#!/usr/bin/env python3
# Compile-time regression check.
#
# Times `ail check FILE` and `ail build --opt=-O0 FILE` over a curated
# corpus, drops the slowest run, takes the median of the rest, diffs
# against bench/baseline_compile.json. Exits 0 if every metric is within
# tolerance, 1 on any regression, 2 on infrastructure failure (missing
# fixture, ail spawn error).
#
# Methodology caveats this bench cannot work around:
# - Wall time of `ail check` is dominated by subprocess spawn (~5-10ms
# on Linux); typechecker work below that scale is invisible.
# - Wall time of `ail build` is dominated by clang's link step
# (~100ms+); the AILang IR-emit + codegen contribution is a
# small slice of the total.
# Both are still useful as catastrophe detectors (10x slowdowns visible)
# even if subtler regressions need a profiler. The right tool for finer-
# grained codegen perf is the runtime bench (bench/run.sh), not this one.
#
# Usage:
# bench/compile_check.py # run bench, exit 0/1/2
# bench/compile_check.py -n 10 # more runs, tighter median
# bench/compile_check.py --update-baseline
# bench/compile_check.py --baseline path # alternate baseline file
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import tempfile
import time
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
DEFAULT_BASELINE = ROOT / "bench" / "baseline_compile.json"
AIL = ROOT / "target" / "release" / "ail"
EXAMPLES = ROOT / "examples"
# Curated corpus. Pinned by name so the baseline keys are stable across
# example-directory churn. Each entry must have a corresponding
# `<name>.ail.json` under examples/.
CORPUS = [
# Surface coverage: each fixture exercises a different feature set.
"hello", # IO baseline
"list_map_poly", # polymorphism + HOF
"local_rec_capture", # let-rec capture, closure path
"borrow_own_demo", # explicit modes
"nested_pat", # nested pattern matching
# Bench fixtures (correlation with bench/check.py runtime baseline).
"bench_list_sum",
"bench_tree_walk",
"bench_closure_chain",
"bench_hof_pipeline",
]
def time_one(args: list[str]) -> float:
"""Run a command, return wall-time in milliseconds. Raises if exit != 0."""
t0 = time.monotonic()
proc = subprocess.run(args, capture_output=True)
t1 = time.monotonic()
if proc.returncode != 0:
raise RuntimeError(
f"command failed (exit {proc.returncode}): {' '.join(args)}\n"
f"stderr: {proc.stderr.decode(errors='replace')}"
)
return (t1 - t0) * 1000.0
def median_drop_slowest(runs: list[float]) -> float:
"""Drop the slowest run, return median of the rest."""
if len(runs) < 2:
return runs[0] if runs else 0.0
kept = sorted(runs)[:-1]
n = len(kept)
if n % 2 == 1:
return kept[n // 2]
return 0.5 * (kept[n // 2 - 1] + kept[n // 2])
def measure(num_runs: int) -> dict[str, dict[str, float]]:
"""Return {section: {fixture: ms}}. Aborts on missing fixture / spawn fail."""
if not AIL.is_file():
print(f"missing release ail binary at {AIL}; building...", file=sys.stderr)
subprocess.run(
["cargo", "build", "--release", "-p", "ail"],
cwd=str(ROOT), check=True, capture_output=True,
)
out: dict[str, dict[str, float]] = {"check_ms": {}, "build_O0_ms": {}}
with tempfile.NamedTemporaryFile(prefix="bench_compile_", delete=False) as tf:
out_path = tf.name
try:
for fixture in CORPUS:
src = EXAMPLES / f"{fixture}.ail.json"
if not src.is_file():
print(f"missing fixture: {src}", file=sys.stderr)
sys.exit(2)
try:
check_runs = [time_one([str(AIL), "check", str(src)]) for _ in range(num_runs)]
build_runs = [time_one([str(AIL), "build", "--opt=-O0",
str(src), "-o", out_path]) for _ in range(num_runs)]
except RuntimeError as e:
print(f"spawn error on {fixture}: {e}", file=sys.stderr)
sys.exit(2)
out["check_ms"][fixture] = median_drop_slowest(check_runs)
out["build_O0_ms"][fixture] = median_drop_slowest(build_runs)
print(f" {fixture:<32} check={out['check_ms'][fixture]:6.1f}ms "
f"build={out['build_O0_ms'][fixture]:6.1f}ms", file=sys.stderr)
finally:
try:
Path(out_path).unlink()
except FileNotFoundError:
pass
return out
def diff_report(measured: dict, baseline: dict) -> tuple[str, bool]:
rows = []
has_regression = False
for section in ("check_ms", "build_O0_ms"):
for fixture, spec in baseline.get(section, {}).items():
actual = measured[section].get(fixture)
if actual is None:
rows.append((f"{section}.{fixture}", spec["baseline_ms"], None,
None, spec["tolerance_pct"], "MISSING"))
has_regression = True
continue
base = spec["baseline_ms"]
tol = spec["tolerance_pct"]
diff = 100.0 * (actual - base) / base if base else 0.0
if diff > tol:
status = "REGRESSION"
has_regression = True
elif diff < -tol:
status = "improvement"
else:
status = "ok"
rows.append((f"{section}.{fixture}", base, actual, diff, tol, status))
lines = []
lines.append(f"{'metric':<48} {'baseline':>10} {'actual':>10} {'diff':>9} {'tol':>6} status")
lines.append("-" * 100)
for metric, base, actual, diff, tol, status in rows:
if actual is None:
lines.append(f"{metric:<48} {base:>10.1f} {'-':>10} {'-':>9} {tol:>5.1f}% {status}")
else:
lines.append(
f"{metric:<48} {base:>10.1f} {actual:>10.1f} "
f"{diff:>+8.2f}% {tol:>5.1f}% {status}"
)
regressed = sum(1 for r in rows if r[5] == "REGRESSION")
improved = sum(1 for r in rows if r[5] == "improvement")
stable = len(rows) - regressed - improved
lines.append("")
lines.append(f"summary: {len(rows)} metrics; "
f"{regressed} regressed, {improved} improved beyond tolerance, "
f"{stable} stable")
return "\n".join(lines), has_regression
def write_baseline(measured: dict, baseline_path: Path) -> None:
today = subprocess.check_output(["date", "+%Y-%m-%d"], text=True).strip()
if baseline_path.exists():
existing = json.loads(baseline_path.read_text())
check_tols = {f: spec.get("tolerance_pct", 25)
for f, spec in existing.get("check_ms", {}).items()}
build_tols = {f: spec.get("tolerance_pct", 20)
for f, spec in existing.get("build_O0_ms", {}).items()}
else:
check_tols = {}
build_tols = {}
new = {
"version": 1,
"captured": today,
"captured_via": "bench/compile_check.py",
"note": "Compile-time bench. `check_ms` is `ail check FILE`; `build_O0_ms` is `ail build --opt=-O0 FILE`. Wall-clock includes subprocess spawn (~5-10 ms on Linux) and, for build, the clang link step. Tolerances are tuned for noise on a quiet developer machine, not as the language correctness bar.",
"check_ms": {},
"build_O0_ms": {},
}
for fixture, ms in measured["check_ms"].items():
new["check_ms"][fixture] = {
"baseline_ms": round(ms, 2),
"tolerance_pct": check_tols.get(fixture, 25),
}
for fixture, ms in measured["build_O0_ms"].items():
new["build_O0_ms"][fixture] = {
"baseline_ms": round(ms, 2),
"tolerance_pct": build_tols.get(fixture, 20),
}
baseline_path.write_text(json.dumps(new, indent=2) + "\n")
print(f">>> wrote new baseline to {baseline_path}", file=sys.stderr)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("-n", "--runs", type=int, default=5,
help="runs per fixture per op; min 2, default 5")
ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE)
ap.add_argument("--update-baseline", action="store_true",
help="re-measure, then overwrite baseline_compile.json")
args = ap.parse_args()
if args.runs < 2:
print("--runs must be >= 2", file=sys.stderr)
return 2
print(f">>> measuring {len(CORPUS)} fixtures, {args.runs} runs each "
f"(check + build_O0)", file=sys.stderr)
measured = measure(args.runs)
if args.update_baseline:
write_baseline(measured, args.baseline)
return 0
if not args.baseline.exists():
print(f"no baseline at {args.baseline}; create one with --update-baseline",
file=sys.stderr)
return 2
baseline = json.loads(args.baseline.read_text())
report, has_regression = diff_report(measured, baseline)
print(report)
return 1 if has_regression else 0
if __name__ == "__main__":
sys.exit(main())
docs/JOURNAL.md
+115
View File
@@ -10134,3 +10134,118 @@ fixture and baseline-file additions only.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork.
## 2026-05-09 — Iter 21'c: compile-time regression bench
Closes the second axis the user explicitly named — until this
iter, every typechecker / codegen perf change was invisible to the
tidy-iter gate. Family 21 typeclasses (queued) plus 21'b's poly-
ADT additions both push on the typechecker; without a tripwire,
naive substitution loops or O(n²) constraint resolution would
land silently and decay the whole compile path.
### What shipped
**`bench/compile_check.py`** — separate from `bench/check.py`
because the methodology is different (sub-process spawn timing
on small workloads vs. allocator-stress on large ones) and the
relevant tolerances differ by an order of magnitude. Two ops
per fixture: `ail check FILE` and `ail build --opt=-O0 FILE -o T`.
Same drop-slowest-of-N, median-of-rest convention as
`bench/run.sh`.
**`bench/baseline_compile.json`** — 18 metrics (9 fixtures × 2
ops). Curated corpus: 5 surface-coverage examples (`hello`,
`list_map_poly`, `local_rec_capture`, `borrow_own_demo`,
`nested_pat`) + 4 bench-throughput fixtures (correlation with
`bench/check.py`).
**Baselines on this machine**:
```
fixture | check(ms) | build(ms)
hello | 0.8 | 65.0
list_map_poly | 1.1 | 67.3
local_rec_capture | 0.9 | 65.3
borrow_own_demo | 1.0 | 64.3
nested_pat | 1.7 | 67.8
bench_list_sum | 0.9 | 63.6
bench_tree_walk | 0.9 | 65.8
bench_closure_chain | 0.9 | 69.0
bench_hof_pipeline | 1.0 | 66.6
```
### What the data tells us
`ail check` runs at **sub-millisecond per fixture** for everything
except `nested_pat` (1.7ms — its deeper pattern tree marginally
exceeds the noise floor). The typechecker is genuinely fast at
the current corpus scale; on this hardware the wall-clock is
dominated by subprocess spawn (~5-10ms on Linux), not by check
work. The bench detects catastrophes (10× slowdowns visible),
not subtler regressions — those want a profiler, not wall-clock.
`ail build --opt=-O0` runs at **63-69ms per fixture**, dominated
by clang's link step. The variance across fixtures is small —
~9% spread between fastest (`bench_list_sum` 63.6) and slowest
(`bench_closure_chain` 69.0). This is fine for catastrophe-
detection but not informative about codegen-quality differences.
For codegen-quality questions the runtime bench (rc/bump ratios)
remains the right tool.
### Tolerances
- **`check_ms`**: 25% per fixture. Justified empirically: a
re-run captured a +17.35% diff on `bench_hof_pipeline check`
with no code changes. Sub-millisecond timing is noisy.
- **`build_O0_ms`**: 20% per fixture. Build noise is materially
lower; observed re-run drift was ≤7% on every fixture.
These are catastrophe-detector tolerances. Tightening them
would mean false-positives on quiet-machine noise.
### CLAUDE.md update
The `Performance regressions` section now lists both
`bench/check.py` (runtime) and `bench/compile_check.py` (compile)
as co-equal tidy-iter gates alongside the architect drift report.
Exit 0 / 1 / 2 semantics are uniform across both scripts.
### What this iter does NOT do
- **No latency-harness methodology upgrade.** The wide
explicit_at_rc.p99 dispersion observed across today's three
captures (357.5 / 294.6 / 251.5) is a runtime-bench problem;
the compile bench is a different axis. Methodology upgrade
(n>=10 captures or tighter latency fixture) stays queued.
- **No O2 build bench.** `--opt=-O2` includes additional clang
passes that 2x-3x the build time. Useful for catching codegen
blowup-induced build slowdowns; not useful for detecting
AILang-side regressions, which are amply covered by the O0
pass. Future addition if/when warranted.
- **No incremental check bench.** Today every `ail check`
rebuilds the entire context. If incremental compilation is
added later (no current plan), re-baseline.
### Test state
288 / 0 / 3, unchanged. No Rust changes; the iter is bench-
infrastructure additions only.
### JOURNAL queue (updated)
- **21'd — pure-compute fixtures.** Mandelbrot / N-body / integer-
loop workloads. Heap-light, codegen-quality-heavy. Pairs
naturally with 21'e.
- **21'e — cross-language reference.** Hand-C variants of the
bench corpus, compiled with `clang -O2`. AILang/C ratio is the
honest answer to CLAUDE.md's "LLVM-linkable, performance is
extremely important" claim, which today is unbacked by data.
- **Latency methodology upgrade** — n=10+ captures or tighter
fixture for `explicit_at_rc.p99`. Could fold into 21'd or be
its own short iter.
- **`FnDef::synthetic(...)` factor-out** — unchanged.
- **Boehm full retirement** — unchanged.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork.