bench: 21'e — cross-language reference, AILang/C ratios
Closes the question CLAUDE.md has carried since day one ("LLVM-
linkable, performance is extremely important") with data. Hand-C
variants of the four bench fixtures, compiled with clang -O2,
each carefully matching the AILang algorithm and explicitly
documenting representation differences (cell width, leak policy)
that affect the ratio.
Three substantive findings:
1. Pure-compute parity with C: bench_compute_collatz runs at
AILang/C = 0.99x across both allocators. AILang's IR composes
with LLVM's optimizer at the same level a hand-C source does.
This is the LLVM-linkable performance claim, backed by data
for the first time. bench_compute_intsum (1.05-1.18x) confirms.
2. AILang bump beats glibc malloc 2x on linear allocation:
bench_list_sum.bump/c = 0.50x. Bump's two-instruction inline
fastpath outperforms glibc's free-list-managed malloc on
no-free workloads. Quantitatively measured for the first time.
3. RC overhead vs C malloc quantified: bench_list_sum.rc/c =
1.49x, bench_tree_walk.rc/c = 2.61x. The 8-byte refcount
header + zero-init + libc backing add 50-160% over glibc
malloc on these implicit-mode workloads. Explicit-mode + a
free()-adding C variant (21'f, queued) will close the
apples-to-apples gap on dec-cost.
CLAUDE.md updated to list bench/cross_lang.py as the third
tidy-iter gate alongside bench/check.py and bench/compile_check.py.
20 new metrics in bench/baseline_cross_lang.json with 12-15%
tolerances (cross-language ratios are inherently noisier than
within-AILang ratios — two compiler stacks contribute variance).
This commit is contained in:
@@ -183,7 +183,7 @@ sibling family is blocking, or the user has asked to defer).
|
|||||||
|
|
||||||
### Performance regressions
|
### Performance regressions
|
||||||
|
|
||||||
Two scripts gate the tidy-iter alongside the architect drift
|
Three scripts gate the tidy-iter alongside the architect drift
|
||||||
report:
|
report:
|
||||||
|
|
||||||
- **`bench/check.py`** — runtime regressions (gc/bump/rc throughput
|
- **`bench/check.py`** — runtime regressions (gc/bump/rc throughput
|
||||||
@@ -192,8 +192,14 @@ report:
|
|||||||
- **`bench/compile_check.py`** — compile-time regressions
|
- **`bench/compile_check.py`** — compile-time regressions
|
||||||
(`ail check` and `ail build --opt=-O0` wall-time over a curated
|
(`ail check` and `ail build --opt=-O0` wall-time over a curated
|
||||||
example corpus, baselined in `bench/baseline_compile.json`).
|
example corpus, baselined in `bench/baseline_compile.json`).
|
||||||
|
- **`bench/cross_lang.py`** — cross-language ratios (AILang at
|
||||||
|
`--alloc=rc` and `--alloc=bump` vs. hand-C reference compiled
|
||||||
|
with `clang -O2` over the same workloads, baselined in
|
||||||
|
`bench/baseline_cross_lang.json`). The headline answer to
|
||||||
|
"LLVM-linkable, performance is extremely important" — guards
|
||||||
|
against AILang/C ratios drifting upward over time.
|
||||||
|
|
||||||
Run both at every family close. The exit code is the gate:
|
Run all three at every family close. The exit code is the gate:
|
||||||
|
|
||||||
- **Exit 0 (green).** All metrics within their per-metric
|
- **Exit 0 (green).** All metrics within their per-metric
|
||||||
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
|
tolerance vs. `bench/baseline.json`. Tidy-iter can close.
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
{
|
||||||
|
"version": 1,
|
||||||
|
"captured": "2026-05-09",
|
||||||
|
"captured_via": "bench/cross_lang.py",
|
||||||
|
"note": "Cross-language wall-time baseline. Per-fixture: AILang at --alloc=rc, AILang at --alloc=bump, hand-C at clang -O2. Ratios rc/c and bump/c are the headline answer to CLAUDE.md's LLVM-linkable performance claim. The C reference uses malloc-and-leak to mirror AILang's implicit-mode RC; an explicit-mode + free() variant is queued.",
|
||||||
|
"fixtures": {
|
||||||
|
"bench_list_sum": {
|
||||||
|
"ail_rc_s": {
|
||||||
|
"baseline": 0.141597,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"ail_bump_s": {
|
||||||
|
"baseline": 0.048038,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"c_s": {
|
||||||
|
"baseline": 0.095312,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"rc_over_c": {
|
||||||
|
"baseline": 1.485614,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
},
|
||||||
|
"bump_over_c": {
|
||||||
|
"baseline": 0.504003,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"bench_tree_walk": {
|
||||||
|
"ail_rc_s": {
|
||||||
|
"baseline": 0.097028,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"ail_bump_s": {
|
||||||
|
"baseline": 0.038905,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"c_s": {
|
||||||
|
"baseline": 0.037159,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"rc_over_c": {
|
||||||
|
"baseline": 2.611185,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
},
|
||||||
|
"bump_over_c": {
|
||||||
|
"baseline": 1.046998,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"bench_compute_intsum": {
|
||||||
|
"ail_rc_s": {
|
||||||
|
"baseline": 0.00044,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"ail_bump_s": {
|
||||||
|
"baseline": 0.000393,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"c_s": {
|
||||||
|
"baseline": 0.000374,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"rc_over_c": {
|
||||||
|
"baseline": 1.177715,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
},
|
||||||
|
"bump_over_c": {
|
||||||
|
"baseline": 1.050856,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"bench_compute_collatz": {
|
||||||
|
"ail_rc_s": {
|
||||||
|
"baseline": 0.056878,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"ail_bump_s": {
|
||||||
|
"baseline": 0.056547,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"c_s": {
|
||||||
|
"baseline": 0.05748,
|
||||||
|
"tolerance_pct": 15
|
||||||
|
},
|
||||||
|
"rc_over_c": {
|
||||||
|
"baseline": 0.989523,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
},
|
||||||
|
"bump_over_c": {
|
||||||
|
"baseline": 0.983756,
|
||||||
|
"tolerance_pct": 12
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Executable
+241
@@ -0,0 +1,241 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# Cross-language reference bench.
|
||||||
|
#
|
||||||
|
# For each fixture in CORPUS, builds:
|
||||||
|
# - AILang binary at -O2 --alloc=rc (canonical default)
|
||||||
|
# - AILang binary at -O2 --alloc=bump (no-free upper bound)
|
||||||
|
# - hand-C binary via clang -O2 from bench/reference/<fixture>.c
|
||||||
|
# Times each, drops the slowest of N runs, takes the median, computes
|
||||||
|
# AILang/C ratios. Diffs against bench/baseline_cross_lang.json or, with
|
||||||
|
# --update-baseline, captures fresh numbers as the new baseline.
|
||||||
|
#
|
||||||
|
# What this answers (and what it does not): the AILang/C ratio is the
|
||||||
|
# honest answer to CLAUDE.md's "LLVM-linkable, performance is extremely
|
||||||
|
# important" claim. Ratios near 1.0 mean AILang's IR composes with
|
||||||
|
# LLVM's optimizer at the same level as a hand-C source; ratios of
|
||||||
|
# 2-3x are JIT-quality territory; ratios of 10x+ are a real problem.
|
||||||
|
# The bench DOES NOT distinguish "AILang's IR is slow" from "the
|
||||||
|
# discriminated-union representation is wider than C's hand-tuned
|
||||||
|
# struct"; that's a representation question, not an optimizer one.
|
||||||
|
# Each fixture's .c file documents its representation choices so the
|
||||||
|
# orchestrator can read the ratio with that context in mind.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bench/cross_lang.py # run + diff
|
||||||
|
# bench/cross_lang.py -n 10 # tighter median
|
||||||
|
# bench/cross_lang.py --update-baseline
|
||||||
|
# bench/cross_lang.py --baseline path
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import resource
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
DEFAULT_BASELINE = ROOT / "bench" / "baseline_cross_lang.json"
|
||||||
|
AIL = ROOT / "target" / "release" / "ail"
|
||||||
|
EXAMPLES = ROOT / "examples"
|
||||||
|
REFERENCE = ROOT / "bench" / "reference"
|
||||||
|
OUTDIR = ROOT / "target" / "cross_lang"
|
||||||
|
|
||||||
|
# (ailang fixture stem, C reference stem) — both must exist as
|
||||||
|
# `examples/<ail>.ail.json` and `bench/reference/<c>.c`.
|
||||||
|
CORPUS = [
|
||||||
|
("bench_list_sum", "list_sum"),
|
||||||
|
("bench_tree_walk", "tree_walk"),
|
||||||
|
("bench_compute_intsum", "compute_intsum"),
|
||||||
|
("bench_compute_collatz", "compute_collatz"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def time_one(bin_path: Path) -> float:
|
||||||
|
"""Run binary, return wall-time in seconds. Stdout discarded."""
|
||||||
|
t0 = time.monotonic()
|
||||||
|
proc = subprocess.run([str(bin_path)], stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL)
|
||||||
|
t1 = time.monotonic()
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise RuntimeError(f"{bin_path} exited {proc.returncode}")
|
||||||
|
return t1 - t0
|
||||||
|
|
||||||
|
|
||||||
|
def median_drop_slowest(runs: list[float]) -> float:
|
||||||
|
if len(runs) < 2:
|
||||||
|
return runs[0] if runs else 0.0
|
||||||
|
kept = sorted(runs)[:-1]
|
||||||
|
n = len(kept)
|
||||||
|
if n % 2 == 1:
|
||||||
|
return kept[n // 2]
|
||||||
|
return 0.5 * (kept[n // 2 - 1] + kept[n // 2])
|
||||||
|
|
||||||
|
|
||||||
|
def build_ailang(stem: str, alloc: str) -> Path:
|
||||||
|
src = EXAMPLES / f"{stem}.ail.json"
|
||||||
|
if not src.is_file():
|
||||||
|
print(f"missing AILang fixture: {src}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
bin_path = OUTDIR / f"{stem}_{alloc}"
|
||||||
|
subprocess.run(
|
||||||
|
[str(AIL), "build", "--opt=-O2", f"--alloc={alloc}", str(src), "-o", str(bin_path)],
|
||||||
|
check=True, capture_output=True,
|
||||||
|
)
|
||||||
|
return bin_path
|
||||||
|
|
||||||
|
|
||||||
|
def build_c(stem: str) -> Path:
|
||||||
|
src = REFERENCE / f"{stem}.c"
|
||||||
|
if not src.is_file():
|
||||||
|
print(f"missing C reference: {src}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
bin_path = OUTDIR / f"{stem}_c"
|
||||||
|
subprocess.run(["clang", "-O2", "-o", str(bin_path), str(src)],
|
||||||
|
check=True, capture_output=True)
|
||||||
|
return bin_path
|
||||||
|
|
||||||
|
|
||||||
|
def measure(num_runs: int) -> dict[str, dict[str, float]]:
|
||||||
|
if not AIL.is_file():
|
||||||
|
subprocess.run(["cargo", "build", "--release", "-p", "ail"],
|
||||||
|
cwd=str(ROOT), check=True, capture_output=True)
|
||||||
|
OUTDIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
out: dict[str, dict[str, float]] = {}
|
||||||
|
for ail_stem, c_stem in CORPUS:
|
||||||
|
print(f">>> {ail_stem}", file=sys.stderr)
|
||||||
|
ail_rc = build_ailang(ail_stem, "rc")
|
||||||
|
ail_bump = build_ailang(ail_stem, "bump")
|
||||||
|
c_bin = build_c(c_stem)
|
||||||
|
|
||||||
|
rc_runs = [time_one(ail_rc) for _ in range(num_runs)]
|
||||||
|
bump_runs = [time_one(ail_bump) for _ in range(num_runs)]
|
||||||
|
c_runs = [time_one(c_bin) for _ in range(num_runs)]
|
||||||
|
|
||||||
|
rc_s = median_drop_slowest(rc_runs)
|
||||||
|
bump_s = median_drop_slowest(bump_runs)
|
||||||
|
c_s = median_drop_slowest(c_runs)
|
||||||
|
|
||||||
|
out[ail_stem] = {
|
||||||
|
"ail_rc_s": rc_s,
|
||||||
|
"ail_bump_s": bump_s,
|
||||||
|
"c_s": c_s,
|
||||||
|
"rc_over_c": (rc_s / c_s) if c_s > 0 else 0.0,
|
||||||
|
"bump_over_c": (bump_s / c_s) if c_s > 0 else 0.0,
|
||||||
|
}
|
||||||
|
print(f" ail_rc={rc_s*1000:7.2f}ms ail_bump={bump_s*1000:7.2f}ms "
|
||||||
|
f"c={c_s*1000:7.2f}ms rc/c={out[ail_stem]['rc_over_c']:5.2f}× "
|
||||||
|
f"bump/c={out[ail_stem]['bump_over_c']:5.2f}×", file=sys.stderr)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def diff_report(measured: dict, baseline: dict) -> tuple[str, bool]:
|
||||||
|
rows = []
|
||||||
|
has_regression = False
|
||||||
|
for fixture, spec_dict in baseline.get("fixtures", {}).items():
|
||||||
|
actual = measured.get(fixture)
|
||||||
|
if actual is None:
|
||||||
|
continue
|
||||||
|
for metric, spec in spec_dict.items():
|
||||||
|
base = spec["baseline"]
|
||||||
|
tol = spec["tolerance_pct"]
|
||||||
|
a = actual.get(metric)
|
||||||
|
if a is None:
|
||||||
|
continue
|
||||||
|
diff = 100.0 * (a - base) / base if base else 0.0
|
||||||
|
if diff > tol:
|
||||||
|
status = "REGRESSION"
|
||||||
|
has_regression = True
|
||||||
|
elif diff < -tol:
|
||||||
|
status = "improvement"
|
||||||
|
else:
|
||||||
|
status = "ok"
|
||||||
|
rows.append((f"{fixture}.{metric}", base, a, diff, tol, status))
|
||||||
|
|
||||||
|
lines = []
|
||||||
|
lines.append(f"{'metric':<48} {'baseline':>10} {'actual':>10} {'diff':>9} {'tol':>6} status")
|
||||||
|
lines.append("-" * 100)
|
||||||
|
for m, b, a, d, t, s in rows:
|
||||||
|
lines.append(f"{m:<48} {b:>10.4f} {a:>10.4f} {d:>+8.2f}% {t:>5.1f}% {s}")
|
||||||
|
regressed = sum(1 for r in rows if r[5] == "REGRESSION")
|
||||||
|
improved = sum(1 for r in rows if r[5] == "improvement")
|
||||||
|
stable = len(rows) - regressed - improved
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"summary: {len(rows)} metrics; "
|
||||||
|
f"{regressed} regressed, {improved} improved beyond tolerance, "
|
||||||
|
f"{stable} stable")
|
||||||
|
return "\n".join(lines), has_regression
|
||||||
|
|
||||||
|
|
||||||
|
def write_baseline(measured: dict, path: Path) -> None:
|
||||||
|
today = subprocess.check_output(["date", "+%Y-%m-%d"], text=True).strip()
|
||||||
|
if path.exists():
|
||||||
|
existing = json.loads(path.read_text())
|
||||||
|
existing_tols = existing.get("fixtures", {})
|
||||||
|
else:
|
||||||
|
existing_tols = {}
|
||||||
|
|
||||||
|
DEFAULT_TOLS = {
|
||||||
|
"ail_rc_s": 15,
|
||||||
|
"ail_bump_s": 15,
|
||||||
|
"c_s": 15,
|
||||||
|
"rc_over_c": 12,
|
||||||
|
"bump_over_c": 12,
|
||||||
|
}
|
||||||
|
|
||||||
|
new = {
|
||||||
|
"version": 1,
|
||||||
|
"captured": today,
|
||||||
|
"captured_via": "bench/cross_lang.py",
|
||||||
|
"note": "Cross-language wall-time baseline. Per-fixture: AILang at --alloc=rc, AILang at --alloc=bump, hand-C at clang -O2. Ratios rc/c and bump/c are the headline answer to CLAUDE.md's LLVM-linkable performance claim. The C reference uses malloc-and-leak to mirror AILang's implicit-mode RC; an explicit-mode + free() variant is queued.",
|
||||||
|
"fixtures": {},
|
||||||
|
}
|
||||||
|
for fixture, metrics in measured.items():
|
||||||
|
existing_fix = existing_tols.get(fixture, {})
|
||||||
|
new["fixtures"][fixture] = {
|
||||||
|
metric: {
|
||||||
|
"baseline": round(value, 6),
|
||||||
|
"tolerance_pct": existing_fix.get(metric, {}).get(
|
||||||
|
"tolerance_pct", DEFAULT_TOLS.get(metric, 15)
|
||||||
|
),
|
||||||
|
}
|
||||||
|
for metric, value in metrics.items()
|
||||||
|
}
|
||||||
|
path.write_text(json.dumps(new, indent=2) + "\n")
|
||||||
|
print(f">>> wrote new baseline to {path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__)
|
||||||
|
ap.add_argument("-n", "--runs", type=int, default=5,
|
||||||
|
help="runs per binary; min 2, default 5")
|
||||||
|
ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE)
|
||||||
|
ap.add_argument("--update-baseline", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
if args.runs < 2:
|
||||||
|
print("--runs must be >= 2", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
measured = measure(args.runs)
|
||||||
|
|
||||||
|
if args.update_baseline:
|
||||||
|
write_baseline(measured, args.baseline)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if not args.baseline.exists():
|
||||||
|
print(f"no baseline at {args.baseline}; create one with --update-baseline",
|
||||||
|
file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
baseline = json.loads(args.baseline.read_text())
|
||||||
|
report, has_regression = diff_report(measured, baseline)
|
||||||
|
print(report)
|
||||||
|
return 1 if has_regression else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
// Hand-C reference for bench_compute_collatz.
|
||||||
|
//
|
||||||
|
// Same algorithm as examples/bench_compute_collatz.ailx — for each
|
||||||
|
// starting value in [1..N], count Collatz steps to reach 1, sum.
|
||||||
|
// Three sizes: 10k / 100k / 500k starting values.
|
||||||
|
//
|
||||||
|
// Data-dependent control flow (n % 2 branch) prevents LLVM from
|
||||||
|
// reducing this to closed form. The AILang/C wall-time ratio
|
||||||
|
// directly reflects integer-arithmetic + branch-prediction codegen
|
||||||
|
// quality.
|
||||||
|
//
|
||||||
|
// Build: clang -O2 -o compute_collatz compute_collatz.c
|
||||||
|
// Expected stdout (one int per line):
|
||||||
|
// 849666
|
||||||
|
// 10753840
|
||||||
|
// 62134795
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
|
||||||
|
static long collatz_steps(long n) {
|
||||||
|
long steps = 0;
|
||||||
|
while (n != 1) {
|
||||||
|
if ((n % 2) == 0) {
|
||||||
|
n = n / 2;
|
||||||
|
} else {
|
||||||
|
n = n * 3 + 1;
|
||||||
|
}
|
||||||
|
steps += 1;
|
||||||
|
}
|
||||||
|
return steps;
|
||||||
|
}
|
||||||
|
|
||||||
|
static long sum_steps(long n) {
|
||||||
|
long total = 0;
|
||||||
|
for (long i = n; i > 0; i--) {
|
||||||
|
total += collatz_steps(i);
|
||||||
|
}
|
||||||
|
return total;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void run_one(long n) {
|
||||||
|
printf("%ld\n", sum_steps(n));
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
run_one(10000);
|
||||||
|
run_one(100000);
|
||||||
|
run_one(500000);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
// Hand-C reference for bench_compute_intsum.
|
||||||
|
//
|
||||||
|
// Same algorithm as examples/bench_compute_intsum.ailx — accumulate
|
||||||
|
// `i * 7` for i in [n, n-1, ..., 1], printing the final acc.
|
||||||
|
// Three sizes: 1M / 10M / 50M iterations.
|
||||||
|
//
|
||||||
|
// Just like AILang's version under -O2, this loop is closed-form
|
||||||
|
// reducible (sum_{i=1..N} i*7 = 7*N*(N+1)/2). clang -O2 will likely
|
||||||
|
// fold it. The AILang/C wall-time ratio at this fixture answers
|
||||||
|
// "does AILang's IR enable the same constant fold C's source does"
|
||||||
|
// — both should be startup-dominated.
|
||||||
|
//
|
||||||
|
// Build: clang -O2 -o compute_intsum compute_intsum.c
|
||||||
|
// Expected stdout (one int per line):
|
||||||
|
// 3500003500000
|
||||||
|
// 350000035000000
|
||||||
|
// 8750000175000000
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
|
||||||
|
static long intsum_loop(long n) {
|
||||||
|
long acc = 0;
|
||||||
|
while (n > 0) {
|
||||||
|
acc += n * 7;
|
||||||
|
n -= 1;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void run_one(long n) {
|
||||||
|
printf("%ld\n", intsum_loop(n));
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
run_one(1000000);
|
||||||
|
run_one(10000000);
|
||||||
|
run_one(50000000);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
// Hand-C reference for bench_list_sum.
|
||||||
|
//
|
||||||
|
// Same algorithm as examples/bench_list_sum.ailx — build a linked list
|
||||||
|
// of [0, 1, ..., N-1] via prepending, then sum by linear traversal.
|
||||||
|
// Three workload sizes: 100k / 1M / 3M cells, matching the AILang
|
||||||
|
// fixture exactly so the AILang/C wall-time ratio is fair.
|
||||||
|
//
|
||||||
|
// Cell layout: { long head; struct cell *tail; } — 16 bytes (8 head
|
||||||
|
// + 8 pointer). AILang's IntList ICons cell is wider (tag + payload +
|
||||||
|
// tail = 24 bytes) because the runtime carries a constructor tag for
|
||||||
|
// the discriminated-union. The 1.5x size difference is one of the
|
||||||
|
// real costs of the discriminated-union representation; quoting the
|
||||||
|
// raw ratio without naming this is the wrong comparison.
|
||||||
|
//
|
||||||
|
// Memory policy: this reference uses malloc and DELIBERATELY DOES
|
||||||
|
// NOT FREE. That matches AILang's bench_list_sum running under
|
||||||
|
// --alloc=rc with implicit-mode params (cells leak by design — the
|
||||||
|
// 18c.3 known debt). The fair comparison is therefore:
|
||||||
|
// AILang --alloc=rc (implicit-mode, leaks) vs. this C (leaks)
|
||||||
|
// A future iter that ships explicit-mode bench_list_sum + a free()-
|
||||||
|
// adding C variant would close the apples-to-apples gap on the
|
||||||
|
// dec-cost axis.
|
||||||
|
//
|
||||||
|
// Build: clang -O2 -o list_sum list_sum.c
|
||||||
|
// Expected stdout (one int per line):
|
||||||
|
// 4999950000
|
||||||
|
// 499999500000
|
||||||
|
// 4499998500000
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
|
||||||
|
typedef struct cell {
|
||||||
|
long head;
|
||||||
|
struct cell *tail;
|
||||||
|
} cell_t;
|
||||||
|
|
||||||
|
static cell_t *cons_n(long n) {
|
||||||
|
cell_t *acc = NULL;
|
||||||
|
for (long i = n - 1; i >= 0; i--) {
|
||||||
|
cell_t *c = (cell_t *) malloc(sizeof(cell_t));
|
||||||
|
c->head = i;
|
||||||
|
c->tail = acc;
|
||||||
|
acc = c;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static long sum_list(const cell_t *xs) {
|
||||||
|
long acc = 0;
|
||||||
|
while (xs) {
|
||||||
|
acc += xs->head;
|
||||||
|
xs = xs->tail;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void run_one(long n) {
|
||||||
|
const cell_t *xs = cons_n(n);
|
||||||
|
printf("%ld\n", sum_list(xs));
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
run_one(100000);
|
||||||
|
run_one(1000000);
|
||||||
|
run_one(3000000);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
// Hand-C reference for bench_tree_walk.
|
||||||
|
//
|
||||||
|
// Same algorithm as examples/bench_tree_walk.ailx — build a balanced
|
||||||
|
// binary tree of given depth (every value = 1) and sum every node.
|
||||||
|
// Three depths: 16 / 18 / 20 (= 65535 / 262143 / 1048575 nodes).
|
||||||
|
//
|
||||||
|
// Cell layout: { long value; struct node *left; struct node *right; }
|
||||||
|
// — 24 bytes. AILang's Tree Node cell is 32 bytes (tag + value + l
|
||||||
|
// + r) due to the discriminated-union tag. The Leaf variant is also
|
||||||
|
// boxed in AILang (tag-only, ~8 bytes). For C, we use NULL pointers
|
||||||
|
// for leaves (no allocation), which is a representation choice that
|
||||||
|
// favors C; a fair-er comparison would tag leaves explicitly.
|
||||||
|
//
|
||||||
|
// Memory policy: malloc, DELIBERATELY no free. Matches AILang's
|
||||||
|
// bench_tree_walk under --alloc=rc (implicit-mode, leaks).
|
||||||
|
//
|
||||||
|
// Build: clang -O2 -o tree_walk tree_walk.c
|
||||||
|
// Expected stdout (one int per line):
|
||||||
|
// 65535
|
||||||
|
// 262143
|
||||||
|
// 1048575
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
|
||||||
|
typedef struct node {
|
||||||
|
long value;
|
||||||
|
struct node *left;
|
||||||
|
struct node *right;
|
||||||
|
} node_t;
|
||||||
|
|
||||||
|
static node_t *build_tree(long depth) {
|
||||||
|
if (depth == 0) return NULL;
|
||||||
|
node_t *n = (node_t *) malloc(sizeof(node_t));
|
||||||
|
n->value = 1;
|
||||||
|
n->left = build_tree(depth - 1);
|
||||||
|
n->right = build_tree(depth - 1);
|
||||||
|
return n;
|
||||||
|
}
|
||||||
|
|
||||||
|
static long sum_tree(const node_t *t) {
|
||||||
|
if (t == NULL) return 0;
|
||||||
|
return t->value + sum_tree(t->left) + sum_tree(t->right);
|
||||||
|
}
|
||||||
|
|
||||||
|
static void run_one(long depth) {
|
||||||
|
const node_t *t = build_tree(depth);
|
||||||
|
printf("%ld\n", sum_tree(t));
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
run_one(16);
|
||||||
|
run_one(18);
|
||||||
|
run_one(20);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
+120
@@ -10135,6 +10135,126 @@ fixture and baseline-file additions only.
|
|||||||
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||||
pattern-binding generalisation. Orchestrator-level fork.
|
pattern-binding generalisation. Orchestrator-level fork.
|
||||||
|
|
||||||
|
## 2026-05-09 — Iter 21'e: cross-language reference + AILang/C ratios
|
||||||
|
|
||||||
|
Closes the question CLAUDE.md has carried since day one — *"the
|
||||||
|
language must, in the end, be linkable to LLVM. Performance is
|
||||||
|
extremely important."* — by adding hand-C variants of the bench
|
||||||
|
corpus, building both with `clang -O2`, and comparing wall times
|
||||||
|
directly. Until this iter, every performance number AILang shipped
|
||||||
|
was internal (gc vs. bump vs. rc); none of them said anything
|
||||||
|
about absolute competitiveness.
|
||||||
|
|
||||||
|
### Hand-C corpus
|
||||||
|
|
||||||
|
`bench/reference/` — four C sources, one per fixture, each
|
||||||
|
carefully matching the AILang algorithm and explicitly
|
||||||
|
documenting representation choices (cell width, leak policy)
|
||||||
|
that affect the ratio:
|
||||||
|
|
||||||
|
- **`list_sum.c`** — linked list, malloc-and-leak (matches
|
||||||
|
AILang implicit-mode RC). 16-byte cell vs. AILang's 24-byte
|
||||||
|
ICons (tag overhead).
|
||||||
|
- **`tree_walk.c`** — balanced tree, malloc-and-leak. 24-byte
|
||||||
|
cell vs. AILang's 32-byte Tree::Node. NULL leaves (no alloc)
|
||||||
|
vs. AILang's tag-only Leaf cells.
|
||||||
|
- **`compute_intsum.c`** — pure-compute affine recurrence.
|
||||||
|
- **`compute_collatz.c`** — pure-compute, data-dependent control
|
||||||
|
flow.
|
||||||
|
|
||||||
|
### Headline numbers (5-run, drop-slowest, median of 4)
|
||||||
|
|
||||||
|
```
|
||||||
|
fixture | AILang_rc | AILang_bump | C | rc/c | bump/c
|
||||||
|
-----------------------+-----------+-------------+--------+-------+-------
|
||||||
|
bench_list_sum | 141.6 ms | 48.0 ms | 95.3 ms| 1.49× | 0.50×
|
||||||
|
bench_tree_walk | 97.0 ms | 38.9 ms | 37.2 ms| 2.61× | 1.05×
|
||||||
|
bench_compute_intsum | 0.4 ms | 0.4 ms | 0.4 ms| 1.18× | 1.05×
|
||||||
|
bench_compute_collatz | 56.9 ms | 56.6 ms | 57.5 ms| 0.99× | 0.98×
|
||||||
|
```
|
||||||
|
|
||||||
|
### Three substantive findings
|
||||||
|
|
||||||
|
**1. Pure-compute parity with C is real.** `bench_compute_collatz`
|
||||||
|
runs at AILang/C = 0.98–0.99× across both allocators. Same
|
||||||
|
algorithm, same `clang -O2`, same wall time. The IR AILang's
|
||||||
|
codegen emits composes with LLVM's optimizer at the same level
|
||||||
|
a hand-written C source does — both tail-recursive iteration,
|
||||||
|
both data-dependent branch prediction. This is the canonical
|
||||||
|
"LLVM-linkable, performance is extremely important" claim,
|
||||||
|
backed by data for the first time. `bench_compute_intsum` (1.05–
|
||||||
|
1.18×) confirms the pattern; both fixtures get LLVM-folded /
|
||||||
|
optimized symmetrically.
|
||||||
|
|
||||||
|
**2. AILang bump beats glibc malloc on linear allocation.**
|
||||||
|
`bench_list_sum.bump/c = 0.50×` — AILang's `bump_malloc`
|
||||||
|
(`ptr += size; return old`) is twice as fast as glibc's
|
||||||
|
`malloc()` on dense Cons-cell allocation. Expected
|
||||||
|
qualitatively (bump is 2 instructions inline; glibc malloc has
|
||||||
|
free-list management even on the alloc path), but the
|
||||||
|
quantitative result is the first time it's been measured. No-
|
||||||
|
free workloads (bench fixtures) are exactly where bump shines;
|
||||||
|
production workloads that actually free are a separate story.
|
||||||
|
|
||||||
|
**3. RC overhead vs C malloc is now quantified.** Linear:
|
||||||
|
`bench_list_sum.rc/c = 1.49×` — RC's per-call cost (8-byte
|
||||||
|
refcount header + zero-init + libc malloc backing) is ~50%
|
||||||
|
above glibc malloc on this workload. Tree: 2.61×, larger
|
||||||
|
because the per-node fixed cost amortizes over a smaller
|
||||||
|
working set. These ratios are *implicit-mode* RC (no dec-tax);
|
||||||
|
explicit-mode would add the dec-cost on top, but the hand-C
|
||||||
|
reference also has no free, so the apples-to-apples comparison
|
||||||
|
needs an explicit-mode AILang fixture + a free()-adding C
|
||||||
|
variant to be honest about both sides. Queued.
|
||||||
|
|
||||||
|
### 20 new baselined metrics
|
||||||
|
|
||||||
|
`bench/baseline_cross_lang.json` — 4 fixtures × 5 metrics
|
||||||
|
(ail_rc_s, ail_bump_s, c_s, rc_over_c, bump_over_c). Tolerances
|
||||||
|
12–15% across the board: cross-language ratios are inherently
|
||||||
|
less stable than within-AILang ratios because two compiler
|
||||||
|
stacks contribute noise.
|
||||||
|
|
||||||
|
### CLAUDE.md update
|
||||||
|
|
||||||
|
`Performance regressions` now lists three tidy-iter gates:
|
||||||
|
`bench/check.py`, `bench/compile_check.py`, and
|
||||||
|
`bench/cross_lang.py`. The cross-lang script is the
|
||||||
|
heaviest of the three (12 binary builds + 60 timed runs at
|
||||||
|
n=5), but it's the only mechanism that catches AILang/C ratios
|
||||||
|
drifting upward over time. Worth the seconds.
|
||||||
|
|
||||||
|
### What this iter does NOT do
|
||||||
|
|
||||||
|
- **No explicit-mode bench fixture pair.** `bench_list_sum`
|
||||||
|
and `bench_tree_walk` are implicit-mode-only; the C reference
|
||||||
|
is malloc-and-leak. To honestly compare RC's full alloc+dec
|
||||||
|
cost vs. C's full malloc+free cost would need a paired
|
||||||
|
explicit-mode AILang fixture + a `free()`-adding C variant.
|
||||||
|
Queued as 21'f.
|
||||||
|
- **No multi-platform reference.** Single x86-64 Linux
|
||||||
|
measurement. Cross-platform ratios may differ; not in scope.
|
||||||
|
- **No JIT comparison.** `clang -O2` AOT, AILang AOT — apples
|
||||||
|
to apples. JIT (LuaJIT, V8, etc.) is a different question.
|
||||||
|
|
||||||
|
### Test state
|
||||||
|
|
||||||
|
288 / 0 / 3, unchanged.
|
||||||
|
|
||||||
|
### JOURNAL queue (updated)
|
||||||
|
|
||||||
|
- **21'f — explicit-mode cross-lang pair.** Add `bench_list_sum_
|
||||||
|
explicit.ailx` (with `(borrow)` / `(own)` / `(drop-iterative)`)
|
||||||
|
and `list_sum_explicit_free.c` (matching `free()` calls).
|
||||||
|
Re-run cross_lang, capture rc-with-dec / c-with-free ratios.
|
||||||
|
Closes the apples-to-apples gap on the dec-cost axis.
|
||||||
|
- **`FnDef::synthetic(...)` factor-out** — unchanged.
|
||||||
|
- **Boehm full retirement** — unchanged.
|
||||||
|
- **Latency methodology upgrade** (n=10+ captures) — unchanged.
|
||||||
|
- **Deferred richer integration paths** (from 20f) — unchanged.
|
||||||
|
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||||
|
pattern-binding generalisation. Orchestrator-level fork.
|
||||||
|
|
||||||
## 2026-05-09 — Iter 21'd: pure-compute fixtures + harness hardening
|
## 2026-05-09 — Iter 21'd: pure-compute fixtures + harness hardening
|
||||||
|
|
||||||
Closes a third bench-corpus blind spot: every fixture so far has
|
Closes a third bench-corpus blind spot: every fixture so far has
|
||||||
|
|||||||
Reference in New Issue
Block a user