Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

bench: regression harness — baseline.json + check.py

Browse Source 
Closes the structural gap between bench/run.sh (one-shot capture
into JOURNAL) and a continuous tripwire. baseline.json records 31
metrics (16 throughput, 15 latency) with per-metric one-sided
tolerances tuned to absorb run-to-run noise on a quiet developer
machine; check.py spawns run.sh, parses both the throughput
pipe-table and the per-arm latency stanzas, diffs against the
baseline, prints a per-metric report, and exits non-zero on any
regression beyond tolerance.

User-facing flags: --from-file, --stdin, --baseline, and
--update-baseline (re-run + overwrite baseline.json after
intentional improvements).

Validation: captured the baseline, then re-ran end-to-end. All 31
metrics within tolerance. The harness also caught a single-capture
explicit-rc p99 spike (357.5 us) that was first read as drift vs.
yesterday's 18g.tidy.fu2 numbers but came in at 294.6 us on the
follow-up run — exactly the kind of noise the tolerance band is
there to absorb. Without 21'a we would have either chased a
phantom or buried the signal; with it, single noisy runs are data
points, not verdicts.

Tidy-iter discipline addition (run check.py at every family close
alongside the architect drift report) is recommended in JOURNAL
but not enacted in this iter — that's an orchestrator-level
update to CLAUDE.md.
This commit is contained in:
Brummel
2026-05-09 00:36:06 +02:00
parent 7e5c95f6c6
commit a1b0ad5723
3 changed files with 499 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bench/baseline.json
+53
View File
@@ -0,0 +1,53 @@
{
"version": 1,
"captured": "2026-05-09",
"captured_via": "bench/run.sh -n 5",
"note": "Baseline for bench/check.py regression detection. Decision-10 thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are LANGUAGE invariants, not regression-check tolerances. The tolerances below are tuned to absorb run-to-run noise on a quiet developer machine; they are NOT the correctness bar. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the JOURNAL entry that ships the baseline bump.",
"throughput": {
"bench_list_sum": {
"gc_s": { "baseline": 0.137, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.046, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.133, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.98, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 2.89, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 103980, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 97696, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 193448, "tolerance_pct": 5 }
},
"bench_tree_walk": {
"gc_s": { "baseline": 0.103, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.038, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.095, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.71, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 2.50, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 73260, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 55208, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 108956, "tolerance_pct": 5 }
}
},
"latency": {
"implicit_at_gc": {
"median_us": { "baseline": 96.4, "tolerance_pct": 15 },
"p99_us": { "baseline": 7130.0, "tolerance_pct": 20 },
"p99_9_us": { "baseline": 8131.2, "tolerance_pct": 25 },
"max_us": { "baseline": 8343.7, "tolerance_pct": 25 },
"p99_over_median": { "baseline": 73.92, "tolerance_pct": 20 }
},
"explicit_at_rc": {
"median_us": { "baseline": 213.9, "tolerance_pct": 15 },
"p99_us": { "baseline": 357.5, "tolerance_pct": 25 },
"p99_9_us": { "baseline": 404.1, "tolerance_pct": 25 },
"max_us": { "baseline": 413.0, "tolerance_pct": 25 },
"p99_over_median": { "baseline": 1.66, "tolerance_pct": 25 }
},
"implicit_at_rc": {
"median_us": { "baseline": 285.7, "tolerance_pct": 15 },
"p99_us": { "baseline": 407.1, "tolerance_pct": 20 },
"p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },
"max_us": { "baseline": 477.3, "tolerance_pct": 25 },
"p99_over_median": { "baseline": 1.43, "tolerance_pct": 20 }
}
}
}
bench/check.py
Executable
+322
View File
@@ -0,0 +1,322 @@
#!/usr/bin/env python3
# Performance regression check.
#
# Runs bench/run.sh (or consumes its output from stdin / a file), parses
# the throughput table and the per-arm latency stanzas, and diffs every
# metric against bench/baseline.json. Exits 0 if every metric is within
# its per-metric tolerance; exits 1 with a per-metric diff table on any
# regression.
#
# Tolerances are one-sided: only metrics that got *worse* than baseline
# (slower wall-time, larger RSS, larger latency, larger ratio) trigger a
# regression. Improvements never fire — they show in the diff with a "+"
# sign and a note, so an intentional improvement can be ratified into
# the baseline with `--update-baseline`.
#
# Usage:
# bench/check.py # spawn bench/run.sh -n 5
# bench/check.py --from-file out.txt # consume captured output
# cat out.txt | bench/check.py --stdin
# bench/check.py --update-baseline # re-run, write new baseline
# bench/check.py --baseline path.json # alternate baseline location
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
DEFAULT_BASELINE = ROOT / "bench" / "baseline.json"
RUN_SH = ROOT / "bench" / "run.sh"
@dataclass
class Measurement:
metric_id: str
baseline: float
tolerance_pct: float
actual: float
@property
def diff_pct(self) -> float:
if self.baseline == 0:
return 0.0
return 100.0 * (self.actual - self.baseline) / self.baseline
@property
def regressed(self) -> bool:
return self.diff_pct > self.tolerance_pct
def parse_throughput_table(text: str) -> dict[str, dict[str, float]]:
# The throughput table has a header row, a separator row, then one row
# per workload. Columns are pipe-separated.
out: dict[str, dict[str, float]] = {}
in_table = False
for line in text.splitlines():
if line.startswith("workload") and "gc(s)" in line:
in_table = True
continue
if in_table and line.startswith("---"):
continue
if in_table:
if not line.strip() or "|" not in line:
in_table = False
continue
cells = [c.strip() for c in line.split("|")]
if len(cells) != 9:
continue
workload = cells[0]
try:
out[workload] = {
"gc_s": float(cells[1]),
"bump_s": float(cells[2]),
"rc_s": float(cells[3]),
"gc_over_bump": float(cells[4].rstrip("x")),
"rc_over_bump": float(cells[5].rstrip("x")),
"gc_rss_kb": float(cells[6]),
"bump_rss_kb": float(cells[7]),
"rc_rss_kb": float(cells[8]),
}
except ValueError:
continue
return out
# Latency arm header looks like: "=== implicit @ gc (Boehm-fair) ===".
# We map the leading prose label to the canonical arm key in baseline.json.
ARM_LABEL_TO_KEY = {
"implicit @ gc": "implicit_at_gc",
"explicit @ rc": "explicit_at_rc",
"implicit @ rc": "implicit_at_rc",
}
# Inside each arm stanza, lines that report a metric look like:
# " median median= 96.4 range=[..., ...] us"
# or:
# " p99/median median=73.92x range=[...]"
# We pick out the metric name and its "median=NN" value.
LINE_RE = re.compile(
r"^\s+(?P<metric>[a-zA-Z0-9./_]+)\s+median=\s*(?P<value>[\d.]+)x?"
)
HEADER_RE = re.compile(r"^=== (?P<label>[^(]+?)\s*\(")
LATENCY_METRIC_RENAMES = {
"median": "median_us",
"p99": "p99_us",
"p99.9": "p99_9_us",
"max": "max_us",
"p99/median": "p99_over_median",
}
def parse_latency_stanzas(text: str) -> dict[str, dict[str, float]]:
out: dict[str, dict[str, float]] = {}
current_key: str | None = None
current: dict[str, float] = {}
def flush() -> None:
if current_key is not None and current:
out.setdefault(current_key, {}).update(current)
for line in text.splitlines():
m = HEADER_RE.match(line)
if m:
flush()
label = m.group("label").strip()
current_key = ARM_LABEL_TO_KEY.get(label)
current = {}
continue
if current_key is None:
continue
m = LINE_RE.match(line)
if not m:
continue
raw_name = m.group("metric")
canonical = LATENCY_METRIC_RENAMES.get(raw_name)
if canonical is None:
continue
try:
current[canonical] = float(m.group("value"))
except ValueError:
continue
flush()
return out
def run_bench() -> str:
cmd = [str(RUN_SH), "-n", "5"]
print(f">>> running: {' '.join(cmd)}", file=sys.stderr)
proc = subprocess.run(
cmd,
cwd=str(ROOT),
capture_output=True,
text=True,
check=False,
)
sys.stderr.write(proc.stderr)
if proc.returncode != 0:
print(f"bench/run.sh exited with code {proc.returncode}", file=sys.stderr)
sys.exit(2)
return proc.stdout
def collect_measurements(
parsed_throughput: dict[str, dict[str, float]],
parsed_latency: dict[str, dict[str, float]],
baseline: dict,
) -> list[Measurement]:
out: list[Measurement] = []
missing: list[str] = []
for workload, metrics in baseline.get("throughput", {}).items():
actuals = parsed_throughput.get(workload)
if actuals is None:
missing.append(f"throughput.{workload}")
continue
for name, spec in metrics.items():
actual = actuals.get(name)
if actual is None:
missing.append(f"throughput.{workload}.{name}")
continue
out.append(Measurement(
metric_id=f"throughput.{workload}.{name}",
baseline=spec["baseline"],
tolerance_pct=spec["tolerance_pct"],
actual=actual,
))
for arm, metrics in baseline.get("latency", {}).items():
actuals = parsed_latency.get(arm)
if actuals is None:
missing.append(f"latency.{arm}")
continue
for name, spec in metrics.items():
actual = actuals.get(name)
if actual is None:
missing.append(f"latency.{arm}.{name}")
continue
out.append(Measurement(
metric_id=f"latency.{arm}.{name}",
baseline=spec["baseline"],
tolerance_pct=spec["tolerance_pct"],
actual=actual,
))
if missing:
print("error: parser did not find these baselined metrics in the bench output:",
file=sys.stderr)
for m in missing:
print(f" {m}", file=sys.stderr)
print("Likely cause: bench/run.sh output format changed, or a fixture / latency arm was renamed. Update bench/check.py parsers or bench/baseline.json before claiming a regression.",
file=sys.stderr)
sys.exit(2)
return out
def render_report(measurements: list[Measurement]) -> tuple[str, bool]:
regressed = [m for m in measurements if m.regressed]
improved = [m for m in measurements if m.diff_pct < -m.tolerance_pct]
lines = []
lines.append(f"{'metric':<48} {'baseline':>12} {'actual':>12} {'diff':>9} {'tol':>6} status")
lines.append("-" * 100)
for m in measurements:
if m.regressed:
status = "REGRESSION"
elif m.diff_pct < -m.tolerance_pct:
status = "improvement"
else:
status = "ok"
lines.append(
f"{m.metric_id:<48} {m.baseline:>12.3f} {m.actual:>12.3f} "
f"{m.diff_pct:>+8.2f}% {m.tolerance_pct:>5.1f}% {status}"
)
lines.append("")
lines.append(f"summary: {len(measurements)} metrics; "
f"{len(regressed)} regressed, "
f"{len(improved)} improved beyond tolerance, "
f"{len(measurements) - len(regressed) - len(improved)} stable")
return "\n".join(lines), bool(regressed)
def write_new_baseline(
parsed_throughput: dict[str, dict[str, float]],
parsed_latency: dict[str, dict[str, float]],
baseline_path: Path,
captured_via: str,
) -> None:
today = subprocess.check_output(["date", "+%Y-%m-%d"], text=True).strip()
existing = json.loads(baseline_path.read_text())
new = {
"version": existing.get("version", 1),
"captured": today,
"captured_via": captured_via,
"note": existing.get("note", ""),
"throughput": {},
"latency": {},
}
for workload, metrics in existing.get("throughput", {}).items():
actuals = parsed_throughput.get(workload, {})
new["throughput"][workload] = {
name: {
"baseline": actuals.get(name, spec["baseline"]),
"tolerance_pct": spec["tolerance_pct"],
}
for name, spec in metrics.items()
}
for arm, metrics in existing.get("latency", {}).items():
actuals = parsed_latency.get(arm, {})
new["latency"][arm] = {
name: {
"baseline": actuals.get(name, spec["baseline"]),
"tolerance_pct": spec["tolerance_pct"],
}
for name, spec in metrics.items()
}
baseline_path.write_text(json.dumps(new, indent=2) + "\n")
print(f">>> wrote new baseline to {baseline_path}", file=sys.stderr)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
src = ap.add_mutually_exclusive_group()
src.add_argument("--from-file", type=Path, help="read bench output from this file")
src.add_argument("--stdin", action="store_true", help="read bench output from stdin")
ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE)
ap.add_argument("--update-baseline", action="store_true",
help="re-run, then overwrite baseline.json with the fresh numbers")
args = ap.parse_args()
if args.from_file:
bench_output = args.from_file.read_text()
captured_via = f"bench/run.sh (captured to {args.from_file})"
elif args.stdin:
bench_output = sys.stdin.read()
captured_via = "bench/run.sh (piped to bench/check.py)"
else:
bench_output = run_bench()
captured_via = "bench/run.sh -n 5"
parsed_throughput = parse_throughput_table(bench_output)
parsed_latency = parse_latency_stanzas(bench_output)
baseline = json.loads(args.baseline.read_text())
if args.update_baseline:
write_new_baseline(parsed_throughput, parsed_latency, args.baseline, captured_via)
return 0
measurements = collect_measurements(parsed_throughput, parsed_latency, baseline)
report, has_regression = render_report(measurements)
print(report)
return 1 if has_regression else 0
if __name__ == "__main__":
sys.exit(main())
docs/JOURNAL.md
+124
View File
@@ -9869,3 +9869,127 @@ no other corpus regressions.
schemas, MCP server, LSP.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation.
## 2026-05-09 — Iter 21'a: bench-regression harness shipped
### Why
User asked: *"Mich würde interessieren, ob ailang wirklich wie
erwartet performt. Und vor allem wäre es wichtig, Performance
regressions zu erwischen."* Two distinct concerns. The first is
a one-shot validation question; the second is a structural gap.
Going in: `bench/run.sh` and `bench/latency_harness.py` exist,
but every run was a one-shot manual capture into JOURNAL — no
baseline file, no diff, no tripwire. Five iter families had
shipped since the last canonical numbers (18f / 18g.tidy.fu2). A
perf regression that didn't break correctness would have gone
unnoticed until the next manual bencher invocation, with the
blame surface spread across every intervening commit. The
validation question gets answered as a side effect of building
the harness: capture a fresh baseline, then anything that drifts
from it gets caught immediately rather than weeks later.
### What shipped
- **`bench/baseline.json`** — flat per-metric record with
`baseline` + `tolerance_pct`. 31 metrics: 16 throughput (2
fixtures × 8: gc_s / bump_s / rc_s / gc_over_bump / rc_over_bump
+ 3 RSS values), 15 latency (3 arms × 5: median / p99 / p99.9 /
max / p99/median).
- **`bench/check.py`** — argparse front-end. Default behaviour:
spawn `bench/run.sh -n 5`, parse the throughput pipe-table and
the latency stanzas, diff against baseline, print a per-metric
report, exit 0 if every metric is within its one-sided tolerance,
exit 1 on any regression, exit 2 on parser misalignment (output
format changed). Flags: `--from-file PATH`, `--stdin`,
`--baseline PATH`, `--update-baseline` (re-run + overwrite
`baseline.json` with fresh numbers, used after intentional
improvements).
Tolerances: throughput wall-time 10%, RSS 5%, ratios 8%; latency
median 15%, p99 20–25%, p99/median 20–25%. Tuned to absorb run-
to-run noise on a quiet developer machine, not as the language
correctness bar — Decision-10 thresholds (rc/bump ≤ 1.3× /
p99/median ≤ 5×) are a separate concern and continue to be
evaluated against absolute numbers.
### Validation against the user's first question
Captured the baseline (n=5), then re-ran the entire harness
back-to-back. First-run numbers vs. JOURNAL's last canonical
captures:
```
| 18f | 18g.tidy | now (1) | now (2)
list_sum.rc/bump | 2.86× | - | 2.89× | 2.96×
tree_walk.rc/bump | 2.53× | - | 2.50× | 2.59×
explicit_at_rc.median | - | 226.1 | 213.9 | 213.5
explicit_at_rc.p99 | - | 296.4 | 357.5 | 294.6
explicit_at_rc.p99/med | - | 1.31× | 1.66× | 1.37×
```
Throughput is stable across 5 iter families. Latency Boehm-arm
is stable. The explicit-rc p99 swings between captures
(296 → 357 → 295) — consistent with the 18g.tidy.fu2-recorded
p99 range `[288.7, 311.3]` expanding to `[273.2, 366.0]` today.
That is run-to-run dispersion on a single fixture, not a
regression. The all-green second run confirmed: AILang performs
as expected, no shift since 18g.
Without 21'a we would have had one capture today, seen the
+20.6% p99 number, and either spent an iter chasing a phantom
or written it off without evidence. With 21'a, a single noisy
run is one data point inside a tolerance band, and the tripwire
fires only when a real regression accumulates.
### What this iter does NOT do
- **No corpus widening.** Same 4 fixtures (2 throughput,
2 latency-implicit/explicit). Closure-with-escape-captures
(the 18c.4 trigger), polymorphic ADT pipelines, function-call-
heavy workloads — all queued as 21'b. The 18c.4 doubled-braces
bug remains the canonical demonstration that the corpus has
gaps; baseline-locking the existing 4 fixtures does not close
that.
- **No compile-time bench.** Typechecker / IR-builder slowdowns
from future iters (Family 21 typeclasses likely) would still
be invisible. Queued as 21'c.
- **No CI wiring.** Project has no CI today. `bench/check.py`
is invocable as a tidy-iter gate by hand or by the orchestrator
at family close.
- **No `cargo bench` / criterion.** Hot path for AILang perf
lives outside the Rust crate boundary (compiled AILang
binaries running their own runtime); criterion would be the
wrong instrument. Stays the right call until a Rust-side
hotpath becomes dominant.
### Tidy-iter discipline addition (proposal, not yet enacted)
Recommended: `bench/check.py` runs at every tidy-iter, alongside
the architect drift report. Green → family closes. Red → family
does not close until the regression is either fixed (revert /
refactor) or ratified (`--update-baseline` with a JOURNAL entry
naming what got intentionally slower and why). Orchestrator's
call to add to `CLAUDE.md` or the `agents/README.md` tidy-iter
checklist; not enacted in this iter.
### Test state
288 passed / 0 failed / 3 ignored. No Rust changes; the addition
is `bench/`-only.
### JOURNAL queue (updated)
- **21'b — bench corpus widening.** Closure-capture-escape fixture
(would have caught 18c.4), polymorphic-ADT-pipeline fixture,
call/return-churn fixture. Re-baseline after.
- **21'c — compile-time bench (optional).** Median `ail check` +
`ail build` over the corpus, with its own baseline. Catches
typechecker-complexity regressions before Family 21 lands.
- **`FnDef::synthetic(...)` factor-out** — unchanged.
- **Boehm full retirement** — unchanged.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork that
still wants direct user input.