bench: regression harness — baseline.json + check.py

Closes the structural gap between bench/run.sh (one-shot capture
into JOURNAL) and a continuous tripwire. baseline.json records 31
metrics (16 throughput, 15 latency) with per-metric one-sided
tolerances tuned to absorb run-to-run noise on a quiet developer
machine; check.py spawns run.sh, parses both the throughput
pipe-table and the per-arm latency stanzas, diffs against the
baseline, prints a per-metric report, and exits non-zero on any
regression beyond tolerance.

User-facing flags: --from-file, --stdin, --baseline, and
--update-baseline (re-run + overwrite baseline.json after
intentional improvements).

Validation: captured the baseline, then re-ran end-to-end. All 31
metrics within tolerance. The harness also caught a single-capture
explicit-rc p99 spike (357.5 us) that was first read as drift vs.
yesterday's 18g.tidy.fu2 numbers but came in at 294.6 us on the
follow-up run — exactly the kind of noise the tolerance band is
there to absorb. Without 21'a we would have either chased a
phantom or buried the signal; with it, single noisy runs are data
points, not verdicts.

Tidy-iter discipline addition (run check.py at every family close
alongside the architect drift report) is recommended in JOURNAL
but not enacted in this iter — that's an orchestrator-level
update to CLAUDE.md.
This commit is contained in:
2026-05-09 00:36:06 +02:00
parent 7e5c95f6c6
commit a1b0ad5723
3 changed files with 499 additions and 0 deletions
+124
View File
@@ -9869,3 +9869,127 @@ no other corpus regressions.
schemas, MCP server, LSP.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation.
## 2026-05-09 — Iter 21'a: bench-regression harness shipped
### Why
User asked: *"Mich würde interessieren, ob ailang wirklich wie
erwartet performt. Und vor allem wäre es wichtig, Performance
regressions zu erwischen."* Two distinct concerns. The first is
a one-shot validation question; the second is a structural gap.
Going in: `bench/run.sh` and `bench/latency_harness.py` exist,
but every run was a one-shot manual capture into JOURNAL — no
baseline file, no diff, no tripwire. Five iter families had
shipped since the last canonical numbers (18f / 18g.tidy.fu2). A
perf regression that didn't break correctness would have gone
unnoticed until the next manual bencher invocation, with the
blame surface spread across every intervening commit. The
validation question gets answered as a side effect of building
the harness: capture a fresh baseline, then anything that drifts
from it gets caught immediately rather than weeks later.
### What shipped
- **`bench/baseline.json`** — flat per-metric record with
`baseline` + `tolerance_pct`. 31 metrics: 16 throughput (2
fixtures × 8: gc_s / bump_s / rc_s / gc_over_bump / rc_over_bump
+ 3 RSS values), 15 latency (3 arms × 5: median / p99 / p99.9 /
max / p99/median).
- **`bench/check.py`** — argparse front-end. Default behaviour:
spawn `bench/run.sh -n 5`, parse the throughput pipe-table and
the latency stanzas, diff against baseline, print a per-metric
report, exit 0 if every metric is within its one-sided tolerance,
exit 1 on any regression, exit 2 on parser misalignment (output
format changed). Flags: `--from-file PATH`, `--stdin`,
`--baseline PATH`, `--update-baseline` (re-run + overwrite
`baseline.json` with fresh numbers, used after intentional
improvements).
Tolerances: throughput wall-time 10%, RSS 5%, ratios 8%; latency
median 15%, p99 2025%, p99/median 2025%. Tuned to absorb run-
to-run noise on a quiet developer machine, not as the language
correctness bar — Decision-10 thresholds (rc/bump ≤ 1.3× /
p99/median ≤ 5×) are a separate concern and continue to be
evaluated against absolute numbers.
### Validation against the user's first question
Captured the baseline (n=5), then re-ran the entire harness
back-to-back. First-run numbers vs. JOURNAL's last canonical
captures:
```
| 18f | 18g.tidy | now (1) | now (2)
list_sum.rc/bump | 2.86× | - | 2.89× | 2.96×
tree_walk.rc/bump | 2.53× | - | 2.50× | 2.59×
explicit_at_rc.median | - | 226.1 | 213.9 | 213.5
explicit_at_rc.p99 | - | 296.4 | 357.5 | 294.6
explicit_at_rc.p99/med | - | 1.31× | 1.66× | 1.37×
```
Throughput is stable across 5 iter families. Latency Boehm-arm
is stable. The explicit-rc p99 swings between captures
(296 → 357 → 295) — consistent with the 18g.tidy.fu2-recorded
p99 range `[288.7, 311.3]` expanding to `[273.2, 366.0]` today.
That is run-to-run dispersion on a single fixture, not a
regression. The all-green second run confirmed: AILang performs
as expected, no shift since 18g.
Without 21'a we would have had one capture today, seen the
+20.6% p99 number, and either spent an iter chasing a phantom
or written it off without evidence. With 21'a, a single noisy
run is one data point inside a tolerance band, and the tripwire
fires only when a real regression accumulates.
### What this iter does NOT do
- **No corpus widening.** Same 4 fixtures (2 throughput,
2 latency-implicit/explicit). Closure-with-escape-captures
(the 18c.4 trigger), polymorphic ADT pipelines, function-call-
heavy workloads — all queued as 21'b. The 18c.4 doubled-braces
bug remains the canonical demonstration that the corpus has
gaps; baseline-locking the existing 4 fixtures does not close
that.
- **No compile-time bench.** Typechecker / IR-builder slowdowns
from future iters (Family 21 typeclasses likely) would still
be invisible. Queued as 21'c.
- **No CI wiring.** Project has no CI today. `bench/check.py`
is invocable as a tidy-iter gate by hand or by the orchestrator
at family close.
- **No `cargo bench` / criterion.** Hot path for AILang perf
lives outside the Rust crate boundary (compiled AILang
binaries running their own runtime); criterion would be the
wrong instrument. Stays the right call until a Rust-side
hotpath becomes dominant.
### Tidy-iter discipline addition (proposal, not yet enacted)
Recommended: `bench/check.py` runs at every tidy-iter, alongside
the architect drift report. Green → family closes. Red → family
does not close until the regression is either fixed (revert /
refactor) or ratified (`--update-baseline` with a JOURNAL entry
naming what got intentionally slower and why). Orchestrator's
call to add to `CLAUDE.md` or the `agents/README.md` tidy-iter
checklist; not enacted in this iter.
### Test state
288 passed / 0 failed / 3 ignored. No Rust changes; the addition
is `bench/`-only.
### JOURNAL queue (updated)
- **21'b — bench corpus widening.** Closure-capture-escape fixture
(would have caught 18c.4), polymorphic-ADT-pipeline fixture,
call/return-churn fixture. Re-baseline after.
- **21'c — compile-time bench (optional).** Median `ail check` +
`ail build` over the corpus, with its own baseline. Catches
typechecker-complexity regressions before Family 21 lands.
- **`FnDef::synthetic(...)` factor-out** — unchanged.
- **Boehm full retirement** — unchanged.
- **Deferred richer integration paths** (from 20f) — unchanged.
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
pattern-binding generalisation. Orchestrator-level fork that
still wants direct user input.