bench: regression harness — baseline.json + check.py
Closes the structural gap between bench/run.sh (one-shot capture into JOURNAL) and a continuous tripwire. baseline.json records 31 metrics (16 throughput, 15 latency) with per-metric one-sided tolerances tuned to absorb run-to-run noise on a quiet developer machine; check.py spawns run.sh, parses both the throughput pipe-table and the per-arm latency stanzas, diffs against the baseline, prints a per-metric report, and exits non-zero on any regression beyond tolerance. User-facing flags: --from-file, --stdin, --baseline, and --update-baseline (re-run + overwrite baseline.json after intentional improvements). Validation: captured the baseline, then re-ran end-to-end. All 31 metrics within tolerance. The harness also caught a single-capture explicit-rc p99 spike (357.5 us) that was first read as drift vs. yesterday's 18g.tidy.fu2 numbers but came in at 294.6 us on the follow-up run — exactly the kind of noise the tolerance band is there to absorb. Without 21'a we would have either chased a phantom or buried the signal; with it, single noisy runs are data points, not verdicts. Tidy-iter discipline addition (run check.py at every family close alongside the architect drift report) is recommended in JOURNAL but not enacted in this iter — that's an orchestrator-level update to CLAUDE.md.
This commit is contained in:
+124
@@ -9869,3 +9869,127 @@ no other corpus regressions.
|
||||
schemas, MCP server, LSP.
|
||||
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||
pattern-binding generalisation.
|
||||
|
||||
## 2026-05-09 — Iter 21'a: bench-regression harness shipped
|
||||
|
||||
### Why
|
||||
|
||||
User asked: *"Mich würde interessieren, ob ailang wirklich wie
|
||||
erwartet performt. Und vor allem wäre es wichtig, Performance
|
||||
regressions zu erwischen."* Two distinct concerns. The first is
|
||||
a one-shot validation question; the second is a structural gap.
|
||||
|
||||
Going in: `bench/run.sh` and `bench/latency_harness.py` exist,
|
||||
but every run was a one-shot manual capture into JOURNAL — no
|
||||
baseline file, no diff, no tripwire. Five iter families had
|
||||
shipped since the last canonical numbers (18f / 18g.tidy.fu2). A
|
||||
perf regression that didn't break correctness would have gone
|
||||
unnoticed until the next manual bencher invocation, with the
|
||||
blame surface spread across every intervening commit. The
|
||||
validation question gets answered as a side effect of building
|
||||
the harness: capture a fresh baseline, then anything that drifts
|
||||
from it gets caught immediately rather than weeks later.
|
||||
|
||||
### What shipped
|
||||
|
||||
- **`bench/baseline.json`** — flat per-metric record with
|
||||
`baseline` + `tolerance_pct`. 31 metrics: 16 throughput (2
|
||||
fixtures × 8: gc_s / bump_s / rc_s / gc_over_bump / rc_over_bump
|
||||
+ 3 RSS values), 15 latency (3 arms × 5: median / p99 / p99.9 /
|
||||
max / p99/median).
|
||||
- **`bench/check.py`** — argparse front-end. Default behaviour:
|
||||
spawn `bench/run.sh -n 5`, parse the throughput pipe-table and
|
||||
the latency stanzas, diff against baseline, print a per-metric
|
||||
report, exit 0 if every metric is within its one-sided tolerance,
|
||||
exit 1 on any regression, exit 2 on parser misalignment (output
|
||||
format changed). Flags: `--from-file PATH`, `--stdin`,
|
||||
`--baseline PATH`, `--update-baseline` (re-run + overwrite
|
||||
`baseline.json` with fresh numbers, used after intentional
|
||||
improvements).
|
||||
|
||||
Tolerances: throughput wall-time 10%, RSS 5%, ratios 8%; latency
|
||||
median 15%, p99 20–25%, p99/median 20–25%. Tuned to absorb run-
|
||||
to-run noise on a quiet developer machine, not as the language
|
||||
correctness bar — Decision-10 thresholds (rc/bump ≤ 1.3× /
|
||||
p99/median ≤ 5×) are a separate concern and continue to be
|
||||
evaluated against absolute numbers.
|
||||
|
||||
### Validation against the user's first question
|
||||
|
||||
Captured the baseline (n=5), then re-ran the entire harness
|
||||
back-to-back. First-run numbers vs. JOURNAL's last canonical
|
||||
captures:
|
||||
|
||||
```
|
||||
| 18f | 18g.tidy | now (1) | now (2)
|
||||
list_sum.rc/bump | 2.86× | - | 2.89× | 2.96×
|
||||
tree_walk.rc/bump | 2.53× | - | 2.50× | 2.59×
|
||||
explicit_at_rc.median | - | 226.1 | 213.9 | 213.5
|
||||
explicit_at_rc.p99 | - | 296.4 | 357.5 | 294.6
|
||||
explicit_at_rc.p99/med | - | 1.31× | 1.66× | 1.37×
|
||||
```
|
||||
|
||||
Throughput is stable across 5 iter families. Latency Boehm-arm
|
||||
is stable. The explicit-rc p99 swings between captures
|
||||
(296 → 357 → 295) — consistent with the 18g.tidy.fu2-recorded
|
||||
p99 range `[288.7, 311.3]` expanding to `[273.2, 366.0]` today.
|
||||
That is run-to-run dispersion on a single fixture, not a
|
||||
regression. The all-green second run confirmed: AILang performs
|
||||
as expected, no shift since 18g.
|
||||
|
||||
Without 21'a we would have had one capture today, seen the
|
||||
+20.6% p99 number, and either spent an iter chasing a phantom
|
||||
or written it off without evidence. With 21'a, a single noisy
|
||||
run is one data point inside a tolerance band, and the tripwire
|
||||
fires only when a real regression accumulates.
|
||||
|
||||
### What this iter does NOT do
|
||||
|
||||
- **No corpus widening.** Same 4 fixtures (2 throughput,
|
||||
2 latency-implicit/explicit). Closure-with-escape-captures
|
||||
(the 18c.4 trigger), polymorphic ADT pipelines, function-call-
|
||||
heavy workloads — all queued as 21'b. The 18c.4 doubled-braces
|
||||
bug remains the canonical demonstration that the corpus has
|
||||
gaps; baseline-locking the existing 4 fixtures does not close
|
||||
that.
|
||||
- **No compile-time bench.** Typechecker / IR-builder slowdowns
|
||||
from future iters (Family 21 typeclasses likely) would still
|
||||
be invisible. Queued as 21'c.
|
||||
- **No CI wiring.** Project has no CI today. `bench/check.py`
|
||||
is invocable as a tidy-iter gate by hand or by the orchestrator
|
||||
at family close.
|
||||
- **No `cargo bench` / criterion.** Hot path for AILang perf
|
||||
lives outside the Rust crate boundary (compiled AILang
|
||||
binaries running their own runtime); criterion would be the
|
||||
wrong instrument. Stays the right call until a Rust-side
|
||||
hotpath becomes dominant.
|
||||
|
||||
### Tidy-iter discipline addition (proposal, not yet enacted)
|
||||
|
||||
Recommended: `bench/check.py` runs at every tidy-iter, alongside
|
||||
the architect drift report. Green → family closes. Red → family
|
||||
does not close until the regression is either fixed (revert /
|
||||
refactor) or ratified (`--update-baseline` with a JOURNAL entry
|
||||
naming what got intentionally slower and why). Orchestrator's
|
||||
call to add to `CLAUDE.md` or the `agents/README.md` tidy-iter
|
||||
checklist; not enacted in this iter.
|
||||
|
||||
### Test state
|
||||
|
||||
288 passed / 0 failed / 3 ignored. No Rust changes; the addition
|
||||
is `bench/`-only.
|
||||
|
||||
### JOURNAL queue (updated)
|
||||
|
||||
- **21'b — bench corpus widening.** Closure-capture-escape fixture
|
||||
(would have caught 18c.4), polymorphic-ADT-pipeline fixture,
|
||||
call/return-churn fixture. Re-baseline after.
|
||||
- **21'c — compile-time bench (optional).** Median `ail check` +
|
||||
`ail build` over the corpus, with its own baseline. Catches
|
||||
typechecker-complexity regressions before Family 21 lands.
|
||||
- **`FnDef::synthetic(...)` factor-out** — unchanged.
|
||||
- **Boehm full retirement** — unchanged.
|
||||
- **Deferred richer integration paths** (from 20f) — unchanged.
|
||||
- **Family 21+** — typeclasses, polymorphic ADTs at runtime,
|
||||
pattern-binding generalisation. Orchestrator-level fork that
|
||||
still wants direct user input.
|
||||
|
||||
Reference in New Issue
Block a user