Brummel
a1b0ad5723
Closes the structural gap between bench/run.sh (one-shot capture into JOURNAL) and a continuous tripwire. baseline.json records 31 metrics (16 throughput, 15 latency) with per-metric one-sided tolerances tuned to absorb run-to-run noise on a quiet developer machine; check.py spawns run.sh, parses both the throughput pipe-table and the per-arm latency stanzas, diffs against the baseline, prints a per-metric report, and exits non-zero on any regression beyond tolerance. User-facing flags: --from-file, --stdin, --baseline, and --update-baseline (re-run + overwrite baseline.json after intentional improvements). Validation: captured the baseline, then re-ran end-to-end. All 31 metrics within tolerance. The harness also caught a single-capture explicit-rc p99 spike (357.5 us) that was first read as drift vs. yesterday's 18g.tidy.fu2 numbers but came in at 294.6 us on the follow-up run — exactly the kind of noise the tolerance band is there to absorb. Without 21'a we would have either chased a phantom or buried the signal; with it, single noisy runs are data points, not verdicts. Tidy-iter discipline addition (run check.py at every family close alongside the architect drift report) is recommended in JOURNAL but not enacted in this iter — that's an orchestrator-level update to CLAUDE.md.
54 lines
2.8 KiB
JSON
54 lines
2.8 KiB
JSON
{
|
|
"version": 1,
|
|
"captured": "2026-05-09",
|
|
"captured_via": "bench/run.sh -n 5",
|
|
"note": "Baseline for bench/check.py regression detection. Decision-10 thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are LANGUAGE invariants, not regression-check tolerances. The tolerances below are tuned to absorb run-to-run noise on a quiet developer machine; they are NOT the correctness bar. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the JOURNAL entry that ships the baseline bump.",
|
|
|
|
"throughput": {
|
|
"bench_list_sum": {
|
|
"gc_s": { "baseline": 0.137, "tolerance_pct": 10 },
|
|
"bump_s": { "baseline": 0.046, "tolerance_pct": 10 },
|
|
"rc_s": { "baseline": 0.133, "tolerance_pct": 10 },
|
|
"gc_over_bump": { "baseline": 2.98, "tolerance_pct": 8 },
|
|
"rc_over_bump": { "baseline": 2.89, "tolerance_pct": 8 },
|
|
"gc_rss_kb": { "baseline": 103980, "tolerance_pct": 5 },
|
|
"bump_rss_kb": { "baseline": 97696, "tolerance_pct": 5 },
|
|
"rc_rss_kb": { "baseline": 193448, "tolerance_pct": 5 }
|
|
},
|
|
"bench_tree_walk": {
|
|
"gc_s": { "baseline": 0.103, "tolerance_pct": 10 },
|
|
"bump_s": { "baseline": 0.038, "tolerance_pct": 10 },
|
|
"rc_s": { "baseline": 0.095, "tolerance_pct": 10 },
|
|
"gc_over_bump": { "baseline": 2.71, "tolerance_pct": 8 },
|
|
"rc_over_bump": { "baseline": 2.50, "tolerance_pct": 8 },
|
|
"gc_rss_kb": { "baseline": 73260, "tolerance_pct": 5 },
|
|
"bump_rss_kb": { "baseline": 55208, "tolerance_pct": 5 },
|
|
"rc_rss_kb": { "baseline": 108956, "tolerance_pct": 5 }
|
|
}
|
|
},
|
|
|
|
"latency": {
|
|
"implicit_at_gc": {
|
|
"median_us": { "baseline": 96.4, "tolerance_pct": 15 },
|
|
"p99_us": { "baseline": 7130.0, "tolerance_pct": 20 },
|
|
"p99_9_us": { "baseline": 8131.2, "tolerance_pct": 25 },
|
|
"max_us": { "baseline": 8343.7, "tolerance_pct": 25 },
|
|
"p99_over_median": { "baseline": 73.92, "tolerance_pct": 20 }
|
|
},
|
|
"explicit_at_rc": {
|
|
"median_us": { "baseline": 213.9, "tolerance_pct": 15 },
|
|
"p99_us": { "baseline": 357.5, "tolerance_pct": 25 },
|
|
"p99_9_us": { "baseline": 404.1, "tolerance_pct": 25 },
|
|
"max_us": { "baseline": 413.0, "tolerance_pct": 25 },
|
|
"p99_over_median": { "baseline": 1.66, "tolerance_pct": 25 }
|
|
},
|
|
"implicit_at_rc": {
|
|
"median_us": { "baseline": 285.7, "tolerance_pct": 15 },
|
|
"p99_us": { "baseline": 407.1, "tolerance_pct": 20 },
|
|
"p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },
|
|
"max_us": { "baseline": 477.3, "tolerance_pct": 25 },
|
|
"p99_over_median": { "baseline": 1.43, "tolerance_pct": 20 }
|
|
}
|
|
}
|
|
}
|