check.py RC-latency *.max_us is a structural false-positive at -n 5 #16

New Issue

2026-05-20T13:21:58+02:00

Brummel commented

2026-05-20 13:21:58 +02:00

Distinct from the *.bump_s staleness above (different metric family, different root cause). latency.{explicit,implicit}_at_rc.max_us is the single worst of ~5000 samples over only 5 runs, dominated by OS scheduling / THP / IRQ jitter on a shared host, not allocator behaviour. It has fired a false REGRESSION on three consecutive no-runtime-change milestones (iteration-discipline-revert, prose-loop-binders, remove-mut-var-assign) and vanished on identical-code re-run every time (remove-mut-var-assign close: HEAD vs 48e7774 bench binaries cmp-byte-identical, explicit_at_rc.max_us collapsed +108%→-2.30% ok by rerun3) — a ~3-for-3 false-positive rate on null changes, while the median/p99/p99.9 of the same benchmarks held across all re-runs.

Pick one mitigation: drop *.max_us from the gating set (keep median/p99/p99.9, which are the trustworthy signal), or raise -n substantially for the tail metrics, or pin the latency arm to an isolated cpuset. Pure bench-harness change; no language change.

context: remove-mut-var-assign close audit, bencher localisation (2026-05-18) — byte-identical-binary causal exoneration + the 3-milestone false-positive history. Plus the bencher localisation evidence in the iteration-discipline-revert audit commit.

**Distinct** from the `*.bump_s` staleness above (different metric family, different root cause). `latency.{explicit,implicit}_at_rc.max_us` is the single worst of ~5000 samples over only 5 runs, dominated by OS scheduling / THP / IRQ jitter on a shared host, not allocator behaviour. It has fired a false `REGRESSION` on **three consecutive no-runtime-change milestones** (iteration-discipline-revert, prose-loop-binders, remove-mut-var-assign) and vanished on identical-code re-run every time (remove-mut-var-assign close: HEAD vs `48e7774` bench binaries `cmp`-byte-identical, `explicit_at_rc.max_us` collapsed +108%→-2.30% ok by rerun3) — a ~3-for-3 false-positive rate on null changes, while the median/p99/p99.9 of the same benchmarks held across all re-runs. Pick one mitigation: drop `*.max_us` from the gating set (keep median/p99/p99.9, which are the trustworthy signal), or raise `-n` substantially for the tail metrics, or pin the latency arm to an isolated cpuset. Pure bench-harness change; no language change. **context:** remove-mut-var-assign close audit, bencher localisation (2026-05-18) — byte-identical-binary causal exoneration + the 3-milestone false-positive history. Plus the bencher localisation evidence in the iteration-discipline-revert audit commit.

Brummel added the bug label 2026-05-20 15:29:34 +02:00

Brummel referenced this issue from a commit

2026-05-20 16:23:53 +02:00

workflow: add `bug` Gitea label + tag 6 existing bugs

Brummel referenced this issue from a commit

2026-05-20 16:23:53 +02:00

spec: bench-harness-recalibration — degate max_us/p99_9, recapture baselines

Brummel referenced this issue from a commit

2026-05-20 16:23:53 +02:00

plan: bench-harness-recalibration.1 — drop 6 latency entries + recapture

Brummel referenced this issue from a commit

2026-05-20 16:23:53 +02:00