Stale compile_check baseline: check_ms drift +35-47% (pre-existing, confirmed at b151990) #45

New Issue

2026-05-30T15:26:30+02:00

Brummel commented

2026-05-30 15:26:30 +02:00

Found during the #44 cycle-close audit. The audit commit body (7321826) initially attributed the compile_check.py exit-1 to transient machine load — that attribution is WRONG and this issue corrects the record.

Finding (verified)

bench/compile_check.py exits 1: all 12 check_ms.* metrics (type-check wall-clock) are +31..49% over baseline, uniformly across every fixture (hello +47%, nested_pat +35%, etc.). build_O0_ms.* are within tolerance.

It is NOT noise and NOT caused by #44

Re-ran 4x this session: the check_ms drift is STABLE at +32..49%, never flipping in/out of REGRESSION (genuine load-noise flips).
Measured the PARENT commit b151990 (before #44) in an isolated git worktree: identical +34..47% check_ms elevation. So the drift predates the cycle entirely.
#44 is physically excluded anyway: its checker-path cost is one raw.contains('$') per token (~microseconds), three orders short of the ~+0.8ms absolute shift.

Root cause

bench/baseline_compile.json was last ratified 2026-05-15 (20add51, mut.4-tidy). Since then ~6 milestones landed (floats, raw-buf, kernel-extension-mechanics, intrinsic-bodies, #43 unique-binder-names, #44). The accumulated type-checker cost has drifted ~+0.8ms on a ~2ms baseline — past the 25% tolerance on the tiny-baseline check_ms metrics. The throughput baseline (bench/baseline.json) was re-ratified 2026-05-28 (de4399d) but the compile baseline was not.

Decision needed (ratify vs investigate)

RATIFY: re-baseline compile_check.py (--update-baseline) with a ratify paragraph. Cheapest, but bakes in ~0.8ms of unexplained accumulated check-time cost as the new normal.
INVESTIGATE: bisect which milestone(s) added the check-time cost before re-baselining — the +0.8ms may itself be a fixable type-checker regression worth understanding, not just accepting.

depends on: nothing
context: surfaced by the #44 audit; main HEAD (7321826) carries the now-corrected wrong attribution and cannot be amended (main sacrosanct), so this issue is the forward record. Baseline deliberately left UNTOUCHED pending this decision.

Found during the #44 cycle-close audit. The audit commit body (7321826) initially attributed the compile_check.py exit-1 to transient machine load — that attribution is WRONG and this issue corrects the record. ## Finding (verified) bench/compile_check.py exits 1: all 12 check_ms.* metrics (type-check wall-clock) are +31..49% over baseline, uniformly across every fixture (hello +47%, nested_pat +35%, etc.). build_O0_ms.* are within tolerance. ## It is NOT noise and NOT caused by #44 - Re-ran 4x this session: the check_ms drift is STABLE at +32..49%, never flipping in/out of REGRESSION (genuine load-noise flips). - Measured the PARENT commit b151990 (before #44) in an isolated git worktree: identical +34..47% check_ms elevation. So the drift predates the cycle entirely. - #44 is physically excluded anyway: its checker-path cost is one raw.contains('$') per token (~microseconds), three orders short of the ~+0.8ms absolute shift. ## Root cause bench/baseline_compile.json was last ratified 2026-05-15 (20add51, mut.4-tidy). Since then ~6 milestones landed (floats, raw-buf, kernel-extension-mechanics, intrinsic-bodies, #43 unique-binder-names, #44). The accumulated type-checker cost has drifted ~+0.8ms on a ~2ms baseline — past the 25% tolerance on the tiny-baseline check_ms metrics. The throughput baseline (bench/baseline.json) was re-ratified 2026-05-28 (de4399d) but the compile baseline was not. ## Decision needed (ratify vs investigate) - RATIFY: re-baseline compile_check.py (--update-baseline) with a ratify paragraph. Cheapest, but bakes in ~0.8ms of unexplained accumulated check-time cost as the new normal. - INVESTIGATE: bisect which milestone(s) added the check-time cost before re-baselining — the +0.8ms may itself be a fixable type-checker regression worth understanding, not just accepting. depends on: nothing context: surfaced by the #44 audit; main HEAD (7321826) carries the now-corrected wrong attribution and cannot be amended (main sacrosanct), so this issue is the forward record. Baseline deliberately left UNTOUCHED pending this decision.

Brummel added the bug label 2026-05-30 15:26:30 +02:00

Brummel referenced this issue from a commit

2026-05-30 15:33:10 +02:00

bench: re-ratify compile_check baseline (closes #45)

Brummel closed this issue

2026-05-30 15:33:10 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/AILang#45

Stale compile_check baseline: check_ms drift +35-47% (pre-existing, confirmed at b151990) #45

Finding (verified)

It is NOT noise and NOT caused by #44

Root cause

Decision needed (ratify vs investigate)

Stale compile_check baseline: check_ms drift +35-47% (pre-existing, confirmed at `b151990`) #45