# Bench harness recalibration — degate `max_us` / `p99_9_us`, re-capture baselines

**Date:** 2026-05-20
**Status:** Draft — awaiting user spec review
**Authors:** Brummel (orchestrator) + Claude

## Goal

End the recurring bench false-positive on no-op milestones by:

1. Removing the two structurally jitter-prone latency metrics
   (`max_us`, `p99_9_us`) from `bench/baseline.json`, so
   `bench/check.py` no longer gates them.
2. Re-capturing the remaining baselines from current HEAD, so the
   environmental drift accumulated since 2026-05-09 is absorbed in
   one honest cut rather than hidden under widened tolerances.

Closes Gitea #15 (`*.bump_s` baseline stale) and #16 (`*.max_us`
structural false-positive). The two issues frame distinct symptoms,
but reproduction on 2026-05-20 (HEAD `19321d8`, two consecutive
`bench/check.py -n 5` runs, both regression-firing) showed:

- The latency false-positive in `implicit_at_rc.max_us` is no longer
  the stochastic +108%→ok-on-rerun spike #16 documented; it now
  reproduces deterministically at +47% / +49% across runs.
  `p99_9_us` on the same arm shows the same pattern (+28% / +42%).
  The metrics are now drift-elevated as well as jitter-prone.
- The throughput drift is wider than #15 framed it: `gc_s` on
  `bench_hof_pipeline` also fires (+10.6% / +10.9% across runs),
  and `bench_list_sum.gc_s` sits at +9.9% (just under the gate).
  Drift correlates with memory pressure / allocation density, not
  with metric family — `bench_tree_walk` and
  `bench_compute_collatz` are clean.

Both issues are therefore one milestone, addressed by one cohesive
change.

## Architecture

No code change. Two artefacts in the working tree:

| File | Change |
|------|--------|
| `bench/baseline.json` | Drop 6 entries (`max_us` + `p99_9_us` × 3 latency arms); regenerate every remaining `baseline` value from current HEAD via `bench/check.py --update-baseline`; update the top-level `note` field |

`bench/check.py` is unmodified. The harness already iterates only
the entries present in `baseline.json` (`collect_measurements`
function, lines 177–220) — removing an entry removes it from both
gating and the per-metric report, no further wiring needed.

`bench/run.sh` and `bench/latency_harness.py` are unmodified. The
removed metrics continue to appear upstream in `bench/run.sh`
output (the latency harness computes them independently of
`baseline.json`); only the regression-check layer no longer sees
them. A developer triaging a regression can still read `max_us`
and `p99.9` directly from the bench-run table.

## Concrete code shapes

### The artefact this milestone delivers — `bench/baseline.json` delta

Per latency arm, two entries are removed. Before (current
`implicit_at_rc`, lines 85–91):

```json
"implicit_at_rc": {
  "median_us":       { "baseline": 285.7,  "tolerance_pct": 15 },
  "p99_us":          { "baseline": 407.1,  "tolerance_pct": 20 },
  "p99_9_us":        { "baseline": 452.0,  "tolerance_pct": 25 },
  "max_us":          { "baseline": 477.3,  "tolerance_pct": 30 },
  "p99_over_median": { "baseline": 1.43,   "tolerance_pct": 20 }
}
```

After (same arm, the two unreliable metrics dropped; remaining
baselines re-captured on 2026-05-20 hardware):

```json
"implicit_at_rc": {
  "median_us":       { "baseline": <recaptured>, "tolerance_pct": 15 },
  "p99_us":          { "baseline": <recaptured>, "tolerance_pct": 20 },
  "p99_over_median": { "baseline": <recaptured>, "tolerance_pct": 20 }
}
```

Same shape for `implicit_at_gc` and `explicit_at_rc`. Total: 6
entries removed, all remaining latency + throughput baseline
numbers refreshed.

Top-level `note` field updated to record the recapture date, the
two removed metric families, and a one-line pointer to this spec
and the closed issues.

### The bench programs this milestone exists to serve

The infrastructure measures the existing AILang fixtures under
`examples/`. Two representative ones — `bench_list_sum.ail`
(allocation-intensive list workload, where the throughput drift
reproduces) and `bench_latency_implicit.ail` (the RC-drop-latency
workload, built by `run.sh` into the `_rc` variant where the
removed `max_us` / `p99_9_us` live) — are the north-star programs
this harness reports on. They are unchanged by this milestone;
only the gating around them shifts.

## Components

Only `bench/baseline.json` is touched. No Rust crate change, no
runtime change, no CLI change.

## Data flow

`bench/run.sh` → text table on stdout → `bench/check.py` parses →
diffs against `bench/baseline.json` → exit code. The path is
identical before and after. Removing entries from `baseline.json`
means `collect_measurements` does not emit `Measurement` records
for those metric names, so they neither appear in the report nor
contribute to the regression count.

## Error handling

None new. `bench/check.py`'s existing "baselined metric not found
in bench output" sys.exit(2) path continues to cover the case
where the bench output format breaks. The removed metrics are not
baselined, so their absence in output is not an error.

## Testing strategy

Two acceptance checks, both run by hand at iter close:

1. **Replay-pass.** After the recapture commit lands, immediate
   `bench/check.py` run on the same HEAD returns exit 0 with zero
   `REGRESSION` rows. (The recapture writes baselines to today's
   measurement; a same-HEAD replay should fall inside every
   per-metric tolerance.)
2. **Detection-still-works (synthetic injection).** Temporarily
   patch `bench/baseline.json` to drop `bench_list_sum.bump_s`
   baseline by 50% (e.g. `0.046` → `0.023`), re-run
   `bench/check.py`, confirm exit 1 and one `REGRESSION` row on
   that metric. Revert the patch. This protects against a
   recapture that accidentally inflates tolerances or removes
   gating beyond the intended two metric families.

No new automated test is introduced — the bench harness is an
out-of-tree Python tool with no Rust test surface, and adding a
test crate for it is scope-disproportionate for this iteration.

## Acceptance criteria

- `bench/baseline.json` no longer contains any `max_us` or
  `p99_9_us` entry (verifiable: `jq '.. | objects | keys' <
  bench/baseline.json | grep -E "max_us|p99_9_us"` returns nothing).
- `bench/baseline.json` `captured` field reads `2026-05-20`;
  `note` documents the removed metrics and links this spec.
- `bench/check.py` replay on the recapture HEAD exits 0 with
  `0 regressed`.
- Synthetic injection test (Acceptance §2) confirms detection
  still fires on a plausible real regression.

Out of scope, explicitly:

- Histogram-based latency methodology rework (Gitea #19) — a
  separate brainstorm; this milestone is a stop-gap that buys back
  the noise floor.
- Latency-arm cpuset / `isolcpus=` pinning — sysadmin-layer fix
  that does not survive CI-move or new dev machine.
- `tolerance_pct` widening as a substitute for recapture — hides
  the drift rather than absorbing it honestly.
- Changes to `bench/check.py`, `bench/run.sh`, or
  `bench/latency_harness.py`.