AILang/docs/plans/bench-harness-recalibration.1.md

# Bench harness recalibration — Implementation Plan (iteration 1, terminal)

> **Parent spec:** `docs/specs/2026-05-20-bench-harness-recalibration.md`
>
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
> to run this plan. Steps use `- [ ]` checkboxes for tracking.

**Goal:** Close Gitea #15 + #16 by removing `max_us` and `p99_9_us`
from `bench/baseline.json` (6 entries across 3 latency arms) and
regenerating the remaining baselines from current HEAD, so
`bench/check.py` stops false-positive-firing on environmentally
drifted no-op milestones.

**Architecture:** Single artefact (`bench/baseline.json`); no code
change. `bench/check.py`'s `--update-baseline` mode preserves
`version` / `note` / per-metric `tolerance_pct` and overwrites only
the `baseline` numeric + `captured` date + `captured_via`. The 6
unreliable entries are dropped *before* `--update-baseline` (the
recapture iterates `existing.get(...).items()` and would faithfully
re-baseline any entry left in the file). The rewritten `note` field
is committed in the same pre-recapture edit so `--update-baseline`'s
note-preservation logic carries the new text forward.

**Tech Stack:** `bench/baseline.json` (data file only).
`bench/check.py` (read-only; invoked twice — once to recapture,
once to replay-verify) and `bench/run.sh` (read-only; invoked by
`bench/check.py` for both runs). No Rust crate touched.

---

**Files this plan creates or modifies:**

- Modify: `bench/baseline.json` — the sole artefact.
  - `note` (line 5): rewrite to drop stale references (JOURNAL workflow retired 8e586f4, `*.max_us` tolerance convention now-moot) and add forward pointer to this spec + closed issues #15 / #16.
  - `latency.implicit_at_gc` (lines 71–77): delete `p99_9_us` (line 74) and `max_us` (line 75) entries.
  - `latency.explicit_at_rc` (lines 78–84): delete `p99_9_us` (line 81) and `max_us` (line 82) entries.
  - `latency.implicit_at_rc` (lines 85–91): delete `p99_9_us` (line 88) and `max_us` (line 89) entries.
  - `throughput.*.baseline` (lines 7–67): regenerated by `bench/check.py --update-baseline` against today's bench run.
  - `latency.*.baseline` (remaining 3 metrics × 3 arms = 9 values after drops): regenerated by the same recapture.
  - `captured` (line 3): `2026-05-09` → `2026-05-20` (auto, via `bench/check.py:255`).
  - `captured_via` (line 4): set to `bench/run.sh -n 5` (auto, via `bench/check.py:305`).
- Read-only verification target: `bench/check.py:177-220` (`collect_measurements` — iterates only baseline.json entries; the JSON drop is sufficient).
- Read-only verification target: `bench/check.py:249-284` (`write_new_baseline` — preserves `version` / `note` / `tolerance_pct`).

No other file changes. No Rust crate touched.

---

### Task 1: Pre-recapture JSON edit — drop 6 entries + rewrite `note`

**Files:**
- Modify: `bench/baseline.json` (lines 5, 74–75, 81–82, 88–89)

The recapture preserves the existing `note` field as-is, so the
`note` rewrite happens here, *before* `bench/check.py --update-baseline`
runs. The 6 latency entries must also be dropped here — leaving them
in the file would cause `write_new_baseline` to faithfully re-baseline
them on today's measurements (recon Open Q).

- [ ] **Step 1: Rewrite the `note` field**

Edit `bench/baseline.json` line 5 — the entire `"note": "..."` value —
to the following exact string (single line in the JSON; newlines below
are for readability only; the JSON value must NOT contain literal
newlines, escape them as needed or keep it one logical sentence):

```
Baseline for bench/check.py regression detection. The language-invariant thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are NOT the regression-check tolerances; the per-metric tolerances below are tuned to absorb run-to-run noise on a quiet developer machine. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the commit body that ships the baseline bump. The latency arms gate only on median / p99 / p99_over_median — max_us and p99_9_us were removed on the 2026-05-20 recapture (Gitea #15 / #16) because tail-of-distribution latency metrics are dominated by OS-level jitter (THP defrag, scheduler preemption, IRQ load), not allocator behaviour, and produced 3+ consecutive false-positive REGRESSION rows on byte-identical no-op milestones. See docs/specs/2026-05-20-bench-harness-recalibration.md.
```

- [ ] **Step 2: Delete the 6 latency entries**

Open `bench/baseline.json` and delete these six lines (line numbers
relative to the pre-edit file; the editor's "find this exact JSON
key" is sufficient — line numbers are guidance only):

- Line 74: `      "p99_9_us":        { "baseline": 8131.2, "tolerance_pct": 25 },`
- Line 75: `      "max_us":          { "baseline": 8343.7, "tolerance_pct": 25 },`
- Line 81: `      "p99_9_us":        { "baseline": 404.1,  "tolerance_pct": 25 },`
- Line 82: `      "max_us":          { "baseline": 413.0,  "tolerance_pct": 25 },`
- Line 88: `      "p99_9_us":        { "baseline": 452.0,  "tolerance_pct": 25 },`
- Line 89: `      "max_us":          { "baseline": 477.3,  "tolerance_pct": 30 },`

For each removed line, also remove its trailing comma from the
previous line if needed to keep the JSON valid (the entry above
`p99_9_us` is `p99_us`, which currently has a trailing comma; after
the deletion, `p99_us` is fine as-is because `p99_over_median`
follows it — re-check the comma chain per arm after edit).

- [ ] **Step 3: Verify JSON validity**

Run:

```bash
python3 -m json.tool bench/baseline.json > /dev/null && echo OK
```

Expected stdout: `OK`. Any other output (a Python traceback) means
the JSON is malformed — fix the comma chain.

- [ ] **Step 4: Verify the 6 entries are gone**

Run:

```bash
jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json
```

Expected stdout:

```
[
  "median_us",
  "p99_over_median",
  "p99_us"
]
```

If `max_us` or `p99_9_us` is in the list, Step 2 was incomplete.

- [ ] **Step 5: Verify the `note` field rewrite landed**

Run:

```bash
jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'
```

Expected stdout: `1`. Zero hits means the rewrite from Step 1 was
not saved.

---

### Task 2: Recapture remaining baselines via `--update-baseline`

**Files:**
- Modify: `bench/baseline.json` (all `baseline` numerics + `captured` + `captured_via`)

- [ ] **Step 1: Confirm HEAD is the bench-relevant clean state**

Run:

```bash
git status --short && git log -1 --oneline
```

Expected stdout includes `M  bench/baseline.json` (or `M bench/baseline.json` — staged or unstaged from Task 1) as the only change. HEAD should be a workflow / spec / plan commit, NOT a runtime-touching commit. If any `crates/` or `runtime/` file is modified or HEAD is a runtime-touching commit, STOP and re-evaluate — the recapture must run from a clean bench-relevant state.

- [ ] **Step 2: Run the recapture**

Run:

```bash
bench/check.py --update-baseline
```

Expected: takes ~3-4 minutes (release-mode build of all bench fixtures + `-n 5` runs across throughput + latency arms). Final stderr line: `>>> wrote new baseline to .../bench/baseline.json`. Exit code 0.

- [ ] **Step 3: Verify the recapture preserved structure**

Run:

```bash
jq '.captured, .captured_via' bench/baseline.json
```

Expected stdout:

```
"2026-05-20"
"bench/run.sh -n 5"
```

(Or whatever today's date is when the implement run executes — the field is `$(date +%Y-%m-%d)` per `bench/check.py:255`.)

Also confirm note + entry deletions still hold:

```bash
jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'
jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json
```

Expected: `1` and the 3-element list `["median_us", "p99_over_median", "p99_us"]`. If either of those checks fails, `--update-baseline` somehow regenerated the dropped entries (would mean the Task-1 deletion didn't actually land before recapture) — STOP, restart from Task 1.

---

### Task 3: Acceptance §1 — same-HEAD replay → exit 0

**Files:** None modified. Read-only check.

- [ ] **Step 1: Run check.py and capture exit code**

Run:

```bash
bench/check.py; echo "EXIT=$?"
```

Expected: takes ~3 minutes (build cached from Task 2). Final summary line of stdout matches the pattern:

```
summary: 57 metrics; 0 regressed, <N> improved beyond tolerance, <M> stable
```

(57 = pre-edit 63 minus 6 removed entries; verify the actual reported count matches.)

Final line: `EXIT=0`.

Any other code is a failure — investigate the per-metric REGRESSION rows shown by `bench/check.py` and decide whether to (a) re-run once more (single-run noise) or (b) widen the offending tolerance after fresh investigation. Two consecutive non-zero exits mean the recapture itself was on a noisy run; redo Task 2.

---

### Task 4: Acceptance §2 — synthetic injection on bump_s → REGRESSION fires

**Files:** `bench/baseline.json` (temporary patch, reverted at end of task).

- [ ] **Step 1: Capture the current `bump_s` baseline value**

Run:

```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```

Expected stdout: a single float, the value captured in Task 2 (≈ 0.05 — exact number from the recapture). Save this number; call it `$ORIG`.

- [ ] **Step 2: Patch the baseline to half its value**

Run:

```bash
jq '.throughput.bench_list_sum.bump_s.baseline *= 0.5' bench/baseline.json > /tmp/baseline_injection.json && mv /tmp/baseline_injection.json bench/baseline.json
```

Verify the patch landed:

```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```

Expected stdout: approximately half of `$ORIG`.

- [ ] **Step 3: Run check.py against the patched baseline**

Run:

```bash
bench/check.py; echo "EXIT=$?"
```

Expected stdout includes a row matching this pattern (exact diff and tolerance vary):

```
throughput.bench_list_sum.bump_s                       <halved>    <actual>  +<~100>% 10.0%  REGRESSION
```

And the last line is `EXIT=1`. If `EXIT=0`, the injection did not fire — investigate the gating path; the synthetic injection acceptance check has failed and the iter is BLOCKED. If `EXIT=1` and the row shows REGRESSION on the patched metric, the gate is functioning correctly.

- [ ] **Step 4: Restore the baseline to `$ORIG`**

Run:

```bash
jq --argjson v "$ORIG" '.throughput.bench_list_sum.bump_s.baseline = $v' bench/baseline.json > /tmp/baseline_restored.json && mv /tmp/baseline_restored.json bench/baseline.json
```

Where `$ORIG` is the value from Step 1. Verify:

```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```

Expected stdout: `$ORIG` (the pre-patch value, recaptured in Task 2).

- [ ] **Step 5: Final replay to confirm restoration**

Run:

```bash
bench/check.py; echo "EXIT=$?"
```

Expected last line: `EXIT=0` and `0 regressed` in the summary. Confirms the synthetic injection was fully reverted.

---

## Closing notes for `implement`

This is a one-task-conceptually iter split into four task-templates
for review granularity (each Task is one self-contained
spec-compliance unit). All four Tasks land in a single working-tree
diff (only `bench/baseline.json` is modified). Boss commits the
whole thing as one iter-level commit at the end.

Acceptance criteria from the spec are mapped to tasks:
- Spec §"max_us / p99_9_us removed" → Task 1.
- Spec §"captured field reads 2026-05-20" → Task 2.
- Spec §"replay exit 0" → Task 3.
- Spec §"synthetic injection fires REGRESSION" → Task 4.