Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

plan: bench-harness-recalibration.1 — drop 6 latency entries + recapture

Browse Source 
One terminal iteration covering the whole spec.  Four tasks, all on
`bench/baseline.json`:

- Task 1: pre-recapture JSON edit — drop `max_us` + `p99_9_us` × 3
  latency arms (6 entries) and rewrite the `note` field forward-
  looking.  The 6 entries must go before `--update-baseline` runs
  because `write_new_baseline` faithfully re-baselines anything
  left in the file (recon Open Q).  The note rewrite is folded in
  here because `--update-baseline` preserves the existing note as-
  is, so any change has to happen before recapture (or as a
  separate post-recapture edit; one-edit is cleaner).
- Task 2: `bench/check.py --update-baseline` regenerates every
  remaining `baseline` value from a fresh `bench/run.sh -n 5`,
  updating `captured` → 2026-05-20 and `captured_via`.
- Task 3: acceptance §1 — fresh-HEAD replay → exit 0, 0 regressed,
  57 metrics in summary (= pre-edit 63 minus the 6 drops).
- Task 4: acceptance §2 — synthetic injection (halve
  `bench_list_sum.bump_s.baseline` via `jq`) → exit 1 + REGRESSION
  row on that metric, then restore via `jq` and confirm a final
  exit-0 replay.

Recon adjudications:

- Replay-source (recon Open Q1): fresh `bench/run.sh -n 5` for
  replay, not the same output that produced the recapture.
  Realistic acceptance scenario; with 10% throughput tolerance vs.
  ~1-2% measured run-to-run variance on the affected metrics
  (run-1 / run-2 reproduction data in spec body), there is ample
  margin.
- Note rewrite (recon Open Q2): forward-looking — drop stale
  refs (JOURNAL workflow retired 8e586f4; the `*.max_us`
  tolerance convention is now moot since the metric is gone),
  replace with pointer to docs/specs/2026-05-20-bench-harness-
  recalibration.md + closed issues #15 / #16.  Spec lives where
  the rationale lives; the note carries the gate-policy
  one-liner only.

No Rust crate touched.  `bench/check.py` / `bench/run.sh` /
`bench/latency_harness.py` unmodified.

Plan self-review (all 8 checklist items): clean.  Step granularity
checked; Task 3 collapsed from two `bench/check.py` runs to one
(single 3-minute run captures both summary and exit code).

Ready for handoff to `skills/implement`.
This commit is contained in:
Brummel
2026-05-20 16:07:53 +02:00
parent a97aaebd45
commit ddb50c3cb3
1 changed files with 299 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/bench-harness-recalibration.1.md
+299
View File
@@ -0,0 +1,299 @@
# Bench harness recalibration — Implementation Plan (iteration 1, terminal)
> **Parent spec:** `docs/specs/2026-05-20-bench-harness-recalibration.md`
>
> **For agentic workers:** REQUIRED SUB-SKILL: use `skills/implement`
> to run this plan. Steps use `- [ ]` checkboxes for tracking.
**Goal:** Close Gitea #15 + #16 by removing `max_us` and `p99_9_us`
from `bench/baseline.json` (6 entries across 3 latency arms) and
regenerating the remaining baselines from current HEAD, so
`bench/check.py` stops false-positive-firing on environmentally
drifted no-op milestones.
**Architecture:** Single artefact (`bench/baseline.json`); no code
change. `bench/check.py`'s `--update-baseline` mode preserves
`version` / `note` / per-metric `tolerance_pct` and overwrites only
the `baseline` numeric + `captured` date + `captured_via`. The 6
unreliable entries are dropped *before* `--update-baseline` (the
recapture iterates `existing.get(...).items()` and would faithfully
re-baseline any entry left in the file). The rewritten `note` field
is committed in the same pre-recapture edit so `--update-baseline`'s
note-preservation logic carries the new text forward.
**Tech Stack:** `bench/baseline.json` (data file only).
`bench/check.py` (read-only; invoked twice — once to recapture,
once to replay-verify) and `bench/run.sh` (read-only; invoked by
`bench/check.py` for both runs). No Rust crate touched.
---
**Files this plan creates or modifies:**
- Modify: `bench/baseline.json` — the sole artefact.
- `note` (line 5): rewrite to drop stale references (JOURNAL workflow retired 8e586f4, `*.max_us` tolerance convention now-moot) and add forward pointer to this spec + closed issues #15 / #16.
- `latency.implicit_at_gc` (lines 71–77): delete `p99_9_us` (line 74) and `max_us` (line 75) entries.
- `latency.explicit_at_rc` (lines 78–84): delete `p99_9_us` (line 81) and `max_us` (line 82) entries.
- `latency.implicit_at_rc` (lines 85–91): delete `p99_9_us` (line 88) and `max_us` (line 89) entries.
- `throughput.*.baseline` (lines 7–67): regenerated by `bench/check.py --update-baseline` against today's bench run.
- `latency.*.baseline` (remaining 3 metrics × 3 arms = 9 values after drops): regenerated by the same recapture.
- `captured` (line 3): `2026-05-09` → `2026-05-20` (auto, via `bench/check.py:255`).
- `captured_via` (line 4): set to `bench/run.sh -n 5` (auto, via `bench/check.py:305`).
- Read-only verification target: `bench/check.py:177-220` (`collect_measurements` — iterates only baseline.json entries; the JSON drop is sufficient).
- Read-only verification target: `bench/check.py:249-284` (`write_new_baseline` — preserves `version` / `note` / `tolerance_pct`).
No other file changes. No Rust crate touched.
---
### Task 1: Pre-recapture JSON edit — drop 6 entries + rewrite `note`
**Files:**
- Modify: `bench/baseline.json` (lines 5, 74–75, 81–82, 88–89)
The recapture preserves the existing `note` field as-is, so the
`note` rewrite happens here, *before* `bench/check.py --update-baseline`
runs. The 6 latency entries must also be dropped here — leaving them
in the file would cause `write_new_baseline` to faithfully re-baseline
them on today's measurements (recon Open Q).
- [ ] **Step 1: Rewrite the `note` field**
Edit `bench/baseline.json` line 5 — the entire `"note": "..."` value —
to the following exact string (single line in the JSON; newlines below
are for readability only; the JSON value must NOT contain literal
newlines, escape them as needed or keep it one logical sentence):
```
Baseline for bench/check.py regression detection. The language-invariant thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are NOT the regression-check tolerances; the per-metric tolerances below are tuned to absorb run-to-run noise on a quiet developer machine. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the commit body that ships the baseline bump. The latency arms gate only on median / p99 / p99_over_median — max_us and p99_9_us were removed on the 2026-05-20 recapture (Gitea #15 / #16) because tail-of-distribution latency metrics are dominated by OS-level jitter (THP defrag, scheduler preemption, IRQ load), not allocator behaviour, and produced 3+ consecutive false-positive REGRESSION rows on byte-identical no-op milestones. See docs/specs/2026-05-20-bench-harness-recalibration.md.
```
- [ ] **Step 2: Delete the 6 latency entries**
Open `bench/baseline.json` and delete these six lines (line numbers
relative to the pre-edit file; the editor's "find this exact JSON
key" is sufficient — line numbers are guidance only):
- Line 74: ` "p99_9_us": { "baseline": 8131.2, "tolerance_pct": 25 },`
- Line 75: ` "max_us": { "baseline": 8343.7, "tolerance_pct": 25 },`
- Line 81: ` "p99_9_us": { "baseline": 404.1, "tolerance_pct": 25 },`
- Line 82: ` "max_us": { "baseline": 413.0, "tolerance_pct": 25 },`
- Line 88: ` "p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },`
- Line 89: ` "max_us": { "baseline": 477.3, "tolerance_pct": 30 },`
For each removed line, also remove its trailing comma from the
previous line if needed to keep the JSON valid (the entry above
`p99_9_us` is `p99_us`, which currently has a trailing comma; after
the deletion, `p99_us` is fine as-is because `p99_over_median`
follows it — re-check the comma chain per arm after edit).
- [ ] **Step 3: Verify JSON validity**
Run:
```bash
python3 -m json.tool bench/baseline.json > /dev/null && echo OK
```
Expected stdout: `OK`. Any other output (a Python traceback) means
the JSON is malformed — fix the comma chain.
- [ ] **Step 4: Verify the 6 entries are gone**
Run:
```bash
jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json
```
Expected stdout:
```
[
"median_us",
"p99_over_median",
"p99_us"
]
```
If `max_us` or `p99_9_us` is in the list, Step 2 was incomplete.
- [ ] **Step 5: Verify the `note` field rewrite landed**
Run:
```bash
jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'
```
Expected stdout: `1`. Zero hits means the rewrite from Step 1 was
not saved.
---
### Task 2: Recapture remaining baselines via `--update-baseline`
**Files:**
- Modify: `bench/baseline.json` (all `baseline` numerics + `captured` + `captured_via`)
- [ ] **Step 1: Confirm HEAD is the bench-relevant clean state**
Run:
```bash
git status --short && git log -1 --oneline
```
Expected stdout includes `M bench/baseline.json` (or `M bench/baseline.json` — staged or unstaged from Task 1) as the only change. HEAD should be a workflow / spec / plan commit, NOT a runtime-touching commit. If any `crates/` or `runtime/` file is modified or HEAD is a runtime-touching commit, STOP and re-evaluate — the recapture must run from a clean bench-relevant state.
- [ ] **Step 2: Run the recapture**
Run:
```bash
bench/check.py --update-baseline
```
Expected: takes ~3-4 minutes (release-mode build of all bench fixtures + `-n 5` runs across throughput + latency arms). Final stderr line: `>>> wrote new baseline to .../bench/baseline.json`. Exit code 0.
- [ ] **Step 3: Verify the recapture preserved structure**
Run:
```bash
jq '.captured, .captured_via' bench/baseline.json
```
Expected stdout:
```
"2026-05-20"
"bench/run.sh -n 5"
```
(Or whatever today's date is when the implement run executes — the field is `$(date +%Y-%m-%d)` per `bench/check.py:255`.)
Also confirm note + entry deletions still hold:
```bash
jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'
jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json
```
Expected: `1` and the 3-element list `["median_us", "p99_over_median", "p99_us"]`. If either of those checks fails, `--update-baseline` somehow regenerated the dropped entries (would mean the Task-1 deletion didn't actually land before recapture) — STOP, restart from Task 1.
---
### Task 3: Acceptance §1 — same-HEAD replay → exit 0
**Files:** None modified. Read-only check.
- [ ] **Step 1: Run check.py and capture exit code**
Run:
```bash
bench/check.py; echo "EXIT=$?"
```
Expected: takes ~3 minutes (build cached from Task 2). Final summary line of stdout matches the pattern:
```
summary: 57 metrics; 0 regressed, <N> improved beyond tolerance, <M> stable
```
(57 = pre-edit 63 minus 6 removed entries; verify the actual reported count matches.)
Final line: `EXIT=0`.
Any other code is a failure — investigate the per-metric REGRESSION rows shown by `bench/check.py` and decide whether to (a) re-run once more (single-run noise) or (b) widen the offending tolerance after fresh investigation. Two consecutive non-zero exits mean the recapture itself was on a noisy run; redo Task 2.
---
### Task 4: Acceptance §2 — synthetic injection on bump_s → REGRESSION fires
**Files:** `bench/baseline.json` (temporary patch, reverted at end of task).
- [ ] **Step 1: Capture the current `bump_s` baseline value**
Run:
```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```
Expected stdout: a single float, the value captured in Task 2 (≈ 0.05 — exact number from the recapture). Save this number; call it `$ORIG`.
- [ ] **Step 2: Patch the baseline to half its value**
Run:
```bash
jq '.throughput.bench_list_sum.bump_s.baseline *= 0.5' bench/baseline.json > /tmp/baseline_injection.json && mv /tmp/baseline_injection.json bench/baseline.json
```
Verify the patch landed:
```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```
Expected stdout: approximately half of `$ORIG`.
- [ ] **Step 3: Run check.py against the patched baseline**
Run:
```bash
bench/check.py; echo "EXIT=$?"
```
Expected stdout includes a row matching this pattern (exact diff and tolerance vary):
```
throughput.bench_list_sum.bump_s <halved> <actual> +<~100>% 10.0% REGRESSION
```
And the last line is `EXIT=1`. If `EXIT=0`, the injection did not fire — investigate the gating path; the synthetic injection acceptance check has failed and the iter is BLOCKED. If `EXIT=1` and the row shows REGRESSION on the patched metric, the gate is functioning correctly.
- [ ] **Step 4: Restore the baseline to `$ORIG`**
Run:
```bash
jq --argjson v "$ORIG" '.throughput.bench_list_sum.bump_s.baseline = $v' bench/baseline.json > /tmp/baseline_restored.json && mv /tmp/baseline_restored.json bench/baseline.json
```
Where `$ORIG` is the value from Step 1. Verify:
```bash
jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json
```
Expected stdout: `$ORIG` (the pre-patch value, recaptured in Task 2).
- [ ] **Step 5: Final replay to confirm restoration**
Run:
```bash
bench/check.py; echo "EXIT=$?"
```
Expected last line: `EXIT=0` and `0 regressed` in the summary. Confirms the synthetic injection was fully reverted.
---
## Closing notes for `implement`
This is a one-task-conceptually iter split into four task-templates
for review granularity (each Task is one self-contained
spec-compliance unit). All four Tasks land in a single working-tree
diff (only `bench/baseline.json` is modified). Boss commits the
whole thing as one iter-level commit at the end.
Acceptance criteria from the spec are mapped to tasks:
- Spec §"max_us / p99_9_us removed" → Task 1.
- Spec §"captured field reads 2026-05-20" → Task 2.
- Spec §"replay exit 0" → Task 3.
- Spec §"synthetic injection fires REGRESSION" → Task 4.