AILang/docs/plans/bench-harness-recalibration.1.md

Bench harness recalibration — Implementation Plan (iteration 1, terminal)

Parent spec: docs/specs/2026-05-20-bench-harness-recalibration.md

For agentic workers: REQUIRED SUB-SKILL: use skills/implement to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Close Gitea #15 + #16 by removing max_us and p99_9_us from bench/baseline.json (6 entries across 3 latency arms) and regenerating the remaining baselines from current HEAD, so bench/check.py stops false-positive-firing on environmentally drifted no-op milestones.

Architecture: Single artefact (bench/baseline.json); no code change. bench/check.py's --update-baseline mode preserves version / note / per-metric tolerance_pct and overwrites only the baseline numeric + captured date + captured_via. The 6 unreliable entries are dropped before --update-baseline (the recapture iterates existing.get(...).items() and would faithfully re-baseline any entry left in the file). The rewritten note field is committed in the same pre-recapture edit so --update-baseline's note-preservation logic carries the new text forward.

Tech Stack: bench/baseline.json (data file only). bench/check.py (read-only; invoked twice — once to recapture, once to replay-verify) and bench/run.sh (read-only; invoked by bench/check.py for both runs). No Rust crate touched.

---

Files this plan creates or modifies:

• Modify: bench/baseline.json — the sole artefact.
  • note (line 5): rewrite to drop stale references (JOURNAL workflow retired 8e586f4, *.max_us tolerance convention now-moot) and add forward pointer to this spec + closed issues #15 / #16.
  • latency.implicit_at_gc (lines 71–77): delete p99_9_us (line 74) and max_us (line 75) entries.
  • latency.explicit_at_rc (lines 78–84): delete p99_9_us (line 81) and max_us (line 82) entries.
  • latency.implicit_at_rc (lines 85–91): delete p99_9_us (line 88) and max_us (line 89) entries.
  • throughput.*.baseline (lines 7–67): regenerated by bench/check.py --update-baseline against today's bench run.
  • latency.*.baseline (remaining 3 metrics × 3 arms = 9 values after drops): regenerated by the same recapture.
  • captured (line 3): 2026-05-09 → 2026-05-20 (auto, via bench/check.py:255).
  • captured_via (line 4): set to bench/run.sh -n 5 (auto, via bench/check.py:305).
• Read-only verification target: bench/check.py:177-220 (collect_measurements — iterates only baseline.json entries; the JSON drop is sufficient).
• Read-only verification target: bench/check.py:249-284 (write_new_baseline — preserves version / note / tolerance_pct).

No other file changes. No Rust crate touched.

---

Task 1: Pre-recapture JSON edit — drop 6 entries + rewrite note

Files:

• Modify: bench/baseline.json (lines 5, 74–75, 81–82, 88–89)

The recapture preserves the existing note field as-is, so the note rewrite happens here, before bench/check.py --update-baseline runs. The 6 latency entries must also be dropped here — leaving them in the file would cause write_new_baseline to faithfully re-baseline them on today's measurements (recon Open Q).

• Step 1: Rewrite the note field

Edit bench/baseline.json line 5 — the entire "note": "..." value — to the following exact string (single line in the JSON; newlines below are for readability only; the JSON value must NOT contain literal newlines, escape them as needed or keep it one logical sentence):

Baseline for bench/check.py regression detection. The language-invariant thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are NOT the regression-check tolerances; the per-metric tolerances below are tuned to absorb run-to-run noise on a quiet developer machine. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the commit body that ships the baseline bump. The latency arms gate only on median / p99 / p99_over_median — max_us and p99_9_us were removed on the 2026-05-20 recapture (Gitea #15 / #16) because tail-of-distribution latency metrics are dominated by OS-level jitter (THP defrag, scheduler preemption, IRQ load), not allocator behaviour, and produced 3+ consecutive false-positive REGRESSION rows on byte-identical no-op milestones. See docs/specs/2026-05-20-bench-harness-recalibration.md.

• Step 2: Delete the 6 latency entries

Open bench/baseline.json and delete these six lines (line numbers relative to the pre-edit file; the editor's "find this exact JSON key" is sufficient — line numbers are guidance only):

• Line 74: "p99_9_us": { "baseline": 8131.2, "tolerance_pct": 25 },
• Line 75: "max_us": { "baseline": 8343.7, "tolerance_pct": 25 },
• Line 81: "p99_9_us": { "baseline": 404.1, "tolerance_pct": 25 },
• Line 82: "max_us": { "baseline": 413.0, "tolerance_pct": 25 },
• Line 88: "p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },
• Line 89: "max_us": { "baseline": 477.3, "tolerance_pct": 30 },

For each removed line, also remove its trailing comma from the previous line if needed to keep the JSON valid (the entry above p99_9_us is p99_us, which currently has a trailing comma; after the deletion, p99_us is fine as-is because p99_over_median follows it — re-check the comma chain per arm after edit).

• Step 3: Verify JSON validity

Run:

python3 -m json.tool bench/baseline.json > /dev/null && echo OK

Expected stdout: OK. Any other output (a Python traceback) means the JSON is malformed — fix the comma chain.

• Step 4: Verify the 6 entries are gone

Run:

jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json

Expected stdout:

[
  "median_us",
  "p99_over_median",
  "p99_us"
]

If max_us or p99_9_us is in the list, Step 2 was incomplete.

• Step 5: Verify the note field rewrite landed

Run:

jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'

Expected stdout: 1. Zero hits means the rewrite from Step 1 was not saved.

---

Task 2: Recapture remaining baselines via --update-baseline

Files:

• Modify: bench/baseline.json (all baseline numerics + captured + captured_via)

• Step 1: Confirm HEAD is the bench-relevant clean state

Run:

git status --short && git log -1 --oneline

Expected stdout includes M bench/baseline.json (or M bench/baseline.json — staged or unstaged from Task 1) as the only change. HEAD should be a workflow / spec / plan commit, NOT a runtime-touching commit. If any crates/ or runtime/ file is modified or HEAD is a runtime-touching commit, STOP and re-evaluate — the recapture must run from a clean bench-relevant state.

• Step 2: Run the recapture

Run:

bench/check.py --update-baseline

Expected: takes ~3-4 minutes (release-mode build of all bench fixtures + -n 5 runs across throughput + latency arms). Final stderr line: >>> wrote new baseline to .../bench/baseline.json. Exit code 0.

• Step 3: Verify the recapture preserved structure

Run:

jq '.captured, .captured_via' bench/baseline.json

Expected stdout:

"2026-05-20"
"bench/run.sh -n 5"

(Or whatever today's date is when the implement run executes — the field is $(date +%Y-%m-%d) per bench/check.py:255.)

Also confirm note + entry deletions still hold:

jq -r '.note' bench/baseline.json | grep -c 'docs/specs/2026-05-20-bench-harness-recalibration.md'
jq '[.latency | to_entries[] | .value | keys[]] | unique' bench/baseline.json

Expected: 1 and the 3-element list ["median_us", "p99_over_median", "p99_us"]. If either of those checks fails, --update-baseline somehow regenerated the dropped entries (would mean the Task-1 deletion didn't actually land before recapture) — STOP, restart from Task 1.

---

Task 3: Acceptance §1 — same-HEAD replay → exit 0

Files: None modified. Read-only check.

• Step 1: Run check.py and capture exit code

Run:

bench/check.py; echo "EXIT=$?"

Expected: takes ~3 minutes (build cached from Task 2). Final summary line of stdout matches the pattern:

summary: 57 metrics; 0 regressed, <N> improved beyond tolerance, <M> stable

(57 = pre-edit 63 minus 6 removed entries; verify the actual reported count matches.)

Final line: EXIT=0.

Any other code is a failure — investigate the per-metric REGRESSION rows shown by bench/check.py and decide whether to (a) re-run once more (single-run noise) or (b) widen the offending tolerance after fresh investigation. Two consecutive non-zero exits mean the recapture itself was on a noisy run; redo Task 2.

---

Task 4: Acceptance §2 — synthetic injection on bump_s → REGRESSION fires

Files: bench/baseline.json (temporary patch, reverted at end of task).

• Step 1: Capture the current bump_s baseline value

Run:

jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json

Expected stdout: a single float, the value captured in Task 2 (≈ 0.05 — exact number from the recapture). Save this number; call it $ORIG.

• Step 2: Patch the baseline to half its value

Run:

jq '.throughput.bench_list_sum.bump_s.baseline *= 0.5' bench/baseline.json > /tmp/baseline_injection.json && mv /tmp/baseline_injection.json bench/baseline.json

Verify the patch landed:

jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json

Expected stdout: approximately half of $ORIG.

• Step 3: Run check.py against the patched baseline

Run:

bench/check.py; echo "EXIT=$?"

Expected stdout includes a row matching this pattern (exact diff and tolerance vary):

throughput.bench_list_sum.bump_s                       <halved>    <actual>  +<~100>% 10.0%  REGRESSION

And the last line is EXIT=1. If EXIT=0, the injection did not fire — investigate the gating path; the synthetic injection acceptance check has failed and the iter is BLOCKED. If EXIT=1 and the row shows REGRESSION on the patched metric, the gate is functioning correctly.

• Step 4: Restore the baseline to $ORIG

Run:

jq --argjson v "$ORIG" '.throughput.bench_list_sum.bump_s.baseline = $v' bench/baseline.json > /tmp/baseline_restored.json && mv /tmp/baseline_restored.json bench/baseline.json

Where $ORIG is the value from Step 1. Verify:

jq '.throughput.bench_list_sum.bump_s.baseline' bench/baseline.json

Expected stdout: $ORIG (the pre-patch value, recaptured in Task 2).

• Step 5: Final replay to confirm restoration

Run:

bench/check.py; echo "EXIT=$?"

Expected last line: EXIT=0 and 0 regressed in the summary. Confirms the synthetic injection was fully reverted.

---

Closing notes for implement

This is a one-task-conceptually iter split into four task-templates for review granularity (each Task is one self-contained spec-compliance unit). All four Tasks land in a single working-tree diff (only bench/baseline.json is modified). Boss commits the whole thing as one iter-level commit at the end.

Acceptance criteria from the spec are mapped to tasks:

• Spec §"max_us / p99_9_us removed" → Task 1.
• Spec §"captured field reads 2026-05-20" → Task 2.
• Spec §"replay exit 0" → Task 3.
• Spec §"synthetic injection fires REGRESSION" → Task 4.