Brummel
832375f2ac
All 176 files in the four accumulating directories now use a zero-padded 4-digit counter prefix that reflects creation order (`NNNN-slug.md`). The counter is assigned per directory in strict git-log creation order; ties broken alphabetically by original name. The old `YYYY-MM-DD-` prefix on docs/specs/ and docs/plans/ files is dropped — the date is recoverable from git log and the counter carries the ordering. A file's counter is stable for the life of the file: never reassigned, never reused, never compacted. Deleted files retire their counter; subsequent files do not fill the gap. This is the property that lets cross-references stay literal — refs use the full filename including the counter (`design/contracts/0007-honesty-rule.md`) so they grep cleanly and resolve directly without a glob step. 313 cross-references updated across .md/.rs/.toml/.c/.json files (test pins, include_str! paths, design-INDEX entries, baseline notes, runtime C comments, inter-contract markdown links incl. bare basename and `../models/foo.md` forms). CLAUDE.md gets a new "File-naming convention" section spelling out the rule and rationale. skills/brainstorm/SKILL.md and skills/planner/SKILL.md updated so new spec/plan creation produces counter-prefixed names from the start. The full test suite (cargo test --workspace) passes.
170 lines
6.9 KiB
Markdown
170 lines
6.9 KiB
Markdown
# Bench harness recalibration — degate `max_us` / `p99_9_us`, re-capture baselines
|
||
|
||
**Date:** 2026-05-20
|
||
**Status:** Draft — awaiting user spec review
|
||
**Authors:** Brummel (orchestrator) + Claude
|
||
|
||
## Goal
|
||
|
||
End the recurring bench false-positive on no-op milestones by:
|
||
|
||
1. Removing the two structurally jitter-prone latency metrics
|
||
(`max_us`, `p99_9_us`) from `bench/baseline.json`, so
|
||
`bench/check.py` no longer gates them.
|
||
2. Re-capturing the remaining baselines from current HEAD, so the
|
||
environmental drift accumulated since 2026-05-09 is absorbed in
|
||
one honest cut rather than hidden under widened tolerances.
|
||
|
||
Closes Gitea #15 (`*.bump_s` baseline stale) and #16 (`*.max_us`
|
||
structural false-positive). The two issues frame distinct symptoms,
|
||
but reproduction on 2026-05-20 (HEAD `19321d8`, two consecutive
|
||
`bench/check.py -n 5` runs, both regression-firing) showed:
|
||
|
||
- The latency false-positive in `implicit_at_rc.max_us` is no longer
|
||
the stochastic +108%→ok-on-rerun spike #16 documented; it now
|
||
reproduces deterministically at +47% / +49% across runs.
|
||
`p99_9_us` on the same arm shows the same pattern (+28% / +42%).
|
||
The metrics are now drift-elevated as well as jitter-prone.
|
||
- The throughput drift is wider than #15 framed it: `gc_s` on
|
||
`bench_hof_pipeline` also fires (+10.6% / +10.9% across runs),
|
||
and `bench_list_sum.gc_s` sits at +9.9% (just under the gate).
|
||
Drift correlates with memory pressure / allocation density, not
|
||
with metric family — `bench_tree_walk` and
|
||
`bench_compute_collatz` are clean.
|
||
|
||
Both issues are therefore one milestone, addressed by one cohesive
|
||
change.
|
||
|
||
## Architecture
|
||
|
||
No code change. Two artefacts in the working tree:
|
||
|
||
| File | Change |
|
||
|------|--------|
|
||
| `bench/baseline.json` | Drop 6 entries (`max_us` + `p99_9_us` × 3 latency arms); regenerate every remaining `baseline` value from current HEAD via `bench/check.py --update-baseline`; update the top-level `note` field |
|
||
|
||
`bench/check.py` is unmodified. The harness already iterates only
|
||
the entries present in `baseline.json` (`collect_measurements`
|
||
function, lines 177–220) — removing an entry removes it from both
|
||
gating and the per-metric report, no further wiring needed.
|
||
|
||
`bench/run.sh` and `bench/latency_harness.py` are unmodified. The
|
||
removed metrics continue to appear upstream in `bench/run.sh`
|
||
output (the latency harness computes them independently of
|
||
`baseline.json`); only the regression-check layer no longer sees
|
||
them. A developer triaging a regression can still read `max_us`
|
||
and `p99.9` directly from the bench-run table.
|
||
|
||
## Concrete code shapes
|
||
|
||
### The artefact this milestone delivers — `bench/baseline.json` delta
|
||
|
||
Per latency arm, two entries are removed. Before (current
|
||
`implicit_at_rc`, lines 85–91):
|
||
|
||
```json
|
||
"implicit_at_rc": {
|
||
"median_us": { "baseline": 285.7, "tolerance_pct": 15 },
|
||
"p99_us": { "baseline": 407.1, "tolerance_pct": 20 },
|
||
"p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },
|
||
"max_us": { "baseline": 477.3, "tolerance_pct": 30 },
|
||
"p99_over_median": { "baseline": 1.43, "tolerance_pct": 20 }
|
||
}
|
||
```
|
||
|
||
After (same arm, the two unreliable metrics dropped; remaining
|
||
baselines re-captured on 2026-05-20 hardware):
|
||
|
||
```json
|
||
"implicit_at_rc": {
|
||
"median_us": { "baseline": <recaptured>, "tolerance_pct": 15 },
|
||
"p99_us": { "baseline": <recaptured>, "tolerance_pct": 20 },
|
||
"p99_over_median": { "baseline": <recaptured>, "tolerance_pct": 20 }
|
||
}
|
||
```
|
||
|
||
Same shape for `implicit_at_gc` and `explicit_at_rc`. Total: 6
|
||
entries removed, all remaining latency + throughput baseline
|
||
numbers refreshed.
|
||
|
||
Top-level `note` field updated to record the recapture date, the
|
||
two removed metric families, and a one-line pointer to this spec
|
||
and the closed issues.
|
||
|
||
### The bench programs this milestone exists to serve
|
||
|
||
The infrastructure measures the existing AILang fixtures under
|
||
`examples/`. Two representative ones — `bench_list_sum.ail`
|
||
(allocation-intensive list workload, where the throughput drift
|
||
reproduces) and `bench_latency_implicit.ail` (the RC-drop-latency
|
||
workload, built by `run.sh` into the `_rc` variant where the
|
||
removed `max_us` / `p99_9_us` live) — are the north-star programs
|
||
this harness reports on. They are unchanged by this milestone;
|
||
only the gating around them shifts.
|
||
|
||
## Components
|
||
|
||
Only `bench/baseline.json` is touched. No Rust crate change, no
|
||
runtime change, no CLI change.
|
||
|
||
## Data flow
|
||
|
||
`bench/run.sh` → text table on stdout → `bench/check.py` parses →
|
||
diffs against `bench/baseline.json` → exit code. The path is
|
||
identical before and after. Removing entries from `baseline.json`
|
||
means `collect_measurements` does not emit `Measurement` records
|
||
for those metric names, so they neither appear in the report nor
|
||
contribute to the regression count.
|
||
|
||
## Error handling
|
||
|
||
None new. `bench/check.py`'s existing "baselined metric not found
|
||
in bench output" sys.exit(2) path continues to cover the case
|
||
where the bench output format breaks. The removed metrics are not
|
||
baselined, so their absence in output is not an error.
|
||
|
||
## Testing strategy
|
||
|
||
Two acceptance checks, both run by hand at iter close:
|
||
|
||
1. **Replay-pass.** After the recapture commit lands, immediate
|
||
`bench/check.py` run on the same HEAD returns exit 0 with zero
|
||
`REGRESSION` rows. (The recapture writes baselines to today's
|
||
measurement; a same-HEAD replay should fall inside every
|
||
per-metric tolerance.)
|
||
2. **Detection-still-works (synthetic injection).** Temporarily
|
||
patch `bench/baseline.json` to drop `bench_list_sum.bump_s`
|
||
baseline by 50% (e.g. `0.046` → `0.023`), re-run
|
||
`bench/check.py`, confirm exit 1 and one `REGRESSION` row on
|
||
that metric. Revert the patch. This protects against a
|
||
recapture that accidentally inflates tolerances or removes
|
||
gating beyond the intended two metric families.
|
||
|
||
No new automated test is introduced — the bench harness is an
|
||
out-of-tree Python tool with no Rust test surface, and adding a
|
||
test crate for it is scope-disproportionate for this iteration.
|
||
|
||
## Acceptance criteria
|
||
|
||
- `bench/baseline.json` no longer contains any `max_us` or
|
||
`p99_9_us` entry (verifiable: `jq '.. | objects | keys' <
|
||
bench/baseline.json | grep -E "max_us|p99_9_us"` returns nothing).
|
||
- `bench/baseline.json` `captured` field reads `2026-05-20`;
|
||
`note` documents the removed metrics and links this spec.
|
||
- `bench/check.py` replay on the recapture HEAD exits 0 with
|
||
`0 regressed`.
|
||
- Synthetic injection test (Acceptance §2) confirms detection
|
||
still fires on a plausible real regression.
|
||
|
||
Out of scope, explicitly:
|
||
|
||
- Histogram-based latency methodology rework (Gitea #19) — a
|
||
separate brainstorm; this milestone is a stop-gap that buys back
|
||
the noise floor.
|
||
- Latency-arm cpuset / `isolcpus=` pinning — sysadmin-layer fix
|
||
that does not survive CI-move or new dev machine.
|
||
- `tolerance_pct` widening as a substitute for recapture — hides
|
||
the drift rather than absorbing it honestly.
|
||
- Changes to `bench/check.py`, `bench/run.sh`, or
|
||
`bench/latency_harness.py`.
|