Brummel/AILang
1
0
Fork 0
Code Issues 29 Pull Requests Actions Packages Projects Releases Wiki Activity

iter bench-harness-recalibration.1 (DONE 4/4): drop 6 latency entries + recapture baselines

Browse Source 
One-artefact iter on `bench/baseline.json`.  Closes Gitea #15 + #16
as documented in the spec commit `a97aaeb`.

Changes:

- `note` field rewritten forward-looking: drops the retired-JOURNAL
  workflow reference (Journal system deleted in 54f0ced) and the
  now-moot `*.max_us` tolerance convention (the metric being
  described is being removed in the same edit).  Points readers at
  `docs/specs/2026-05-20-bench-harness-recalibration.md` + closed
  issues #15 / #16 for rationale.
- 6 latency entries removed (`max_us` + `p99_9_us` × 3 arms:
  `implicit_at_gc`, `explicit_at_rc`, `implicit_at_rc`).  After the
  edit, `bench/check.py` gates only on `median_us` / `p99_us` /
  `p99_over_median` across the three latency arms.
- All remaining baselines (48 throughput + 9 latency = 57 metrics)
  regenerated by `bench/check.py --update-baseline` on HEAD
  `ddb50c3` from a fresh `bench/run.sh -n 5`.  `captured` field
  bumped 2026-05-09 → 2026-05-20.  Per-metric `tolerance_pct` and
  `version` preserved verbatim by `write_new_baseline`
  (`bench/check.py:249-284`).

Verification:

- Acceptance §1 (same-HEAD replay): `bench/check.py` exit 0, summary
  `57 metrics; 0 regressed, 0 improved, 57 stable` on first try
  (no single-run noise mitigation triggered).
- Acceptance §2 (synthetic injection): halving
  `throughput.bench_list_sum.bump_s.baseline` 0.053 → 0.026 fires
  REGRESSION row `+99.57% > 10.0%` with exit 1; restoration via
  `jq` returns the value to 0.053 and the next `bench/check.py`
  exits 0 (with 2 `improvement` rows on
  `latency.explicit_at_rc.p99_us` / `p99_over_median` at ≈-33% —
  improvements never gate; this is exactly the tail-distribution
  jitter that motivated dropping `max_us` / `p99_9_us`).

Diff shape:

- `bench/baseline.json` shows +230 / -67 lines.  Most of the line-
  count growth is the `json.dumps(indent=2)` pretty-print emitted
  by `--update-baseline` (`bench/check.py:283`) replacing the
  previous hand-formatted compact entries (each metric was one
  line; now each is six).  No behavioural delta — the file is
  semantically identical to a hand-edited version of the same
  changes.  Future `--update-baseline` runs will keep this expanded
  format.
- `bench/orchestrator-stats/2026-05-20-iter-bench-harness-recalibration.1.json`
  is the iter-stats artefact: outcome DONE, 0 re-loops across all
  4 tasks, the four bench-result fields documenting the acceptance
  outcomes.  Stats files are committed per `skills/implement/SKILL.md`
  Step 3.

No code change to `bench/check.py` / `bench/run.sh` /
`bench/latency_harness.py`.  No Rust crate touched.  `cargo test`
not exercised by this iter — the harness IS the test surface and
its replay+injection ARE the acceptance checks.

Follow-up noted (not addressed here):

- `skills/implement/agents/ailang-implement-orchestrator.md`
  references a Phase-4 per-iter journal write; the
  `docs/journals/` directory was retired in 54f0ced.  The
  orchestrator handled the mismatch by skipping the phase; the
  agent prompt itself is stale and should be edited in a follow-up
  tidy iter.

closes #15
closes #16
This commit is contained in:
Brummel
2026-05-20 16:15:13 +02:00
parent ddb50c3cb3
commit 21cfccf846
2 changed files with 260 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bench/baseline.json
+230 -67
View File
@@ -1,93 +1,256 @@
{
"version": 1,
"captured": "2026-05-09",
"captured": "2026-05-20",
"captured_via": "bench/run.sh -n 5",
"note": "Baseline for bench/check.py regression detection. Decision-10 thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are LANGUAGE invariants, not regression-check tolerances. The tolerances below are tuned to absorb run-to-run noise on a quiet developer machine; they are NOT the correctness bar. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the JOURNAL entry that ships the baseline bump. Tolerance convention: max-of-distribution metrics (e.g. *.max_us) get a wider band than percentile metrics (p99, p99_9) because the maximum of a 1000-sample tail-latency distribution has wider natural run-to-run dispersion than a percentile-of-distribution does — codified after the 21'd implicit_at_rc.max_us 25%->30% recalibration.",
"note": "Baseline for bench/check.py regression detection. The language-invariant thresholds (rc/bump <= 1.3x throughput, p99/median <= 5x latency) are NOT the regression-check tolerances; the per-metric tolerances below are tuned to absorb run-to-run noise on a quiet developer machine. To update after an intentional change, re-run bench/run.sh and replace the values, recording the reason in the commit body that ships the baseline bump. The latency arms gate only on median / p99 / p99_over_median — max_us and p99_9_us were removed on the 2026-05-20 recapture (Gitea #15 / #16) because tail-of-distribution latency metrics are dominated by OS-level jitter (THP defrag, scheduler preemption, IRQ load), not allocator behaviour, and produced 3+ consecutive false-positive REGRESSION rows on byte-identical no-op milestones. See docs/specs/2026-05-20-bench-harness-recalibration.md.",
"throughput": {
"bench_list_sum": {
"gc_s": { "baseline": 0.137, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.046, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.133, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.98, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 2.89, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 137448, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 97696, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 193448, "tolerance_pct": 5 }
"gc_s": {
"baseline": 0.150576,
"tolerance_pct": 10
},
"bump_s": {
"baseline": 0.052977,
"tolerance_pct": 10
},
"rc_s": {
"baseline": 0.143404,
"tolerance_pct": 10
},
"gc_over_bump": {
"baseline": 2.84,
"tolerance_pct": 8
},
"rc_over_bump": {
"baseline": 2.71,
"tolerance_pct": 8
},
"gc_rss_kb": {
"baseline": 137636.0,
"tolerance_pct": 5
},
"bump_rss_kb": {
"baseline": 97436.0,
"tolerance_pct": 5
},
"rc_rss_kb": {
"baseline": 193628.0,
"tolerance_pct": 5
}
},
"bench_tree_walk": {
"gc_s": { "baseline": 0.103, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.038, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.095, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.71, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 2.50, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 73260, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 55208, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 108956, "tolerance_pct": 5 }
"gc_s": {
"baseline": 0.101088,
"tolerance_pct": 10
},
"bump_s": {
"baseline": 0.038371,
"tolerance_pct": 10
},
"rc_s": {
"baseline": 0.098525,
"tolerance_pct": 10
},
"gc_over_bump": {
"baseline": 2.63,
"tolerance_pct": 8
},
"rc_over_bump": {
"baseline": 2.57,
"tolerance_pct": 8
},
"gc_rss_kb": {
"baseline": 73316.0,
"tolerance_pct": 5
},
"bump_rss_kb": {
"baseline": 55196.0,
"tolerance_pct": 5
},
"rc_rss_kb": {
"baseline": 108764.0,
"tolerance_pct": 5
}
},
"bench_closure_chain": {
"gc_s": { "baseline": 0.013, "tolerance_pct": 25 },
"bump_s": { "baseline": 0.007, "tolerance_pct": 25 },
"rc_s": { "baseline": 0.029, "tolerance_pct": 20 },
"gc_over_bump": { "baseline": 1.86, "tolerance_pct": 15 },
"rc_over_bump": { "baseline": 4.14, "tolerance_pct": 15 },
"gc_rss_kb": { "baseline": 13688, "tolerance_pct": 15 },
"bump_rss_kb": { "baseline": 15836, "tolerance_pct": 15 },
"rc_rss_kb": { "baseline": 39644, "tolerance_pct": 10 }
"gc_s": {
"baseline": 0.012135,
"tolerance_pct": 25
},
"bump_s": {
"baseline": 0.007945,
"tolerance_pct": 25
},
"rc_s": {
"baseline": 0.031716,
"tolerance_pct": 20
},
"gc_over_bump": {
"baseline": 1.53,
"tolerance_pct": 15
},
"rc_over_bump": {
"baseline": 3.99,
"tolerance_pct": 15
},
"gc_rss_kb": {
"baseline": 13684.0,
"tolerance_pct": 15
},
"bump_rss_kb": {
"baseline": 15836.0,
"tolerance_pct": 15
},
"rc_rss_kb": {
"baseline": 39652.0,
"tolerance_pct": 10
}
},
"bench_hof_pipeline": {
"gc_s": { "baseline": 0.134, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.048, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.136, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.79, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 2.83, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 137768, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 97448, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 193640, "tolerance_pct": 5 }
"gc_s": {
"baseline": 0.149062,
"tolerance_pct": 10
},
"bump_s": {
"baseline": 0.051993,
"tolerance_pct": 10
},
"rc_s": {
"baseline": 0.143367,
"tolerance_pct": 10
},
"gc_over_bump": {
"baseline": 2.87,
"tolerance_pct": 8
},
"rc_over_bump": {
"baseline": 2.76,
"tolerance_pct": 8
},
"gc_rss_kb": {
"baseline": 137564.0,
"tolerance_pct": 5
},
"bump_rss_kb": {
"baseline": 97628.0,
"tolerance_pct": 5
},
"rc_rss_kb": {
"baseline": 193816.0,
"tolerance_pct": 5
}
},
"bench_compute_collatz": {
"gc_s": { "baseline": 0.057, "tolerance_pct": 12 },
"bump_s": { "baseline": 0.056, "tolerance_pct": 12 },
"rc_s": { "baseline": 0.056, "tolerance_pct": 12 },
"gc_over_bump": { "baseline": 1.02, "tolerance_pct": 10 },
"rc_over_bump": { "baseline": 1.00, "tolerance_pct": 10 },
"gc_rss_kb": { "baseline": 13624, "tolerance_pct": 15 },
"bump_rss_kb": { "baseline": 13860, "tolerance_pct": 15 },
"rc_rss_kb": { "baseline": 13880, "tolerance_pct": 15 }
"gc_s": {
"baseline": 0.056628,
"tolerance_pct": 12
},
"bump_s": {
"baseline": 0.056266,
"tolerance_pct": 12
},
"rc_s": {
"baseline": 0.05629,
"tolerance_pct": 12
},
"gc_over_bump": {
"baseline": 1.01,
"tolerance_pct": 10
},
"rc_over_bump": {
"baseline": 1.0,
"tolerance_pct": 10
},
"gc_rss_kb": {
"baseline": 13860.0,
"tolerance_pct": 15
},
"bump_rss_kb": {
"baseline": 14044.0,
"tolerance_pct": 15
},
"rc_rss_kb": {
"baseline": 14012.0,
"tolerance_pct": 15
}
},
"bench_list_sum_explicit": {
"gc_s": { "baseline": 0.139, "tolerance_pct": 10 },
"bump_s": { "baseline": 0.048, "tolerance_pct": 10 },
"rc_s": { "baseline": 0.152, "tolerance_pct": 10 },
"gc_over_bump": { "baseline": 2.89, "tolerance_pct": 8 },
"rc_over_bump": { "baseline": 3.14, "tolerance_pct": 8 },
"gc_rss_kb": { "baseline": 137636, "tolerance_pct": 5 },
"bump_rss_kb": { "baseline": 97636, "tolerance_pct": 5 },
"rc_rss_kb": { "baseline": 142424, "tolerance_pct": 8 }
"gc_s": {
"baseline": 0.1506,
"tolerance_pct": 10
},
"bump_s": {
"baseline": 0.053389,
"tolerance_pct": 10
},
"rc_s": {
"baseline": 0.15632,
"tolerance_pct": 10
},
"gc_over_bump": {
"baseline": 2.82,
"tolerance_pct": 8
},
"rc_over_bump": {
"baseline": 2.93,
"tolerance_pct": 8
},
"gc_rss_kb": {
"baseline": 137196.0,
"tolerance_pct": 5
},
"bump_rss_kb": {
"baseline": 97628.0,
"tolerance_pct": 5
},
"rc_rss_kb": {
"baseline": 141988.0,
"tolerance_pct": 8
}
}
},
"latency": {
"implicit_at_gc": {
"median_us": { "baseline": 96.4, "tolerance_pct": 15 },
"p99_us": { "baseline": 7130.0, "tolerance_pct": 20 },
"p99_9_us": { "baseline": 8131.2, "tolerance_pct": 25 },
"max_us": { "baseline": 8343.7, "tolerance_pct": 25 },
"p99_over_median": { "baseline": 73.92, "tolerance_pct": 20 }
"median_us": {
"baseline": 97.6,
"tolerance_pct": 15
},
"p99_us": {
"baseline": 7125.8,
"tolerance_pct": 20
},
"p99_over_median": {
"baseline": 72.79,
"tolerance_pct": 20
}
},
"explicit_at_rc": {
"median_us": { "baseline": 213.9, "tolerance_pct": 15 },
"p99_us": { "baseline": 357.5, "tolerance_pct": 25 },
"p99_9_us": { "baseline": 404.1, "tolerance_pct": 25 },
"max_us": { "baseline": 413.0, "tolerance_pct": 25 },
"p99_over_median": { "baseline": 1.66, "tolerance_pct": 25 }
"median_us": {
"baseline": 221.2,
"tolerance_pct": 15
},
"p99_us": {
"baseline": 428.8,
"tolerance_pct": 25
},
"p99_over_median": {
"baseline": 1.93,
"tolerance_pct": 25
}
},
"implicit_at_rc": {
"median_us": { "baseline": 285.7, "tolerance_pct": 15 },
"p99_us": { "baseline": 407.1, "tolerance_pct": 20 },
"p99_9_us": { "baseline": 452.0, "tolerance_pct": 25 },
"max_us": { "baseline": 477.3, "tolerance_pct": 30 },
"p99_over_median": { "baseline": 1.43, "tolerance_pct": 20 }
"median_us": {
"baseline": 301.2,
"tolerance_pct": 15
},
"p99_us": {
"baseline": 461.8,
"tolerance_pct": 20
},
"p99_over_median": {
"baseline": 1.53,
"tolerance_pct": 20
}
}
}
}
bench/orchestrator-stats/2026-05-20-iter-bench-harness-recalibration.1.json
+30
View File
@@ -0,0 +1,30 @@
{
"iter_id": "bench-harness-recalibration.1",
"date": "2026-05-20",
"mode": "standard",
"outcome": "DONE",
"tasks_total": 4,
"tasks_completed": 4,
"reloops_per_task": {
"1": 0,
"2": 0,
"3": 0,
"4": 0
},
"review_loops_spec": 0,
"review_loops_quality": 0,
"blocked_reason": null,
"bench_results": {
"check_py_exit_task3_replay": 0,
"check_py_summary_task3": "57 metrics; 0 regressed, 0 improved beyond tolerance, 57 stable",
"check_py_exit_task4_injection": 1,
"check_py_injection_row": "throughput.bench_list_sum.bump_s 0.026 0.053 +99.57% 10.0% REGRESSION",
"check_py_exit_task4_restored": 0,
"check_py_summary_task4_restored": "57 metrics; 0 regressed, 2 improved beyond tolerance, 55 stable"
},
"notes": [
"Task 3 same-HEAD replay passed on first try (no re-run needed for single-run noise mitigation).",
"Task 4 restored replay shows 2 improvements on latency.explicit_at_rc.p99_us / p99_over_median (-33%) — improvements never gate per bench/check.py header; this is exactly the tail-distribution jitter that motivated dropping max_us / p99_9_us.",
"All four tasks land as a single working-tree diff on bench/baseline.json (single-artefact iter)."
]
}