claude
400105e2e2
Tasks 7-8 of the bench-harness plan plus the orchestrator's sizing pass — this completes the phase-1 harness (closes #251). Driver: run [--surface|--quick|--reps|--out] / pin over five surfaces, report-only comparison (drift NOTICE at >=10%, never a failure), fingerprint mismatch exits 1 with a baseline-vs-measured block, infra errors exit 2, debug builds refuse to measure. The Bench project-facts line and the crate README (discipline: quiet box, warmup+median, deliberate re-pins) land with it. Workload sizing, measured then corrected: the release engine outran the plan's guesses by ~100x, leaving sub-second child walls where spawn jitter reaches the 10% threshold and flaps the report. Full-size workloads now land in whole seconds — engine 10M bars (~0.7s), ingest 24 months x 20 fresh drains (~0.4s), campaigns on a 24-month archive with a ~22-month window and a 5x5 member grid (sweep ~1.4s, heavy ~5.5s), fixed-cost as in-rep medians over spawn batches (20x help, 10x run). Quick mode keeps the small E2E-fixture shape. The campaign fingerprint folds all three realization layers — sweep winner, gate survivor ordinals, bootstrap trade counts (pooled or per-survivor sum) — so a wrong-result regression in any stage fails the bench, not just a wrong sweep winner. Baselines pinned on this host (commit-stamped), verified by a follow-up run: five fingerprints OK at <=2% drift under load. Verification: cargo test --workspace 1379 green (bench tests all quick-mode; the release-binary E2Es are #[ignore]d behind -- --ignored per the suite-wallclock discipline), clippy -D warnings clean, doc build clean.
48 lines
2.4 KiB
Markdown
48 lines
2.4 KiB
Markdown
# aura-bench
|
|
|
|
Wall-clock benchmarks with committed baselines (issue #251, phase 1). Catches
|
|
hot-path regressions the E2E suite structurally cannot see: every surface pins
|
|
its inputs and asserts a determinism fingerprint, so a fast-but-wrong binary
|
|
fails the bench instead of pinning a bogus baseline.
|
|
|
|
## Running
|
|
|
|
```console
|
|
cargo run --release -p aura-bench -- run # measure + compare, report-only
|
|
cargo run --release -p aura-bench -- run --surface engine_throughput
|
|
cargo run --release -p aura-bench -- pin # rewrite committed baselines
|
|
```
|
|
|
|
Debug builds refuse to measure (exit 2) — wall-clock numbers from a debug
|
|
build measure the profile, not the code.
|
|
|
|
## Surfaces
|
|
|
|
| surface | workload | metrics |
|
|
|---|---|---|
|
|
| `engine_throughput` | 10M synthetic bars through an SMA-cross harness (library) | `wall_s`, `bars_per_s` |
|
|
| `ingest_throughput` | 24 synthetic archive months x 20 fresh drains via the ingest window API (library) | `wall_s`, `bars_per_s` |
|
|
| `campaign_sweep` | 8-cell sweep-only campaign, release `aura` binary on a scratch project | `wall_s`, `cpu_percent`, `peak_rss_mb` |
|
|
| `campaign_heavy` | same cells; sweep → gate → walk-forward → 1000-resample bootstrap | `wall_s`, `cpu_percent`, `peak_rss_mb` |
|
|
| `cli_fixed_cost` | spawn floor (`aura --help`) + minimal data-only project run | `help_ms`, `run_ms` |
|
|
|
|
## Discipline
|
|
|
|
- Quiet box: the driver warns at 1-min loadavg > 2.0. Close heavy jobs first.
|
|
- 1 discarded warmup + 3 measured reps (median). Reps must agree on the
|
|
fingerprint — cross-rep drift is reported as nondeterminism, not averaged.
|
|
- Baselines are per-host context (`host` block inside each JSON): comparison on
|
|
a different host prints an informational banner and stays report-only.
|
|
- Timing drift NEVER fails the run (report-only phase, #251 cadence decision);
|
|
a NOTICE line flags drift >= 10%. A fingerprint mismatch exits 1.
|
|
- Re-pin (`pin`) deliberately, after an accepted behaviour or performance
|
|
change; the baseline diff is reviewed like any other commit.
|
|
|
|
## Deferred (follow-up on the tracker)
|
|
|
|
- Instruction-count backend (iai-callgrind) once valgrind is installed on the
|
|
measuring host — slots in as a sibling backend without reshaping baselines.
|
|
- Native-crate project-load sub-cost (dylib SHA-256) — needs a build step in
|
|
the scratch project.
|
|
- Thresholded failure, once recorded variance data justifies thresholds.
|