claude
400105e2e2
Tasks 7-8 of the bench-harness plan plus the orchestrator's sizing pass — this completes the phase-1 harness (closes #251). Driver: run [--surface|--quick|--reps|--out] / pin over five surfaces, report-only comparison (drift NOTICE at >=10%, never a failure), fingerprint mismatch exits 1 with a baseline-vs-measured block, infra errors exit 2, debug builds refuse to measure. The Bench project-facts line and the crate README (discipline: quiet box, warmup+median, deliberate re-pins) land with it. Workload sizing, measured then corrected: the release engine outran the plan's guesses by ~100x, leaving sub-second child walls where spawn jitter reaches the 10% threshold and flaps the report. Full-size workloads now land in whole seconds — engine 10M bars (~0.7s), ingest 24 months x 20 fresh drains (~0.4s), campaigns on a 24-month archive with a ~22-month window and a 5x5 member grid (sweep ~1.4s, heavy ~5.5s), fixed-cost as in-rep medians over spawn batches (20x help, 10x run). Quick mode keeps the small E2E-fixture shape. The campaign fingerprint folds all three realization layers — sweep winner, gate survivor ordinals, bootstrap trade counts (pooled or per-survivor sum) — so a wrong-result regression in any stage fails the bench, not just a wrong sweep winner. Baselines pinned on this host (commit-stamped), verified by a follow-up run: five fingerprints OK at <=2% drift under load. Verification: cargo test --workspace 1379 green (bench tests all quick-mode; the release-binary E2Es are #[ignore]d behind -- --ignored per the suite-wallclock discipline), clippy -D warnings clean, doc build clean.
29 lines
1.5 KiB
Rust
29 lines
1.5 KiB
Rust
//! Integration test: drive the built `aura-bench` binary as a downstream
|
|
//! script would (spawned as a fresh process, no in-process shortcuts).
|
|
|
|
use std::process::Command;
|
|
|
|
/// Path to the freshly-built `aura-bench` binary (Cargo sets this env var
|
|
/// for the test crate; the binary target is named `aura-bench`).
|
|
const BIN: &str = env!("CARGO_BIN_EXE_aura-bench");
|
|
|
|
/// #251: a debug-build wall-clock number measures the build profile, not the
|
|
/// code, so the harness must refuse to run at all rather than silently
|
|
/// producing a number that could later be trusted (or committed) as a
|
|
/// baseline. This pins the profile guard's observable contract — exit code 2
|
|
/// (the usage-error convention aura-bench shares with `aura-cli`) and a
|
|
/// stderr message naming both the cause ("debug build") and the fix
|
|
/// ("--release") — so a future CLI-wiring change (the surfaces glued into
|
|
/// the CLI) cannot silently drop or weaken the guard.
|
|
#[test]
|
|
fn refuses_to_run_in_a_debug_build() {
|
|
// `cargo test` builds test-harness binaries (and their bin-target
|
|
// dependencies) in the dev profile by default, so this binary always
|
|
// has debug_assertions on — deterministic across hosts and CI.
|
|
let out = Command::new(BIN).output().expect("spawn aura-bench");
|
|
let stderr = String::from_utf8_lossy(&out.stderr);
|
|
assert_eq!(out.status.code(), Some(2), "stderr: {stderr}");
|
|
assert!(stderr.contains("debug build"), "stderr: {stderr}");
|
|
assert!(stderr.contains("--release"), "stderr: {stderr}");
|
|
}
|