Add a benchmark harness with committed baselines to catch performance regressions #251
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Motivation
No benchmark surface exists in the workspace: no
benches/directory, no[[bench]]target, and no criterion/iai dev-dependency in anyCargo.toml(verified 2026-07-13, workspace-wide grep). The only wall-clock figure ever recorded for the project is the suite capture quoted in #250 — repo-wide greps forfinished in/test result:overdocs/andfieldtests/andgit log -Sfind nothing older. Consequently the regression question #250 raised ("slow hot path, or slow tests?") was answerable only by an ad-hoc measurement session; the resulting numbers live in the measured breakdown on #250: per-bar marginal cost ~13.8 µs/bar in a debug build, scaling linearly from one month to twelve years of M1 data, withHarness::runallocation-free per tick (crates/aura-engine/src/harness.rs:371-451, unchanged sinced5602ec).The gap this leaves: the E2E suite has no timing assertions anywhere — an engine that silently became 2× slower would still pass 140/140 and only surface as a vaguely slower test suite, which is exactly how #250 was noticed.
Determinism makes the engine unusually benchmarkable: a pinned blueprint + pinned data window + pinned seed reproduce the identical computation (domain invariant 1, CLAUDE.md), so a regression separates cleanly from noise.
Benchmark surface (what a harness would pin)
cargo metadata+ two git spawns (crates/aura-cli/src/project.rs:532-534, :269-273, :320-340), measured at ~124 ms debug for a 4.7 MB dylib in the #250 breakdown.Design forks
benches/baselines/) so drift is visible in diffs. Bench frameworks enter as dev-dependencies only; nothing reaches the frozen deploy artifact (domain invariant 8, CLAUDE.md).Relation to #250
Orthogonal. #250 cuts the E2E suite's wall-clock (test workloads and store serialization); this issue is about detecting future hot-path regressions, which that suite structurally cannot see.
Input from the first real campaign-scaling measurement (2026-07-17)
The parallel-cell-loop delivery was benchmarked ad hoc on real data (full numbers + limitations: issues/277#issuecomment-3707). Parking the reusable findings here, since this issue is where a committed harness would absorb them.
Reference point for the "campaign scaling" surface (the second bullet of this issue's benchmark-surface list): a 4-instrument × 2-strategy real-data campaign (8 cells; sweep + walk-forward + 1000-resample bootstrap + generalize per cell), release builds, 24-core box — sequential cell loop 1:09.5 wall at 65% box utilization; parallel cell loop 0:49.0 at 95% (
--parallel-instruments 4, the default), monotone across K=1/2/4 (1.26×/1.34×/1.42×). Peak RSS 966 MB (K=1) → 1.56 GB (K=4). The campaign-record winner fingerprint was byte-identical across five binaries and all K values — the determinism-makes-it-benchmarkable premise of this issue's Motivation, confirmed on real data.Harness practices that proved themselves in that session (candidate design input):
/usr/bin/time -vcovers wall, CPU%, and max RSS in one tool.Measurement axes worth adding to the surface list:
Parked side-observations from the same session (outside this issue's scope; recorded here at the maintainer's direction rather than as new issues):
aura:stderr prefix; the semantics are documented (authoring guide, exit-code contract), but a visible channel distinction would remove the remaining ambiguity for a user scanning stderr.Design decisions (specify, 2026-07-17)
Derived autonomously under the orchestrator's fork authority; recorded here for after-the-fact audit and veto. Environment probe on the measuring host (2026-07-17): no valgrind, no perf, no hyperfine installed; package installation requires interactive privileges, so none can be added autonomously.
Fork: committed-gate backend — instruction counts vs wall-clock. → Phase 1 ships wall-clock measurement only; the instruction-count backend (the body's "natural split" gate half) is deferred to a follow-up issue contingent on valgrind being installed on the measuring host. Derived: bench code that cannot execute on the measuring host would ship unverified; the cadence decision in this issue's body already fixes phase 1 as report-only, so no gate strength is lost by deferring the deterministic backend; and the measured run-to-run spread on this host is <2% for the campaign workload (issues/277#issuecomment-3707), ample resolution for the "silently 2× slower" regression class the Motivation names.
Fork: framework — criterion vs bespoke driver. → A bespoke driver binary in a new dev-only workspace crate. Derived: the substance of this work — per-surface JSON baselines committed in-repo and compared on each run, a determinism fingerprint asserted inside the bench itself (a fast-but-wrong binary must fail the bench), and child-process measurement of the CLI binary including peak RSS — lies outside criterion's model (statistical micro-sampling with machine-local baselines under target/), and criterion's minimum sampling makes multi-second campaign workloads impractical. The per-case dependency policy admits vetted crates that do the job; of the surveyed candidates (criterion, divan, iai-callgrind, hyperfine) none does this job on this host, so the timing loop (warmup + N measured repetitions, median) is written directly — it is the trivial part, not the substance.
Fork: bench data. → All committed surfaces run on synthetic data the harness generates itself, following the 48-byte-record archive-writer pattern the hostless test fixtures already use; real-archive benches stay possible as a local, uncommitted mode but never carry committed baselines. Derived: engine/project separation (domain invariant 9) forbids project data in this repo, and a committed baseline must be reproducible from the repo alone.
Fork: phase-1 surfaces. → Five: (1) engine throughput — bars/s through a bootstrapped harness, library call, synthetic in-memory source over a committed example blueprint; (2) ingest throughput — synthetic zip archive through the ingest window path, library call; (3+4) campaign wall-clock in two shapes — sweep-only (narrow per-cell fan-out) and walk-forward+bootstrap-heavy (wide fan-out) — run through the CLI binary against a scratch data-only project the driver builds in a temp dir; (5) CLI fixed cost — spawn floor plus data-only project load. Derived: the two campaign shapes implement the parked finding that a single-shape harness misreads regressions in the other shape (this issue's 2026-07-17 comment); the native-crate project-load sub-cost (dylib hashing) is deferred alongside the instruction-count follow-up so the scratch project stays build-step-free and the bench loop never invokes cargo.
Fork: baseline home + cadence. → Baselines as per-surface JSON files committed inside the bench crate; the bench runs at cycle close via a new project-facts entry; comparison is report-only (prints drift percentages and flags fingerprint mismatches, never fails the cycle) until recorded variance justifies thresholds. Derived: follows the leanings already recorded in this issue's body ("committed next to the benches", "cycle close", "report-only until run-to-run variance is known"); baseline files carry the measuring host's identity so a foreign-host comparison is visibly informational.
Autonomous spec sign-off (2026-07-17)
The design spec for the phase-1 benchmark harness — a dev-only workspace crate measuring five wall-clock surfaces (engine throughput, ingest throughput, two campaign shapes, CLI fixed cost) against committed per-surface JSON baselines with in-bench determinism fingerprints, report-only comparison — was signed autonomously: the signature is an independent fresh-context grounding check returning PASS, with every load-bearing assumption about current codebase behaviour traced to a named, currently-green test (engine library idiom, folding-sink finalize row, synthetic-archive campaign E2E, ingest window API, store winner-ordinal reads). No human reviewed the spec at signing time; this comment is the audit trail, and the design decisions it encodes are the ones recorded in the previous decision-log comment (issues/251#issuecomment-3717).
The spec is an uncommitted per-cycle working file; its durable design intent is lifted into the design ledger at cycle close. Status: spec signed — ready for implementation planning.
Delivery record (2026-07-17)
Phase 1 is implemented and cycle-closed on branch
worktree-issue-251-bench-harness(5 commits over basebfb8648), awaiting review and the ratified merge;closes #251rides the final feature commit.What shipped, against the decisions recorded in issues/251#issuecomment-3717: the dev-only workspace crate
aura-benchwith five wall-clock surfaces (engine throughput, ingest throughput, campaign sweep-only + walk-forward/bootstrap-heavy shapes via the release binary on scratch data-only projects, CLI fixed cost), per-surface JSON baselines committed and pinned on the reference host, in-bench determinism fingerprints (a fingerprint mismatch exits 1; timing drift is report-only with a 10% NOTICE line), a Bench project-facts entry, and a README carrying the measurement discipline.One decision was added after implementation, from measurement: the plan's workload sizes left sub-second child walls (the release engine outran the estimates ~100×), where spawn/scheduler jitter reaches the 10% threshold and flaps the report. Full-size workloads were re-sized to land in whole seconds (engine 10M bars ≈0.7 s, ingest 24 months × 20 drains ≈0.4 s, campaigns on a 24-month archive with a 5×5 grid ≈1.4 s/≈5.5 s, fixed-cost as in-rep spawn-batch medians), and the campaign fingerprint was extended to fold gate survivors and bootstrap trade counts alongside sweep winners. Verified: a follow-up comparison run reports five
fingerprint OKat ≤2% drift under load.The cycle-close architecture review found no invariant breach; its three findings are resolved on the branch (honest walk-forward-coverage doc, a lockstep test binding the fingerprint parse to the real registry record types, a test pinning the library-surface list). The walk-forward realization gap and the other deferred items are parked on issues/279. No field test was run for this cycle: the tool's consumer flows (run, pin, re-pin comparison, surface filter, failure exit codes) were exercised for real, repeatedly, as part of the delivery verification itself.