Extend the benchmark harness: instruction-count backend, native-crate load cost, RSS-vs-K sweep, thresholded gate #279

New Issue

2026-07-17T15:39:39+02:00

claude commented

2026-07-17 15:39:39 +02:00

Motivation

The phase-1 benchmark harness (#251) shipped wall-clock-only, report-only, with three deliberate scope cuts recorded in its design decisions (issues/251#issuecomment-3717). Each cut is worth revisiting once its precondition is met; this issue parks them so they survive the cycle close.

Deferred items

Instruction-count backend (the committed-gate half of the original design). Wall-clock baselines are host context; a regression gate wants a deterministic, byte-committable metric. iai-callgrind (valgrind-based) was the intended backend, but valgrind is not installed on the measuring host and package installation needs interactive privileges — bench code that cannot execute on the measuring host would have shipped unverified.
depends on: valgrind installed on the measuring host (one package install)
context: the harness's result/compare model is backend-agnostic — a counting backend slots in as sibling metrics without reshaping the committed baseline files.
Native-crate project-load sub-cost (dylib SHA-256 + cargo metadata + git spawns). The phase-1 cli_fixed_cost surface covers the data-only load path; the native-crate path (~124 ms debug for a 4.7 MB dylib in the #250 breakdown) needs a scratch project with a built node crate, which would put a cargo build step inside the bench loop.
context: needs a prebuilt fixture-crate pointer approach (the suite's [nodes]-pointer tempdir trick) to stay build-step-free per repetition.
Per-K peak-RSS sweep on a wider instrument set. The --parallel-instruments default (4) is derived from the ~1.4 GB/instrument retention estimate, not measured on breadth; a committed RSS-vs-K curve on 8+ synthetic instruments would ground the default in data and catch retention regressions. The harness already captures peak RSS per campaign surface, so this is a workload + report extension, not new machinery.
Thresholded failure mode. Report-only until run-to-run variance is known (the #251 cadence decision). Once a few cycle-close runs have accumulated drift history, promote the NOTICE threshold to a failing gate with a variance-justified margin.

## Motivation The phase-1 benchmark harness (#251) shipped wall-clock-only, report-only, with three deliberate scope cuts recorded in its design decisions (issues/251#issuecomment-3717). Each cut is worth revisiting once its precondition is met; this issue parks them so they survive the cycle close. ## Deferred items 1. **Instruction-count backend (the committed-gate half of the original design).** Wall-clock baselines are host context; a regression *gate* wants a deterministic, byte-committable metric. iai-callgrind (valgrind-based) was the intended backend, but valgrind is not installed on the measuring host and package installation needs interactive privileges — bench code that cannot execute on the measuring host would have shipped unverified. depends on: valgrind installed on the measuring host (one package install) context: the harness's result/compare model is backend-agnostic — a counting backend slots in as sibling metrics without reshaping the committed baseline files. 2. **Native-crate project-load sub-cost (dylib SHA-256 + cargo metadata + git spawns).** The phase-1 `cli_fixed_cost` surface covers the data-only load path; the native-crate path (~124 ms debug for a 4.7 MB dylib in the #250 breakdown) needs a scratch project with a built node crate, which would put a cargo build step inside the bench loop. context: needs a prebuilt fixture-crate pointer approach (the suite's `[nodes]`-pointer tempdir trick) to stay build-step-free per repetition. 3. **Per-K peak-RSS sweep on a wider instrument set.** The `--parallel-instruments` default (4) is derived from the ~1.4 GB/instrument retention estimate, not measured on breadth; a committed RSS-vs-K curve on 8+ synthetic instruments would ground the default in data and catch retention regressions. The harness already captures peak RSS per campaign surface, so this is a workload + report extension, not new machinery. 4. **Thresholded failure mode.** Report-only until run-to-run variance is known (the #251 cadence decision). Once a few cycle-close runs have accumulated drift history, promote the NOTICE threshold to a failing gate with a variance-justified margin.

claude added the idea label 2026-07-17 15:39:48 +02:00

claude commented

2026-07-17 15:47:33 +02:00

Fifth deferred item (from the #251 cycle-close architecture review, 2026-07-17)

Walk-forward realization is invisible to the campaign fingerprint. The campaign executor's walk-forward stage emits an empty realization record — no selection, no survivor ordinals, no bootstrap block (crates/aura-campaign/src/exec.rs, WalkForward arm) — so the bench's campaign_heavy fingerprint covers the walk-forward machinery only indirectly, through the bootstrap trade count that depends on its out-of-sample output. A wrong-but-plausible walk-forward regression that leaves the pooled trade population intact would pass the bench. Fixing this is an aura-campaign change (emit per-window winner ordinals or an aggregate realization for the stage), after which the bench fingerprint picks it up without code changes (its parse already folds any stage's selection/survivors/bootstrap fields).
context: found by the cycle-close drift review; the bench-side doc (winner_fingerprint) states the gap honestly and points here.

## Fifth deferred item (from the #251 cycle-close architecture review, 2026-07-17) 5. **Walk-forward realization is invisible to the campaign fingerprint.** The campaign executor's walk-forward stage emits an empty realization record — no selection, no survivor ordinals, no bootstrap block (`crates/aura-campaign/src/exec.rs`, WalkForward arm) — so the bench's `campaign_heavy` fingerprint covers the walk-forward machinery only indirectly, through the bootstrap trade count that depends on its out-of-sample output. A wrong-but-plausible walk-forward regression that leaves the pooled trade population intact would pass the bench. Fixing this is an aura-campaign change (emit per-window winner ordinals or an aggregate realization for the stage), after which the bench fingerprint picks it up without code changes (its parse already folds any stage's selection/survivors/bootstrap fields). context: found by the cycle-close drift review; the bench-side doc (`winner_fingerprint`) states the gap honestly and points here.

Brummel referenced this issue from a commit

2026-07-17 18:17:28 +02:00

audit: cycle-close tidy for #251 — fingerprint honesty, lockstep + surface-list guards

claude referenced this issue

2026-07-23 20:56:56 +02:00

Build a deployment arena: a standing proving ground where source-blind agents exercise the released binary #322

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/Aura#279