Caught by the harness itself on its first post-commit run: the aura run
record line embeds manifest.commit — the binary's compile-time build sha
(crates/aura-cli/build.rs, the invariant-8 audit trail) — so hashing the raw
line flipped the fingerprint on every rebuild at a new commit and reported a
phantom correctness change (exit 1) at each cycle close. The rebase onto the
#191 main exposed it; nothing in #191 touched the record.
RED-first: run_line_fingerprint_is_invariant_across_build_commits pins the
symptom (two lines differing only in manifest.commit must hash identically);
the fix extracts run_line_fingerprint, which parses the line, blanks that
one volatile field, re-serializes (serde_json maps are sorted, so the bytes
are deterministic), and hashes. A missing manifest.commit is a hard error,
so a future record-schema move fails loudly instead of silently passing.
The cli_fixed_cost baseline is re-pinned under the new definition — ratify:
the fingerprint DEFINITION changed (provenance excluded, results still
covered, pinned by run_line_fingerprint_still_detects_result_changes); no
measured behaviour of aura moved.
refs #251
Architect drift review (cycle bfb8648..0389399): no domain-invariant breach.
What holds: aura-bench is bin-only dev tooling with zero production reverse
deps (invariant 8/C13); every committed surface runs on seeded synthetic
data in scratch temp projects (invariant 9); run_reps enforces cross-rep
fingerprint identity and the campaign fingerprint carries integer ordinals
across the child boundary, consistent with C1 and its cross-command ULP
carve-out; zip/clap per-case dependency notes in place (C16), and the ledger
correctly gains no entry — tooling lives in the project facts + README.
Three drift items, resolved in this commit:
- [medium] the winner_fingerprint doc claimed walk-forward regressions fail
the bench; the walk-forward stage in fact emits an empty realization
record, so coverage is only indirect (via the bootstrap trade count). Doc
now states the gap honestly; the engine-side fix is parked on #279.
- [low] the fingerprint's lenient JSON parse would silently erode on a
registry-schema rename: a new lockstep test builds a real
CampaignRunRecord from the aura-registry/aura-analysis/aura-engine types
(new dev-deps) and asserts the extracted fingerprint, so a rename breaks
this crate's tests instead.
- [low] LIBRARY_SURFACES was hand-synced with measure_surface: a new test
pins that every listed surface measures without a binary path.
Regression gates on the closing tree: cargo test --workspace 1379 green,
clippy -D warnings clean, doc build clean, aura-bench run exit 0 with five
'fingerprint OK' at <=2% drift. No baseline moved in this commit — the
bench baselines were pinned and verified in the feature commit.
refs #251
Tasks 7-8 of the bench-harness plan plus the orchestrator's sizing pass —
this completes the phase-1 harness (closes#251).
Driver: run [--surface|--quick|--reps|--out] / pin over five surfaces,
report-only comparison (drift NOTICE at >=10%, never a failure), fingerprint
mismatch exits 1 with a baseline-vs-measured block, infra errors exit 2,
debug builds refuse to measure. The Bench project-facts line and the crate
README (discipline: quiet box, warmup+median, deliberate re-pins) land with
it.
Workload sizing, measured then corrected: the release engine outran the
plan's guesses by ~100x, leaving sub-second child walls where spawn jitter
reaches the 10% threshold and flaps the report. Full-size workloads now land
in whole seconds — engine 10M bars (~0.7s), ingest 24 months x 20 fresh
drains (~0.4s), campaigns on a 24-month archive with a ~22-month window and
a 5x5 member grid (sweep ~1.4s, heavy ~5.5s), fixed-cost as in-rep medians
over spawn batches (20x help, 10x run). Quick mode keeps the small
E2E-fixture shape.
The campaign fingerprint folds all three realization layers — sweep winner,
gate survivor ordinals, bootstrap trade counts (pooled or per-survivor sum)
— so a wrong-result regression in any stage fails the bench, not just a
wrong sweep winner. Baselines pinned on this host (commit-stamped), verified
by a follow-up run: five fingerprints OK at <=2% drift under load.
Verification: cargo test --workspace 1379 green (bench tests all quick-mode;
the release-binary E2Es are #[ignore]d behind -- --ignored per the
suite-wallclock discipline), clippy -D warnings clean, doc build clean.
Tasks 5-6 of the bench-harness plan (refs #251). Host context probes
(hostname/nproc/loadavg/commit/date, all best-effort), the child runner
(release aura binary resolved from cargo's compiler-artifact JSON messages;
/usr/bin/time -v wraps the child for CPU% + peak RSS, wall measured
in-process), and the campaign surfaces:
- a scratch data-only project per repetition ([paths] only — no [nodes]; the
std-vocabulary fallback covers the SMA blueprints), synthetic archives for
4 instruments, blueprints seeded via tiny synthetic sweeps, process docs
registered through the real CLI;
- campaign_sweep (sweep-only process — narrow per-cell fan-out) and
campaign_heavy (sweep -> gate -> walk-forward -> 1000-resample bootstrap —
wide fan-out): one campaign shape misreads regressions in the other;
- fingerprint = winner ordinals parsed from the always-on final
{"campaign_run":...} stdout line.
The full campaign-sweep determinism E2E (real release binary, real scratch
runs) is #[ignore]d out of the default suite — it costs a release build plus
child campaign runs, which the suite-wallclock discipline forbids on every
cargo test; run it via `cargo test -p aura-bench -- --ignored`. Held
quality note: the heavy/quick process docs stay two verbatim consts (plan
prescription; a format! builder would trade the uniform &str contract for
cosmetic dedup).
Tasks 3-4 of the bench-harness plan (refs #251). The surface framework
(1 discarded warmup + N measured reps, median per metric, cross-rep
fingerprint identity enforced — a surface disagreeing with itself is
nondeterminism, never averaged away) and the two library surfaces:
- engine_throughput: 1M-bar seeded walk through an SMA-cross bias graph
(GraphBuilder -> compile -> bootstrap -> VecSource -> run), fingerprint =
the SeriesReducer finalize row.
- ingest_throughput: synthetic 48-byte M1 archive drained via open_columns
with a FRESH DataServer per rep, so the measured drain pays zip inflate +
parse instead of the in-memory FileCache; fingerprint = count + bit-XOR
of the close column.
Quick-mode determinism is pinned in-suite for both surfaces (C1 — the
harness's premise as currently-green tests); full-size numbers wait for the
release driver glue.
Tasks 1-2 of the bench-harness plan (refs #251): the new dev-only workspace
bin crate with the report-only comparison core (BaselineDoc serde, per-metric
relative drift, 10% NOTICE threshold, fingerprint-equality exit code) and the
deterministic synthetic inputs (seeded LCG price walk, the proven 48-byte
Delphi-record M1 zip archive writer, self-deleting scratch dirs).
The archive writer is deliberately a third copy of the test-fixture packer
(aura-cli tests / aura-ingest each carry one), byte-compatible by
construction and pinned here by a round-trip test through the real ingest
reader. The binary refuses debug builds (exit 2) — wall-clock numbers from a
debug profile measure the profile, not the code — pinned by an E2E spawn
test. Temporary #![allow(dead_code)] markers cover the not-yet-wired core
until the CLI glue lands; the surfaces and driver follow.
claude