Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Cut the E2E suite wall-clock: 247s -> 46s (cli_run 211s -> 25s) #255

Merged
claude merged 8 commits from worktree-250-test-wallclock into main 2026-07-13 16:49:45 +02:00
Conversation 0 Commits 8 Files Changed 12
+1055 -420
claude commented 2026-07-13 15:24:41 +02:00
Collaborator
Copy Link
Copy Source

Merged fast-forward onto main at 84e1075 (2026-07-13), rebased over #248/#249 first — full workspace suite green over the combined state. Cuts the workspace test suite from ~247 s to ~40 s (cli_run: 211 s → ~25 s, now 142 tests; research_docs: 21.1 s → 1.9 s), everything green, clippy clean, every exact-float pin byte-identical. Second round adds the product-side fixes that fell out of the analysis.

closes #250

Test-side commits (the #250 wall-clock work)

  1. 038e170 — window the whale tests. The two no-window dissolved-walkforward tests assert window-independent properties and now use the ~135-day window the reproduce E2E already uses; the mc roller-fit test replaces its full-archive probe sweep with a fixed ~30-day window. ~19.6 s + 19.6 s + 14.5 s → 1.1–1.9 s each.
  2. 00cc2fe — dependencies at opt-level 2 in the dev profile. Workspace crates stay at opt 0; zip inflate / sha2 / serde get optimized (−17 % on a full-archive sweep).
  3. 25b8354 — per-test project dirs, project_lock retired. Every store-mutating E2E test mints a tempdir project pointing at the once-built fixture crate via an absolute [nodes] pointer; libtest's 24-thread parallelism returns. The #223 cross-process race window is structurally gone — cargo-nextest is now safe to consider.
  4. c31e946 — synthetic per-test M1 archive. The two no-window generalize tests generate a deterministic two-symbol archive and drop their local_data_present gates — hostless, ~35 s + ~50 s → 0.74 s + 1.14 s.

Product-side commits (second round)

  1. a04fda3 — fix: relative paths.data resolves against the project root (closes #254), matching runs_root() and the [nodes] pointers; RED-first unit test.
  2. 479c662 — perf: synthetic walkforward pre-flight validates axes without executing members (closes #253). Attribution correction recorded on the issue: this path serves only the synthetic walkforward; the --real trunk's discarded computation is the campaign [Sweep, WalkForward] leading sweep — a design fork, filed as #256.
  3. a895891 — perf: probe_window drain retired (closes #252): archive bounds now derive from the monthly file index via aura-ingest's archive_extent() (boundary-month loads, gap-month walking). The probe itself drops 366 ms → 13 ms (~28×); end-to-end command wall-clock moves only ~2-3 % because the sim loop dominates. Characterization tests pinned the resolved windows against the old implementation and stay green unchanged.
  4. 84e1075 — fix: knob-lab test scaffold no longer leaks into /tmp (closes #257): the pid-suffixed tempdir leaked ~66 MB per test run (159 dirs ≈ 10 GB filled a 16 GB tmpfs); now a fixed name under CARGO_TARGET_TMPDIR, each run reclaims the previous one.

Verification

  • Full cargo test --workspace: green, 39.8 s wall (run twice back-to-back, stable), no newly ignored/skipped tests.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • All exact-float byte-identity anchors pass unchanged; the two #252 characterization tests were captured against the OLD probe implementation first.
  • Hot-path regression ruled out separately (measured breakdown + A/B binary comparison on issue 250): per-bar cost linear, engine loop untouched since 2026-06-29.

Remaining follow-ups

  • #256 — decide whether the dissolved walkforward/mc leading sweep may skip member execution (design fork, ledger territory — the real --real runtime lever)
  • #251 — benchmark harness with committed baselines (idea)

🤖 Generated with Claude Code

Merged fast-forward onto main at `84e1075` (2026-07-13), rebased over #248/#249 first — full workspace suite green over the combined state. Cuts the workspace test suite from ~247 s to **~40 s** (cli_run: 211 s → **~25 s**, now 142 tests; research_docs: 21.1 s → 1.9 s), everything green, clippy clean, every exact-float pin byte-identical. Second round adds the product-side fixes that fell out of the analysis. closes #250 ## Test-side commits (the #250 wall-clock work) 1. **`038e170` — window the whale tests.** The two no-window dissolved-walkforward tests assert window-independent properties and now use the ~135-day window the reproduce E2E already uses; the mc roller-fit test replaces its full-archive probe sweep with a fixed ~30-day window. ~19.6 s + 19.6 s + 14.5 s → 1.1–1.9 s each. 2. **`00cc2fe` — dependencies at opt-level 2 in the dev profile.** Workspace crates stay at opt 0; zip inflate / sha2 / serde get optimized (−17 % on a full-archive sweep). 3. **`25b8354` — per-test project dirs, `project_lock` retired.** Every store-mutating E2E test mints a tempdir project pointing at the once-built fixture crate via an absolute `[nodes]` pointer; libtest's 24-thread parallelism returns. The #223 cross-process race window is structurally gone — `cargo-nextest` is now safe to consider. 4. **`c31e946` — synthetic per-test M1 archive.** The two no-window generalize tests generate a deterministic two-symbol archive and drop their `local_data_present` gates — hostless, ~35 s + ~50 s → 0.74 s + 1.14 s. ## Product-side commits (second round) 5. **`a04fda3` — fix: relative `paths.data` resolves against the project root** (closes #254), matching `runs_root()` and the `[nodes]` pointers; RED-first unit test. 6. **`479c662` — perf: synthetic walkforward pre-flight validates axes without executing members** (closes #253). Attribution correction recorded on the issue: this path serves only the synthetic walkforward; the `--real` trunk's discarded computation is the campaign `[Sweep, WalkForward]` leading sweep — a design fork, filed as #256. 7. **`a895891` — perf: `probe_window` drain retired** (closes #252): archive bounds now derive from the monthly file index via aura-ingest's `archive_extent()` (boundary-month loads, gap-month walking). The probe itself drops 366 ms → 13 ms (~28×); end-to-end command wall-clock moves only ~2-3 % because the sim loop dominates. Characterization tests pinned the resolved windows against the old implementation and stay green unchanged. 8. **`84e1075` — fix: knob-lab test scaffold no longer leaks into `/tmp`** (closes #257): the pid-suffixed tempdir leaked ~66 MB per test run (159 dirs ≈ 10 GB filled a 16 GB tmpfs); now a fixed name under `CARGO_TARGET_TMPDIR`, each run reclaims the previous one. ## Verification - Full `cargo test --workspace`: green, **39.8 s** wall (run twice back-to-back, stable), no newly ignored/skipped tests. - `cargo clippy --workspace --all-targets -- -D warnings`: clean. - All exact-float byte-identity anchors pass unchanged; the two #252 characterization tests were captured against the OLD probe implementation first. - Hot-path regression ruled out separately (measured breakdown + A/B binary comparison on issue 250): per-bar cost linear, engine loop untouched since 2026-06-29. ## Remaining follow-ups - #256 — decide whether the dissolved walkforward/mc leading sweep may skip member execution (design fork, ledger territory — the real `--real` runtime lever) - #251 — benchmark harness with committed baselines (idea) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Brummel added 8 commits 2026-07-13 16:49:26 +02:00
test(cli): bound three whale E2E windows instead of streaming full archives 038e1702b3 
The two dissolved-walkforward flag tests assert window-independent
properties (plateau acceptance, per-flag stop defaulting) and now use
the ~135-day window the reproduce e2e already uses. The mc roller-fit
test replaces its full-archive probe sweep with a fixed ~30-day window
inside that same span: the property needs a data-carrying window far
shorter than the 90+30-day roller, not a live-span derivation, and the
archive only grows forward. Measured on the data-full host: 19.6s,
19.6s and 14.5s+ of serialized real-data compute drop to 1.1-1.9s each.

refs #250
build: compile dependencies at opt-level 2 in the dev profile 00cc2fe927 
The E2E suite spawns the debug aura binary, whose wall-clock is
dominated by dependency code (zip inflate of the tick archive, sha2
over the project dylib, serde). Workspace crates stay at opt-level 0
for fast rebuilds and debugging; every pinned float is computed by
workspace code, so the byte-identity anchors are untouched.

refs #250
test(cli): per-test project dirs via a [nodes] pointer — project_lock retired 25b8354959 
Every E2E test that mutated the shared demo-project fixture store now
mints its own tempdir project whose 2-line Aura.toml points at the
once-built fixture crate by absolute [nodes] pointer; the runs/ store
follows the project root, so no two tests contend and libtest's
default thread parallelism applies. project_load keeps driving the
fixture directly (the suite's only proof of the inline-crate routing
arm); its single store-mutating test is that binary's only one, so no
lock is needed there either. This also closes the documented
cross-process race window (#223): there is no shared mutable store
left to race on.

Measured on the data-full host: cli_run 211s -> 57.2s (140 tests),
research_docs 21.1s -> 2.1s (73 tests), project_load green, clippy
clean.

refs #250
test(cli): synthetic per-test M1 archive — no-window generalize tests go hostless c31e946bb7 
The two generalize E2E tests whose property is no-window span
resolution (shared-window fallback, intersection semantics) no longer
stream the 6.6 GB host archive: fresh_project_with_data() generates a
deterministic two-symbol M1 archive (SYMA 2024-01..08, SYMB 2024-03..06
strictly inside it, so the intersection differs from symbols[0]'s
window on both bounds) in the exact zip/48-byte-record format
data-server reads, plus geometry sidecars, under the per-test project's
paths.data. Both tests drop their local_data_present gates — they now
run on any host, data mount or not.

~35s and ~50s become 0.74s and 1.14s; the full cli_run binary lands at
~25s (211s at the branch base). Full workspace suite: 247s -> 45.7s.

refs #250
fix(cli): resolve a relative paths.data against the project root a04fda3a4c 
Env::data_path() handed paths.data to the data layer verbatim, so a
relative value resolved against the invoking process's cwd — aura run
from a project subdirectory silently read a different archive location
than from the root, while runs/ stayed anchored. Relative values now
join onto the project root, matching runs_root() and the [nodes]
pointer resolution; absolute values and the no-project default pass
through unchanged. RED-first unit test pins the root-anchored
resolution.

closes #254
perf(cli): synthetic walkforward pre-flight validates axes without running members 479c6620e2 
blueprint_walkforward_family's pre-flight executed the full first-IS-
window sweep via blueprint_sweep_over and discarded the result, purely
to surface axis errors once before the parallel window fan-out (#177).
validate_axis_grid now drives the sweep terminal's own resolve/arity/
kind checks with a constant placeholder report — same single-sourced
rejection wording, no member run, no roller derivation needed.

Scope correction against the issue text: this path serves only the
synthetic (non---real) walkforward. The --real trunk routes through
verb_sugar into aura-campaign's [Sweep, WalkForward] process, whose
leading full-window sweep is a separate, larger duplication — re-filed
with corrected attribution in the issue's closing comment.

closes #253
perf(ingest,cli): derive archive bounds from the monthly file index — probe_window drain retired a895891ab1 
probe_window drained a source's entire close column just to learn its
first and last bar timestamp; both live callers (open_real_source and
campaign_window_ms on the campaign trunk) now resolve through
aura-ingest's new archive_extent(): list the symbol's SYM_YYYY_MM.m1
files, load only the boundary months via the existing loader seam, and
walk across gap months forward/backward — O(2 file loads) typical
instead of O(archive). Empty-window refusal semantics are unchanged
(pinned). Two characterization tests captured the resolved windows
against the OLD implementation over the synthetic per-test archive and
stay green unchanged after the swap.

Measured: the probe itself drops 366ms -> 13ms (~28x) on the full
2014-2026 GER40 archive; end-to-end command wall-clock moves only
~2-3% because the sim loop dominates — the fix removes an
O(archive-size) term, not the dominant cost.

closes #252
fix(test): anchor the knob-lab scaffold under CARGO_TARGET_TMPDIR 84e1075176 
built_scale_project() scaffolded ~66 MB (built node crate) into a
pid-suffixed tempdir, so its wipe-before-scaffold never hit a previous
run's copy: one leaked directory per test-binary run. 159 of those
(~10 GB) filled the host's 16 GB /tmp tmpfs to 100%, breaking every
std::env::temp_dir() consumer. A fixed name under CARGO_TARGET_TMPDIR
(real disk, reclaimed by cargo clean) makes the existing wipe reclaim
the previous run — steady state is exactly one directory. Trade-off:
two concurrent cargo-test invocations over the same checkout collide
on the fixed name, the documented pre-existing caveat class for this
suite (#223).

closes #257
Brummel force-pushed worktree-250-test-wallclock from e7f16c179b to 84e1075176 2026-07-13 16:49:26 +02:00 Compare
claude merged commit 84e1075176 into main 2026-07-13 16:49:45 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#255
Delete Branch "worktree-250-test-wallclock"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?