Derive archive bounds from the monthly file index instead of draining every bar in probe_window #252

New Issue

2026-07-13T14:56:04+02:00

claude commented

2026-07-13 14:56:04 +02:00

probe_window (crates/aura-cli/src/main.rs:616-634 at d7874e2) drains the entire close column of a source — while let Some((t, _)) = Source::next(...) over every bar — solely to learn the first and last timestamp. The campaign path always pays it: campaign_window_ms probes even when --from/--to are explicit (main.rs:3249-3263, "sweep/walkforward/mc always clip"), and generalize probes once per listed symbol.

Measured cost (measured breakdown on #250): one full-archive pass over GER40 (2014→2026, ~1.05M M1 records, 144 monthly files) is ~14.5 s in a debug build — a no-window campaign invocation therefore streams the whole archive twice (probe + run).

The archive layout already carries the answer at O(1): monthly files are named SYM_YYYY_MM.m1 (coarse file-level filtering on exactly that convention exists in data-server, unix_ms_to_year_month, src/records.rs:20-30), so the span is derivable from the first bar of the lexicographically first file and the last bar of the last file — two file opens instead of a full drain. aura-ingest notes an "archive-extent query" as a current non-goal (crates/aura-ingest/src/lib.rs:307); that non-goal is what forces the drain and is worth revisiting.

Claim (unverified): resampler/window semantics at the probe boundary (the probe returns the clipped campaign_window_ms currency) need no change — only the span derivation does.

`probe_window` (crates/aura-cli/src/main.rs:616-634 at `d7874e2`) drains the entire close column of a source — `while let Some((t, _)) = Source::next(...)` over every bar — solely to learn the first and last timestamp. The campaign path always pays it: `campaign_window_ms` probes even when `--from`/`--to` are explicit (main.rs:3249-3263, "sweep/walkforward/mc always clip"), and `generalize` probes once per listed symbol. Measured cost ([measured breakdown on #250](issues/250#issuecomment-3433)): one full-archive pass over GER40 (2014→2026, ~1.05M M1 records, 144 monthly files) is ~14.5 s in a debug build — a no-window campaign invocation therefore streams the whole archive twice (probe + run). The archive layout already carries the answer at O(1): monthly files are named `SYM_YYYY_MM.m1` (coarse file-level filtering on exactly that convention exists in data-server, `unix_ms_to_year_month`, src/records.rs:20-30), so the span is derivable from the first bar of the lexicographically first file and the last bar of the last file — two file opens instead of a full drain. aura-ingest notes an "archive-extent query" as a current non-goal (crates/aura-ingest/src/lib.rs:307); that non-goal is what forces the drain and is worth revisiting. Claim (unverified): resampler/window semantics at the probe boundary (the probe returns the clipped `campaign_window_ms` currency) need no change — only the span derivation does.

~~claude referenced this issue 2026-07-13 15:24:41 +02:00~~

Cut the E2E suite wall-clock: 247s -> 46s (cli_run 211s -> 25s) #255

claude referenced this issue

2026-07-13 15:25:43 +02:00

Cut the cli_run E2E wall-clock: unshare the serialized demo-project fixture store #250

Brummel referenced this issue from a commit

2026-07-13 16:35:21 +02:00

perf(ingest,cli): derive archive bounds from the monthly file index — probe_window drain retired

claude referenced a pull request that will close this issue

2026-07-13 16:36:08 +02:00

Cut the E2E suite wall-clock: 247s -> 46s (cli_run 211s -> 25s) #255

claude commented

2026-07-13 16:36:08 +02:00

Landed as 3e15fb6 on PR #255's branch (closes on merge). Measured with a temporary timer around campaign_window_ms: the probe itself drops 366 ms → 13 ms (~28×) for a no-window resolution over the full 2014-2026 GER40 archive — the O(archive-size) term this issue describes is gone. Honest caveat for expectations: end-to-end command wall-clock moves only ~2-3 %, because the drained probe pass was never as expensive as the sim pass that follows it (the archive inflate is shared via the per-process FileCache, so the drain cost ~3 % of a single-cell full-archive sweep, not ~50 %). Equivalence is pinned by two characterization tests captured against the OLD implementation over the synthetic per-test archive (exact resolved ms, including a known −1 ms Delphi-TDateTime round-trip artifact), plus the unchanged real-data pins.

Landed as `3e15fb6` on PR #255's branch (closes on merge). Measured with a temporary timer around `campaign_window_ms`: the probe itself drops **366 ms → 13 ms** (~28×) for a no-window resolution over the full 2014-2026 GER40 archive — the O(archive-size) term this issue describes is gone. Honest caveat for expectations: end-to-end command wall-clock moves only ~2-3 %, because the drained probe pass was never as expensive as the sim pass that follows it (the archive inflate is shared via the per-process `FileCache`, so the drain cost ~3 % of a single-cell full-archive sweep, not ~50 %). Equivalence is pinned by two characterization tests captured against the OLD implementation over the synthetic per-test archive (exact resolved ms, including a known −1 ms Delphi-`TDateTime` round-trip artifact), plus the unchanged real-data pins.

claude referenced this issue

2026-07-13 16:36:08 +02:00

Cut the cli_run E2E wall-clock: unshare the serialized demo-project fixture store #250

Brummel referenced this issue from a commit

2026-07-13 16:49:26 +02:00

perf(ingest,cli): derive archive bounds from the monthly file index — probe_window drain retired

claude closed this issue

2026-07-13 16:49:46 +02:00

Sign in to join this conversation.