Resample/VolTfStop bucket by raw provider timestamps — sub-second M1 stamp jitter mis-buckets bars #280

New Issue

2026-07-17T17:14:16+02:00

claude commented

2026-07-17 17:14:16 +02:00

Evidence (Pepperstone archive, UK100)

Provider M1 stamps carry up to ±1s jitter around minute boundaries. Measured over all 3120 weekdays 2014-08..2026-07 while running an offline session-anchored study: on ~55% of days the 09:00:00-Berlin minute bar is stamped 08:59:59.xxx. A [t0, ..) window load misses it entirely; nearest-minute rounding recovered 1701 of 1709 otherwise-dropped days. Example (2024-07-15, offsets relative to 07:00:00 UTC): first bar stamped -0.6s, next at +60s — the nominal 09:00 candle exists but sits a hair before the minute mark.

Mechanism

Resample decides bucket membership by raw integer division of the cycle stamp: let bucket = ctx.now().0 / self.period_ns; (crates/aura-std/src/resample.rs:107). VolTfStop's internal resampler uses the identical pattern (crates/aura-std/src/vol_tf_stop.rs:93).
Ingestion performs a pure unit conversion ms→ns (crates/aura-ingest/src/lib.rs:41-43); no normalization/snapping exists anywhere in the pipeline.
Session/SessionFrankfurt derive wall-clock from the same raw ctx.now(), so the jitter also flips bars across session-open boundaries (bars_since_open sees 08:59:59.4 as pre-open).

Impact

Coarse bars built inside a real backtest differ from the provider's nominal minutes on a majority of days: a bar stamped 59.4s early is aggregated into the previous 5m bucket, shifting every downstream open/close by one M1 bar around bucket boundaries. Any session-anchored strategy logic inherits the same off-by-one at the open. Silent — no error, plausible-looking bars.

Open design question (ledger-worthy, not prescribing here)

Where should normalization live?

At ingestion (single place, consistent with the one-merge boundary): but it rewrites recorded stamps, touching the record-then-replay contract that the archive replays verbatim.
In the consuming nodes (Resample/Session snap to nearest minute): keeps the stream verbatim, but spreads the obligation across every time-bucketing node, current and future.

Workaround used meanwhile: the scratch-project study node does nearest-minute rounding internally before bucketing.

(Found during a throwaway UK100 study in the external scratch project; the offline analysis and the node live there, not in this repo.)

## Evidence (Pepperstone archive, UK100) Provider M1 stamps carry up to ±1s jitter around minute boundaries. Measured over all 3120 weekdays 2014-08..2026-07 while running an offline session-anchored study: on ~55% of days the 09:00:00-Berlin minute bar is stamped 08:59:59.xxx. A `[t0, ..)` window load misses it entirely; nearest-minute rounding recovered 1701 of 1709 otherwise-dropped days. Example (2024-07-15, offsets relative to 07:00:00 UTC): first bar stamped -0.6s, next at +60s — the nominal 09:00 candle exists but sits a hair before the minute mark. ## Mechanism - `Resample` decides bucket membership by raw integer division of the cycle stamp: `let bucket = ctx.now().0 / self.period_ns;` (`crates/aura-std/src/resample.rs:107`). `VolTfStop`'s internal resampler uses the identical pattern (`crates/aura-std/src/vol_tf_stop.rs:93`). - Ingestion performs a pure unit conversion ms→ns (`crates/aura-ingest/src/lib.rs:41-43`); no normalization/snapping exists anywhere in the pipeline. - `Session`/`SessionFrankfurt` derive wall-clock from the same raw `ctx.now()`, so the jitter also flips bars across session-open boundaries (`bars_since_open` sees 08:59:59.4 as pre-open). ## Impact Coarse bars built *inside* a real backtest differ from the provider's nominal minutes on a majority of days: a bar stamped 59.4s early is aggregated into the *previous* 5m bucket, shifting every downstream open/close by one M1 bar around bucket boundaries. Any session-anchored strategy logic inherits the same off-by-one at the open. Silent — no error, plausible-looking bars. ## Open design question (ledger-worthy, not prescribing here) Where should normalization live? - **At ingestion** (single place, consistent with the one-merge boundary): but it rewrites recorded stamps, touching the record-then-replay contract that the archive replays verbatim. - **In the consuming nodes** (Resample/Session snap to nearest minute): keeps the stream verbatim, but spreads the obligation across every time-bucketing node, current and future. Workaround used meanwhile: the scratch-project study node does nearest-minute rounding internally before bucketing. (Found during a throwaway UK100 study in the external scratch project; the offline analysis and the node live there, not in this repo.)

claude added the bug label 2026-07-17 17:14:16 +02:00

claude referenced this issue

2026-07-17 17:22:08 +02:00

std vocabulary gap: Sign / Select / CumSum primitives #281

claude referenced this issue

2026-07-17 17:54:39 +02:00

Blueprint grammar cannot reach sinks — add declared taps (output-side twin of input_roles) #282

claude added this to the Measurement as a first-class citizen milestone 2026-07-17 18:08:01 +02:00

claude self-assigned this 2026-07-17 18:28:24 +02:00

claude commented

2026-07-17 18:28:36 +02:00

Fork decision (autonomous, 2026-07-17): where time normalization lives.

Options considered:

At ingestion — rewrite stamps once, at the one-merge boundary, next to the existing ms→ns conversion.
In the consuming nodes, hand-rolled per node — each time-bucketing node rounds internally (the scratch-study workaround).
Consumer-side, as one shared core utility — a single tested snap function (nearest-boundary rounding) that every time-bucketing node (Resample, VolTfStop, Session/SessionFrankfurt) applies when computing bucket/session membership; the stream itself stays verbatim.

Chosen: option 3.

Rationale:

The record-then-replay contract wants the archive replayed verbatim; an ingestion-time rewrite mutates the recorded stream and irreversibly hides the raw time axis from every consumer, including future microstructure or latency studies that need the true stamps.
Rewriting stamps at ingestion can also reorder the k-way merge around boundaries (events from different sources near a minute mark may flip), so it touches the chronological merge contract itself — a deeper intervention than the defect warrants.
The milestone frames the fix as "bucket membership computes on normalized time, not raw provider stamps": normalization is an interpretation applied where time is bucketed, not a property of the data.
The spread-obligation objection to consumer-side snapping (the reason option 2 is rejected) dissolves under centralization: one shared, tested utility replaces per-node hand-rolling instead of multiplying it.

Status: fix path settled — proceeding RED-first against the mis-bucketing symptom.

Fork decision (autonomous, 2026-07-17): where time normalization lives. Options considered: 1. **At ingestion** — rewrite stamps once, at the one-merge boundary, next to the existing ms→ns conversion. 2. **In the consuming nodes, hand-rolled per node** — each time-bucketing node rounds internally (the scratch-study workaround). 3. **Consumer-side, as one shared core utility** — a single tested snap function (nearest-boundary rounding) that every time-bucketing node (Resample, VolTfStop, Session/SessionFrankfurt) applies when computing bucket/session membership; the stream itself stays verbatim. Chosen: option 3. Rationale: - The record-then-replay contract wants the archive replayed verbatim; an ingestion-time rewrite mutates the recorded stream and irreversibly hides the raw time axis from every consumer, including future microstructure or latency studies that need the true stamps. - Rewriting stamps at ingestion can also reorder the k-way merge around boundaries (events from different sources near a minute mark may flip), so it touches the chronological merge contract itself — a deeper intervention than the defect warrants. - The milestone frames the fix as "bucket membership computes on normalized time, not raw provider stamps": normalization is an interpretation applied where time is *bucketed*, not a property of the data. - The spread-obligation objection to consumer-side snapping (the reason option 2 is rejected) dissolves under centralization: one shared, tested utility replaces per-node hand-rolling instead of multiplying it. Status: fix path settled — proceeding RED-first against the mis-bucketing symptom.

Brummel referenced this issue from a commit

2026-07-18 14:40:49 +02:00

test: red for #280 — jitter-early boundary bars mis-bucketed by raw stamps

Brummel referenced this issue from a commit

2026-07-18 14:40:49 +02:00

fix(std): snap time-bucket/session reads to the nominal minute

Brummel closed this issue

2026-07-18 14:40:49 +02:00

Brummel referenced this issue from a commit

2026-07-18 14:40:49 +02:00

audit: cycle-close tidy for the measurement milestone (#280/#281/#282)

claude referenced this issue