Refactor data streams to support time windows

This commit introduces time windowing functionality to the M1 and tick
data streams.

Key changes:

- **`DataServer::stream_m1_windowed` and
  `DataServer::stream_tick_windowed`**: New methods added to
  `DataServer` to allow streaming data within a specified `from_ms` and
  `to_ms` range.
- **`SymbolChunkIter` modifications**: The `SymbolChunkIter` struct now
  includes `from_ms`, `to_ms`, and `exhausted` fields to manage time
  window filtering.
- **`filter_chunk` helper function**: A new utility function to
  efficiently filter individual data chunks based on the time window.
  This function also handles early termination when `to_ms` is exceeded.
- **`HasTimestamp` trait**: Introduced to provide a common interface for
  accessing timestamps across different record types (`M1Parsed`,
  `TickParsed`), enabling generic filtering.
- **`filter_files` method**: Added to `DataServer` to pre-filter file
  keys based on the time window before creating iterators. This
  optimizes file loading by skipping files that fall entirely outside
  the desired range.
- **`create-m1-stream` and `create-tick-stream`**: The native functions
  in `data_streams.rs` are updated to accept optional `DateTime`
  arguments for `from` and `to` timestamps. They now also handle unknown
  symbols by returning `Value::Void` and return an empty stream if the
  time window results in no data.
- **Documentation**: Updated doc comments for the stream creation
  functions to reflect the new time windowing capabilities and to
  clarify behavior for unknown symbols and empty time windows.

These changes significantly enhance the flexibility of data streaming,
allowing users to query specific time ranges of M1 and tick data more
efficiently.
This commit is contained in:
2026-03-29 22:03:33 +02:00
parent 1f68371920
commit e82d48d1cb
3 changed files with 352 additions and 46 deletions
+44
View File
@@ -17,6 +17,26 @@ pub fn delphi_to_unix_ms(dt: f64) -> i64 {
((dt - DELPHI_EPOCH_OFFSET_DAYS) * MS_PER_DAY) as i64
}
/// Extracts the (year, month) from a Unix-millisecond timestamp.
///
/// Used for coarse file-level filtering: data files are named `SYMBOL_YYYY_MM.ext`,
/// so we can skip entire files that fall outside the requested time window.
#[inline]
pub fn unix_ms_to_year_month(ms: i64) -> (u16, u8) {
use chrono::Datelike;
let dt = chrono::DateTime::from_timestamp_millis(ms)
.expect("unix_ms_to_year_month: invalid timestamp");
(dt.year() as u16, dt.month() as u8)
}
/// Common timestamp access for parsed record types.
///
/// Enables generic time-window filtering in [`super::SymbolChunkIter`] without
/// duplicating the logic for M1 and tick records.
pub trait HasTimestamp {
fn time_ms(&self) -> i64;
}
/// Raw M1 (minute) record as stored on disk. 48 bytes, packed.
///
/// Matches the Delphi `TM1Record = packed record` layout exactly:
@@ -75,6 +95,20 @@ pub enum DataFormat {
Tick,
}
impl HasTimestamp for M1Parsed {
#[inline]
fn time_ms(&self) -> i64 {
self.time_ms
}
}
impl HasTimestamp for TickParsed {
#[inline]
fn time_ms(&self) -> i64 {
self.time_ms
}
}
impl M1Parsed {
/// Converts a raw on-disk record to the parsed representation.
///
@@ -136,6 +170,16 @@ mod tests {
assert_eq!(unix_ms, 0);
}
#[test]
fn test_unix_ms_to_year_month() {
// 2017-03-01 00:00:00 UTC = 1488326400000 ms
assert_eq!(unix_ms_to_year_month(1_488_326_400_000), (2017, 3));
// 2020-12-31 23:59:59.999 UTC
assert_eq!(unix_ms_to_year_month(1_609_459_199_999), (2020, 12));
// Unix epoch
assert_eq!(unix_ms_to_year_month(0), (1970, 1));
}
#[test]
fn test_raw_m1_record_size() {
assert_eq!(std::mem::size_of::<RawM1Record>(), 48);