Refactor data streams to support time windows
This commit introduces time windowing functionality to the M1 and tick data streams. Key changes: - **`DataServer::stream_m1_windowed` and `DataServer::stream_tick_windowed`**: New methods added to `DataServer` to allow streaming data within a specified `from_ms` and `to_ms` range. - **`SymbolChunkIter` modifications**: The `SymbolChunkIter` struct now includes `from_ms`, `to_ms`, and `exhausted` fields to manage time window filtering. - **`filter_chunk` helper function**: A new utility function to efficiently filter individual data chunks based on the time window. This function also handles early termination when `to_ms` is exceeded. - **`HasTimestamp` trait**: Introduced to provide a common interface for accessing timestamps across different record types (`M1Parsed`, `TickParsed`), enabling generic filtering. - **`filter_files` method**: Added to `DataServer` to pre-filter file keys based on the time window before creating iterators. This optimizes file loading by skipping files that fall entirely outside the desired range. - **`create-m1-stream` and `create-tick-stream`**: The native functions in `data_streams.rs` are updated to accept optional `DateTime` arguments for `from` and `to` timestamps. They now also handle unknown symbols by returning `Value::Void` and return an empty stream if the time window results in no data. - **Documentation**: Updated doc comments for the stream creation functions to reflect the new time windowing capabilities and to clarify behavior for unknown symbols and empty time windows. These changes significantly enhance the flexibility of data streaming, allowing users to query specific time ranges of M1 and tick data more efficiently.
This commit is contained in:
@@ -17,6 +17,26 @@ pub fn delphi_to_unix_ms(dt: f64) -> i64 {
|
||||
((dt - DELPHI_EPOCH_OFFSET_DAYS) * MS_PER_DAY) as i64
|
||||
}
|
||||
|
||||
/// Extracts the (year, month) from a Unix-millisecond timestamp.
|
||||
///
|
||||
/// Used for coarse file-level filtering: data files are named `SYMBOL_YYYY_MM.ext`,
|
||||
/// so we can skip entire files that fall outside the requested time window.
|
||||
#[inline]
|
||||
pub fn unix_ms_to_year_month(ms: i64) -> (u16, u8) {
|
||||
use chrono::Datelike;
|
||||
let dt = chrono::DateTime::from_timestamp_millis(ms)
|
||||
.expect("unix_ms_to_year_month: invalid timestamp");
|
||||
(dt.year() as u16, dt.month() as u8)
|
||||
}
|
||||
|
||||
/// Common timestamp access for parsed record types.
|
||||
///
|
||||
/// Enables generic time-window filtering in [`super::SymbolChunkIter`] without
|
||||
/// duplicating the logic for M1 and tick records.
|
||||
pub trait HasTimestamp {
|
||||
fn time_ms(&self) -> i64;
|
||||
}
|
||||
|
||||
/// Raw M1 (minute) record as stored on disk. 48 bytes, packed.
|
||||
///
|
||||
/// Matches the Delphi `TM1Record = packed record` layout exactly:
|
||||
@@ -75,6 +95,20 @@ pub enum DataFormat {
|
||||
Tick,
|
||||
}
|
||||
|
||||
impl HasTimestamp for M1Parsed {
|
||||
#[inline]
|
||||
fn time_ms(&self) -> i64 {
|
||||
self.time_ms
|
||||
}
|
||||
}
|
||||
|
||||
impl HasTimestamp for TickParsed {
|
||||
#[inline]
|
||||
fn time_ms(&self) -> i64 {
|
||||
self.time_ms
|
||||
}
|
||||
}
|
||||
|
||||
impl M1Parsed {
|
||||
/// Converts a raw on-disk record to the parsed representation.
|
||||
///
|
||||
@@ -136,6 +170,16 @@ mod tests {
|
||||
assert_eq!(unix_ms, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_unix_ms_to_year_month() {
|
||||
// 2017-03-01 00:00:00 UTC = 1488326400000 ms
|
||||
assert_eq!(unix_ms_to_year_month(1_488_326_400_000), (2017, 3));
|
||||
// 2020-12-31 23:59:59.999 UTC
|
||||
assert_eq!(unix_ms_to_year_month(1_609_459_199_999), (2020, 12));
|
||||
// Unix epoch
|
||||
assert_eq!(unix_ms_to_year_month(0), (1970, 1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_raw_m1_record_size() {
|
||||
assert_eq!(std::mem::size_of::<RawM1Record>(), 48);
|
||||
|
||||
Reference in New Issue
Block a user