This commit introduces time windowing functionality to the M1 and tick
data streams.
Key changes:
- **`DataServer::stream_m1_windowed` and
`DataServer::stream_tick_windowed`**: New methods added to
`DataServer` to allow streaming data within a specified `from_ms` and
`to_ms` range.
- **`SymbolChunkIter` modifications**: The `SymbolChunkIter` struct now
includes `from_ms`, `to_ms`, and `exhausted` fields to manage time
window filtering.
- **`filter_chunk` helper function**: A new utility function to
efficiently filter individual data chunks based on the time window.
This function also handles early termination when `to_ms` is exceeded.
- **`HasTimestamp` trait**: Introduced to provide a common interface for
accessing timestamps across different record types (`M1Parsed`,
`TickParsed`), enabling generic filtering.
- **`filter_files` method**: Added to `DataServer` to pre-filter file
keys based on the time window before creating iterators. This
optimizes file loading by skipping files that fall entirely outside
the desired range.
- **`create-m1-stream` and `create-tick-stream`**: The native functions
in `data_streams.rs` are updated to accept optional `DateTime`
arguments for `from` and `to` timestamps. They now also handle unknown
symbols by returning `Value::Void` and return an empty stream if the
time window results in no data.
- **Documentation**: Updated doc comments for the stream creation
functions to reflect the new time windowing capabilities and to
clarify behavior for unknown symbols and empty time windows.
These changes significantly enhance the flexibility of data streaming,
allowing users to query specific time ranges of M1 and tick data more
efficiently.
This commit introduces a new data server module (`src/ast/data_server`)
designed for high-performance, thread-safe loading and caching of market
data.
Key components:
- `DataServer`: Manages symbol indexing and delegates loading/caching.
- `SymbolIndex`: Scans the data directory and maintains a sorted index
of data files per symbol.
- `FileCache`: Implements a thread-safe cache using `RwLock` and a
loading guard to prevent duplicate work.
- `loader`: Handles ZIP decompression and binary record parsing from
`.bin` files.
- `records`: Defines raw and parsed data structures for M1 and tick
data.
- `SymbolChunkIter`: Provides an iterator over pre-parsed data chunks,
prefetching subsequent files.
This architecture allows multiple VM threads to access market data
concurrently without redundant I/O, mirroring the strategy used in the
original Delphi `TDataServer`.
Brummel