diff --git a/README.md b/README.md index 872525f..5ab048f 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,102 @@ # data-server High-performance, thread-safe loader and cache for binary market data files -(Pepperstone M1 / tick format). +(Pepperstone **M1** / **tick** format). Each file is a ZIP archive containing +one `.bin` entry of tightly packed records; it is read and parsed exactly once, +then shared lock-free across many readers via `Arc<[T]>` chunks. -Extracted from the `myc` project as a standalone, dependency-free leaf crate -(only `chrono`, `regex`, `zip`). +Standalone leaf crate — depends only on `chrono`, `regex`, `zip`. -## Usage +## Add as dependency ```toml -# Cargo.toml of the consuming project data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" } ``` -For local co-development, override with a per-machine `.cargo/config.toml`: +## Interface -```toml -paths = ["/home/brummel/dev/libs/data-server"] +### Entry points + +| Item | Description | +|---|---| +| `DataServer::new(base_path) -> DataServer` | Scan a directory and build the symbol index. Wrap in `Arc` to stream (see below). | +| `init_data_server(base_path) -> Arc` | Initialize the **global** singleton. Panics if called twice. | +| `init_default_data_server()` | Initialize the global from `DEFAULT_DATA_PATH` if that dir exists. Idempotent / safe to call repeatedly. | +| `get_data_server() -> Option>` | The global instance, or `None` if not initialized. | +| `DEFAULT_DATA_PATH: &str` | `"/mnt/tickdata/Pepperstone"`. | + +### Querying & streaming (`DataServer`) + +| Method | Returns | +|---|---| +| `has_symbol(symbol) -> bool` | Whether the symbol has any files. | +| `symbols() -> Vec>` | All known symbols, sorted. | +| `file_count(symbol, DataFormat) -> Option` | Number of files for that symbol/format. | +| `stream_m1(symbol)` / `stream_tick(symbol)` | `Option>` over all data. | +| `stream_m1_windowed(symbol, from_ms, to_ms)` / `stream_tick_windowed(...)` | Same, restricted to an inclusive `[from_ms, to_ms]` Unix-millisecond window (`Option` bounds; files outside are skipped without I/O). | + +The `stream_*` methods take `self: &Arc` (background prefetch needs a +shared owner), so the server **must be an `Arc`**. They return +`None` if the symbol is unknown (windowed: also if no file falls in the window). + +### Consuming chunks (`SymbolChunkIter`) + +`next_chunk(&mut self) -> Option>` yields successive chunks +(≤ `CHUNK_SIZE` = 1024 records) in chronological order, prefetching the next +file in the background. **This is an inherent method, not the `Iterator` +trait** — drive it with a `while let` loop. While the iterator is alive the +symbol's data is pinned in the cache; on `Drop` the symbol is released and, if +no other consumer holds it, evicted. + +### Record types (`data_server::records`) + +- `M1Parsed { time_ms: i64, open, high, low, close, spread: f64, volume: i64 }` +- `TickParsed { time_ms: i64, ask: f64, bid: f64 }` +- `enum DataFormat { M1, Tick }` +- `trait HasTimestamp { fn time_ms(&self) -> i64; }` (implemented by both parsed types) +- `delphi_to_unix_ms(f64) -> i64`, `unix_ms_to_year_month(i64) -> (u16, u8)` +- `RawM1Record` / `RawTickRecord`: the on-disk `#[repr(C, packed)]` layouts, + plus `M1Parsed::from_raw` / `TickParsed::from_raw`. + +All timestamps are **Unix milliseconds** (converted from the Delphi +`TDateTime` epoch on load). + +### Lower-level modules + +- `data_server::loader` — `load_m1_file` / `load_tick_file` (ZIP → parsed + chunks), `CHUNK_SIZE`. +- `data_server::cache` — `FileCache`, `FileKey`, `ChunkVec`. The + many-readers / rare-writer cache used internally; rarely needed directly. + +## Example + +```rust +use std::sync::Arc; +use data_server::DataServer; + +let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone")); + +if server.has_symbol("EURUSD") { + let mut it = server + .stream_m1_windowed("EURUSD", Some(from_ms), Some(to_ms)) + .expect("symbol has data in window"); + + while let Some(chunk) = it.next_chunk() { + for r in chunk.iter() { + // r: M1Parsed — r.time_ms, r.open, r.high, r.low, r.close, ... + } + } +} ``` + +## Concurrency + +Designed to back hundreds of single-threaded VMs in parallel: `DataServer` is +`Send + Sync`, shared as `Arc`. Each file loads once; concurrent +requests for the same file block on a `Condvar` rather than loading twice. + +## Data files + +Files must be named `SYMBOL_YYYY_MM.m1` or `SYMBOL_YYYY_MM.tick` (the symbol +may contain dots, e.g. `AAPL.US_2006_08.m1`) and be ZIP archives containing a +single `*.bin` entry of packed records.