Brummel
be2489ac95
Implements #3: DataServer::symbol_meta(symbol) -> Option<InstrumentGeometry> loads a <SYMBOL>.meta.json sidecar from the owned data directory and returns neutral, broker-agnostic price geometry (digits, pip_size, tick_size, lot_size, base, quote). A consumer can now obtain a symbol's bars and its price-interpretation geometry from one server. Design (forks logged on #3): - Neutral output is the struct's shape/semantics, not currency-code normalization: base/quote pass through verbatim. The live sidecars carry wildly heterogeneous base codes (ISO, index codes, descriptive names, stock tickers) with no definable neutral target; the downstream consumer cross-checks them against its own vetted table. - Format quirks are absorbed in a pure parser (meta::from_sidecar_bytes): scientific notation parses natively; a null / non-finite / missing core field or an unsupported schemaVersion yields None, never a partial struct. - A missing sidecar is a normal None (silent); a present-but-unusable one warns to stderr, matching the existing loader's degrade-and-warn style. - serde_json (Value extraction, no derive) is the new direct dependency, chosen over a hand-rolled parser for robust number/escape handling; the README dependency statement is updated to match. - No caching and no Arc<Self>: geometry files are tiny and read rarely, and the read needs no prefetch. Lookup is not gated on bar-index membership. Verified: cargo build and clippy clean (0 warnings); full suite green (7 parser unit tests + 5 symbol_meta integration tests pin each absorbed quirk and the index-independent lookup). Cargo.lock stays untracked per the library convention. closes #3
104 lines
4.3 KiB
Markdown
104 lines
4.3 KiB
Markdown
# data-server
|
|
|
|
High-performance, thread-safe loader and cache for binary market data files
|
|
(Pepperstone **M1** / **tick** format). Each file is a ZIP archive containing
|
|
one `.bin` entry of tightly packed records; it is read and parsed exactly once,
|
|
then shared lock-free across many readers via `Arc<[T]>` chunks.
|
|
|
|
Leaf crate — depends only on `chrono`, `regex`, `serde_json`, `zip`.
|
|
|
|
## Add as dependency
|
|
|
|
```toml
|
|
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }
|
|
```
|
|
|
|
## Interface
|
|
|
|
### Entry points
|
|
|
|
| Item | Description |
|
|
|---|---|
|
|
| `DataServer::new(base_path) -> DataServer` | Scan a directory and build the symbol index. Wrap in `Arc` to stream (see below). |
|
|
| `init_data_server(base_path) -> Arc<DataServer>` | Initialize the **global** singleton. Panics if called twice. |
|
|
| `init_default_data_server()` | Initialize the global from `DEFAULT_DATA_PATH` if that dir exists. Idempotent / safe to call repeatedly. |
|
|
| `get_data_server() -> Option<Arc<DataServer>>` | The global instance, or `None` if not initialized. |
|
|
| `DEFAULT_DATA_PATH: &str` | `"/mnt/tickdata/Pepperstone"`. |
|
|
|
|
### Querying & streaming (`DataServer`)
|
|
|
|
| Method | Returns |
|
|
|---|---|
|
|
| `has_symbol(symbol) -> bool` | Whether the symbol has any files. |
|
|
| `symbols() -> Vec<Arc<str>>` | All known symbols, sorted. |
|
|
| `file_count(symbol, DataFormat) -> Option<usize>` | Number of files for that symbol/format. |
|
|
| `symbol_meta(symbol) -> Option<meta::InstrumentGeometry>` | Neutral price geometry from the symbol's `<SYMBOL>.meta.json` sidecar; `None` if no usable sidecar exists. |
|
|
| `stream_m1(symbol)` / `stream_tick(symbol)` | `Option<SymbolChunkIter<_>>` over all data. |
|
|
| `stream_m1_windowed(symbol, from_ms, to_ms)` / `stream_tick_windowed(...)` | Same, restricted to an inclusive `[from_ms, to_ms]` Unix-millisecond window (`Option<i64>` bounds; files outside are skipped without I/O). |
|
|
|
|
The `stream_*` methods take `self: &Arc<Self>` (background prefetch needs a
|
|
shared owner), so the server **must be an `Arc<DataServer>`**. They return
|
|
`None` if the symbol is unknown (windowed: also if no file falls in the window).
|
|
|
|
### Consuming chunks (`SymbolChunkIter<T>`)
|
|
|
|
`next_chunk(&mut self) -> Option<Arc<[T]>>` yields successive chunks
|
|
(≤ `CHUNK_SIZE` = 1024 records) in chronological order, prefetching the next
|
|
file in the background. **This is an inherent method, not the `Iterator`
|
|
trait** — drive it with a `while let` loop. While the iterator is alive the
|
|
symbol's data is pinned in the cache; on `Drop` the symbol is released and, if
|
|
no other consumer holds it, evicted.
|
|
|
|
### Record types (`data_server::records`)
|
|
|
|
- `M1Parsed { time_ms: i64, open, high, low, close, spread: f64, volume: i64 }`
|
|
- `TickParsed { time_ms: i64, ask: f64, bid: f64 }`
|
|
- `enum DataFormat { M1, Tick }`
|
|
- `trait HasTimestamp { fn time_ms(&self) -> i64; }` (implemented by both parsed types)
|
|
- `delphi_to_unix_ms(f64) -> i64`, `unix_ms_to_year_month(i64) -> (u16, u8)`
|
|
- `RawM1Record` / `RawTickRecord`: the on-disk `#[repr(C, packed)]` layouts,
|
|
plus `M1Parsed::from_raw` / `TickParsed::from_raw`.
|
|
|
|
All timestamps are **Unix milliseconds** (converted from the Delphi
|
|
`TDateTime` epoch on load).
|
|
|
|
### Lower-level modules
|
|
|
|
- `data_server::loader` — `load_m1_file` / `load_tick_file` (ZIP → parsed
|
|
chunks), `CHUNK_SIZE`.
|
|
- `data_server::cache` — `FileCache`, `FileKey`, `ChunkVec<T>`. The
|
|
many-readers / rare-writer cache used internally; rarely needed directly.
|
|
|
|
## Example
|
|
|
|
```rust
|
|
use std::sync::Arc;
|
|
use data_server::DataServer;
|
|
|
|
let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));
|
|
|
|
if server.has_symbol("EURUSD") {
|
|
let mut it = server
|
|
.stream_m1_windowed("EURUSD", Some(from_ms), Some(to_ms))
|
|
.expect("symbol has data in window");
|
|
|
|
while let Some(chunk) = it.next_chunk() {
|
|
for r in chunk.iter() {
|
|
// r: M1Parsed — r.time_ms, r.open, r.high, r.low, r.close, ...
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Concurrency
|
|
|
|
Designed to back hundreds of single-threaded VMs in parallel: `DataServer` is
|
|
`Send + Sync`, shared as `Arc<DataServer>`. Each file loads once; concurrent
|
|
requests for the same file block on a `Condvar` rather than loading twice.
|
|
|
|
## Data files
|
|
|
|
Files must be named `SYMBOL_YYYY_MM.m1` or `SYMBOL_YYYY_MM.tick` (the symbol
|
|
may contain dots, e.g. `AAPL.US_2006_08.m1`) and be ZIP archives containing a
|
|
single `*.bin` entry of packed records.
|