Brummel
694f96f2b2
Cycle-close for the per-symbol geometry sidecar reader (#3). Architect drift review (0f5e665..be2489a): code matches the spec verbatim; the README "## Interface" contract was already updated in lockstep (symbol_meta row + serde_json dependency line). Two drift items, both resolved here: - [medium] Retire the ephemeral active-cycle artefacts: git-rm the spec and plan (recoverable from history via `git show <rev>:<path>`), per the docs/specs + docs/plans lifecycle convention. - [low] Document meta::InstrumentGeometry to parity with the other public record types: add a "data_server::meta" subsection enumerating its six fields and SUPPORTED_SCHEMA_VERSION. Regression gate (all green): - cargo build -> exit 0, finished clean - cargo clippy -> exit 0, 0 warnings - cargo test -> exit 0; 36 lib + 4 integration + 0 doc tests passed No baseline to update (the project has no metric baseline; cargo is the gate). refs #3
115 lines
5.0 KiB
Markdown
115 lines
5.0 KiB
Markdown
# data-server
|
|
|
|
High-performance, thread-safe loader and cache for binary market data files
|
|
(Pepperstone **M1** / **tick** format). Each file is a ZIP archive containing
|
|
one `.bin` entry of tightly packed records; it is read and parsed exactly once,
|
|
then shared lock-free across many readers via `Arc<[T]>` chunks.
|
|
|
|
Leaf crate — depends only on `chrono`, `regex`, `serde_json`, `zip`.
|
|
|
|
## Add as dependency
|
|
|
|
```toml
|
|
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }
|
|
```
|
|
|
|
## Interface
|
|
|
|
### Entry points
|
|
|
|
| Item | Description |
|
|
|---|---|
|
|
| `DataServer::new(base_path) -> DataServer` | Scan a directory and build the symbol index. Wrap in `Arc` to stream (see below). |
|
|
| `init_data_server(base_path) -> Arc<DataServer>` | Initialize the **global** singleton. Panics if called twice. |
|
|
| `init_default_data_server()` | Initialize the global from `DEFAULT_DATA_PATH` if that dir exists. Idempotent / safe to call repeatedly. |
|
|
| `get_data_server() -> Option<Arc<DataServer>>` | The global instance, or `None` if not initialized. |
|
|
| `DEFAULT_DATA_PATH: &str` | `"/mnt/tickdata/Pepperstone"`. |
|
|
|
|
### Querying & streaming (`DataServer`)
|
|
|
|
| Method | Returns |
|
|
|---|---|
|
|
| `has_symbol(symbol) -> bool` | Whether the symbol has any files. |
|
|
| `symbols() -> Vec<Arc<str>>` | All known symbols, sorted. |
|
|
| `file_count(symbol, DataFormat) -> Option<usize>` | Number of files for that symbol/format. |
|
|
| `symbol_meta(symbol) -> Option<meta::InstrumentGeometry>` | Neutral price geometry from the symbol's `<SYMBOL>.meta.json` sidecar; `None` if no usable sidecar exists. |
|
|
| `stream_m1(symbol)` / `stream_tick(symbol)` | `Option<SymbolChunkIter<_>>` over all data. |
|
|
| `stream_m1_windowed(symbol, from_ms, to_ms)` / `stream_tick_windowed(...)` | Same, restricted to an inclusive `[from_ms, to_ms]` Unix-millisecond window (`Option<i64>` bounds; files outside are skipped without I/O). |
|
|
|
|
The `stream_*` methods take `self: &Arc<Self>` (background prefetch needs a
|
|
shared owner), so the server **must be an `Arc<DataServer>`**. They return
|
|
`None` if the symbol is unknown (windowed: also if no file falls in the window).
|
|
|
|
### Consuming chunks (`SymbolChunkIter<T>`)
|
|
|
|
`next_chunk(&mut self) -> Option<Arc<[T]>>` yields successive chunks
|
|
(≤ `CHUNK_SIZE` = 1024 records) in chronological order, prefetching the next
|
|
file in the background. **This is an inherent method, not the `Iterator`
|
|
trait** — drive it with a `while let` loop. While the iterator is alive the
|
|
symbol's data is pinned in the cache; on `Drop` the symbol is released and, if
|
|
no other consumer holds it, evicted.
|
|
|
|
### Record types (`data_server::records`)
|
|
|
|
- `M1Parsed { time_ms: i64, open, high, low, close, spread: f64, volume: i64 }`
|
|
- `TickParsed { time_ms: i64, ask: f64, bid: f64 }`
|
|
- `enum DataFormat { M1, Tick }`
|
|
- `trait HasTimestamp { fn time_ms(&self) -> i64; }` (implemented by both parsed types)
|
|
- `delphi_to_unix_ms(f64) -> i64`, `unix_ms_to_year_month(i64) -> (u16, u8)`
|
|
- `RawM1Record` / `RawTickRecord`: the on-disk `#[repr(C, packed)]` layouts,
|
|
plus `M1Parsed::from_raw` / `TickParsed::from_raw`.
|
|
|
|
All timestamps are **Unix milliseconds** (converted from the Delphi
|
|
`TDateTime` epoch on load).
|
|
|
|
### Instrument geometry (`data_server::meta`)
|
|
|
|
- `InstrumentGeometry { digits: u32, pip_size: f64, tick_size: f64, lot_size: f64, base: String, quote: String }`
|
|
— neutral, broker-agnostic price geometry for one symbol, loaded by
|
|
`DataServer::symbol_meta` from the symbol's `<SYMBOL>.meta.json` sidecar.
|
|
`base` / `quote` are passed through verbatim (the consumer resolves provider
|
|
codes against its own reference table). A missing sidecar, an unsupported
|
|
schema version, or any `null` / non-finite field yields `None` — never a
|
|
partial struct.
|
|
- `SUPPORTED_SCHEMA_VERSION: u64` — the sidecar `schemaVersion` the reader accepts.
|
|
|
|
### Lower-level modules
|
|
|
|
- `data_server::loader` — `load_m1_file` / `load_tick_file` (ZIP → parsed
|
|
chunks), `CHUNK_SIZE`.
|
|
- `data_server::cache` — `FileCache`, `FileKey`, `ChunkVec<T>`. The
|
|
many-readers / rare-writer cache used internally; rarely needed directly.
|
|
|
|
## Example
|
|
|
|
```rust
|
|
use std::sync::Arc;
|
|
use data_server::DataServer;
|
|
|
|
let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));
|
|
|
|
if server.has_symbol("EURUSD") {
|
|
let mut it = server
|
|
.stream_m1_windowed("EURUSD", Some(from_ms), Some(to_ms))
|
|
.expect("symbol has data in window");
|
|
|
|
while let Some(chunk) = it.next_chunk() {
|
|
for r in chunk.iter() {
|
|
// r: M1Parsed — r.time_ms, r.open, r.high, r.low, r.close, ...
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Concurrency
|
|
|
|
Designed to back hundreds of single-threaded VMs in parallel: `DataServer` is
|
|
`Send + Sync`, shared as `Arc<DataServer>`. Each file loads once; concurrent
|
|
requests for the same file block on a `Condvar` rather than loading twice.
|
|
|
|
## Data files
|
|
|
|
Files must be named `SYMBOL_YYYY_MM.m1` or `SYMBOL_YYYY_MM.tick` (the symbol
|
|
may contain dots, e.g. `AAPL.US_2006_08.m1`) and be ZIP archives containing a
|
|
single `*.bin` entry of packed records.
|