data-server/README.md

data-server

High-performance, thread-safe loader and cache for binary market data files (Pepperstone M1 / tick format). Each file is a ZIP archive containing one .bin entry of tightly packed records; it is read and parsed exactly once, then shared lock-free across many readers via Arc<[T]> chunks.

Leaf crate — depends only on chrono, regex, serde_json, zip.

Add as dependency

data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }

Interface

Entry points

Item	Description
DataServer::new(base_path) -> DataServer	Scan a directory and build the symbol index. Wrap in Arc to stream (see below).
init_data_server(base_path) -> Arc<DataServer>	Initialize the global singleton. Panics if called twice.
init_default_data_server()	Initialize the global from DEFAULT_DATA_PATH if that dir exists. Idempotent / safe to call repeatedly.
get_data_server() -> Option<Arc<DataServer>>	The global instance, or None if not initialized.
DEFAULT_DATA_PATH: &str	"/mnt/tickdata/Pepperstone".

Querying & streaming (DataServer)

Method	Returns
has_symbol(symbol) -> bool	Whether the symbol has any files.
symbols() -> Vec<Arc<str>>	All known symbols, sorted.
file_count(symbol, DataFormat) -> Option<usize>	Number of files for that symbol/format.
symbol_meta(symbol) -> Option<meta::InstrumentGeometry>	Neutral price geometry from the symbol's <SYMBOL>.meta.json sidecar; None if no usable sidecar exists.
stream_m1(symbol) / stream_tick(symbol)	Option<SymbolChunkIter<_>> over all data.
stream_m1_windowed(symbol, from_ms, to_ms) / stream_tick_windowed(...)	Same, restricted to an inclusive [from_ms, to_ms] Unix-millisecond window (Option<i64> bounds; files outside are skipped without I/O).

The stream_* methods take self: &Arc<Self> (background prefetch needs a shared owner), so the server must be an Arc<DataServer>. They return None if the symbol is unknown (windowed: also if no file falls in the window).

Consuming chunks (SymbolChunkIter<T>)

next_chunk(&mut self) -> Option<Arc<[T]>> yields successive chunks (≤ CHUNK_SIZE = 1024 records) in chronological order, prefetching the next file in the background. This is an inherent method, not the Iterator trait — drive it with a while let loop. While the iterator is alive the symbol's data is pinned in the cache; on Drop the symbol is released and, if no other consumer holds it, evicted.

Record types (data_server::records)

• M1Parsed { time_ms: i64, open, high, low, close, spread: f64, volume: i64 }
• TickParsed { time_ms: i64, ask: f64, bid: f64 }
• enum DataFormat { M1, Tick }
• trait HasTimestamp { fn time_ms(&self) -> i64; } (implemented by both parsed types)
• delphi_to_unix_ms(f64) -> i64, unix_ms_to_year_month(i64) -> (u16, u8)
• RawM1Record / RawTickRecord: the on-disk #[repr(C, packed)] layouts, plus M1Parsed::from_raw / TickParsed::from_raw.

All timestamps are Unix milliseconds (converted from the Delphi TDateTime epoch on load).

Instrument geometry (data_server::meta)

• InstrumentGeometry { digits: u32, pip_size: f64, tick_size: f64, lot_size: f64, base: String, quote: String } — neutral, broker-agnostic price geometry for one symbol, loaded by DataServer::symbol_meta from the symbol's <SYMBOL>.meta.json sidecar. base / quote are passed through verbatim (the consumer resolves provider codes against its own reference table). A missing sidecar, an unsupported schema version, or any null / non-finite field yields None — never a partial struct.
• SUPPORTED_SCHEMA_VERSION: u64 — the sidecar schemaVersion the reader accepts.

Lower-level modules

• data_server::loader — load_m1_file / load_tick_file (ZIP → parsed chunks), CHUNK_SIZE.
• data_server::cache — FileCache, FileKey, ChunkVec<T>. The many-readers / rare-writer cache used internally; rarely needed directly.

Example

use std::sync::Arc;
use data_server::DataServer;

let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));

if server.has_symbol("EURUSD") {
    let mut it = server
        .stream_m1_windowed("EURUSD", Some(from_ms), Some(to_ms))
        .expect("symbol has data in window");

    while let Some(chunk) = it.next_chunk() {
        for r in chunk.iter() {
            // r: M1Parsed — r.time_ms, r.open, r.high, r.low, r.close, ...
        }
    }
}

Concurrency

Designed to back hundreds of single-threaded VMs in parallel: DataServer is Send + Sync, shared as Arc<DataServer>. Each file loads once; concurrent requests for the same file block on a Condvar rather than loading twice.

Data files

Files must be named SYMBOL_YYYY_MM.m1 or SYMBOL_YYYY_MM.tick (the symbol may contain dots, e.g. AAPL.US_2006_08.m1) and be ZIP archives containing a single *.bin entry of packed records.