# data-server

High-performance, thread-safe loader and cache for binary market data files
(Pepperstone **M1** / **tick** format). Each file is a ZIP archive containing
one `.bin` entry of tightly packed records; it is read and parsed exactly once,
then shared lock-free across many readers via `Arc<[T]>` chunks.

Leaf crate — depends only on `chrono`, `regex`, `serde_json`, `zip`.

## Add as dependency

```toml
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }
```

## Interface

### Entry points

| Item | Description |
|---|---|
| `DataServer::new(base_path) -> DataServer` | Scan a directory and build the symbol index. Wrap in `Arc` to stream (see below). |
| `init_data_server(base_path) -> Arc<DataServer>` | Initialize the **global** singleton. Panics if called twice. |
| `init_default_data_server()` | Initialize the global from `DEFAULT_DATA_PATH` if that dir exists. Idempotent / safe to call repeatedly. |
| `get_data_server() -> Option<Arc<DataServer>>` | The global instance, or `None` if not initialized. |
| `DEFAULT_DATA_PATH: &str` | `"/mnt/tickdata/Pepperstone"`. |

### Querying & streaming (`DataServer`)

| Method | Returns |
|---|---|
| `has_symbol(symbol) -> bool` | Whether the symbol has any files. |
| `symbols() -> Vec<Arc<str>>` | All known symbols, sorted. |
| `file_count(symbol, DataFormat) -> Option<usize>` | Number of files for that symbol/format. |
| `symbol_meta(symbol) -> Option<meta::InstrumentGeometry>` | Neutral price geometry from the symbol's `<SYMBOL>.meta.json` sidecar; `None` if no usable sidecar exists. |
| `stream_m1(symbol)` / `stream_tick(symbol)` | `Option<SymbolChunkIter<_>>` over all data. |
| `stream_m1_windowed(symbol, from_ms, to_ms)` / `stream_tick_windowed(...)` | Same, restricted to an inclusive `[from_ms, to_ms]` Unix-millisecond window (`Option<i64>` bounds; files outside are skipped without I/O). |

The `stream_*` methods take `self: &Arc<Self>` (background prefetch needs a
shared owner), so the server **must be an `Arc<DataServer>`**. They return
`None` if the symbol is unknown (windowed: also if no file falls in the window).

### Consuming chunks (`SymbolChunkIter<T>`)

`next_chunk(&mut self) -> Option<Arc<[T]>>` yields successive chunks
(≤ `CHUNK_SIZE` = 1024 records) in chronological order, prefetching the next
file in the background. **This is an inherent method, not the `Iterator`
trait** — drive it with a `while let` loop. While the iterator is alive the
symbol's data is pinned in the cache; on `Drop` the symbol is released and, if
no other consumer holds it, evicted.

### Record types (`data_server::records`)

- `M1Parsed { time_ms: i64, open, high, low, close, spread: f64, volume: i64 }`
- `TickParsed { time_ms: i64, ask: f64, bid: f64 }`
- `enum DataFormat { M1, Tick }`
- `trait HasTimestamp { fn time_ms(&self) -> i64; }` (implemented by both parsed types)
- `delphi_to_unix_ms(f64) -> i64`, `unix_ms_to_year_month(i64) -> (u16, u8)`
- `RawM1Record` / `RawTickRecord`: the on-disk `#[repr(C, packed)]` layouts,
  plus `M1Parsed::from_raw` / `TickParsed::from_raw`.

All timestamps are **Unix milliseconds** (converted from the Delphi
`TDateTime` epoch on load).

### Instrument geometry (`data_server::meta`)

- `InstrumentGeometry { digits: u32, pip_size: f64, tick_size: f64, lot_size: f64, base: String, quote: String }`
  — neutral, broker-agnostic price geometry for one symbol, loaded by
  `DataServer::symbol_meta` from the symbol's `<SYMBOL>.meta.json` sidecar.
  `base` / `quote` are passed through verbatim (the consumer resolves provider
  codes against its own reference table). A missing sidecar, an unsupported
  schema version, or any `null` / non-finite field yields `None` — never a
  partial struct.
- `SUPPORTED_SCHEMA_VERSION: u64` — the sidecar `schemaVersion` the reader accepts.

### Lower-level modules

- `data_server::loader` — `load_m1_file` / `load_tick_file` (ZIP → parsed
  chunks), `CHUNK_SIZE`.
- `data_server::cache` — `FileCache`, `FileKey`, `ChunkVec<T>`. The
  many-readers / rare-writer cache used internally; rarely needed directly.

## Example

```rust
use std::sync::Arc;
use data_server::DataServer;

let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));

if server.has_symbol("EURUSD") {
    let mut it = server
        .stream_m1_windowed("EURUSD", Some(from_ms), Some(to_ms))
        .expect("symbol has data in window");

    while let Some(chunk) = it.next_chunk() {
        for r in chunk.iter() {
            // r: M1Parsed — r.time_ms, r.open, r.high, r.low, r.close, ...
        }
    }
}
```

## Concurrency

Designed to back hundreds of single-threaded VMs in parallel: `DataServer` is
`Send + Sync`, shared as `Arc<DataServer>`. Each file loads once; concurrent
requests for the same file block on a `Condvar` rather than loading twice.

## Data files

Files must be named `SYMBOL_YYYY_MM.m1` or `SYMBOL_YYYY_MM.tick` (the symbol
may contain dots, e.g. `AAPL.US_2006_08.m1`) and be ZIP archives containing a
single `*.bin` entry of packed records.