# Per-symbol geometry sidecar reader — Design Spec **Date:** 2026-06-25 **Status:** Draft — awaiting user spec review **Authors:** orchestrator + Claude Tracker: Brummel/data-server#3 (reference issue; fork decisions logged there). ## Goal Add a reader that loads a per-symbol `.meta.json` sidecar from the data directory this server already owns and returns **neutral** price-interpretation geometry — `digits`, `pip_size`, `tick_size`, `lot_size`, `base`, `quote` — so a downstream consumer can ask one server for a symbol's bars *and* its geometry without parsing provider JSON itself. The sidecar carries the price-interpretation facts the bars lack (how many decimals a price has, what a pip is, what one lot is worth, which currency the price is quoted in). Keeping the sidecar's filename, schema version, and format quirks (scientific-notation numbers, `null` for a non-finite value, an unknown schema version) inside this broker-facing library is what lets the consumer stay broker-agnostic: it receives a plain Rust struct or nothing, never raw JSON. Non-goals: time-varying / account-dependent conditions (commission, swap, deposit-currency pip value, volume limits, trading status) — a separate, time-indexed concern, not this reader's job. No currency-code normalization (see §Error handling, base/quote). No caching (see §Components). ## Architecture A new leaf module `src/meta.rs` owns the neutral type and the pure parser; a new one-shot method on `DataServer` owns the filesystem read. ```text DataServer ┌───────────────────────────────────────┐ │ index: SymbolIndex { base_path, … } │ │ cache: Arc (bars only) │ │ │ │ symbol_meta(&self, symbol) ────────────┼──► fs::read(base_path/.meta.json) └───────────────────────────────────────┘ │ ▼ meta::InstrumentGeometry::from_sidecar_bytes(&[u8]) -> Option<…> ``` The reader is independent of the bar `FileCache` and the streaming path: it does not touch the cache, spawns no thread, and needs no `Arc`. It reaches the data directory through the existing `SymbolIndex::base_path` — the same field the bar file paths are built from (ratified by `test_file_path_construction` in `src/lib.rs`, which pins `base_path` as the data directory). Membership in the bar index is **not** required: a symbol with a sidecar but no bar files, or bar files but no sidecar, is handled by the file existing or not. ## Concrete code shapes ### Worked consumer program (the acceptance evidence) What an Aura-side consumer (the counterpart, Brummel/Aura#143) writes — bars and geometry from one server, no provider JSON in sight: ```rust use std::sync::Arc; use data_server::DataServer; let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone")); if let Some(geo) = server.symbol_meta("EURUSD") { // geo: InstrumentGeometry — neutral, plain fields assert_eq!(geo.digits, 5); assert_eq!(geo.pip_size, 0.0001); assert_eq!(geo.tick_size, 1e-05); // sidecar wrote it as 1E-05 assert_eq!(geo.lot_size, 100_000.0); assert_eq!(geo.quote, "USD"); // The consumer derives the rest; nothing provider-specific reached it. let pip_value_per_lot = geo.lot_size * geo.pip_size; // 100000 * 0.0001 = 10 USD } // Pairs with the existing symbol surface. for sym in server.symbols() { if let Some(geo) = server.symbol_meta(&sym) { // … cross-check geo.base / geo.quote against the consumer's vetted table. } } ``` `symbol_meta` returns `None` for: no sidecar file, malformed JSON, an unsupported schema version, or any missing / `null` / non-finite core field (see §Error handling). The consumer never sees a partial geometry. ### Implementation shapes (secondary) New file `src/meta.rs` — before: does not exist. After: ```rust //! Reader for per-symbol `.meta.json` geometry sidecars. use serde_json::Value; /// The only sidecar schema version this reader understands. pub const SUPPORTED_SCHEMA_VERSION: u64 = 2; /// Neutral, broker-agnostic price-interpretation geometry for one symbol. /// /// Intrinsic to the historical price series and effectively constant over its /// span. Derived/account-dependent conditions are deliberately excluded. #[derive(Debug, Clone, PartialEq)] pub struct InstrumentGeometry { /// Number of decimal places in a quoted price (e.g. 5 for EURUSD). pub digits: u32, /// Price increment of one pip (e.g. 0.0001 for EURUSD). pub pip_size: f64, /// Smallest representable price increment (e.g. 1e-05 for EURUSD). pub tick_size: f64, /// Contract size of one lot in base units (e.g. 100000 for EURUSD). pub lot_size: f64, /// Base asset code, passed through verbatim from the sidecar. pub base: String, /// Quote (price) currency code, passed through verbatim. pub quote: String, } impl InstrumentGeometry { /// Parse sidecar JSON bytes into neutral geometry. /// /// Returns `None` if the JSON is malformed, the schema version is /// unsupported, or any core field is missing / `null` / non-finite. pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option { let v: Value = serde_json::from_slice(bytes).ok()?; if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION { return None; } Some(Self { digits: v.get("digits").and_then(Value::as_u64)? as u32, pip_size: finite(v.get("pipSize"))?, tick_size: finite(v.get("tickSize"))?, lot_size: finite(v.get("lotSize"))?, base: v.get("baseAsset").and_then(Value::as_str)?.to_string(), quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(), }) } } /// A finite `f64` from a JSON number, or `None` for missing / null / non-finite. /// `Value::as_f64` already rejects `null` and non-numbers; this also rejects /// `NaN`/`Inf` defensively. fn finite(v: Option<&Value>) -> Option { let n = v?.as_f64()?; n.is_finite().then_some(n) } ``` New method on `DataServer` in `src/lib.rs` — before: absent. After: ```rust /// Loads the neutral price geometry for a symbol from its /// `.meta.json` sidecar, or `None` if no usable sidecar exists. /// /// A missing sidecar is normal (`None`, silent). A present-but-unusable /// sidecar (malformed, unsupported version, incomplete) returns `None` /// after a one-line stderr warning. pub fn symbol_meta(&self, symbol: &str) -> Option { let path = self.index.base_path.join(format!("{symbol}.meta.json")); let bytes = std::fs::read(&path).ok()?; // missing file -> silent None let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes); if geo.is_none() { eprintln!("Warning: {} present but unusable (bad/incomplete geometry)", path.display()); } geo } ``` Module wiring in `src/lib.rs` — add `pub mod meta;` alongside the existing `pub mod cache; pub mod loader; pub mod records;`. `base_path` is already a field of the private `SymbolIndex`; `symbol_meta` is a `DataServer` method in the same crate, so it reads `self.index.base_path` directly (no visibility change). `Cargo.toml` — add one direct dependency: ```toml serde_json = "1" ``` ## Components - **`meta::InstrumentGeometry`** — the public neutral struct (six fields, `Debug + Clone + PartialEq`). Excludes `symbol`, `description`, `generator`, `generatedAtUtc` (the most provider-specific fields; `symbol` is already known to the caller). - **`meta::InstrumentGeometry::from_sidecar_bytes`** — `pub(crate)` pure parser, bytes → `Option`. No filesystem, fully unit-testable. Absorbs every format quirk here. - **`meta::SUPPORTED_SCHEMA_VERSION`** — the `u64` gate (currently `2`). - **`DataServer::symbol_meta`** — `&self` one-shot read; the only filesystem touch. No caching: geometry files are tiny and read rarely (typically once per symbol); a cache map + invalidation is complexity for no stated need, and the bar `FileCache` exists for a different problem (large, repeatedly-streamed files). ## Data flow 1. Consumer calls `server.symbol_meta("EURUSD")`. 2. Path built: `/EURUSD.meta.json` (symbols may contain dots, e.g. `AAPL.US.meta.json`; the literal join handles them). 3. `fs::read` → on error (missing/unreadable) return `None` silently. 4. Bytes → `from_sidecar_bytes`: parse JSON; gate `schemaVersion == 2`; extract the six fields with the null/non-finite guard. 5. `Some(geometry)` on success; `None` + one stderr warning on a present but unusable file. ## Error handling The reader absorbs every documented provider quirk so none reaches a consumer: - **Scientific notation** (`1E-05`, in 7 of 30 live files) — valid JSON, parses as a number; `Value::as_f64` handles it. No special code. - **`null` for a non-finite broker value** — `Value::as_f64`/`as_u64` return `None` for `null`; that propagates to `symbol_meta` returning `None`. The consumer never sees a hole. (No `null` exists in the current 30 samples; this is defensive.) `finite()` additionally rejects `NaN`/`Inf`. - **Unknown `schemaVersion`** — a reader-side decision, not a consumer's: an unsupported version returns `None` (+ the call-site warning), never an error type the consumer must handle. - **base / quote provider codes** — passed through **verbatim** as `String`. `baseAsset` is wildly heterogeneous across the 30 samples (ISO `EUR`/`USD`, index codes `DE40`/`USTEC`/`JP225`, descriptive `Bitcoin`/`Copper`, stock tickers `AAPL.US`); there is no well-defined "neutral" target and no mapping table, so normalization is not even definable here. The consumer (Aura#143) cross-checks the value against its own hand-authored vetted table and never reads the raw JSON — resolution is the consumer's job by design. "Neutral" means the struct's shape and semantics are broker-agnostic (a plain Rust type, parsed numbers, no `serde_json::Value`, no raw JSON, no provider filename leaking), not that every string is canonicalized. - **Incomplete object** (a missing core field) — same channel as `null`: `None`. A *missing* sidecar (a symbol with bars but no `.meta.json`) is an expected, non-error condition → silent `None`. Only a *present* file that cannot yield a complete geometry warns. ## Testing strategy Unit tests in `src/meta.rs` against the pure `from_sidecar_bytes` (no filesystem) — each pins one absorbed quirk: - Canonical `EURUSD` sidecar → `Some` with exact fields (`digits 5, pip_size 0.0001, tick_size 1e-05, lot_size 100000.0, base "EUR", quote "USD"`). Pins scientific-notation `tickSize` and integer `lotSize`. - Integer-valued geometry (`GER40`: `pipSize 1`, `digits 1`) → `pip_size 1.0`. - Provider base code passes through unchanged (`NAS100` → `base == "USTEC"`; dotted `AAPL.US` → `base == "AAPL.US"`). - Unsupported `schemaVersion` (e.g. `99`) → `None`. - `null` core field (e.g. `"lotSize": null`) → `None`. - Missing core field → `None`. - Malformed JSON → `None`. Integration test in `tests/data_server.rs` (or `src/lib.rs` tests) using a `tempfile::tempdir`: - Write `EURUSD.meta.json` into the dir → `server.symbol_meta("EURUSD")` is `Some` with the expected geometry. - A symbol with a bar file but no sidecar → `symbol_meta` is `None`. - An unknown symbol → `symbol_meta` is `None`. All tests are deterministic and self-contained (no `/mnt` dependency), matching the existing test suite's tempdir pattern. ## Acceptance criteria - A consumer obtains a symbol's bars (existing `stream_*`) and its neutral geometry (`symbol_meta`) from one `DataServer`, pairing with `has_symbol`/`symbols`. - The public API exposes only the plain `InstrumentGeometry` struct and `Option` — no `serde_json` type, no raw JSON, no provider filename, no error type. - Every documented quirk (scientific notation, `null`-for-non-finite, unknown schema version, heterogeneous base codes) is absorbed: the consumer sees a complete neutral geometry or `None`, never a partial struct or a provider artefact. - `pip_value_per_lot = lot_size * pip_size` reproduces the issue's verified values (EURUSD 10 USD; USDJPY 1000 JPY; XAUUSD 10 USD) from the returned fields. - The new `serde_json` dependency is the only addition to the dependency set; the README's dependency statement is updated to reflect it. - The full suite stays green; new tests pin each absorbed quirk.