data-server/docs/specs/0001-symbol-meta-geometry-reader.md

Per-symbol geometry sidecar reader — Design Spec

Date: 2026-06-25 Status: Draft — awaiting user spec review Authors: orchestrator + Claude

Tracker: Brummel/data-server#3 (reference issue; fork decisions logged there).

Goal

Add a reader that loads a per-symbol <SYMBOL>.meta.json sidecar from the data directory this server already owns and returns neutral price-interpretation geometry — digits, pip_size, tick_size, lot_size, base, quote — so a downstream consumer can ask one server for a symbol's bars and its geometry without parsing provider JSON itself.

The sidecar carries the price-interpretation facts the bars lack (how many decimals a price has, what a pip is, what one lot is worth, which currency the price is quoted in). Keeping the sidecar's filename, schema version, and format quirks (scientific-notation numbers, null for a non-finite value, an unknown schema version) inside this broker-facing library is what lets the consumer stay broker-agnostic: it receives a plain Rust struct or nothing, never raw JSON.

Non-goals: time-varying / account-dependent conditions (commission, swap, deposit-currency pip value, volume limits, trading status) — a separate, time-indexed concern, not this reader's job. No currency-code normalization (see §Error handling, base/quote). No caching (see §Components).

Architecture

A new leaf module src/meta.rs owns the neutral type and the pure parser; a new one-shot method on DataServer owns the filesystem read.

        DataServer
        ┌───────────────────────────────────────┐
        │ index: SymbolIndex { base_path, … }    │
        │ cache: Arc<FileCache>   (bars only)    │
        │                                        │
        │ symbol_meta(&self, symbol) ────────────┼──► fs::read(base_path/<symbol>.meta.json)
        └───────────────────────────────────────┘            │
                                                              ▼
                         meta::InstrumentGeometry::from_sidecar_bytes(&[u8]) -> Option<…>

The reader is independent of the bar FileCache and the streaming path: it does not touch the cache, spawns no thread, and needs no Arc<Self>. It reaches the data directory through the existing SymbolIndex::base_path — the same field the bar file paths are built from (ratified by test_file_path_construction in src/lib.rs, which pins base_path as the data directory). Membership in the bar index is not required: a symbol with a sidecar but no bar files, or bar files but no sidecar, is handled by the file existing or not.

Concrete code shapes

Worked consumer program (the acceptance evidence)

What an Aura-side consumer (the counterpart, Brummel/Aura#143) writes — bars and geometry from one server, no provider JSON in sight:

use std::sync::Arc;
use data_server::DataServer;

let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));

if let Some(geo) = server.symbol_meta("EURUSD") {
    // geo: InstrumentGeometry — neutral, plain fields
    assert_eq!(geo.digits, 5);
    assert_eq!(geo.pip_size, 0.0001);
    assert_eq!(geo.tick_size, 1e-05);   // sidecar wrote it as 1E-05
    assert_eq!(geo.lot_size, 100_000.0);
    assert_eq!(geo.quote, "USD");

    // The consumer derives the rest; nothing provider-specific reached it.
    let pip_value_per_lot = geo.lot_size * geo.pip_size; // 100000 * 0.0001 = 10 USD
}

// Pairs with the existing symbol surface.
for sym in server.symbols() {
    if let Some(geo) = server.symbol_meta(&sym) {
        // … cross-check geo.base / geo.quote against the consumer's vetted table.
    }
}

symbol_meta returns None for: no sidecar file, malformed JSON, an unsupported schema version, or any missing / null / non-finite core field (see §Error handling). The consumer never sees a partial geometry.

Implementation shapes (secondary)

New file src/meta.rs — before: does not exist. After:

//! Reader for per-symbol `<SYMBOL>.meta.json` geometry sidecars.

use serde_json::Value;

/// The only sidecar schema version this reader understands.
pub const SUPPORTED_SCHEMA_VERSION: u64 = 2;

/// Neutral, broker-agnostic price-interpretation geometry for one symbol.
///
/// Intrinsic to the historical price series and effectively constant over its
/// span. Derived/account-dependent conditions are deliberately excluded.
#[derive(Debug, Clone, PartialEq)]
pub struct InstrumentGeometry {
    /// Number of decimal places in a quoted price (e.g. 5 for EURUSD).
    pub digits: u32,
    /// Price increment of one pip (e.g. 0.0001 for EURUSD).
    pub pip_size: f64,
    /// Smallest representable price increment (e.g. 1e-05 for EURUSD).
    pub tick_size: f64,
    /// Contract size of one lot in base units (e.g. 100000 for EURUSD).
    pub lot_size: f64,
    /// Base asset code, passed through verbatim from the sidecar.
    pub base: String,
    /// Quote (price) currency code, passed through verbatim.
    pub quote: String,
}

impl InstrumentGeometry {
    /// Parse sidecar JSON bytes into neutral geometry.
    ///
    /// Returns `None` if the JSON is malformed, the schema version is
    /// unsupported, or any core field is missing / `null` / non-finite.
    pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option<Self> {
        let v: Value = serde_json::from_slice(bytes).ok()?;
        if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION {
            return None;
        }
        Some(Self {
            digits: v.get("digits").and_then(Value::as_u64)? as u32,
            pip_size: finite(v.get("pipSize"))?,
            tick_size: finite(v.get("tickSize"))?,
            lot_size: finite(v.get("lotSize"))?,
            base: v.get("baseAsset").and_then(Value::as_str)?.to_string(),
            quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(),
        })
    }
}

/// A finite `f64` from a JSON number, or `None` for missing / null / non-finite.
/// `Value::as_f64` already rejects `null` and non-numbers; this also rejects
/// `NaN`/`Inf` defensively.
fn finite(v: Option<&Value>) -> Option<f64> {
    let n = v?.as_f64()?;
    n.is_finite().then_some(n)
}

New method on DataServer in src/lib.rs — before: absent. After:

/// Loads the neutral price geometry for a symbol from its
/// `<SYMBOL>.meta.json` sidecar, or `None` if no usable sidecar exists.
///
/// A missing sidecar is normal (`None`, silent). A present-but-unusable
/// sidecar (malformed, unsupported version, incomplete) returns `None`
/// after a one-line stderr warning.
pub fn symbol_meta(&self, symbol: &str) -> Option<meta::InstrumentGeometry> {
    let path = self.index.base_path.join(format!("{symbol}.meta.json"));
    let bytes = std::fs::read(&path).ok()?; // missing file -> silent None
    let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes);
    if geo.is_none() {
        eprintln!("Warning: {} present but unusable (bad/incomplete geometry)", path.display());
    }
    geo
}

Module wiring in src/lib.rs — add pub mod meta; alongside the existing pub mod cache; pub mod loader; pub mod records;. base_path is already a field of the private SymbolIndex; symbol_meta is a DataServer method in the same crate, so it reads self.index.base_path directly (no visibility change).

Cargo.toml — add one direct dependency:

serde_json = "1"

Components

• meta::InstrumentGeometry — the public neutral struct (six fields, Debug + Clone + PartialEq). Excludes symbol, description, generator, generatedAtUtc (the most provider-specific fields; symbol is already known to the caller).
• meta::InstrumentGeometry::from_sidecar_bytes — pub(crate) pure parser, bytes → Option<Self>. No filesystem, fully unit-testable. Absorbs every format quirk here.
• meta::SUPPORTED_SCHEMA_VERSION — the u64 gate (currently 2).
• DataServer::symbol_meta — &self one-shot read; the only filesystem touch. No caching: geometry files are tiny and read rarely (typically once per symbol); a cache map + invalidation is complexity for no stated need, and the bar FileCache exists for a different problem (large, repeatedly-streamed files).

Data flow

1. Consumer calls server.symbol_meta("EURUSD").
2. Path built: <base_path>/EURUSD.meta.json (symbols may contain dots, e.g. AAPL.US.meta.json; the literal join handles them).
3. fs::read → on error (missing/unreadable) return None silently.
4. Bytes → from_sidecar_bytes: parse JSON; gate schemaVersion == 2; extract the six fields with the null/non-finite guard.
5. Some(geometry) on success; None + one stderr warning on a present but unusable file.

Error handling

The reader absorbs every documented provider quirk so none reaches a consumer:

• Scientific notation (1E-05, in 7 of 30 live files) — valid JSON, parses as a number; Value::as_f64 handles it. No special code.
• null for a non-finite broker value — Value::as_f64/as_u64 return None for null; that propagates to symbol_meta returning None. The consumer never sees a hole. (No null exists in the current 30 samples; this is defensive.) finite() additionally rejects NaN/Inf.
• Unknown schemaVersion — a reader-side decision, not a consumer's: an unsupported version returns None (+ the call-site warning), never an error type the consumer must handle.
• base / quote provider codes — passed through verbatim as String. baseAsset is wildly heterogeneous across the 30 samples (ISO EUR/USD, index codes DE40/USTEC/JP225, descriptive Bitcoin/Copper, stock tickers AAPL.US); there is no well-defined "neutral" target and no mapping table, so normalization is not even definable here. The consumer (Aura#143) cross-checks the value against its own hand-authored vetted table and never reads the raw JSON — resolution is the consumer's job by design. "Neutral" means the struct's shape and semantics are broker-agnostic (a plain Rust type, parsed numbers, no serde_json::Value, no raw JSON, no provider filename leaking), not that every string is canonicalized.
• Incomplete object (a missing core field) — same channel as null: None.

A missing sidecar (a symbol with bars but no .meta.json) is an expected, non-error condition → silent None. Only a present file that cannot yield a complete geometry warns.

Testing strategy

Unit tests in src/meta.rs against the pure from_sidecar_bytes (no filesystem) — each pins one absorbed quirk:

• Canonical EURUSD sidecar → Some with exact fields (digits 5, pip_size 0.0001, tick_size 1e-05, lot_size 100000.0, base "EUR", quote "USD"). Pins scientific-notation tickSize and integer lotSize.
• Integer-valued geometry (GER40: pipSize 1, digits 1) → pip_size 1.0.
• Provider base code passes through unchanged (NAS100 → base == "USTEC"; dotted AAPL.US → base == "AAPL.US").
• Unsupported schemaVersion (e.g. 99) → None.
• null core field (e.g. "lotSize": null) → None.
• Missing core field → None.
• Malformed JSON → None.

Integration test in tests/data_server.rs (or src/lib.rs tests) using a tempfile::tempdir:

• Write EURUSD.meta.json into the dir → server.symbol_meta("EURUSD") is Some with the expected geometry.
• A symbol with a bar file but no sidecar → symbol_meta is None.
• An unknown symbol → symbol_meta is None.

All tests are deterministic and self-contained (no /mnt dependency), matching the existing test suite's tempdir pattern.

Acceptance criteria

• A consumer obtains a symbol's bars (existing stream_*) and its neutral geometry (symbol_meta) from one DataServer, pairing with has_symbol/symbols.
• The public API exposes only the plain InstrumentGeometry struct and Option — no serde_json type, no raw JSON, no provider filename, no error type.
• Every documented quirk (scientific notation, null-for-non-finite, unknown schema version, heterogeneous base codes) is absorbed: the consumer sees a complete neutral geometry or None, never a partial struct or a provider artefact.
• pip_value_per_lot = lot_size * pip_size reproduces the issue's verified values (EURUSD 10 USD; USDJPY 1000 JPY; XAUUSD 10 USD) from the returned fields.
• The new serde_json dependency is the only addition to the dependency set; the README's dependency statement is updated to reflect it.
• The full suite stays green; new tests pin each absorbed quirk.