data-server/docs/specs/0001-symbol-meta-geometry-reader.md

# Per-symbol geometry sidecar reader — Design Spec

**Date:** 2026-06-25
**Status:** Draft — awaiting user spec review
**Authors:** orchestrator + Claude

Tracker: Brummel/data-server#3 (reference issue; fork decisions logged there).

## Goal

Add a reader that loads a per-symbol `<SYMBOL>.meta.json` sidecar from the data
directory this server already owns and returns **neutral** price-interpretation
geometry — `digits`, `pip_size`, `tick_size`, `lot_size`, `base`, `quote` — so a
downstream consumer can ask one server for a symbol's bars *and* its geometry
without parsing provider JSON itself.

The sidecar carries the price-interpretation facts the bars lack (how many
decimals a price has, what a pip is, what one lot is worth, which currency the
price is quoted in). Keeping the sidecar's filename, schema version, and format
quirks (scientific-notation numbers, `null` for a non-finite value, an unknown
schema version) inside this broker-facing library is what lets the consumer stay
broker-agnostic: it receives a plain Rust struct or nothing, never raw JSON.

Non-goals: time-varying / account-dependent conditions (commission, swap,
deposit-currency pip value, volume limits, trading status) — a separate,
time-indexed concern, not this reader's job. No currency-code normalization (see
§Error handling, base/quote). No caching (see §Components).

## Architecture

A new leaf module `src/meta.rs` owns the neutral type and the pure parser; a new
one-shot method on `DataServer` owns the filesystem read.

```text
        DataServer
        ┌───────────────────────────────────────┐
        │ index: SymbolIndex { base_path, … }    │
        │ cache: Arc<FileCache>   (bars only)    │
        │                                        │
        │ symbol_meta(&self, symbol) ────────────┼──► fs::read(base_path/<symbol>.meta.json)
        └───────────────────────────────────────┘            │
                                                              ▼
                         meta::InstrumentGeometry::from_sidecar_bytes(&[u8]) -> Option<…>
```

The reader is independent of the bar `FileCache` and the streaming path: it does
not touch the cache, spawns no thread, and needs no `Arc<Self>`. It reaches the
data directory through the existing `SymbolIndex::base_path` — the same field
the bar file paths are built from (ratified by `test_file_path_construction` in
`src/lib.rs`, which pins `base_path` as the data directory). Membership in the
bar index is **not** required: a symbol with a sidecar but no bar files, or bar
files but no sidecar, is handled by the file existing or not.

## Concrete code shapes

### Worked consumer program (the acceptance evidence)

What an Aura-side consumer (the counterpart, Brummel/Aura#143) writes — bars and
geometry from one server, no provider JSON in sight:

```rust
use std::sync::Arc;
use data_server::DataServer;

let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone"));

if let Some(geo) = server.symbol_meta("EURUSD") {
    // geo: InstrumentGeometry — neutral, plain fields
    assert_eq!(geo.digits, 5);
    assert_eq!(geo.pip_size, 0.0001);
    assert_eq!(geo.tick_size, 1e-05);   // sidecar wrote it as 1E-05
    assert_eq!(geo.lot_size, 100_000.0);
    assert_eq!(geo.quote, "USD");

    // The consumer derives the rest; nothing provider-specific reached it.
    let pip_value_per_lot = geo.lot_size * geo.pip_size; // 100000 * 0.0001 = 10 USD
}

// Pairs with the existing symbol surface.
for sym in server.symbols() {
    if let Some(geo) = server.symbol_meta(&sym) {
        // … cross-check geo.base / geo.quote against the consumer's vetted table.
    }
}
```

`symbol_meta` returns `None` for: no sidecar file, malformed JSON, an
unsupported schema version, or any missing / `null` / non-finite core field
(see §Error handling). The consumer never sees a partial geometry.

### Implementation shapes (secondary)

New file `src/meta.rs` — before: does not exist. After:

```rust
//! Reader for per-symbol `<SYMBOL>.meta.json` geometry sidecars.

use serde_json::Value;

/// The only sidecar schema version this reader understands.
pub const SUPPORTED_SCHEMA_VERSION: u64 = 2;

/// Neutral, broker-agnostic price-interpretation geometry for one symbol.
///
/// Intrinsic to the historical price series and effectively constant over its
/// span. Derived/account-dependent conditions are deliberately excluded.
#[derive(Debug, Clone, PartialEq)]
pub struct InstrumentGeometry {
    /// Number of decimal places in a quoted price (e.g. 5 for EURUSD).
    pub digits: u32,
    /// Price increment of one pip (e.g. 0.0001 for EURUSD).
    pub pip_size: f64,
    /// Smallest representable price increment (e.g. 1e-05 for EURUSD).
    pub tick_size: f64,
    /// Contract size of one lot in base units (e.g. 100000 for EURUSD).
    pub lot_size: f64,
    /// Base asset code, passed through verbatim from the sidecar.
    pub base: String,
    /// Quote (price) currency code, passed through verbatim.
    pub quote: String,
}

impl InstrumentGeometry {
    /// Parse sidecar JSON bytes into neutral geometry.
    ///
    /// Returns `None` if the JSON is malformed, the schema version is
    /// unsupported, or any core field is missing / `null` / non-finite.
    pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option<Self> {
        let v: Value = serde_json::from_slice(bytes).ok()?;
        if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION {
            return None;
        }
        Some(Self {
            digits: v.get("digits").and_then(Value::as_u64)? as u32,
            pip_size: finite(v.get("pipSize"))?,
            tick_size: finite(v.get("tickSize"))?,
            lot_size: finite(v.get("lotSize"))?,
            base: v.get("baseAsset").and_then(Value::as_str)?.to_string(),
            quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(),
        })
    }
}

/// A finite `f64` from a JSON number, or `None` for missing / null / non-finite.
/// `Value::as_f64` already rejects `null` and non-numbers; this also rejects
/// `NaN`/`Inf` defensively.
fn finite(v: Option<&Value>) -> Option<f64> {
    let n = v?.as_f64()?;
    n.is_finite().then_some(n)
}
```

New method on `DataServer` in `src/lib.rs` — before: absent. After:

```rust
/// Loads the neutral price geometry for a symbol from its
/// `<SYMBOL>.meta.json` sidecar, or `None` if no usable sidecar exists.
///
/// A missing sidecar is normal (`None`, silent). A present-but-unusable
/// sidecar (malformed, unsupported version, incomplete) returns `None`
/// after a one-line stderr warning.
pub fn symbol_meta(&self, symbol: &str) -> Option<meta::InstrumentGeometry> {
    let path = self.index.base_path.join(format!("{symbol}.meta.json"));
    let bytes = std::fs::read(&path).ok()?; // missing file -> silent None
    let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes);
    if geo.is_none() {
        eprintln!("Warning: {} present but unusable (bad/incomplete geometry)", path.display());
    }
    geo
}
```

Module wiring in `src/lib.rs` — add `pub mod meta;` alongside the existing
`pub mod cache; pub mod loader; pub mod records;`. `base_path` is already a field
of the private `SymbolIndex`; `symbol_meta` is a `DataServer` method in the same
crate, so it reads `self.index.base_path` directly (no visibility change).

`Cargo.toml` — add one direct dependency:

```toml
serde_json = "1"
```

## Components

- **`meta::InstrumentGeometry`** — the public neutral struct (six fields,
  `Debug + Clone + PartialEq`). Excludes `symbol`, `description`, `generator`,
  `generatedAtUtc` (the most provider-specific fields; `symbol` is already known
  to the caller).
- **`meta::InstrumentGeometry::from_sidecar_bytes`** — `pub(crate)` pure parser,
  bytes → `Option<Self>`. No filesystem, fully unit-testable. Absorbs every
  format quirk here.
- **`meta::SUPPORTED_SCHEMA_VERSION`** — the `u64` gate (currently `2`).
- **`DataServer::symbol_meta`** — `&self` one-shot read; the only filesystem
  touch. No caching: geometry files are tiny and read rarely (typically once per
  symbol); a cache map + invalidation is complexity for no stated need, and the
  bar `FileCache` exists for a different problem (large, repeatedly-streamed
  files).

## Data flow

1. Consumer calls `server.symbol_meta("EURUSD")`.
2. Path built: `<base_path>/EURUSD.meta.json` (symbols may contain dots, e.g.
   `AAPL.US.meta.json`; the literal join handles them).
3. `fs::read` → on error (missing/unreadable) return `None` silently.
4. Bytes → `from_sidecar_bytes`: parse JSON; gate `schemaVersion == 2`; extract
   the six fields with the null/non-finite guard.
5. `Some(geometry)` on success; `None` + one stderr warning on a present but
   unusable file.

## Error handling

The reader absorbs every documented provider quirk so none reaches a consumer:

- **Scientific notation** (`1E-05`, in 7 of 30 live files) — valid JSON, parses
  as a number; `Value::as_f64` handles it. No special code.
- **`null` for a non-finite broker value** — `Value::as_f64`/`as_u64` return
  `None` for `null`; that propagates to `symbol_meta` returning `None`. The
  consumer never sees a hole. (No `null` exists in the current 30 samples; this
  is defensive.) `finite()` additionally rejects `NaN`/`Inf`.
- **Unknown `schemaVersion`** — a reader-side decision, not a consumer's: an
  unsupported version returns `None` (+ the call-site warning), never an error
  type the consumer must handle.
- **base / quote provider codes** — passed through **verbatim** as `String`.
  `baseAsset` is wildly heterogeneous across the 30 samples (ISO `EUR`/`USD`,
  index codes `DE40`/`USTEC`/`JP225`, descriptive `Bitcoin`/`Copper`, stock
  tickers `AAPL.US`); there is no well-defined "neutral" target and no mapping
  table, so normalization is not even definable here. The consumer (Aura#143)
  cross-checks the value against its own hand-authored vetted table and never
  reads the raw JSON — resolution is the consumer's job by design. "Neutral"
  means the struct's shape and semantics are broker-agnostic (a plain Rust type,
  parsed numbers, no `serde_json::Value`, no raw JSON, no provider filename
  leaking), not that every string is canonicalized.
- **Incomplete object** (a missing core field) — same channel as `null`:
  `None`.

A *missing* sidecar (a symbol with bars but no `.meta.json`) is an expected,
non-error condition → silent `None`. Only a *present* file that cannot yield a
complete geometry warns.

## Testing strategy

Unit tests in `src/meta.rs` against the pure `from_sidecar_bytes` (no
filesystem) — each pins one absorbed quirk:

- Canonical `EURUSD` sidecar → `Some` with exact fields
  (`digits 5, pip_size 0.0001, tick_size 1e-05, lot_size 100000.0, base "EUR",
  quote "USD"`). Pins scientific-notation `tickSize` and integer `lotSize`.
- Integer-valued geometry (`GER40`: `pipSize 1`, `digits 1`) → `pip_size 1.0`.
- Provider base code passes through unchanged (`NAS100` → `base == "USTEC"`;
  dotted `AAPL.US` → `base == "AAPL.US"`).
- Unsupported `schemaVersion` (e.g. `99`) → `None`.
- `null` core field (e.g. `"lotSize": null`) → `None`.
- Missing core field → `None`.
- Malformed JSON → `None`.

Integration test in `tests/data_server.rs` (or `src/lib.rs` tests) using a
`tempfile::tempdir`:

- Write `EURUSD.meta.json` into the dir → `server.symbol_meta("EURUSD")` is
  `Some` with the expected geometry.
- A symbol with a bar file but no sidecar → `symbol_meta` is `None`.
- An unknown symbol → `symbol_meta` is `None`.

All tests are deterministic and self-contained (no `/mnt` dependency), matching
the existing test suite's tempdir pattern.

## Acceptance criteria

- A consumer obtains a symbol's bars (existing `stream_*`) and its neutral
  geometry (`symbol_meta`) from one `DataServer`, pairing with
  `has_symbol`/`symbols`.
- The public API exposes only the plain `InstrumentGeometry` struct and `Option`
  — no `serde_json` type, no raw JSON, no provider filename, no error type.
- Every documented quirk (scientific notation, `null`-for-non-finite, unknown
  schema version, heterogeneous base codes) is absorbed: the consumer sees a
  complete neutral geometry or `None`, never a partial struct or a provider
  artefact.
- `pip_value_per_lot = lot_size * pip_size` reproduces the issue's verified
  values (EURUSD 10 USD; USDJPY 1000 JPY; XAUUSD 10 USD) from the returned
  fields.
- The new `serde_json` dependency is the only addition to the dependency set;
  the README's dependency statement is updated to reflect it.
- The full suite stays green; new tests pin each absorbed quirk.