data-server/docs/plans/0001-symbol-meta-geometry-reader.md

Per-symbol geometry sidecar reader — Implementation Plan

Parent spec: docs/specs/0001-symbol-meta-geometry-reader.md

For agentic workers: REQUIRED SUB-SKILL: use the implement skill to run this plan. Steps use - [ ] checkboxes for tracking.

Goal: Add DataServer::symbol_meta(symbol) -> Option<InstrumentGeometry>, a reader that loads neutral price geometry from a <SYMBOL>.meta.json sidecar.

Architecture: A new leaf module src/meta.rs owns the neutral InstrumentGeometry struct and a pure from_sidecar_bytes parser (serde_json Value extraction, absorbing scientific notation, null-for-non-finite, and an unknown schema version). A new one-shot &self method on DataServer does the single filesystem read via the existing SymbolIndex::base_path. The reader is independent of the bar FileCache and streaming path.

Tech Stack: Rust, serde_json (new direct dep), tempfile (existing dev-dep) for the integration tests.

Files this plan creates or modifies:

• Create: src/meta.rs — InstrumentGeometry, SUPPORTED_SCHEMA_VERSION, from_sidecar_bytes, private finite, and #[cfg(test)] mod tests.
• Modify: src/lib.rs:21-23 — add pub mod meta; to the module-declaration block.
• Modify: src/lib.rs:196-201 — add symbol_meta method to impl DataServer, after file_count, before stream_m1.
• Modify: src/lib.rs (#[cfg(test)] mod tests, near line 792) — add a write_meta_file helper and two integration tests.
• Modify: Cargo.toml:7-10 — add serde_json = "1" to [dependencies].
• Modify: README.md:8 — update the dependency statement to include serde_json.
• Test: src/meta.rs unit tests — pin each absorbed parser quirk.
• Test: src/lib.rs integration tests — symbol_meta over a tempdir.

Out-of-scope (not touched): tests/data_server.rs (its /mnt-skip-guard idiom is unsuited to the deterministic tempdir tests); Cargo.lock (git-ignored per the library convention — do NOT git add it even though cargo build regenerates it).

---

Task 1: meta module — neutral geometry struct + sidecar parser

Files:

• Modify: Cargo.toml:7-10

• Modify: README.md:8

• Create: src/meta.rs

• Modify: src/lib.rs:21-23

• Step 1: Add the serde_json dependency and update the README

In Cargo.toml, change the [dependencies] block from:

[dependencies]
chrono = "0.4"
regex = "1.10"
zip = "2"

to:

[dependencies]
chrono = "0.4"
regex = "1.10"
serde_json = "1"
zip = "2"

In README.md, change line 8 from:

Standalone leaf crate — depends only on `chrono`, `regex`, `zip`.

to:

Leaf crate — depends only on `chrono`, `regex`, `serde_json`, `zip`.

• Step 2: Write src/meta.rs with the type, a stub parser, and the unit tests

Create src/meta.rs with this exact content (the parser body is a stub at this step so the "parses to Some" tests fail RED):

//! Reader for per-symbol `<SYMBOL>.meta.json` geometry sidecars.
//!
//! Turns the provider's flat JSON sidecar into neutral, broker-agnostic
//! price-interpretation geometry. Every format quirk (scientific-notation
//! numbers, `null` for a non-finite value, an unknown schema version) is
//! absorbed here so nothing provider-specific reaches a consumer.

use serde_json::Value;

/// The only sidecar schema version this reader understands.
pub const SUPPORTED_SCHEMA_VERSION: u64 = 2;

/// Neutral, broker-agnostic price-interpretation geometry for one symbol.
///
/// Intrinsic to the historical price series and effectively constant over its
/// span. Time-varying / account-dependent conditions are deliberately excluded.
#[derive(Debug, Clone, PartialEq)]
pub struct InstrumentGeometry {
    /// Number of decimal places in a quoted price (e.g. 5 for EURUSD).
    pub digits: u32,
    /// Price increment of one pip (e.g. 0.0001 for EURUSD).
    pub pip_size: f64,
    /// Smallest representable price increment (e.g. 1e-05 for EURUSD).
    pub tick_size: f64,
    /// Contract size of one lot in base units (e.g. 100000 for EURUSD).
    pub lot_size: f64,
    /// Base asset code, passed through verbatim from the sidecar.
    pub base: String,
    /// Quote (price) currency code, passed through verbatim.
    pub quote: String,
}

impl InstrumentGeometry {
    /// Parse sidecar JSON bytes into neutral geometry.
    ///
    /// Returns `None` if the JSON is malformed, the schema version is
    /// unsupported, or any core field is missing / `null` / non-finite.
    pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option<Self> {
        // STUB — replaced in Step 4.
        let _ = bytes;
        None
    }
}

/// A finite `f64` from a JSON number, or `None` for missing / null / non-finite.
fn finite(v: Option<&Value>) -> Option<f64> {
    let n = v?.as_f64()?;
    n.is_finite().then_some(n)
}

#[cfg(test)]
mod tests {
    use super::*;

    const EURUSD: &str = r#"{
      "schemaVersion": 2,
      "symbol": "EURUSD",
      "generatedAtUtc": "2026-06-25T11:26:37Z",
      "generator": "MS_SimpleExport (cTrader.Automate)",
      "description": "Euro vs US Dollar",
      "digits": 5,
      "pipSize": 0.0001,
      "tickSize": 1E-05,
      "lotSize": 100000,
      "baseAsset": "EUR",
      "quoteAsset": "USD"
    }"#;

    #[test]
    fn parses_canonical_sidecar_with_scientific_notation_and_integer_lot() {
        let g = InstrumentGeometry::from_sidecar_bytes(EURUSD.as_bytes()).unwrap();
        assert_eq!(
            g,
            InstrumentGeometry {
                digits: 5,
                pip_size: 0.0001,
                tick_size: 1e-05,
                lot_size: 100_000.0,
                base: "EUR".to_string(),
                quote: "USD".to_string(),
            }
        );
    }

    #[test]
    fn integer_valued_geometry_parses_as_float() {
        // GER40-shaped: pipSize 1, digits 1, lotSize 1.
        let json = r#"{"schemaVersion":2,"digits":1,"pipSize":1,"tickSize":0.1,"lotSize":1,"baseAsset":"DE40","quoteAsset":"EUR"}"#;
        let g = InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).unwrap();
        assert_eq!(g.pip_size, 1.0);
        assert_eq!(g.digits, 1);
        assert_eq!(g.lot_size, 1.0);
    }

    #[test]
    fn provider_base_codes_pass_through_verbatim() {
        let nas = r#"{"schemaVersion":2,"digits":1,"pipSize":1,"tickSize":0.1,"lotSize":1,"baseAsset":"USTEC","quoteAsset":"USD"}"#;
        assert_eq!(
            InstrumentGeometry::from_sidecar_bytes(nas.as_bytes()).unwrap().base,
            "USTEC"
        );
        let aapl = r#"{"schemaVersion":2,"digits":2,"pipSize":0.01,"tickSize":0.01,"lotSize":1,"baseAsset":"AAPL.US","quoteAsset":"USD"}"#;
        assert_eq!(
            InstrumentGeometry::from_sidecar_bytes(aapl.as_bytes()).unwrap().base,
            "AAPL.US"
        );
    }

    #[test]
    fn unsupported_schema_version_yields_none() {
        let json = r#"{"schemaVersion":99,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"lotSize":100000,"baseAsset":"EUR","quoteAsset":"USD"}"#;
        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
    }

    #[test]
    fn null_core_field_yields_none() {
        let json = r#"{"schemaVersion":2,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"lotSize":null,"baseAsset":"EUR","quoteAsset":"USD"}"#;
        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
    }

    #[test]
    fn missing_core_field_yields_none() {
        let json = r#"{"schemaVersion":2,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"baseAsset":"EUR","quoteAsset":"USD"}"#;
        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
    }

    #[test]
    fn malformed_json_yields_none() {
        assert!(InstrumentGeometry::from_sidecar_bytes(b"not json").is_none());
    }
}

Then add the module declaration in src/lib.rs, changing the block at lines 21-23 from:

pub mod cache;
pub mod loader;
pub mod records;

to:

pub mod cache;
pub mod loader;
pub mod meta;
pub mod records;

• Step 3: Run the meta unit tests — verify RED

Run: cargo test -p data-server --lib meta:: Expected: the suite COMPILES and the 7 meta::tests::* tests run; the three that expect Some FAIL (parses_canonical_sidecar_with_scientific_notation_and_integer_lot, integer_valued_geometry_parses_as_float, provider_base_codes_pass_through_verbatim) with called Option::unwrap() on a None value (the stub returns None). The four *_yields_none / malformed_* tests pass against the stub.

• Step 4: Implement the real parser

In src/meta.rs, replace the stub body of from_sidecar_bytes:

        // STUB — replaced in Step 4.
        let _ = bytes;
        None

with:

        let v: Value = serde_json::from_slice(bytes).ok()?;
        if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION {
            return None;
        }
        Some(Self {
            digits: v.get("digits").and_then(Value::as_u64)? as u32,
            pip_size: finite(v.get("pipSize"))?,
            tick_size: finite(v.get("tickSize"))?,
            lot_size: finite(v.get("lotSize"))?,
            base: v.get("baseAsset").and_then(Value::as_str)?.to_string(),
            quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(),
        })

• Step 5: Run the meta unit tests — verify GREEN

Run: cargo test -p data-server --lib meta:: Expected: PASS — all 7 meta::tests::* tests green.

---

Task 2: DataServer::symbol_meta method + integration tests

Files:

• Modify: src/lib.rs:196-201 (add method to impl DataServer)

• Modify: src/lib.rs (#[cfg(test)] mod tests, near line 792)

• Step 1: Write the integration tests (RED)

In src/lib.rs, inside #[cfg(test)] mod tests (after the existing write_m1_file helper, before the first #[test]), add a helper:

    /// Writes a `<SYMBOL>.meta.json` sidecar into `dir` so the reader can
    /// load it. The body is written verbatim.
    fn write_meta_file(dir: &Path, symbol: &str, body: &str) {
        std::fs::write(dir.join(format!("{symbol}.meta.json")), body).unwrap();
    }

Then add two tests at the end of the same mod tests (before its closing }):

    #[test]
    fn test_symbol_meta_reads_sidecar() {
        let dir = tempfile::tempdir().unwrap();
        write_meta_file(
            dir.path(),
            "EURUSD",
            r#"{
              "schemaVersion": 2, "digits": 5, "pipSize": 0.0001,
              "tickSize": 1E-05, "lotSize": 100000,
              "baseAsset": "EUR", "quoteAsset": "USD"
            }"#,
        );
        let server = DataServer::new(dir.path());
        let g = server.symbol_meta("EURUSD").unwrap();
        assert_eq!(g.digits, 5);
        assert_eq!(g.pip_size, 0.0001);
        assert_eq!(g.tick_size, 1e-05);
        assert_eq!(g.lot_size, 100_000.0);
        assert_eq!(g.base, "EUR");
        assert_eq!(g.quote, "USD");
        // Issue's worked derivation: pip_value_per_lot = lot_size * pip_size.
        assert!((g.lot_size * g.pip_size - 10.0).abs() < 1e-9);
    }

    #[test]
    fn test_symbol_meta_missing_sidecar_is_none() {
        let dir = tempfile::tempdir().unwrap();
        // A symbol with a bar file but no sidecar -> None.
        write_m1_file(dir.path(), "GBPUSD", 2020, 6);
        let server = DataServer::new(dir.path());
        assert!(server.symbol_meta("GBPUSD").is_none());
        // An entirely unknown symbol -> None.
        assert!(server.symbol_meta("NOPE").is_none());
    }

• Step 2: Run the integration tests — verify RED

Run: cargo test -p data-server --lib test_symbol_meta Expected: COMPILE ERROR — no method named symbol_meta found for ... DataServer. (The method does not exist yet; this is the RED state. The two test_symbol_meta_* tests are the ones gated.)

• Step 3: Add the symbol_meta method

In src/lib.rs, inside impl DataServer, immediately after the file_count method (which ends at line 200, before stream_m1 at line 202), insert:

    /// Loads the neutral price geometry for a symbol from its
    /// `<SYMBOL>.meta.json` sidecar, or `None` if no usable sidecar exists.
    ///
    /// A missing sidecar is normal (a symbol may have bars but no sidecar) and
    /// returns `None` silently. A present-but-unusable sidecar (malformed,
    /// unsupported schema version, or an incomplete / `null` / non-finite core
    /// field) returns `None` after a one-line stderr warning. The returned
    /// geometry is always complete; the consumer never sees a partial struct.
    pub fn symbol_meta(&self, symbol: &str) -> Option<meta::InstrumentGeometry> {
        let path = self.index.base_path.join(format!("{symbol}.meta.json"));
        let bytes = std::fs::read(&path).ok()?; // missing / unreadable -> silent None
        let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes);
        if geo.is_none() {
            eprintln!(
                "Warning: {} present but unusable (bad/incomplete geometry)",
                path.display()
            );
        }
        geo
    }

• Step 4: Run the integration tests — verify GREEN

Run: cargo test -p data-server --lib test_symbol_meta Expected: PASS — test_symbol_meta_reads_sidecar and test_symbol_meta_missing_sidecar_is_none both green.

• Step 5: Run the full suite — verify no regression

Run: cargo test -p data-server Expected: PASS — all tests green (existing suite + 7 meta unit tests + 2 symbol_meta integration tests). No test is filtered out; the result line reports the full count.