plan: per-symbol geometry sidecar reader

Two-task plan for #3: (1) the meta module — InstrumentGeometry + from_sidecar_bytes parser + serde_json dep + README update + unit tests; (2) DataServer::symbol_meta + deterministic tempdir integration tests. refs #3
2026-06-25 14:25:01 +02:00
parent 4bda01e627
commit a1ce14ccc7
1 changed files with 387 additions and 0 deletions
@@ -0,0 +1,387 @@
+# Per-symbol geometry sidecar reader — Implementation Plan
+
+> **Parent spec:** `docs/specs/0001-symbol-meta-geometry-reader.md`
+>
+> **For agentic workers:** REQUIRED SUB-SKILL: use the `implement`
+> skill to run this plan. Steps use `- [ ]` checkboxes for tracking.
+
+**Goal:** Add `DataServer::symbol_meta(symbol) -> Option<InstrumentGeometry>`,
+a reader that loads neutral price geometry from a `<SYMBOL>.meta.json` sidecar.
+
+**Architecture:** A new leaf module `src/meta.rs` owns the neutral
+`InstrumentGeometry` struct and a pure `from_sidecar_bytes` parser (serde_json
+`Value` extraction, absorbing scientific notation, `null`-for-non-finite, and
+an unknown schema version). A new one-shot `&self` method on `DataServer` does
+the single filesystem read via the existing `SymbolIndex::base_path`. The
+reader is independent of the bar `FileCache` and streaming path.
+
+**Tech Stack:** Rust, `serde_json` (new direct dep), `tempfile` (existing
+dev-dep) for the integration tests.
+
+**Files this plan creates or modifies:**
+
+- Create: `src/meta.rs` — `InstrumentGeometry`, `SUPPORTED_SCHEMA_VERSION`,
+  `from_sidecar_bytes`, private `finite`, and `#[cfg(test)] mod tests`.
+- Modify: `src/lib.rs:21-23` — add `pub mod meta;` to the module-declaration block.
+- Modify: `src/lib.rs:196-201` — add `symbol_meta` method to `impl DataServer`,
+  after `file_count`, before `stream_m1`.
+- Modify: `src/lib.rs` (`#[cfg(test)] mod tests`, near line 792) — add a
+  `write_meta_file` helper and two integration tests.
+- Modify: `Cargo.toml:7-10` — add `serde_json = "1"` to `[dependencies]`.
+- Modify: `README.md:8` — update the dependency statement to include `serde_json`.
+- Test: `src/meta.rs` unit tests — pin each absorbed parser quirk.
+- Test: `src/lib.rs` integration tests — `symbol_meta` over a tempdir.
+
+**Out-of-scope (not touched):** `tests/data_server.rs` (its `/mnt`-skip-guard
+idiom is unsuited to the deterministic tempdir tests); `Cargo.lock` (git-ignored
+per the library convention — do NOT `git add` it even though `cargo build`
+regenerates it).
+
+---
+
+### Task 1: `meta` module — neutral geometry struct + sidecar parser
+
+**Files:**
+- Modify: `Cargo.toml:7-10`
+- Modify: `README.md:8`
+- Create: `src/meta.rs`
+- Modify: `src/lib.rs:21-23`
+
+- [ ] **Step 1: Add the `serde_json` dependency and update the README**
+
+In `Cargo.toml`, change the `[dependencies]` block from:
+
+```toml
+[dependencies]
+chrono = "0.4"
+regex = "1.10"
+zip = "2"
+```
+
+to:
+
+```toml
+[dependencies]
+chrono = "0.4"
+regex = "1.10"
+serde_json = "1"
+zip = "2"
+```
+
+In `README.md`, change line 8 from:
+
+```markdown
+Standalone leaf crate — depends only on `chrono`, `regex`, `zip`.
+```
+
+to:
+
+```markdown
+Leaf crate — depends only on `chrono`, `regex`, `serde_json`, `zip`.
+```
+
+- [ ] **Step 2: Write `src/meta.rs` with the type, a stub parser, and the unit tests**
+
+Create `src/meta.rs` with this exact content (the parser body is a stub at this
+step so the "parses to `Some`" tests fail RED):
+
+```rust
+//! Reader for per-symbol `<SYMBOL>.meta.json` geometry sidecars.
+//!
+//! Turns the provider's flat JSON sidecar into neutral, broker-agnostic
+//! price-interpretation geometry. Every format quirk (scientific-notation
+//! numbers, `null` for a non-finite value, an unknown schema version) is
+//! absorbed here so nothing provider-specific reaches a consumer.
+
+use serde_json::Value;
+
+/// The only sidecar schema version this reader understands.
+pub const SUPPORTED_SCHEMA_VERSION: u64 = 2;
+
+/// Neutral, broker-agnostic price-interpretation geometry for one symbol.
+///
+/// Intrinsic to the historical price series and effectively constant over its
+/// span. Time-varying / account-dependent conditions are deliberately excluded.
+#[derive(Debug, Clone, PartialEq)]
+pub struct InstrumentGeometry {
+    /// Number of decimal places in a quoted price (e.g. 5 for EURUSD).
+    pub digits: u32,
+    /// Price increment of one pip (e.g. 0.0001 for EURUSD).
+    pub pip_size: f64,
+    /// Smallest representable price increment (e.g. 1e-05 for EURUSD).
+    pub tick_size: f64,
+    /// Contract size of one lot in base units (e.g. 100000 for EURUSD).
+    pub lot_size: f64,
+    /// Base asset code, passed through verbatim from the sidecar.
+    pub base: String,
+    /// Quote (price) currency code, passed through verbatim.
+    pub quote: String,
+}
+
+impl InstrumentGeometry {
+    /// Parse sidecar JSON bytes into neutral geometry.
+    ///
+    /// Returns `None` if the JSON is malformed, the schema version is
+    /// unsupported, or any core field is missing / `null` / non-finite.
+    pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option<Self> {
+        // STUB — replaced in Step 4.
+        let _ = bytes;
+        None
+    }
+}
+
+/// A finite `f64` from a JSON number, or `None` for missing / null / non-finite.
+fn finite(v: Option<&Value>) -> Option<f64> {
+    let n = v?.as_f64()?;
+    n.is_finite().then_some(n)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const EURUSD: &str = r#"{
+      "schemaVersion": 2,
+      "symbol": "EURUSD",
+      "generatedAtUtc": "2026-06-25T11:26:37Z",
+      "generator": "MS_SimpleExport (cTrader.Automate)",
+      "description": "Euro vs US Dollar",
+      "digits": 5,
+      "pipSize": 0.0001,
+      "tickSize": 1E-05,
+      "lotSize": 100000,
+      "baseAsset": "EUR",
+      "quoteAsset": "USD"
+    }"#;
+
+    #[test]
+    fn parses_canonical_sidecar_with_scientific_notation_and_integer_lot() {
+        let g = InstrumentGeometry::from_sidecar_bytes(EURUSD.as_bytes()).unwrap();
+        assert_eq!(
+            g,
+            InstrumentGeometry {
+                digits: 5,
+                pip_size: 0.0001,
+                tick_size: 1e-05,
+                lot_size: 100_000.0,
+                base: "EUR".to_string(),
+                quote: "USD".to_string(),
+            }
+        );
+    }
+
+    #[test]
+    fn integer_valued_geometry_parses_as_float() {
+        // GER40-shaped: pipSize 1, digits 1, lotSize 1.
+        let json = r#"{"schemaVersion":2,"digits":1,"pipSize":1,"tickSize":0.1,"lotSize":1,"baseAsset":"DE40","quoteAsset":"EUR"}"#;
+        let g = InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).unwrap();
+        assert_eq!(g.pip_size, 1.0);
+        assert_eq!(g.digits, 1);
+        assert_eq!(g.lot_size, 1.0);
+    }
+
+    #[test]
+    fn provider_base_codes_pass_through_verbatim() {
+        let nas = r#"{"schemaVersion":2,"digits":1,"pipSize":1,"tickSize":0.1,"lotSize":1,"baseAsset":"USTEC","quoteAsset":"USD"}"#;
+        assert_eq!(
+            InstrumentGeometry::from_sidecar_bytes(nas.as_bytes()).unwrap().base,
+            "USTEC"
+        );
+        let aapl = r#"{"schemaVersion":2,"digits":2,"pipSize":0.01,"tickSize":0.01,"lotSize":1,"baseAsset":"AAPL.US","quoteAsset":"USD"}"#;
+        assert_eq!(
+            InstrumentGeometry::from_sidecar_bytes(aapl.as_bytes()).unwrap().base,
+            "AAPL.US"
+        );
+    }
+
+    #[test]
+    fn unsupported_schema_version_yields_none() {
+        let json = r#"{"schemaVersion":99,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"lotSize":100000,"baseAsset":"EUR","quoteAsset":"USD"}"#;
+        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
+    }
+
+    #[test]
+    fn null_core_field_yields_none() {
+        let json = r#"{"schemaVersion":2,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"lotSize":null,"baseAsset":"EUR","quoteAsset":"USD"}"#;
+        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
+    }
+
+    #[test]
+    fn missing_core_field_yields_none() {
+        let json = r#"{"schemaVersion":2,"digits":5,"pipSize":0.0001,"tickSize":1E-05,"baseAsset":"EUR","quoteAsset":"USD"}"#;
+        assert!(InstrumentGeometry::from_sidecar_bytes(json.as_bytes()).is_none());
+    }
+
+    #[test]
+    fn malformed_json_yields_none() {
+        assert!(InstrumentGeometry::from_sidecar_bytes(b"not json").is_none());
+    }
+}
+```
+
+Then add the module declaration in `src/lib.rs`, changing the block at lines 21-23
+from:
+
+```rust
+pub mod cache;
+pub mod loader;
+pub mod records;
+```
+
+to:
+
+```rust
+pub mod cache;
+pub mod loader;
+pub mod meta;
+pub mod records;
+```
+
+- [ ] **Step 3: Run the meta unit tests — verify RED**
+
+Run: `cargo test -p data-server --lib meta::`
+Expected: the suite COMPILES and the 7 `meta::tests::*` tests run; the three
+that expect `Some` FAIL (`parses_canonical_sidecar_with_scientific_notation_and_integer_lot`,
+`integer_valued_geometry_parses_as_float`, `provider_base_codes_pass_through_verbatim`)
+with `called Option::unwrap() on a None value` (the stub returns `None`). The
+four `*_yields_none` / `malformed_*` tests pass against the stub.
+
+- [ ] **Step 4: Implement the real parser**
+
+In `src/meta.rs`, replace the stub body of `from_sidecar_bytes`:
+
+```rust
+        // STUB — replaced in Step 4.
+        let _ = bytes;
+        None
+```
+
+with:
+
+```rust
+        let v: Value = serde_json::from_slice(bytes).ok()?;
+        if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION {
+            return None;
+        }
+        Some(Self {
+            digits: v.get("digits").and_then(Value::as_u64)? as u32,
+            pip_size: finite(v.get("pipSize"))?,
+            tick_size: finite(v.get("tickSize"))?,
+            lot_size: finite(v.get("lotSize"))?,
+            base: v.get("baseAsset").and_then(Value::as_str)?.to_string(),
+            quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(),
+        })
+```
+
+- [ ] **Step 5: Run the meta unit tests — verify GREEN**
+
+Run: `cargo test -p data-server --lib meta::`
+Expected: PASS — all 7 `meta::tests::*` tests green.
+
+---
+
+### Task 2: `DataServer::symbol_meta` method + integration tests
+
+**Files:**
+- Modify: `src/lib.rs:196-201` (add method to `impl DataServer`)
+- Modify: `src/lib.rs` (`#[cfg(test)] mod tests`, near line 792)
+
+- [ ] **Step 1: Write the integration tests (RED)**
+
+In `src/lib.rs`, inside `#[cfg(test)] mod tests` (after the existing
+`write_m1_file` helper, before the first `#[test]`), add a helper:
+
+```rust
+    /// Writes a `<SYMBOL>.meta.json` sidecar into `dir` so the reader can
+    /// load it. The body is written verbatim.
+    fn write_meta_file(dir: &Path, symbol: &str, body: &str) {
+        std::fs::write(dir.join(format!("{symbol}.meta.json")), body).unwrap();
+    }
+```
+
+Then add two tests at the end of the same `mod tests` (before its closing `}`):
+
+```rust
+    #[test]
+    fn test_symbol_meta_reads_sidecar() {
+        let dir = tempfile::tempdir().unwrap();
+        write_meta_file(
+            dir.path(),
+            "EURUSD",
+            r#"{
+              "schemaVersion": 2, "digits": 5, "pipSize": 0.0001,
+              "tickSize": 1E-05, "lotSize": 100000,
+              "baseAsset": "EUR", "quoteAsset": "USD"
+            }"#,
+        );
+        let server = DataServer::new(dir.path());
+        let g = server.symbol_meta("EURUSD").unwrap();
+        assert_eq!(g.digits, 5);
+        assert_eq!(g.pip_size, 0.0001);
+        assert_eq!(g.tick_size, 1e-05);
+        assert_eq!(g.lot_size, 100_000.0);
+        assert_eq!(g.base, "EUR");
+        assert_eq!(g.quote, "USD");
+        // Issue's worked derivation: pip_value_per_lot = lot_size * pip_size.
+        assert!((g.lot_size * g.pip_size - 10.0).abs() < 1e-9);
+    }
+
+    #[test]
+    fn test_symbol_meta_missing_sidecar_is_none() {
+        let dir = tempfile::tempdir().unwrap();
+        // A symbol with a bar file but no sidecar -> None.
+        write_m1_file(dir.path(), "GBPUSD", 2020, 6);
+        let server = DataServer::new(dir.path());
+        assert!(server.symbol_meta("GBPUSD").is_none());
+        // An entirely unknown symbol -> None.
+        assert!(server.symbol_meta("NOPE").is_none());
+    }
+```
+
+- [ ] **Step 2: Run the integration tests — verify RED**
+
+Run: `cargo test -p data-server --lib test_symbol_meta`
+Expected: COMPILE ERROR — `no method named symbol_meta found for ... DataServer`.
+(The method does not exist yet; this is the RED state. The two
+`test_symbol_meta_*` tests are the ones gated.)
+
+- [ ] **Step 3: Add the `symbol_meta` method**
+
+In `src/lib.rs`, inside `impl DataServer`, immediately after the `file_count`
+method (which ends at line 200, before `stream_m1` at line 202), insert:
+
+```rust
+    /// Loads the neutral price geometry for a symbol from its
+    /// `<SYMBOL>.meta.json` sidecar, or `None` if no usable sidecar exists.
+    ///
+    /// A missing sidecar is normal (a symbol may have bars but no sidecar) and
+    /// returns `None` silently. A present-but-unusable sidecar (malformed,
+    /// unsupported schema version, or an incomplete / `null` / non-finite core
+    /// field) returns `None` after a one-line stderr warning. The returned
+    /// geometry is always complete; the consumer never sees a partial struct.
+    pub fn symbol_meta(&self, symbol: &str) -> Option<meta::InstrumentGeometry> {
+        let path = self.index.base_path.join(format!("{symbol}.meta.json"));
+        let bytes = std::fs::read(&path).ok()?; // missing / unreadable -> silent None
+        let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes);
+        if geo.is_none() {
+            eprintln!(
+                "Warning: {} present but unusable (bad/incomplete geometry)",
+                path.display()
+            );
+        }
+        geo
+    }
+```
+
+- [ ] **Step 4: Run the integration tests — verify GREEN**
+
+Run: `cargo test -p data-server --lib test_symbol_meta`
+Expected: PASS — `test_symbol_meta_reads_sidecar` and
+`test_symbol_meta_missing_sidecar_is_none` both green.
+
+- [ ] **Step 5: Run the full suite — verify no regression**
+
+Run: `cargo test -p data-server`
+Expected: PASS — all tests green (existing suite + 7 meta unit tests + 2
+`symbol_meta` integration tests). No test is filtered out; the result line
+reports the full count.