From 4bda01e62759ba36a9cbeb539fa822d88374981b Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 25 Jun 2026 14:19:42 +0200 Subject: [PATCH] spec: per-symbol geometry sidecar reader (boss-signed) Settled prose design from #3: a reader returning neutral InstrumentGeometry (digits, pip_size, tick_size, lot_size, base, quote) from .meta.json. Signed autonomously under /boss on the grounding-check PASS; fork decisions (base/quote pass-through, unknown-version -> None, serde_json, no caching) are logged on the reference issue. refs #3 --- .../specs/0001-symbol-meta-geometry-reader.md | 284 ++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 docs/specs/0001-symbol-meta-geometry-reader.md diff --git a/docs/specs/0001-symbol-meta-geometry-reader.md b/docs/specs/0001-symbol-meta-geometry-reader.md new file mode 100644 index 0000000..0c14e3b --- /dev/null +++ b/docs/specs/0001-symbol-meta-geometry-reader.md @@ -0,0 +1,284 @@ +# Per-symbol geometry sidecar reader — Design Spec + +**Date:** 2026-06-25 +**Status:** Draft — awaiting user spec review +**Authors:** orchestrator + Claude + +Tracker: Brummel/data-server#3 (reference issue; fork decisions logged there). + +## Goal + +Add a reader that loads a per-symbol `.meta.json` sidecar from the data +directory this server already owns and returns **neutral** price-interpretation +geometry — `digits`, `pip_size`, `tick_size`, `lot_size`, `base`, `quote` — so a +downstream consumer can ask one server for a symbol's bars *and* its geometry +without parsing provider JSON itself. + +The sidecar carries the price-interpretation facts the bars lack (how many +decimals a price has, what a pip is, what one lot is worth, which currency the +price is quoted in). Keeping the sidecar's filename, schema version, and format +quirks (scientific-notation numbers, `null` for a non-finite value, an unknown +schema version) inside this broker-facing library is what lets the consumer stay +broker-agnostic: it receives a plain Rust struct or nothing, never raw JSON. + +Non-goals: time-varying / account-dependent conditions (commission, swap, +deposit-currency pip value, volume limits, trading status) — a separate, +time-indexed concern, not this reader's job. No currency-code normalization (see +§Error handling, base/quote). No caching (see §Components). + +## Architecture + +A new leaf module `src/meta.rs` owns the neutral type and the pure parser; a new +one-shot method on `DataServer` owns the filesystem read. + +```text + DataServer + ┌───────────────────────────────────────┐ + │ index: SymbolIndex { base_path, … } │ + │ cache: Arc (bars only) │ + │ │ + │ symbol_meta(&self, symbol) ────────────┼──► fs::read(base_path/.meta.json) + └───────────────────────────────────────┘ │ + ▼ + meta::InstrumentGeometry::from_sidecar_bytes(&[u8]) -> Option<…> +``` + +The reader is independent of the bar `FileCache` and the streaming path: it does +not touch the cache, spawns no thread, and needs no `Arc`. It reaches the +data directory through the existing `SymbolIndex::base_path` — the same field +the bar file paths are built from (ratified by `test_file_path_construction` in +`src/lib.rs`, which pins `base_path` as the data directory). Membership in the +bar index is **not** required: a symbol with a sidecar but no bar files, or bar +files but no sidecar, is handled by the file existing or not. + +## Concrete code shapes + +### Worked consumer program (the acceptance evidence) + +What an Aura-side consumer (the counterpart, Brummel/Aura#143) writes — bars and +geometry from one server, no provider JSON in sight: + +```rust +use std::sync::Arc; +use data_server::DataServer; + +let server = Arc::new(DataServer::new("/mnt/tickdata/Pepperstone")); + +if let Some(geo) = server.symbol_meta("EURUSD") { + // geo: InstrumentGeometry — neutral, plain fields + assert_eq!(geo.digits, 5); + assert_eq!(geo.pip_size, 0.0001); + assert_eq!(geo.tick_size, 1e-05); // sidecar wrote it as 1E-05 + assert_eq!(geo.lot_size, 100_000.0); + assert_eq!(geo.quote, "USD"); + + // The consumer derives the rest; nothing provider-specific reached it. + let pip_value_per_lot = geo.lot_size * geo.pip_size; // 100000 * 0.0001 = 10 USD +} + +// Pairs with the existing symbol surface. +for sym in server.symbols() { + if let Some(geo) = server.symbol_meta(&sym) { + // … cross-check geo.base / geo.quote against the consumer's vetted table. + } +} +``` + +`symbol_meta` returns `None` for: no sidecar file, malformed JSON, an +unsupported schema version, or any missing / `null` / non-finite core field +(see §Error handling). The consumer never sees a partial geometry. + +### Implementation shapes (secondary) + +New file `src/meta.rs` — before: does not exist. After: + +```rust +//! Reader for per-symbol `.meta.json` geometry sidecars. + +use serde_json::Value; + +/// The only sidecar schema version this reader understands. +pub const SUPPORTED_SCHEMA_VERSION: u64 = 2; + +/// Neutral, broker-agnostic price-interpretation geometry for one symbol. +/// +/// Intrinsic to the historical price series and effectively constant over its +/// span. Derived/account-dependent conditions are deliberately excluded. +#[derive(Debug, Clone, PartialEq)] +pub struct InstrumentGeometry { + /// Number of decimal places in a quoted price (e.g. 5 for EURUSD). + pub digits: u32, + /// Price increment of one pip (e.g. 0.0001 for EURUSD). + pub pip_size: f64, + /// Smallest representable price increment (e.g. 1e-05 for EURUSD). + pub tick_size: f64, + /// Contract size of one lot in base units (e.g. 100000 for EURUSD). + pub lot_size: f64, + /// Base asset code, passed through verbatim from the sidecar. + pub base: String, + /// Quote (price) currency code, passed through verbatim. + pub quote: String, +} + +impl InstrumentGeometry { + /// Parse sidecar JSON bytes into neutral geometry. + /// + /// Returns `None` if the JSON is malformed, the schema version is + /// unsupported, or any core field is missing / `null` / non-finite. + pub(crate) fn from_sidecar_bytes(bytes: &[u8]) -> Option { + let v: Value = serde_json::from_slice(bytes).ok()?; + if v.get("schemaVersion").and_then(Value::as_u64)? != SUPPORTED_SCHEMA_VERSION { + return None; + } + Some(Self { + digits: v.get("digits").and_then(Value::as_u64)? as u32, + pip_size: finite(v.get("pipSize"))?, + tick_size: finite(v.get("tickSize"))?, + lot_size: finite(v.get("lotSize"))?, + base: v.get("baseAsset").and_then(Value::as_str)?.to_string(), + quote: v.get("quoteAsset").and_then(Value::as_str)?.to_string(), + }) + } +} + +/// A finite `f64` from a JSON number, or `None` for missing / null / non-finite. +/// `Value::as_f64` already rejects `null` and non-numbers; this also rejects +/// `NaN`/`Inf` defensively. +fn finite(v: Option<&Value>) -> Option { + let n = v?.as_f64()?; + n.is_finite().then_some(n) +} +``` + +New method on `DataServer` in `src/lib.rs` — before: absent. After: + +```rust +/// Loads the neutral price geometry for a symbol from its +/// `.meta.json` sidecar, or `None` if no usable sidecar exists. +/// +/// A missing sidecar is normal (`None`, silent). A present-but-unusable +/// sidecar (malformed, unsupported version, incomplete) returns `None` +/// after a one-line stderr warning. +pub fn symbol_meta(&self, symbol: &str) -> Option { + let path = self.index.base_path.join(format!("{symbol}.meta.json")); + let bytes = std::fs::read(&path).ok()?; // missing file -> silent None + let geo = meta::InstrumentGeometry::from_sidecar_bytes(&bytes); + if geo.is_none() { + eprintln!("Warning: {} present but unusable (bad/incomplete geometry)", path.display()); + } + geo +} +``` + +Module wiring in `src/lib.rs` — add `pub mod meta;` alongside the existing +`pub mod cache; pub mod loader; pub mod records;`. `base_path` is already a field +of the private `SymbolIndex`; `symbol_meta` is a `DataServer` method in the same +crate, so it reads `self.index.base_path` directly (no visibility change). + +`Cargo.toml` — add one direct dependency: + +```toml +serde_json = "1" +``` + +## Components + +- **`meta::InstrumentGeometry`** — the public neutral struct (six fields, + `Debug + Clone + PartialEq`). Excludes `symbol`, `description`, `generator`, + `generatedAtUtc` (the most provider-specific fields; `symbol` is already known + to the caller). +- **`meta::InstrumentGeometry::from_sidecar_bytes`** — `pub(crate)` pure parser, + bytes → `Option`. No filesystem, fully unit-testable. Absorbs every + format quirk here. +- **`meta::SUPPORTED_SCHEMA_VERSION`** — the `u64` gate (currently `2`). +- **`DataServer::symbol_meta`** — `&self` one-shot read; the only filesystem + touch. No caching: geometry files are tiny and read rarely (typically once per + symbol); a cache map + invalidation is complexity for no stated need, and the + bar `FileCache` exists for a different problem (large, repeatedly-streamed + files). + +## Data flow + +1. Consumer calls `server.symbol_meta("EURUSD")`. +2. Path built: `/EURUSD.meta.json` (symbols may contain dots, e.g. + `AAPL.US.meta.json`; the literal join handles them). +3. `fs::read` → on error (missing/unreadable) return `None` silently. +4. Bytes → `from_sidecar_bytes`: parse JSON; gate `schemaVersion == 2`; extract + the six fields with the null/non-finite guard. +5. `Some(geometry)` on success; `None` + one stderr warning on a present but + unusable file. + +## Error handling + +The reader absorbs every documented provider quirk so none reaches a consumer: + +- **Scientific notation** (`1E-05`, in 7 of 30 live files) — valid JSON, parses + as a number; `Value::as_f64` handles it. No special code. +- **`null` for a non-finite broker value** — `Value::as_f64`/`as_u64` return + `None` for `null`; that propagates to `symbol_meta` returning `None`. The + consumer never sees a hole. (No `null` exists in the current 30 samples; this + is defensive.) `finite()` additionally rejects `NaN`/`Inf`. +- **Unknown `schemaVersion`** — a reader-side decision, not a consumer's: an + unsupported version returns `None` (+ the call-site warning), never an error + type the consumer must handle. +- **base / quote provider codes** — passed through **verbatim** as `String`. + `baseAsset` is wildly heterogeneous across the 30 samples (ISO `EUR`/`USD`, + index codes `DE40`/`USTEC`/`JP225`, descriptive `Bitcoin`/`Copper`, stock + tickers `AAPL.US`); there is no well-defined "neutral" target and no mapping + table, so normalization is not even definable here. The consumer (Aura#143) + cross-checks the value against its own hand-authored vetted table and never + reads the raw JSON — resolution is the consumer's job by design. "Neutral" + means the struct's shape and semantics are broker-agnostic (a plain Rust type, + parsed numbers, no `serde_json::Value`, no raw JSON, no provider filename + leaking), not that every string is canonicalized. +- **Incomplete object** (a missing core field) — same channel as `null`: + `None`. + +A *missing* sidecar (a symbol with bars but no `.meta.json`) is an expected, +non-error condition → silent `None`. Only a *present* file that cannot yield a +complete geometry warns. + +## Testing strategy + +Unit tests in `src/meta.rs` against the pure `from_sidecar_bytes` (no +filesystem) — each pins one absorbed quirk: + +- Canonical `EURUSD` sidecar → `Some` with exact fields + (`digits 5, pip_size 0.0001, tick_size 1e-05, lot_size 100000.0, base "EUR", + quote "USD"`). Pins scientific-notation `tickSize` and integer `lotSize`. +- Integer-valued geometry (`GER40`: `pipSize 1`, `digits 1`) → `pip_size 1.0`. +- Provider base code passes through unchanged (`NAS100` → `base == "USTEC"`; + dotted `AAPL.US` → `base == "AAPL.US"`). +- Unsupported `schemaVersion` (e.g. `99`) → `None`. +- `null` core field (e.g. `"lotSize": null`) → `None`. +- Missing core field → `None`. +- Malformed JSON → `None`. + +Integration test in `tests/data_server.rs` (or `src/lib.rs` tests) using a +`tempfile::tempdir`: + +- Write `EURUSD.meta.json` into the dir → `server.symbol_meta("EURUSD")` is + `Some` with the expected geometry. +- A symbol with a bar file but no sidecar → `symbol_meta` is `None`. +- An unknown symbol → `symbol_meta` is `None`. + +All tests are deterministic and self-contained (no `/mnt` dependency), matching +the existing test suite's tempdir pattern. + +## Acceptance criteria + +- A consumer obtains a symbol's bars (existing `stream_*`) and its neutral + geometry (`symbol_meta`) from one `DataServer`, pairing with + `has_symbol`/`symbols`. +- The public API exposes only the plain `InstrumentGeometry` struct and `Option` + — no `serde_json` type, no raw JSON, no provider filename, no error type. +- Every documented quirk (scientific notation, `null`-for-non-finite, unknown + schema version, heterogeneous base codes) is absorbed: the consumer sees a + complete neutral geometry or `None`, never a partial struct or a provider + artefact. +- `pip_value_per_lot = lot_size * pip_size` reproduces the issue's verified + values (EURUSD 10 USD; USDJPY 1000 JPY; XAUUSD 10 USD) from the returned + fields. +- The new `serde_json` dependency is the only addition to the dependency set; + the README's dependency statement is updated to reflect it. +- The full suite stays green; new tests pin each absorbed quirk.