Add a reader for per-symbol geometry sidecars (<SYMBOL>.meta.json) returning neutral InstrumentGeometry #3

New Issue

Closed

opened 2026-06-25 13:58:33 +02:00 by Brummel · 1 comment

Brummel commented 2026-06-25 13:58:33 +02:00

Owner

Copy Link

Copy Source

The export pipeline that produces this server's SYMBOL_YYYY_MM.m1 / .tick
files now also writes one small per-symbol JSON sidecar next to them in the data
directory: <SYMBOL>.meta.json. Verified: 30 such files in
/mnt/tickdata/Pepperstone, all schemaVersion 2, mtimes 2026-06-25 13:26–13:32.
They carry the price-interpretation geometry the bars themselves lack — how many
decimals a price has, what a pip is, what one lot is worth, which currency the
price is quoted in.

This server already owns that directory, the symbol index over it, and the
binary price format. The sidecar is the price stream's own units/geometry, so it
belongs behind the same reader rather than being parsed by each downstream
consumer.

What

Add a reader (e.g. symbol_meta(symbol) -> Option<InstrumentGeometry>) that loads
<SYMBOL>.meta.json and returns neutral geometry — nothing provider-specific
escapes the API. Pair it with the existing has_symbol / symbols surface so a
consumer can ask one server for a symbol's bars and its geometry.

Sidecar schema (verified, schemaVersion 2)

{
  "schemaVersion": 2,
  "symbol": "EURUSD",
  "generatedAtUtc": "2026-06-25T11:26:37Z",
  "generator": "MS_SimpleExport (cTrader.Automate)",
  "description": "Euro vs US Dollar",
  "digits": 5,
  "pipSize": 0.0001,
  "tickSize": 1E-05,
  "lotSize": 100000,
  "baseAsset": "EUR",
  "quoteAsset": "USD"
}

A flat JSON object, one per symbol. Provider quirks the reader must absorb so they
never reach a consumer:

• Numbers may be in scientific notation (1E-05); valid JSON, parses as a number.
• A non-finite broker value is emitted as JSON null.
• schemaVersion gates compatibility (currently 2); an unknown version is a
  reader-side decision, not a consumer's.
• baseAsset is sometimes a provider-specific code, not clean ISO: GER40 → DE40, FRA40 → F40, NAS100 → USTEC, JPN225 → JP225. quoteAsset is a real
  currency across the 30 samples (EUR/USD/JPY/HKD/GBP).

Suggested neutral output

A flat geometry struct: digits, pip_size, tick_size, lot_size, base,
quote. A consumer derives the rest, e.g. pip_value_per_lot = lot_size * pip_size (in quote): EURUSD 100000 * 0.0001 = 10 USD, USDJPY 100000 * 0.01 = 1000 JPY, XAUUSD 100 * 0.1 = 10 USD — all verified against the samples.

Scope boundary — geometry only

The sidecar deliberately carries only facts intrinsic to the historical price
series and effectively constant over its multi-year span. It excludes
time-varying / account-dependent conditions (commission, swap, deposit-currency
pip value, volume limits, trading status). Those are a separate, time-indexed
concern and are not this reader's job.

Why here, not in the consumer

Keeping the sidecar's filename, schema version, and provider quirks (the DE40 /
USTEC naming, the 1E-05 formatting, null-for-NaN) inside this broker-facing
library is what lets the downstream consumer stay broker-agnostic. Splitting it —
bars read here, geometry read by the consumer — would put two readers with two
independent path assumptions on one directory, free to drift.

Dependency note

Parsing the flat object adds a JSON dependency (serde_json) or a small
hand-rolled parse. Minor; serde_json is the straightforward call.

Consumer counterpart

The aura-side consumer of this neutral geometry is
http://192.168.178.103:3000/Brummel/Aura/issues/143. This server is the
upstream provider; that issue slots the neutral geometry into aura's
instrument-reference resolution hierarchy (the recorded-metadata tier of
http://192.168.178.103:3000/Brummel/Aura/issues/124), cross-checks it against
the hand-authored vetted table, and never reads the raw JSON itself.

The export pipeline that produces this server's `SYMBOL_YYYY_MM.m1` / `.tick` files now also writes one small per-symbol JSON sidecar next to them in the data directory: `<SYMBOL>.meta.json`. Verified: 30 such files in `/mnt/tickdata/Pepperstone`, all `schemaVersion` 2, mtimes 2026-06-25 13:26–13:32. They carry the price-interpretation geometry the bars themselves lack — how many decimals a price has, what a pip is, what one lot is worth, which currency the price is quoted in. This server already owns that directory, the symbol index over it, and the binary price format. The sidecar is the price stream's own units/geometry, so it belongs behind the same reader rather than being parsed by each downstream consumer. ## What Add a reader (e.g. `symbol_meta(symbol) -> Option<InstrumentGeometry>`) that loads `<SYMBOL>.meta.json` and returns **neutral** geometry — nothing provider-specific escapes the API. Pair it with the existing `has_symbol` / `symbols` surface so a consumer can ask one server for a symbol's bars *and* its geometry. ### Sidecar schema (verified, schemaVersion 2) ```json { "schemaVersion": 2, "symbol": "EURUSD", "generatedAtUtc": "2026-06-25T11:26:37Z", "generator": "MS_SimpleExport (cTrader.Automate)", "description": "Euro vs US Dollar", "digits": 5, "pipSize": 0.0001, "tickSize": 1E-05, "lotSize": 100000, "baseAsset": "EUR", "quoteAsset": "USD" } ``` A flat JSON object, one per symbol. Provider quirks the reader must absorb so they never reach a consumer: - Numbers may be in scientific notation (`1E-05`); valid JSON, parses as a number. - A non-finite broker value is emitted as JSON `null`. - `schemaVersion` gates compatibility (currently `2`); an unknown version is a reader-side decision, not a consumer's. - `baseAsset` is sometimes a provider-specific code, not clean ISO: `GER40 → DE40`, `FRA40 → F40`, `NAS100 → USTEC`, `JPN225 → JP225`. `quoteAsset` is a real currency across the 30 samples (EUR/USD/JPY/HKD/GBP). ### Suggested neutral output A flat geometry struct: `digits`, `pip_size`, `tick_size`, `lot_size`, `base`, `quote`. A consumer derives the rest, e.g. `pip_value_per_lot = lot_size * pip_size` (in `quote`): EURUSD `100000 * 0.0001 = 10` USD, USDJPY `100000 * 0.01 = 1000` JPY, XAUUSD `100 * 0.1 = 10` USD — all verified against the samples. ## Scope boundary — geometry only The sidecar deliberately carries only facts intrinsic to the historical price series and effectively constant over its multi-year span. It excludes time-varying / account-dependent conditions (commission, swap, deposit-currency pip value, volume limits, trading status). Those are a separate, time-indexed concern and are not this reader's job. ## Why here, not in the consumer Keeping the sidecar's filename, schema version, and provider quirks (the `DE40` / `USTEC` naming, the `1E-05` formatting, `null`-for-NaN) inside this broker-facing library is what lets the downstream consumer stay broker-agnostic. Splitting it — bars read here, geometry read by the consumer — would put two readers with two independent path assumptions on one directory, free to drift. ## Dependency note Parsing the flat object adds a JSON dependency (`serde_json`) or a small hand-rolled parse. Minor; `serde_json` is the straightforward call. ## Consumer counterpart The aura-side consumer of this neutral geometry is `http://192.168.178.103:3000/Brummel/Aura/issues/143`. This server is the upstream provider; that issue slots the neutral geometry into aura's instrument-reference resolution hierarchy (the recorded-metadata tier of `http://192.168.178.103:3000/Brummel/Aura/issues/124`), cross-checks it against the hand-authored vetted table, and never reads the raw JSON itself.

Brummel commented 2026-06-25 14:15:50 +02:00

Author

Owner

Copy Link

Copy Source

Design reconciliation (specify)

Spec: docs/specs/0001-symbol-meta-geometry-reader.md. The forks below are
still listed open on this issue; this records their resolution. All are
derived orchestrator decisions (autonomous /boss run), not minuted user
decisions — each carries its rationale and is vetoable.

• Fork: base/quote — normalize provider codes vs pass through → pass the
  raw baseAsset / quoteAsset strings through unchanged.
  Basis: derived. The 30 live sidecars show baseAsset is wildly
  heterogeneous — clean ISO (EUR, USD), index codes (DE40, USTEC,
  JP225, US500, VIX), descriptive names (Bitcoin, Copper,
  Soybeans), and stock tickers (AAPL.US, GOOG.US). There is no
  well-defined "neutral" target for an index/stock/commodity base, so
  normalization is not even definable here, and no mapping table exists. The
  consumer (Aura #143) explicitly cross-checks the value against a
  hand-authored vetted table and never reads the raw JSON — resolution is the
  consumer's job by design. "Neutral" is therefore read as: the struct's
  shape and semantics are broker-agnostic (a plain Rust type, parsed numbers,
  no serde_json::Value, no raw JSON, no provider filename leaking), not that
  every string is canonicalized.

• Fork: unknown schemaVersion → return None (and log a one-line stderr
  warning). Basis: derived. The API is already Option; the issue states the
  version decision is "reader-side, not a consumer's". House style degrades
  failures to None/empty with an eprintln warning (see lib.rs load
  helpers). A missing sidecar is normal (a symbol may have bars but no
  sidecar) → silent None; a present but unsupported/unparseable sidecar →
  None + warning.

• Fork: null / non-finite numeric fields → any null or non-finite core
  numeric field makes the geometry incomplete → symbol_meta returns None
  (never a holed struct). Basis: derived. The issue frames a flat geometry
  struct (not per-field options) and the consumer derives arithmetic
  (pip_value_per_lot = lot_size * pip_size) needing every field. "Absorb
  null-for-NaN so it never reaches a consumer" is honored most literally by
  emitting a complete geometry or None. (No nulls exist in the current 30
  samples; this is a defensive policy with no present observable effect.)

• Fork: field types → InstrumentGeometry { digits: u32, pip_size: f64, tick_size: f64, lot_size: f64, base: String, quote: String }. Basis:
  derived. digits is a small precision count (samples 1/2/5); lot_size is
  f64 because the consumer multiplies it by pip_size and f64 represents
  the integer sizes (1/100/100000) exactly. The six fields are exactly the
  issue's "suggested neutral output"; symbol / description / generator /
  generatedAtUtc are deliberately excluded (the most provider-specific
  strings, and symbol is already known to the caller).

• Fork: JSON parsing — serde_json vs hand-rolled → add serde_json (only;
  no serde-derive), parse via serde_json::Value field extraction. Basis:
  derived + issue-endorsed. The issue calls serde_json "the straightforward
  call"; scientific notation (1E-05, present in 7 files), string escaping,
  and robust number parsing are exactly what a hand-rolled parser botches.
  Value extraction keeps direct deps minimal (one new direct dependency, no
  proc-macro). This expands the README's "depends only on chrono, regex, zip"
  boast — a deliberate, issue-sanctioned change; the README dependency line is
  updated as part of the work.

• Fork: caching vs read-on-demand → read + parse the file fresh on each
  symbol_meta call (no cache). Basis: derived. Geometry files are tiny and
  read rarely (typically once per symbol); the issue requests no caching;
  adding a cache map + invalidation is complexity for no stated need. Contrast
  the bar FileCache, which exists for large, repeatedly-streamed files.

• Fork: receiver & index coupling → DataServer::symbol_meta(&self, symbol: &str) -> Option<InstrumentGeometry>; does not require the symbol to be
  in the bar index. Basis: derived. No prefetch/streaming → no Arc<Self>
  needed. The file is read directly at <base_path>/<symbol>.meta.json
  (symbols with dots like AAPL.US work — the files are literally named that
  way); a missing file → None covers both "unknown symbol" and "known symbol
  without sidecar". Requiring index membership would add coupling the issue
  does not ask for.

New module src/meta.rs holds InstrumentGeometry + the parser; the
symbol_meta method lives on DataServer in lib.rs. Public path
data_server::meta::InstrumentGeometry, consistent with
data_server::records::*.

## Design reconciliation (specify) Spec: `docs/specs/0001-symbol-meta-geometry-reader.md`. The forks below are still listed open on this issue; this records their resolution. All are **derived** orchestrator decisions (autonomous `/boss` run), not minuted user decisions — each carries its rationale and is vetoable. - **Fork: base/quote — normalize provider codes vs pass through** → pass the raw `baseAsset` / `quoteAsset` strings through unchanged. Basis: derived. The 30 live sidecars show `baseAsset` is wildly heterogeneous — clean ISO (`EUR`, `USD`), index codes (`DE40`, `USTEC`, `JP225`, `US500`, `VIX`), descriptive names (`Bitcoin`, `Copper`, `Soybeans`), and stock tickers (`AAPL.US`, `GOOG.US`). There is no well-defined "neutral" target for an index/stock/commodity base, so normalization is not even definable here, and no mapping table exists. The consumer (Aura #143) explicitly *cross-checks the value against a hand-authored vetted table and never reads the raw JSON* — resolution is the consumer's job by design. "Neutral" is therefore read as: the struct's shape and semantics are broker-agnostic (a plain Rust type, parsed numbers, no `serde_json::Value`, no raw JSON, no provider filename leaking), not that every string is canonicalized. - **Fork: unknown `schemaVersion`** → return `None` (and log a one-line stderr warning). Basis: derived. The API is already `Option`; the issue states the version decision is "reader-side, not a consumer's". House style degrades failures to `None`/empty with an `eprintln` warning (see lib.rs load helpers). A *missing* sidecar is normal (a symbol may have bars but no sidecar) → silent `None`; a *present but unsupported/unparseable* sidecar → `None` + warning. - **Fork: null / non-finite numeric fields** → any `null` or non-finite core numeric field makes the geometry incomplete → `symbol_meta` returns `None` (never a holed struct). Basis: derived. The issue frames a flat geometry struct (not per-field options) and the consumer derives arithmetic (`pip_value_per_lot = lot_size * pip_size`) needing every field. "Absorb null-for-NaN so it never reaches a consumer" is honored most literally by emitting a complete geometry or `None`. (No nulls exist in the current 30 samples; this is a defensive policy with no present observable effect.) - **Fork: field types** → `InstrumentGeometry { digits: u32, pip_size: f64, tick_size: f64, lot_size: f64, base: String, quote: String }`. Basis: derived. `digits` is a small precision count (samples 1/2/5); `lot_size` is `f64` because the consumer multiplies it by `pip_size` and `f64` represents the integer sizes (1/100/100000) exactly. The six fields are exactly the issue's "suggested neutral output"; `symbol` / `description` / `generator` / `generatedAtUtc` are deliberately excluded (the most provider-specific strings, and `symbol` is already known to the caller). - **Fork: JSON parsing — serde_json vs hand-rolled** → add `serde_json` (only; no serde-derive), parse via `serde_json::Value` field extraction. Basis: derived + issue-endorsed. The issue calls serde_json "the straightforward call"; scientific notation (`1E-05`, present in 7 files), string escaping, and robust number parsing are exactly what a hand-rolled parser botches. `Value` extraction keeps direct deps minimal (one new direct dependency, no proc-macro). This expands the README's "depends only on chrono, regex, zip" boast — a deliberate, issue-sanctioned change; the README dependency line is updated as part of the work. - **Fork: caching vs read-on-demand** → read + parse the file fresh on each `symbol_meta` call (no cache). Basis: derived. Geometry files are tiny and read rarely (typically once per symbol); the issue requests no caching; adding a cache map + invalidation is complexity for no stated need. Contrast the bar `FileCache`, which exists for large, repeatedly-streamed files. - **Fork: receiver & index coupling** → `DataServer::symbol_meta(&self, symbol: &str) -> Option<InstrumentGeometry>`; does **not** require the symbol to be in the bar index. Basis: derived. No prefetch/streaming → no `Arc<Self>` needed. The file is read directly at `<base_path>/<symbol>.meta.json` (symbols with dots like `AAPL.US` work — the files are literally named that way); a missing file → `None` covers both "unknown symbol" and "known symbol without sidecar". Requiring index membership would add coupling the issue does not ask for. New module `src/meta.rs` holds `InstrumentGeometry` + the parser; the `symbol_meta` method lives on `DataServer` in lib.rs. Public path `data_server::meta::InstrumentGeometry`, consistent with `data_server::records::*`.

Brummel referenced this issue from a commit 2026-06-25 15:29:11 +02:00

spec: per-symbol geometry sidecar reader (boss-signed)

Brummel referenced this issue from a commit 2026-06-25 15:29:11 +02:00

plan: per-symbol geometry sidecar reader

Brummel referenced this issue from a commit 2026-06-25 15:29:11 +02:00

Add symbol_meta reader for per-symbol geometry sidecars

Brummel closed this issue 2026-06-25 15:29:11 +02:00

Brummel referenced this issue from a commit 2026-06-25 15:29:11 +02:00

audit: cycle #3 tidy (clean)

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels

bug

observable misbehaviour

idea

strikeable, no commitment

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

claude (Claude) Brummel

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/data-server#3