Files
cTrader/docs/symbol-metadata.md
T
Brummel cd6874b43f Add per-symbol price-metadata sidecars (<symbol>.meta.json)
The price files carry no facts for interpreting the bars (pip size, digit count,
contract size, quote currency); those are known only to the broker via cTrader's
Symbol object. Produce a JSON sidecar per symbol next to the price files so a
downstream reader does not have to hardcode (and drift on) its own per-symbol table.

Scope is the price-interpretation geometry only — digits, pipSize, tickSize,
lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable
across its whole history. Costs, swaps, volume limits, trading status and the
deposit-currency pip/tick values are intentionally left out: the cBot only ever
sees the current contract, so those are a point-in-time snapshot of today's trading
conditions and would misrepresent a multi-year series (the deposit-currency values
are account-dependent too).

- cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat
  JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror
  the cTrader property names.
- runner: write_meta runs once per symbol (mode-independent), validates the JSON,
  and atomically promotes the sidecar. Best-effort — a metadata failure is logged
  and never aborts the export, and a pre-existing sidecar is kept on failure. A
  freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short;
  EXPORT_META=0 disables it.
- The file is a raw serialization of what cTrader delivers; a reader derives its
  own shape (contract_size, per-lot pip value) from the raw fields. See
  docs/symbol-metadata.md.
- Add a source-level regression test for the freshness guard.

Verified end-to-end against the live broker for all 30 configured symbols (FX,
indices, metals, crypto, commodities, equities); the price corpus was untouched.
2026-06-25 13:32:45 +02:00

5.7 KiB
Raw Blame History

Per-symbol price metadata (<SYMBOL>.meta.json)

Alongside the price files (<SYMBOL>_YYYY_MM.m1 / .tick) the export driver writes one small JSON sidecar per symbol, <SYMBOL>.meta.json. Its sole purpose is to let a reader interpret the historical price data — the price files contain only raw OHLC / spread / volume numbers, with nothing that says how to read them (how many decimals a price has, what a "pip" is, which currency the price is quoted in, what one lot is worth). Those facts are known only to the broker, exposed through cTrader's Symbol object inside a running cBot. This sidecar captures them so a downstream reader does not have to hardcode (and drift on) a per-symbol table. The per-instrument differences are real and easy to get wrong by hand: a JPY pair's pip is 0.01 (not 0.0001); gold's lot size is 100 (not the FX 100000).

Scope: price-interpretation geometry only

The file carries only the geometry needed to interpret the price series — facts that are intrinsic to the data and effectively constant over its whole multi-year span:

Field cTrader source Meaning
digits Symbol.Digits price decimal places
pipSize Symbol.PipSize price units per pip (EURUSD 0.0001, USDJPY 0.01, XAUUSD 0.1)
tickSize Symbol.TickSize smallest price increment
lotSize Symbol.LotSize base-currency units per 1.0 lot — turns a price move into money (FX 100000, XAUUSD 100)
baseAsset Symbol.BaseAsset.Name base asset / currency ISO label
quoteAsset Symbol.QuoteAsset.Name quote asset / currency the price is denominated in
description Symbol.Description human-readable name

Plus provenance: schemaVersion, symbol, generatedAtUtc, generator.

What is deliberately NOT here — and why

cTrader's Symbol also exposes costs (Commission*, Swap*), volume limits (VolumeInUnits*), trading status (TradingMode, IsTradingEnabled) and deposit-currency pip/tick values (PipValue, TickValue). These are intentionally excluded, because they do not describe the historical price series — they are a point-in-time snapshot of today's trading conditions:

  • The cBot only ever sees the current contract. A backtest runs against the symbol as it is defined now, never as it stood in a historical month. So the bot physically cannot emit a historically-accurate swap rate or commission for 2015 — only today's. Swap rates in particular change weekly; stamping a single 2026 value onto a 15-year series would misrepresent it.
  • PipValue / TickValue are additionally account-dependent. cTrader reports them already converted into the exporting account's deposit currency at the export-time FX rate, so they are a property of (account × moment), not of the instrument. The stable, account-free pip value is pure arithmetic: lotSize × pipSize, in the quote currency.

If a consumer needs current trading conditions (for a friction model, position sizing, etc.), those are a separate, time-varying concern and belong in their own time-indexed source — not baked into a per-symbol price-geometry file as if they held across history.

Mechanism

  • MS_SimpleExport cBot, MetaOnly mode (src/MS_SimpleExport/MS_SimpleExport.cs). Run with --MetaOnly=true --MetaFile=<path>, the bot does no price export: in OnStart it reads Symbol, serializes the geometry fields to JSON at MetaFile, prints META_SUCCESS, and stops.
  • export-runner.sh, write_meta. Once per symbol per run (mode-independent), the driver spawns a short MetaOnly backtest, writes <SYMBOL>.meta.json.tmp, validates it is parseable JSON, then renames it to <SYMBOL>.meta.json.
    • Best-effort by design: a metadata failure is logged (⚠️) and never aborts the export run; a pre-existing sidecar is left untouched (overwrite only on a validated success).
    • Cadence: refreshed at most once per META_MAX_AGE_DAYS (default 7) so scheduled runs stay short. EXPORT_META=0 disables; META_MAX_AGE_DAYS=0 forces a refresh every run.

Format

A flat JSON object. Conventions:

  • Numbers use invariant formatting — a pip size is 0.0001, never a locale 0,0001. Round-trip formatting may emit scientific notation (1E-05); valid JSON, parses as a number.
  • A non-finite broker value (NaN / Infinity) is emitted as JSON null.
  • schemaVersion is bumped on any incompatible change to field names or units (currently 2).

Example

{
  "schemaVersion": 2,
  "symbol": "EURUSD",
  "generatedAtUtc": "2026-06-25T11:06:01Z",
  "generator": "MS_SimpleExport (cTrader.Automate)",
  "description": "Euro vs US Dollar",
  "digits": 5,
  "pipSize": 0.0001,
  "tickSize": 1E-05,
  "lotSize": 100000,
  "baseAsset": "EUR",
  "quoteAsset": "USD"
}

Consumer-side derivations

A reader composes whatever shape it wants from these raw fields, e.g.:

contract_size      = lotSize
pip_value_per_lot  = lotSize * pipSize           # in quoteAsset (EURUSD: 100000*0.0001 = 10 USD)
point_size         = 10 ** -digits

Trading constraints a consumer's own instrument spec might want — minimum lot, lot step, a stable instrument id — are not price-interpretation geometry and are not in this file; the consumer authors those itself (an id is a registry choice; min-lot / lot-step are current order limits, not facts about the historical bars).

Worked geometry, verified against the live broker:

symbol digits pipSize tickSize lotSize base/quote → pip_value_per_lot
EURUSD 5 0.0001 1E-05 100000 EUR/USD 10.0 USD
USDJPY 3 0.01 0.001 100000 USD/JPY 1000 JPY
XAUUSD 2 0.1 0.01 100 XAU/USD 10.0 USD