Add per-symbol price-metadata sidecars (<symbol>.meta.json)
The price files carry no facts for interpreting the bars (pip size, digit count, contract size, quote currency); those are known only to the broker via cTrader's Symbol object. Produce a JSON sidecar per symbol next to the price files so a downstream reader does not have to hardcode (and drift on) its own per-symbol table. Scope is the price-interpretation geometry only — digits, pipSize, tickSize, lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable across its whole history. Costs, swaps, volume limits, trading status and the deposit-currency pip/tick values are intentionally left out: the cBot only ever sees the current contract, so those are a point-in-time snapshot of today's trading conditions and would misrepresent a multi-year series (the deposit-currency values are account-dependent too). - cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror the cTrader property names. - runner: write_meta runs once per symbol (mode-independent), validates the JSON, and atomically promotes the sidecar. Best-effort — a metadata failure is logged and never aborts the export, and a pre-existing sidecar is kept on failure. A freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short; EXPORT_META=0 disables it. - The file is a raw serialization of what cTrader delivers; a reader derives its own shape (contract_size, per-lot pip value) from the raw fields. See docs/symbol-metadata.md. - Add a source-level regression test for the freshness guard. Verified end-to-end against the live broker for all 30 configured symbols (FX, indices, metals, crypto, commodities, equities); the price corpus was untouched.
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# Per-symbol price metadata (`<SYMBOL>.meta.json`)
|
||||
|
||||
Alongside the price files (`<SYMBOL>_YYYY_MM.m1` / `.tick`) the export driver writes one
|
||||
small JSON sidecar per symbol, `<SYMBOL>.meta.json`. Its sole purpose is to let a reader
|
||||
**interpret the historical price data** — the price files contain only raw OHLC / spread /
|
||||
volume numbers, with nothing that says how to read them (how many decimals a price has, what
|
||||
a "pip" is, which currency the price is quoted in, what one lot is worth). Those facts are
|
||||
known only to the broker, exposed through cTrader's `Symbol` object inside a running cBot.
|
||||
This sidecar captures them so a downstream reader does not have to hardcode (and drift on) a
|
||||
per-symbol table. The per-instrument differences are real and easy to get wrong by hand: a
|
||||
JPY pair's pip is `0.01` (not `0.0001`); gold's lot size is `100` (not the FX `100000`).
|
||||
|
||||
## Scope: price-interpretation geometry only
|
||||
|
||||
The file carries **only the geometry needed to interpret the price series** — facts that are
|
||||
intrinsic to the data and effectively constant over its whole multi-year span:
|
||||
|
||||
| Field | cTrader source | Meaning |
|
||||
|---|---|---|
|
||||
| `digits` | `Symbol.Digits` | price decimal places |
|
||||
| `pipSize` | `Symbol.PipSize` | price units per pip (EURUSD `0.0001`, USDJPY `0.01`, XAUUSD `0.1`) |
|
||||
| `tickSize` | `Symbol.TickSize` | smallest price increment |
|
||||
| `lotSize` | `Symbol.LotSize` | base-currency units per `1.0` lot — turns a price move into money (FX `100000`, XAUUSD `100`) |
|
||||
| `baseAsset` | `Symbol.BaseAsset.Name` | base asset / currency ISO label |
|
||||
| `quoteAsset` | `Symbol.QuoteAsset.Name` | quote asset / currency the price is denominated in |
|
||||
| `description` | `Symbol.Description` | human-readable name |
|
||||
|
||||
Plus provenance: `schemaVersion`, `symbol`, `generatedAtUtc`, `generator`.
|
||||
|
||||
## What is deliberately NOT here — and why
|
||||
|
||||
cTrader's `Symbol` also exposes costs (`Commission*`, `Swap*`), volume limits
|
||||
(`VolumeInUnits*`), trading status (`TradingMode`, `IsTradingEnabled`) and deposit-currency
|
||||
pip/tick values (`PipValue`, `TickValue`). **These are intentionally excluded**, because they
|
||||
do not describe the historical price series — they are a *point-in-time snapshot of today's
|
||||
trading conditions*:
|
||||
|
||||
- **The cBot only ever sees the current contract.** A backtest runs against the symbol as it
|
||||
is defined *now*, never as it stood in a historical month. So the bot physically cannot
|
||||
emit a historically-accurate swap rate or commission for 2015 — only today's. Swap rates in
|
||||
particular change weekly; stamping a single 2026 value onto a 15-year series would
|
||||
misrepresent it.
|
||||
- **`PipValue` / `TickValue` are additionally account-dependent.** cTrader reports them already
|
||||
converted into the *exporting account's deposit currency* at the export-time FX rate, so they
|
||||
are a property of (account × moment), not of the instrument. The stable, account-free pip
|
||||
value is pure arithmetic: `lotSize × pipSize`, in the quote currency.
|
||||
|
||||
If a consumer needs current trading conditions (for a friction model, position sizing, etc.),
|
||||
those are a separate, time-varying concern and belong in their own time-indexed source — not
|
||||
baked into a per-symbol price-geometry file as if they held across history.
|
||||
|
||||
## Mechanism
|
||||
|
||||
- **`MS_SimpleExport` cBot, `MetaOnly` mode** (`src/MS_SimpleExport/MS_SimpleExport.cs`).
|
||||
Run with `--MetaOnly=true --MetaFile=<path>`, the bot does no price export: in `OnStart` it
|
||||
reads `Symbol`, serializes the geometry fields to JSON at `MetaFile`, prints `META_SUCCESS`,
|
||||
and stops.
|
||||
- **`export-runner.sh`, `write_meta`**. Once per symbol per run (mode-independent), the driver
|
||||
spawns a short `MetaOnly` backtest, writes `<SYMBOL>.meta.json.tmp`, validates it is parseable
|
||||
JSON, then renames it to `<SYMBOL>.meta.json`.
|
||||
- **Best-effort by design**: a metadata failure is logged (`⚠️`) and *never* aborts the
|
||||
export run; a pre-existing sidecar is left untouched (overwrite only on a validated success).
|
||||
- **Cadence**: refreshed at most once per `META_MAX_AGE_DAYS` (default `7`) so scheduled runs
|
||||
stay short. `EXPORT_META=0` disables; `META_MAX_AGE_DAYS=0` forces a refresh every run.
|
||||
|
||||
## Format
|
||||
|
||||
A flat JSON object. Conventions:
|
||||
|
||||
- Numbers use invariant formatting — a pip size is `0.0001`, never a locale `0,0001`. Round-trip
|
||||
formatting may emit scientific notation (`1E-05`); valid JSON, parses as a number.
|
||||
- A non-finite broker value (NaN / Infinity) is emitted as JSON `null`.
|
||||
- `schemaVersion` is bumped on any incompatible change to field names or units (currently `2`).
|
||||
|
||||
### Example
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 2,
|
||||
"symbol": "EURUSD",
|
||||
"generatedAtUtc": "2026-06-25T11:06:01Z",
|
||||
"generator": "MS_SimpleExport (cTrader.Automate)",
|
||||
"description": "Euro vs US Dollar",
|
||||
"digits": 5,
|
||||
"pipSize": 0.0001,
|
||||
"tickSize": 1E-05,
|
||||
"lotSize": 100000,
|
||||
"baseAsset": "EUR",
|
||||
"quoteAsset": "USD"
|
||||
}
|
||||
```
|
||||
|
||||
## Consumer-side derivations
|
||||
|
||||
A reader composes whatever shape it wants from these raw fields, e.g.:
|
||||
|
||||
```
|
||||
contract_size = lotSize
|
||||
pip_value_per_lot = lotSize * pipSize # in quoteAsset (EURUSD: 100000*0.0001 = 10 USD)
|
||||
point_size = 10 ** -digits
|
||||
```
|
||||
|
||||
Trading constraints a consumer's own instrument spec might want — minimum lot, lot step, a
|
||||
stable instrument id — are **not** price-interpretation geometry and are not in this file; the
|
||||
consumer authors those itself (an id is a registry choice; min-lot / lot-step are current order
|
||||
limits, not facts about the historical bars).
|
||||
|
||||
Worked geometry, verified against the live broker:
|
||||
|
||||
| symbol | digits | pipSize | tickSize | lotSize | base/quote | → pip_value_per_lot |
|
||||
|---|---|---|---|---|---|---|
|
||||
| EURUSD | 5 | 0.0001 | 1E-05 | 100000 | EUR/USD | 10.0 USD |
|
||||
| USDJPY | 3 | 0.01 | 0.001 | 100000 | USD/JPY | 1000 JPY |
|
||||
| XAUUSD | 2 | 0.1 | 0.01 | 100 | XAU/USD | 10.0 USD |
|
||||
Reference in New Issue
Block a user