Brummel/cTrader
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Add per-symbol price-metadata sidecars (<symbol>.meta.json)

Browse Source 
The price files carry no facts for interpreting the bars (pip size, digit count,
contract size, quote currency); those are known only to the broker via cTrader's
Symbol object. Produce a JSON sidecar per symbol next to the price files so a
downstream reader does not have to hardcode (and drift on) its own per-symbol table.

Scope is the price-interpretation geometry only — digits, pipSize, tickSize,
lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable
across its whole history. Costs, swaps, volume limits, trading status and the
deposit-currency pip/tick values are intentionally left out: the cBot only ever
sees the current contract, so those are a point-in-time snapshot of today's trading
conditions and would misrepresent a multi-year series (the deposit-currency values
are account-dependent too).

- cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat
  JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror
  the cTrader property names.
- runner: write_meta runs once per symbol (mode-independent), validates the JSON,
  and atomically promotes the sidecar. Best-effort — a metadata failure is logged
  and never aborts the export, and a pre-existing sidecar is kept on failure. A
  freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short;
  EXPORT_META=0 disables it.
- The file is a raw serialization of what cTrader delivers; a reader derives its
  own shape (contract_size, per-lot pip value) from the raw fields. See
  docs/symbol-metadata.md.
- Add a source-level regression test for the freshness guard.

Verified end-to-end against the live broker for all 30 configured symbols (FX,
indices, metals, crypto, commodities, equities); the price corpus was untouched.
This commit is contained in:
Brummel
2026-06-25 13:12:54 +02:00
parent ddd8274fad
commit cd6874b43f
4 changed files with 272 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/symbol-metadata.md
+114
View File
@@ -0,0 +1,114 @@
# Per-symbol price metadata (`<SYMBOL>.meta.json`)
Alongside the price files (`<SYMBOL>_YYYY_MM.m1` / `.tick`) the export driver writes one
small JSON sidecar per symbol, `<SYMBOL>.meta.json`. Its sole purpose is to let a reader
**interpret the historical price data** — the price files contain only raw OHLC / spread /
volume numbers, with nothing that says how to read them (how many decimals a price has, what
a "pip" is, which currency the price is quoted in, what one lot is worth). Those facts are
known only to the broker, exposed through cTrader's `Symbol` object inside a running cBot.
This sidecar captures them so a downstream reader does not have to hardcode (and drift on) a
per-symbol table. The per-instrument differences are real and easy to get wrong by hand: a
JPY pair's pip is `0.01` (not `0.0001`); gold's lot size is `100` (not the FX `100000`).
## Scope: price-interpretation geometry only
The file carries **only the geometry needed to interpret the price series** — facts that are
intrinsic to the data and effectively constant over its whole multi-year span:
| Field | cTrader source | Meaning |
|---|---|---|
| `digits` | `Symbol.Digits` | price decimal places |
| `pipSize` | `Symbol.PipSize` | price units per pip (EURUSD `0.0001`, USDJPY `0.01`, XAUUSD `0.1`) |
| `tickSize` | `Symbol.TickSize` | smallest price increment |
| `lotSize` | `Symbol.LotSize` | base-currency units per `1.0` lot — turns a price move into money (FX `100000`, XAUUSD `100`) |
| `baseAsset` | `Symbol.BaseAsset.Name` | base asset / currency ISO label |
| `quoteAsset` | `Symbol.QuoteAsset.Name` | quote asset / currency the price is denominated in |
| `description` | `Symbol.Description` | human-readable name |
Plus provenance: `schemaVersion`, `symbol`, `generatedAtUtc`, `generator`.
## What is deliberately NOT here — and why
cTrader's `Symbol` also exposes costs (`Commission*`, `Swap*`), volume limits
(`VolumeInUnits*`), trading status (`TradingMode`, `IsTradingEnabled`) and deposit-currency
pip/tick values (`PipValue`, `TickValue`). **These are intentionally excluded**, because they
do not describe the historical price series — they are a *point-in-time snapshot of today's
trading conditions*:
- **The cBot only ever sees the current contract.** A backtest runs against the symbol as it
is defined *now*, never as it stood in a historical month. So the bot physically cannot
emit a historically-accurate swap rate or commission for 2015 — only today's. Swap rates in
particular change weekly; stamping a single 2026 value onto a 15-year series would
misrepresent it.
- **`PipValue` / `TickValue` are additionally account-dependent.** cTrader reports them already
converted into the *exporting account's deposit currency* at the export-time FX rate, so they
are a property of (account × moment), not of the instrument. The stable, account-free pip
value is pure arithmetic: `lotSize × pipSize`, in the quote currency.
If a consumer needs current trading conditions (for a friction model, position sizing, etc.),
those are a separate, time-varying concern and belong in their own time-indexed source — not
baked into a per-symbol price-geometry file as if they held across history.
## Mechanism
- **`MS_SimpleExport` cBot, `MetaOnly` mode** (`src/MS_SimpleExport/MS_SimpleExport.cs`).
Run with `--MetaOnly=true --MetaFile=<path>`, the bot does no price export: in `OnStart` it
reads `Symbol`, serializes the geometry fields to JSON at `MetaFile`, prints `META_SUCCESS`,
and stops.
- **`export-runner.sh`, `write_meta`**. Once per symbol per run (mode-independent), the driver
spawns a short `MetaOnly` backtest, writes `<SYMBOL>.meta.json.tmp`, validates it is parseable
JSON, then renames it to `<SYMBOL>.meta.json`.
- **Best-effort by design**: a metadata failure is logged (`⚠️`) and *never* aborts the
export run; a pre-existing sidecar is left untouched (overwrite only on a validated success).
- **Cadence**: refreshed at most once per `META_MAX_AGE_DAYS` (default `7`) so scheduled runs
stay short. `EXPORT_META=0` disables; `META_MAX_AGE_DAYS=0` forces a refresh every run.
## Format
A flat JSON object. Conventions:
- Numbers use invariant formatting — a pip size is `0.0001`, never a locale `0,0001`. Round-trip
formatting may emit scientific notation (`1E-05`); valid JSON, parses as a number.
- A non-finite broker value (NaN / Infinity) is emitted as JSON `null`.
- `schemaVersion` is bumped on any incompatible change to field names or units (currently `2`).
### Example
```json
{
"schemaVersion": 2,
"symbol": "EURUSD",
"generatedAtUtc": "2026-06-25T11:06:01Z",
"generator": "MS_SimpleExport (cTrader.Automate)",
"description": "Euro vs US Dollar",
"digits": 5,
"pipSize": 0.0001,
"tickSize": 1E-05,
"lotSize": 100000,
"baseAsset": "EUR",
"quoteAsset": "USD"
}
```
## Consumer-side derivations
A reader composes whatever shape it wants from these raw fields, e.g.:
```
contract_size = lotSize
pip_value_per_lot = lotSize * pipSize # in quoteAsset (EURUSD: 100000*0.0001 = 10 USD)
point_size = 10 ** -digits
```
Trading constraints a consumer's own instrument spec might want — minimum lot, lot step, a
stable instrument id — are **not** price-interpretation geometry and are not in this file; the
consumer authors those itself (an id is a registry choice; min-lot / lot-step are current order
limits, not facts about the historical bars).
Worked geometry, verified against the live broker:
| symbol | digits | pipSize | tickSize | lotSize | base/quote | → pip_value_per_lot |
|---|---|---|---|---|---|---|
| EURUSD | 5 | 0.0001 | 1E-05 | 100000 | EUR/USD | 10.0 USD |
| USDJPY | 3 | 0.01 | 0.001 | 100000 | USD/JPY | 1000 JPY |
| XAUUSD | 2 | 0.1 | 0.01 | 100 | XAU/USD | 10.0 USD |
scripts/export-runner.sh
+77 -1
View File
@@ -35,6 +35,8 @@ SYMBOL_FILE="${SYMBOL_FILE:-$REPO_ROOT/runtime/ScrapeSymbols.txt}" # overridabl
RETRIES="${RETRIES:-3}" # attempts before a transient failure aborts
RUN_TIMEOUT="${RUN_TIMEOUT:-600}" # seconds per backtest before it counts as a hang
DISCOVER_MIN_YEAR="${DISCOVER_MIN_YEAR:-2005}" # earliest year the first-data probe scans
EXPORT_META="${EXPORT_META:-1}" # 1 = also produce <symbol>.meta.json sidecars
META_MAX_AGE_DAYS="${META_MAX_AGE_DAYS:-7}" # refresh a sidecar at most this often (0 = every run)
# --- DEPLOYMENT CONFIG (secrets/IDs/paths) ---
ENV_FILE="${ENV_FILE:-$REPO_ROOT/runtime/export.env}"
@@ -207,6 +209,76 @@ probe_run() {
esac
}
# ---------------------------------------------------------------------------
# write_meta SYMBOL — produce the per-symbol contract-metadata sidecar
# (<SYMBOL>.meta.json), placed next to the price files. The cBot's MetaOnly mode
# is the ONLY authoritative source for these values: it reads the broker's Symbol
# object and serializes it verbatim; this driver merely places the file it writes.
#
# BEST-EFFORT BY DESIGN: a metadata failure is logged (⚠️) and NEVER aborts the
# export run — the sidecar is auxiliary, and any pre-existing one is left untouched
# (we only ever overwrite on a validated success). Refreshed at most once per
# META_MAX_AGE_DAYS so scheduled runs stay short.
# ---------------------------------------------------------------------------
write_meta() {
local S=$1
local FINAL="${S}.meta.json"
local TEMP="${S}.meta.json.tmp"
# Freshness guard: skip if a recent sidecar already exists (mtime within the window).
if [[ "$META_MAX_AGE_DAYS" -gt 0 && -f "$HOST_DATA_PATH/$FINAL" \
&& -n "$(find "$HOST_DATA_PATH/$FINAL" -mtime "-$META_MAX_AGE_DAYS" 2>/dev/null)" ]]; then
echo "🏷 Metadata: $S — fresh (<${META_MAX_AGE_DAYS}d), skipping"
META_SKIP=$((META_SKIP+1)); return 0
fi
local C_NAME="ctrader-meta-${S//[^A-Za-z0-9]/-}-$(date +%s%N)"
local s_date e_date
s_date="$(date -d '7 days ago' +%d/%m/%Y)" # tiny window; MetaOnly stops in OnStart anyway
e_date="$(date -d 'yesterday' +%d/%m/%Y)"
echo "🏷 Metadata: $S (MetaOnly)"
CURRENT_CONTAINER="$C_NAME"
docker run -d --name "$C_NAME" --init \
-v "$HOST_ROBOTS_PATH":"/mnt/Robots" \
-v "$HOST_DATA_PATH":"$CONTAINER_DATA_PATH" \
-e TZ="Europe/Berlin" \
"$IMAGE" \
backtest "/mnt/Robots/$BOT_FILE" \
--start="$s_date" --end="$e_date" --symbol="$S" \
--ctid="$CTID" --pwd-file="$PWD_FILE" --account="$ACCOUNT" \
--period="M1" --data-mode="m1" \
--Mode="M1" --IsProbing="false" --MetaOnly="true" \
--MetaFile="${CONTAINER_DATA_PATH}/${TEMP}" --full-access > /dev/null 2>&1 || {
echo " ⚠️ meta: docker run failed to start container for $S"
CURRENT_CONTAINER=""; META_FAIL=$((META_FAIL+1)); return 0; }
while IFS= read -r line; do
case "$line" in
*META_SUCCESS*) break ;;
*META_ERROR*) echo " ⚠️ meta: bot reported${line#*META_ERROR}"; break ;;
"}") break ;;
esac
done < <(timeout "$RUN_TIMEOUT" docker logs -f "$C_NAME" 2>&1)
docker rm -f "$C_NAME" >/dev/null 2>&1
CURRENT_CONTAINER=""
# Promote only a syntactically valid JSON; on any failure keep the existing sidecar.
if [[ -f "$HOST_DATA_PATH/$TEMP" ]] \
&& python3 -c 'import json,sys; json.load(open(sys.argv[1]))' "$HOST_DATA_PATH/$TEMP" 2>/dev/null; then
if mv -f "$HOST_DATA_PATH/$TEMP" "$HOST_DATA_PATH/$FINAL"; then
echo " ✅ meta saved: $FINAL"; META_OK=$((META_OK+1))
else
echo " ⚠️ meta: rename $TEMP → $FINAL failed for $S"; META_FAIL=$((META_FAIL+1))
fi
else
echo " ⚠️ meta: no valid metadata produced for $S (existing $FINAL, if any, kept)"
rm -f "$HOST_DATA_PATH/$TEMP" 2>/dev/null
META_FAIL=$((META_FAIL+1))
fi
}
# discover_start SYMBOL MODE -> echoes "YEAR MONTH" of the first month with data, or
# nothing if the symbol has no data at all. Yearly probe uses December (catches data
# that started any time that year); monthly probe uses the FULL month (catches data
@@ -287,7 +359,7 @@ process_symbol() {
# --- EXECUTION (skipped when the script is sourced, e.g. for unit tests) ---
main() {
SAVED=0; EMPTY=0
SAVED=0; EMPTY=0; META_OK=0; META_FAIL=0; META_SKIP=0
# Clean up stale temp files at start (only *.tmp — never touches real month files).
[[ -d "$HOST_DATA_PATH" ]] && rm -f "$HOST_DATA_PATH"/*.tmp 2>/dev/null
@@ -307,6 +379,9 @@ while IFS= read -r line || [[ -n "$line" ]]; do
read -r S MODES <<< "$line"
[[ -z "$MODES" ]] && MODES="M1"
# Per-symbol contract metadata sidecar (mode-independent — produced once per symbol).
[[ "$EXPORT_META" == "1" ]] && write_meta "$S"
for MODE in $MODES; do
if [[ "$MODE" == "M1" || "$MODE" == "Tick" ]]; then
echo "------------------------------------------"
@@ -321,6 +396,7 @@ done < "$SYMBOL_FILE"
echo "------------------------------------------"
echo "🏁 Done: $(date) — $SYMBOLS_DONE jobs, $SAVED month(s) saved, $EMPTY empty/skipped."
[[ "$EXPORT_META" == "1" ]] && echo "🏷 Metadata: $META_OK written, $META_SKIP fresh-skipped, $META_FAIL failed."
}
# Run main only when executed directly; `source`-ing the script (for tests) defines the
scripts/test-export-runner.sh
+14
View File
@@ -71,6 +71,20 @@ i=$(midx 2026 5)
check "midx/idx_year round-trip" "2026" "$(idx_year "$i")"
check "midx/idx_month round-trip" "5" "$(idx_month "$i")"
# --- write_meta freshness guard (no Docker needed) -------------------------------------
# A recent sidecar must be skipped without spawning a container, so scheduled runs stay
# short. Point HOST_DATA_PATH at a temp dir, drop a just-created meta file, and assert
# write_meta classifies it as a fresh-skip. A docker stub fails loudly if it wrongly runs.
TMPDATA="$(mktemp -d)"; HOST_DATA_PATH="$TMPDATA"
: > "$TMPDATA/EURUSD.meta.json" # mtime = now -> inside any positive window
META_OK=0; META_FAIL=0; META_SKIP=0; META_MAX_AGE_DAYS=7
docker() { echo "UNEXPECTED docker call" >&2; return 1; } # must NOT be reached on a fresh file
write_meta EURUSD >/dev/null 2>&1
unset -f docker
check "write_meta: fresh sidecar -> skipped" "1" "$META_SKIP"
check "write_meta: fresh sidecar -> not produced" "0" "$META_OK"
rm -rf "$TMPDATA"
echo "-----"
echo "$PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]
src/MS_SimpleExport/MS_SimpleExport.cs
+67 -3
View File
@@ -2,6 +2,7 @@ using System;
using System.IO;
using System.IO.Compression;
using System.Collections.Generic;
using System.Globalization;
using System.Runtime.InteropServices;
using cAlgo.API;
@@ -35,16 +36,34 @@ namespace cAlgo.Robots
[Parameter("Is Probing", DefaultValue = false)]
public bool IsProbing { get; set; }
// Metadata sidecar: when MetaOnly is set, the bot does NOT export price data. It
// dumps the broker's authoritative per-symbol contract specs (pip size, lot size,
// currencies, costs, …) as a flat JSON file to MetaFile, then stops. This is the
// ONLY authoritative source for these values — the host-side driver cannot know
// them, it can only place the file the bot produces. See docs/symbol-metadata.md.
[Parameter("Metadata Only", DefaultValue = false)]
public bool MetaOnly { get; set; }
[Parameter("Metadata File", DefaultValue = "")]
public string MetaFile { get; set; }
private List<M1Record> _m1Buffer = new List<M1Record>();
private List<TickRecord> _tickBuffer = new List<TickRecord>();
protected override void OnStart() {
// Metadata run: emit the contract specs and stop — no price export.
if (MetaOnly) {
WriteMetadata();
Stop();
return;
}
// No price factor needed for double/float export
}
protected override void OnTick() {
if (MetaOnly) return;
if (Mode != ExportMode.Tick) return;
if (IsProbing) {
Print("DATA_FOUND");
Stop();
@@ -59,8 +78,9 @@ namespace cAlgo.Robots
}
protected override void OnBar() {
if (MetaOnly) return;
if (Mode != ExportMode.M1) return;
if (IsProbing) {
Print("DATA_FOUND");
Stop();
@@ -80,7 +100,7 @@ namespace cAlgo.Robots
}
protected override void OnStop() {
if (IsProbing || string.IsNullOrEmpty(OutputFile)) return;
if (MetaOnly || IsProbing || string.IsNullOrEmpty(OutputFile)) return;
try {
using var fileStream = new FileStream(OutputFile, FileMode.Create);
@@ -104,5 +124,49 @@ namespace cAlgo.Robots
Print("ERROR: {0}", ex.Message);
}
}
// Serialize the symbol's PRICE-INTERPRETATION GEOMETRY — the facts needed to read the
// historical price files: digit count, pip/tick size, contract size, and the currencies
// the price is quoted in. These are intrinsic to the price series and effectively
// time-stable, so they can be applied across the whole multi-year history.
//
// Deliberately EXCLUDED: costs, swaps, volume limits, trading status, and the
// deposit-currency pip/tick values. Those are a point-in-time snapshot of *today's*
// trading conditions — the cBot only ever sees the current contract, never the contract
// as it stood in a historical month — so stamping them onto a years-long series would
// misrepresent history (and the deposit-currency values further depend on the exporting
// account). See docs/symbol-metadata.md for the rationale.
//
// Keys mirror the cTrader property names; numbers use InvariantCulture (a pip size is
// "0.0001", never a locale "0,0001"); a non-finite value becomes JSON null.
private void WriteMetadata() {
try {
var s = Symbol;
var inv = CultureInfo.InvariantCulture;
string Num(double d) => (double.IsNaN(d) || double.IsInfinity(d)) ? "null" : d.ToString("R", inv);
string Str(string v) => v == null ? "null" : "\"" + v.Replace("\\", "\\\\").Replace("\"", "\\\"") + "\"";
var f = new List<string> {
// provenance of this sidecar
"\"schemaVersion\": 2",
"\"symbol\": " + Str(s.Name),
"\"generatedAtUtc\": " + Str(DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ", inv)),
"\"generator\": " + Str("MS_SimpleExport (cTrader.Automate)"),
// --- price-interpretation geometry (intrinsic, time-stable) ---
"\"description\": " + Str(s.Description),
"\"digits\": " + s.Digits.ToString(inv),
"\"pipSize\": " + Num(s.PipSize),
"\"tickSize\": " + Num(s.TickSize),
"\"lotSize\": " + Num(s.LotSize),
"\"baseAsset\": " + Str(s.BaseAsset?.Name),
"\"quoteAsset\": " + Str(s.QuoteAsset?.Name),
};
System.IO.File.WriteAllText(MetaFile, "{\n " + string.Join(",\n ", f) + "\n}\n");
Print("META_SUCCESS");
} catch (Exception ex) {
Print("META_ERROR: {0}", ex.Message);
}
}
}
}