From cd6874b43feb952158beb7e70eca05d4be2675fe Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 25 Jun 2026 13:12:54 +0200 Subject: [PATCH] Add per-symbol price-metadata sidecars (.meta.json) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The price files carry no facts for interpreting the bars (pip size, digit count, contract size, quote currency); those are known only to the broker via cTrader's Symbol object. Produce a JSON sidecar per symbol next to the price files so a downstream reader does not have to hardcode (and drift on) its own per-symbol table. Scope is the price-interpretation geometry only — digits, pipSize, tickSize, lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable across its whole history. Costs, swaps, volume limits, trading status and the deposit-currency pip/tick values are intentionally left out: the cBot only ever sees the current contract, so those are a point-in-time snapshot of today's trading conditions and would misrepresent a multi-year series (the deposit-currency values are account-dependent too). - cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror the cTrader property names. - runner: write_meta runs once per symbol (mode-independent), validates the JSON, and atomically promotes the sidecar. Best-effort — a metadata failure is logged and never aborts the export, and a pre-existing sidecar is kept on failure. A freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short; EXPORT_META=0 disables it. - The file is a raw serialization of what cTrader delivers; a reader derives its own shape (contract_size, per-lot pip value) from the raw fields. See docs/symbol-metadata.md. - Add a source-level regression test for the freshness guard. Verified end-to-end against the live broker for all 30 configured symbols (FX, indices, metals, crypto, commodities, equities); the price corpus was untouched. --- docs/symbol-metadata.md | 114 +++++++++++++++++++++++++ scripts/export-runner.sh | 78 ++++++++++++++++- scripts/test-export-runner.sh | 14 +++ src/MS_SimpleExport/MS_SimpleExport.cs | 70 ++++++++++++++- 4 files changed, 272 insertions(+), 4 deletions(-) create mode 100644 docs/symbol-metadata.md diff --git a/docs/symbol-metadata.md b/docs/symbol-metadata.md new file mode 100644 index 0000000..1d24d3d --- /dev/null +++ b/docs/symbol-metadata.md @@ -0,0 +1,114 @@ +# Per-symbol price metadata (`.meta.json`) + +Alongside the price files (`_YYYY_MM.m1` / `.tick`) the export driver writes one +small JSON sidecar per symbol, `.meta.json`. Its sole purpose is to let a reader +**interpret the historical price data** — the price files contain only raw OHLC / spread / +volume numbers, with nothing that says how to read them (how many decimals a price has, what +a "pip" is, which currency the price is quoted in, what one lot is worth). Those facts are +known only to the broker, exposed through cTrader's `Symbol` object inside a running cBot. +This sidecar captures them so a downstream reader does not have to hardcode (and drift on) a +per-symbol table. The per-instrument differences are real and easy to get wrong by hand: a +JPY pair's pip is `0.01` (not `0.0001`); gold's lot size is `100` (not the FX `100000`). + +## Scope: price-interpretation geometry only + +The file carries **only the geometry needed to interpret the price series** — facts that are +intrinsic to the data and effectively constant over its whole multi-year span: + +| Field | cTrader source | Meaning | +|---|---|---| +| `digits` | `Symbol.Digits` | price decimal places | +| `pipSize` | `Symbol.PipSize` | price units per pip (EURUSD `0.0001`, USDJPY `0.01`, XAUUSD `0.1`) | +| `tickSize` | `Symbol.TickSize` | smallest price increment | +| `lotSize` | `Symbol.LotSize` | base-currency units per `1.0` lot — turns a price move into money (FX `100000`, XAUUSD `100`) | +| `baseAsset` | `Symbol.BaseAsset.Name` | base asset / currency ISO label | +| `quoteAsset` | `Symbol.QuoteAsset.Name` | quote asset / currency the price is denominated in | +| `description` | `Symbol.Description` | human-readable name | + +Plus provenance: `schemaVersion`, `symbol`, `generatedAtUtc`, `generator`. + +## What is deliberately NOT here — and why + +cTrader's `Symbol` also exposes costs (`Commission*`, `Swap*`), volume limits +(`VolumeInUnits*`), trading status (`TradingMode`, `IsTradingEnabled`) and deposit-currency +pip/tick values (`PipValue`, `TickValue`). **These are intentionally excluded**, because they +do not describe the historical price series — they are a *point-in-time snapshot of today's +trading conditions*: + +- **The cBot only ever sees the current contract.** A backtest runs against the symbol as it + is defined *now*, never as it stood in a historical month. So the bot physically cannot + emit a historically-accurate swap rate or commission for 2015 — only today's. Swap rates in + particular change weekly; stamping a single 2026 value onto a 15-year series would + misrepresent it. +- **`PipValue` / `TickValue` are additionally account-dependent.** cTrader reports them already + converted into the *exporting account's deposit currency* at the export-time FX rate, so they + are a property of (account × moment), not of the instrument. The stable, account-free pip + value is pure arithmetic: `lotSize × pipSize`, in the quote currency. + +If a consumer needs current trading conditions (for a friction model, position sizing, etc.), +those are a separate, time-varying concern and belong in their own time-indexed source — not +baked into a per-symbol price-geometry file as if they held across history. + +## Mechanism + +- **`MS_SimpleExport` cBot, `MetaOnly` mode** (`src/MS_SimpleExport/MS_SimpleExport.cs`). + Run with `--MetaOnly=true --MetaFile=`, the bot does no price export: in `OnStart` it + reads `Symbol`, serializes the geometry fields to JSON at `MetaFile`, prints `META_SUCCESS`, + and stops. +- **`export-runner.sh`, `write_meta`**. Once per symbol per run (mode-independent), the driver + spawns a short `MetaOnly` backtest, writes `.meta.json.tmp`, validates it is parseable + JSON, then renames it to `.meta.json`. + - **Best-effort by design**: a metadata failure is logged (`⚠️`) and *never* aborts the + export run; a pre-existing sidecar is left untouched (overwrite only on a validated success). + - **Cadence**: refreshed at most once per `META_MAX_AGE_DAYS` (default `7`) so scheduled runs + stay short. `EXPORT_META=0` disables; `META_MAX_AGE_DAYS=0` forces a refresh every run. + +## Format + +A flat JSON object. Conventions: + +- Numbers use invariant formatting — a pip size is `0.0001`, never a locale `0,0001`. Round-trip + formatting may emit scientific notation (`1E-05`); valid JSON, parses as a number. +- A non-finite broker value (NaN / Infinity) is emitted as JSON `null`. +- `schemaVersion` is bumped on any incompatible change to field names or units (currently `2`). + +### Example + +```json +{ + "schemaVersion": 2, + "symbol": "EURUSD", + "generatedAtUtc": "2026-06-25T11:06:01Z", + "generator": "MS_SimpleExport (cTrader.Automate)", + "description": "Euro vs US Dollar", + "digits": 5, + "pipSize": 0.0001, + "tickSize": 1E-05, + "lotSize": 100000, + "baseAsset": "EUR", + "quoteAsset": "USD" +} +``` + +## Consumer-side derivations + +A reader composes whatever shape it wants from these raw fields, e.g.: + +``` +contract_size = lotSize +pip_value_per_lot = lotSize * pipSize # in quoteAsset (EURUSD: 100000*0.0001 = 10 USD) +point_size = 10 ** -digits +``` + +Trading constraints a consumer's own instrument spec might want — minimum lot, lot step, a +stable instrument id — are **not** price-interpretation geometry and are not in this file; the +consumer authors those itself (an id is a registry choice; min-lot / lot-step are current order +limits, not facts about the historical bars). + +Worked geometry, verified against the live broker: + +| symbol | digits | pipSize | tickSize | lotSize | base/quote | → pip_value_per_lot | +|---|---|---|---|---|---|---| +| EURUSD | 5 | 0.0001 | 1E-05 | 100000 | EUR/USD | 10.0 USD | +| USDJPY | 3 | 0.01 | 0.001 | 100000 | USD/JPY | 1000 JPY | +| XAUUSD | 2 | 0.1 | 0.01 | 100 | XAU/USD | 10.0 USD | diff --git a/scripts/export-runner.sh b/scripts/export-runner.sh index dc14e47..7620405 100755 --- a/scripts/export-runner.sh +++ b/scripts/export-runner.sh @@ -35,6 +35,8 @@ SYMBOL_FILE="${SYMBOL_FILE:-$REPO_ROOT/runtime/ScrapeSymbols.txt}" # overridabl RETRIES="${RETRIES:-3}" # attempts before a transient failure aborts RUN_TIMEOUT="${RUN_TIMEOUT:-600}" # seconds per backtest before it counts as a hang DISCOVER_MIN_YEAR="${DISCOVER_MIN_YEAR:-2005}" # earliest year the first-data probe scans +EXPORT_META="${EXPORT_META:-1}" # 1 = also produce .meta.json sidecars +META_MAX_AGE_DAYS="${META_MAX_AGE_DAYS:-7}" # refresh a sidecar at most this often (0 = every run) # --- DEPLOYMENT CONFIG (secrets/IDs/paths) --- ENV_FILE="${ENV_FILE:-$REPO_ROOT/runtime/export.env}" @@ -207,6 +209,76 @@ probe_run() { esac } +# --------------------------------------------------------------------------- +# write_meta SYMBOL — produce the per-symbol contract-metadata sidecar +# (.meta.json), placed next to the price files. The cBot's MetaOnly mode +# is the ONLY authoritative source for these values: it reads the broker's Symbol +# object and serializes it verbatim; this driver merely places the file it writes. +# +# BEST-EFFORT BY DESIGN: a metadata failure is logged (⚠️) and NEVER aborts the +# export run — the sidecar is auxiliary, and any pre-existing one is left untouched +# (we only ever overwrite on a validated success). Refreshed at most once per +# META_MAX_AGE_DAYS so scheduled runs stay short. +# --------------------------------------------------------------------------- +write_meta() { + local S=$1 + local FINAL="${S}.meta.json" + local TEMP="${S}.meta.json.tmp" + + # Freshness guard: skip if a recent sidecar already exists (mtime within the window). + if [[ "$META_MAX_AGE_DAYS" -gt 0 && -f "$HOST_DATA_PATH/$FINAL" \ + && -n "$(find "$HOST_DATA_PATH/$FINAL" -mtime "-$META_MAX_AGE_DAYS" 2>/dev/null)" ]]; then + echo "🏷 Metadata: $S — fresh (<${META_MAX_AGE_DAYS}d), skipping" + META_SKIP=$((META_SKIP+1)); return 0 + fi + + local C_NAME="ctrader-meta-${S//[^A-Za-z0-9]/-}-$(date +%s%N)" + local s_date e_date + s_date="$(date -d '7 days ago' +%d/%m/%Y)" # tiny window; MetaOnly stops in OnStart anyway + e_date="$(date -d 'yesterday' +%d/%m/%Y)" + + echo "🏷 Metadata: $S (MetaOnly)" + CURRENT_CONTAINER="$C_NAME" + docker run -d --name "$C_NAME" --init \ + -v "$HOST_ROBOTS_PATH":"/mnt/Robots" \ + -v "$HOST_DATA_PATH":"$CONTAINER_DATA_PATH" \ + -e TZ="Europe/Berlin" \ + "$IMAGE" \ + backtest "/mnt/Robots/$BOT_FILE" \ + --start="$s_date" --end="$e_date" --symbol="$S" \ + --ctid="$CTID" --pwd-file="$PWD_FILE" --account="$ACCOUNT" \ + --period="M1" --data-mode="m1" \ + --Mode="M1" --IsProbing="false" --MetaOnly="true" \ + --MetaFile="${CONTAINER_DATA_PATH}/${TEMP}" --full-access > /dev/null 2>&1 || { + echo " ⚠️ meta: docker run failed to start container for $S" + CURRENT_CONTAINER=""; META_FAIL=$((META_FAIL+1)); return 0; } + + while IFS= read -r line; do + case "$line" in + *META_SUCCESS*) break ;; + *META_ERROR*) echo " ⚠️ meta: bot reported${line#*META_ERROR}"; break ;; + "}") break ;; + esac + done < <(timeout "$RUN_TIMEOUT" docker logs -f "$C_NAME" 2>&1) + + docker rm -f "$C_NAME" >/dev/null 2>&1 + CURRENT_CONTAINER="" + + # Promote only a syntactically valid JSON; on any failure keep the existing sidecar. + if [[ -f "$HOST_DATA_PATH/$TEMP" ]] \ + && python3 -c 'import json,sys; json.load(open(sys.argv[1]))' "$HOST_DATA_PATH/$TEMP" 2>/dev/null; then + if mv -f "$HOST_DATA_PATH/$TEMP" "$HOST_DATA_PATH/$FINAL"; then + echo " ✅ meta saved: $FINAL"; META_OK=$((META_OK+1)) + else + echo " ⚠️ meta: rename $TEMP → $FINAL failed for $S"; META_FAIL=$((META_FAIL+1)) + fi + else + echo " ⚠️ meta: no valid metadata produced for $S (existing $FINAL, if any, kept)" + rm -f "$HOST_DATA_PATH/$TEMP" 2>/dev/null + META_FAIL=$((META_FAIL+1)) + fi +} + # discover_start SYMBOL MODE -> echoes "YEAR MONTH" of the first month with data, or # nothing if the symbol has no data at all. Yearly probe uses December (catches data # that started any time that year); monthly probe uses the FULL month (catches data @@ -287,7 +359,7 @@ process_symbol() { # --- EXECUTION (skipped when the script is sourced, e.g. for unit tests) --- main() { -SAVED=0; EMPTY=0 +SAVED=0; EMPTY=0; META_OK=0; META_FAIL=0; META_SKIP=0 # Clean up stale temp files at start (only *.tmp — never touches real month files). [[ -d "$HOST_DATA_PATH" ]] && rm -f "$HOST_DATA_PATH"/*.tmp 2>/dev/null @@ -307,6 +379,9 @@ while IFS= read -r line || [[ -n "$line" ]]; do read -r S MODES <<< "$line" [[ -z "$MODES" ]] && MODES="M1" + # Per-symbol contract metadata sidecar (mode-independent — produced once per symbol). + [[ "$EXPORT_META" == "1" ]] && write_meta "$S" + for MODE in $MODES; do if [[ "$MODE" == "M1" || "$MODE" == "Tick" ]]; then echo "------------------------------------------" @@ -321,6 +396,7 @@ done < "$SYMBOL_FILE" echo "------------------------------------------" echo "🏁 Done: $(date) — $SYMBOLS_DONE jobs, $SAVED month(s) saved, $EMPTY empty/skipped." +[[ "$EXPORT_META" == "1" ]] && echo "🏷 Metadata: $META_OK written, $META_SKIP fresh-skipped, $META_FAIL failed." } # Run main only when executed directly; `source`-ing the script (for tests) defines the diff --git a/scripts/test-export-runner.sh b/scripts/test-export-runner.sh index 2c5ea99..a62a7d9 100755 --- a/scripts/test-export-runner.sh +++ b/scripts/test-export-runner.sh @@ -71,6 +71,20 @@ i=$(midx 2026 5) check "midx/idx_year round-trip" "2026" "$(idx_year "$i")" check "midx/idx_month round-trip" "5" "$(idx_month "$i")" +# --- write_meta freshness guard (no Docker needed) ------------------------------------- +# A recent sidecar must be skipped without spawning a container, so scheduled runs stay +# short. Point HOST_DATA_PATH at a temp dir, drop a just-created meta file, and assert +# write_meta classifies it as a fresh-skip. A docker stub fails loudly if it wrongly runs. +TMPDATA="$(mktemp -d)"; HOST_DATA_PATH="$TMPDATA" +: > "$TMPDATA/EURUSD.meta.json" # mtime = now -> inside any positive window +META_OK=0; META_FAIL=0; META_SKIP=0; META_MAX_AGE_DAYS=7 +docker() { echo "UNEXPECTED docker call" >&2; return 1; } # must NOT be reached on a fresh file +write_meta EURUSD >/dev/null 2>&1 +unset -f docker +check "write_meta: fresh sidecar -> skipped" "1" "$META_SKIP" +check "write_meta: fresh sidecar -> not produced" "0" "$META_OK" +rm -rf "$TMPDATA" + echo "-----" echo "$PASS passed, $FAIL failed" [[ $FAIL -eq 0 ]] diff --git a/src/MS_SimpleExport/MS_SimpleExport.cs b/src/MS_SimpleExport/MS_SimpleExport.cs index 675d067..206f9a8 100755 --- a/src/MS_SimpleExport/MS_SimpleExport.cs +++ b/src/MS_SimpleExport/MS_SimpleExport.cs @@ -2,6 +2,7 @@ using System; using System.IO; using System.IO.Compression; using System.Collections.Generic; +using System.Globalization; using System.Runtime.InteropServices; using cAlgo.API; @@ -35,16 +36,34 @@ namespace cAlgo.Robots [Parameter("Is Probing", DefaultValue = false)] public bool IsProbing { get; set; } + // Metadata sidecar: when MetaOnly is set, the bot does NOT export price data. It + // dumps the broker's authoritative per-symbol contract specs (pip size, lot size, + // currencies, costs, …) as a flat JSON file to MetaFile, then stops. This is the + // ONLY authoritative source for these values — the host-side driver cannot know + // them, it can only place the file the bot produces. See docs/symbol-metadata.md. + [Parameter("Metadata Only", DefaultValue = false)] + public bool MetaOnly { get; set; } + + [Parameter("Metadata File", DefaultValue = "")] + public string MetaFile { get; set; } + private List _m1Buffer = new List(); private List _tickBuffer = new List(); protected override void OnStart() { + // Metadata run: emit the contract specs and stop — no price export. + if (MetaOnly) { + WriteMetadata(); + Stop(); + return; + } // No price factor needed for double/float export } protected override void OnTick() { + if (MetaOnly) return; if (Mode != ExportMode.Tick) return; - + if (IsProbing) { Print("DATA_FOUND"); Stop(); @@ -59,8 +78,9 @@ namespace cAlgo.Robots } protected override void OnBar() { + if (MetaOnly) return; if (Mode != ExportMode.M1) return; - + if (IsProbing) { Print("DATA_FOUND"); Stop(); @@ -80,7 +100,7 @@ namespace cAlgo.Robots } protected override void OnStop() { - if (IsProbing || string.IsNullOrEmpty(OutputFile)) return; + if (MetaOnly || IsProbing || string.IsNullOrEmpty(OutputFile)) return; try { using var fileStream = new FileStream(OutputFile, FileMode.Create); @@ -104,5 +124,49 @@ namespace cAlgo.Robots Print("ERROR: {0}", ex.Message); } } + + // Serialize the symbol's PRICE-INTERPRETATION GEOMETRY — the facts needed to read the + // historical price files: digit count, pip/tick size, contract size, and the currencies + // the price is quoted in. These are intrinsic to the price series and effectively + // time-stable, so they can be applied across the whole multi-year history. + // + // Deliberately EXCLUDED: costs, swaps, volume limits, trading status, and the + // deposit-currency pip/tick values. Those are a point-in-time snapshot of *today's* + // trading conditions — the cBot only ever sees the current contract, never the contract + // as it stood in a historical month — so stamping them onto a years-long series would + // misrepresent history (and the deposit-currency values further depend on the exporting + // account). See docs/symbol-metadata.md for the rationale. + // + // Keys mirror the cTrader property names; numbers use InvariantCulture (a pip size is + // "0.0001", never a locale "0,0001"); a non-finite value becomes JSON null. + private void WriteMetadata() { + try { + var s = Symbol; + var inv = CultureInfo.InvariantCulture; + string Num(double d) => (double.IsNaN(d) || double.IsInfinity(d)) ? "null" : d.ToString("R", inv); + string Str(string v) => v == null ? "null" : "\"" + v.Replace("\\", "\\\\").Replace("\"", "\\\"") + "\""; + + var f = new List { + // provenance of this sidecar + "\"schemaVersion\": 2", + "\"symbol\": " + Str(s.Name), + "\"generatedAtUtc\": " + Str(DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ", inv)), + "\"generator\": " + Str("MS_SimpleExport (cTrader.Automate)"), + // --- price-interpretation geometry (intrinsic, time-stable) --- + "\"description\": " + Str(s.Description), + "\"digits\": " + s.Digits.ToString(inv), + "\"pipSize\": " + Num(s.PipSize), + "\"tickSize\": " + Num(s.TickSize), + "\"lotSize\": " + Num(s.LotSize), + "\"baseAsset\": " + Str(s.BaseAsset?.Name), + "\"quoteAsset\": " + Str(s.QuoteAsset?.Name), + }; + + System.IO.File.WriteAllText(MetaFile, "{\n " + string.Join(",\n ", f) + "\n}\n"); + Print("META_SUCCESS"); + } catch (Exception ex) { + Print("META_ERROR: {0}", ex.Message); + } + } } } \ No newline at end of file