Add per-symbol price-metadata sidecars (<symbol>.meta.json)

The price files carry no facts for interpreting the bars (pip size, digit count,
contract size, quote currency); those are known only to the broker via cTrader's
Symbol object. Produce a JSON sidecar per symbol next to the price files so a
downstream reader does not have to hardcode (and drift on) its own per-symbol table.

Scope is the price-interpretation geometry only — digits, pipSize, tickSize,
lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable
across its whole history. Costs, swaps, volume limits, trading status and the
deposit-currency pip/tick values are intentionally left out: the cBot only ever
sees the current contract, so those are a point-in-time snapshot of today's trading
conditions and would misrepresent a multi-year series (the deposit-currency values
are account-dependent too).

- cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat
  JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror
  the cTrader property names.
- runner: write_meta runs once per symbol (mode-independent), validates the JSON,
  and atomically promotes the sidecar. Best-effort — a metadata failure is logged
  and never aborts the export, and a pre-existing sidecar is kept on failure. A
  freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short;
  EXPORT_META=0 disables it.
- The file is a raw serialization of what cTrader delivers; a reader derives its
  own shape (contract_size, per-lot pip value) from the raw fields. See
  docs/symbol-metadata.md.
- Add a source-level regression test for the freshness guard.

Verified end-to-end against the live broker for all 30 configured symbols (FX,
indices, metals, crypto, commodities, equities); the price corpus was untouched.

docs/symbol-metadata.md

@@ -0,0 +1,114 @@
+# Per-symbol price metadata (`<SYMBOL>.meta.json`)
+
+Alongside the price files (`<SYMBOL>_YYYY_MM.m1` / `.tick`) the export driver writes one
+small JSON sidecar per symbol, `<SYMBOL>.meta.json`. Its sole purpose is to let a reader
+**interpret the historical price data** — the price files contain only raw OHLC / spread /
+volume numbers, with nothing that says how to read them (how many decimals a price has, what
+a "pip" is, which currency the price is quoted in, what one lot is worth). Those facts are
+known only to the broker, exposed through cTrader's `Symbol` object inside a running cBot.
+This sidecar captures them so a downstream reader does not have to hardcode (and drift on) a
+per-symbol table. The per-instrument differences are real and easy to get wrong by hand: a
+JPY pair's pip is `0.01` (not `0.0001`); gold's lot size is `100` (not the FX `100000`).
+
+## Scope: price-interpretation geometry only
+
+The file carries **only the geometry needed to interpret the price series** — facts that are
+intrinsic to the data and effectively constant over its whole multi-year span:
+
+| Field | cTrader source | Meaning |
+|---|---|---|
+| `digits` | `Symbol.Digits` | price decimal places |
+| `pipSize` | `Symbol.PipSize` | price units per pip (EURUSD `0.0001`, USDJPY `0.01`, XAUUSD `0.1`) |
+| `tickSize` | `Symbol.TickSize` | smallest price increment |
+| `lotSize` | `Symbol.LotSize` | base-currency units per `1.0` lot — turns a price move into money (FX `100000`, XAUUSD `100`) |
+| `baseAsset` | `Symbol.BaseAsset.Name` | base asset / currency ISO label |
+| `quoteAsset` | `Symbol.QuoteAsset.Name` | quote asset / currency the price is denominated in |
+| `description` | `Symbol.Description` | human-readable name |
+
+Plus provenance: `schemaVersion`, `symbol`, `generatedAtUtc`, `generator`.
+
+## What is deliberately NOT here — and why
+
+cTrader's `Symbol` also exposes costs (`Commission*`, `Swap*`), volume limits
+(`VolumeInUnits*`), trading status (`TradingMode`, `IsTradingEnabled`) and deposit-currency
+pip/tick values (`PipValue`, `TickValue`). **These are intentionally excluded**, because they
+do not describe the historical price series — they are a *point-in-time snapshot of today's
+trading conditions*:
+
+- **The cBot only ever sees the current contract.** A backtest runs against the symbol as it
+  is defined *now*, never as it stood in a historical month. So the bot physically cannot
+  emit a historically-accurate swap rate or commission for 2015 — only today's. Swap rates in
+  particular change weekly; stamping a single 2026 value onto a 15-year series would
+  misrepresent it.
+- **`PipValue` / `TickValue` are additionally account-dependent.** cTrader reports them already
+  converted into the *exporting account's deposit currency* at the export-time FX rate, so they
+  are a property of (account × moment), not of the instrument. The stable, account-free pip
+  value is pure arithmetic: `lotSize × pipSize`, in the quote currency.
+
+If a consumer needs current trading conditions (for a friction model, position sizing, etc.),
+those are a separate, time-varying concern and belong in their own time-indexed source — not
+baked into a per-symbol price-geometry file as if they held across history.
+
+## Mechanism
+
+- **`MS_SimpleExport` cBot, `MetaOnly` mode** (`src/MS_SimpleExport/MS_SimpleExport.cs`).
+  Run with `--MetaOnly=true --MetaFile=<path>`, the bot does no price export: in `OnStart` it
+  reads `Symbol`, serializes the geometry fields to JSON at `MetaFile`, prints `META_SUCCESS`,
+  and stops.
+- **`export-runner.sh`, `write_meta`**. Once per symbol per run (mode-independent), the driver
+  spawns a short `MetaOnly` backtest, writes `<SYMBOL>.meta.json.tmp`, validates it is parseable
+  JSON, then renames it to `<SYMBOL>.meta.json`.
+  - **Best-effort by design**: a metadata failure is logged (`⚠️`) and *never* aborts the
+    export run; a pre-existing sidecar is left untouched (overwrite only on a validated success).
+  - **Cadence**: refreshed at most once per `META_MAX_AGE_DAYS` (default `7`) so scheduled runs
+    stay short. `EXPORT_META=0` disables; `META_MAX_AGE_DAYS=0` forces a refresh every run.
+
+## Format
+
+A flat JSON object. Conventions:
+
+- Numbers use invariant formatting — a pip size is `0.0001`, never a locale `0,0001`. Round-trip
+  formatting may emit scientific notation (`1E-05`); valid JSON, parses as a number.
+- A non-finite broker value (NaN / Infinity) is emitted as JSON `null`.
+- `schemaVersion` is bumped on any incompatible change to field names or units (currently `2`).
+
+### Example
+
+```json
+{
+  "schemaVersion": 2,
+  "symbol": "EURUSD",
+  "generatedAtUtc": "2026-06-25T11:06:01Z",
+  "generator": "MS_SimpleExport (cTrader.Automate)",
+  "description": "Euro vs US Dollar",
+  "digits": 5,
+  "pipSize": 0.0001,
+  "tickSize": 1E-05,
+  "lotSize": 100000,
+  "baseAsset": "EUR",
+  "quoteAsset": "USD"
+}
+```
+
+## Consumer-side derivations
+
+A reader composes whatever shape it wants from these raw fields, e.g.:
+
+```
+contract_size      = lotSize
+pip_value_per_lot  = lotSize * pipSize           # in quoteAsset (EURUSD: 100000*0.0001 = 10 USD)
+point_size         = 10 ** -digits
+```
+
+Trading constraints a consumer's own instrument spec might want — minimum lot, lot step, a
+stable instrument id — are **not** price-interpretation geometry and are not in this file; the
+consumer authors those itself (an id is a registry choice; min-lot / lot-step are current order
+limits, not facts about the historical bars).
+
+Worked geometry, verified against the live broker:
+
+| symbol | digits | pipSize | tickSize | lotSize | base/quote | → pip_value_per_lot |
+|---|---|---|---|---|---|---|
+| EURUSD | 5 | 0.0001 | 1E-05 | 100000 | EUR/USD | 10.0 USD |
+| USDJPY | 3 | 0.01 | 0.001 | 100000 | USD/JPY | 1000 JPY |
+| XAUUSD | 2 | 0.1 | 0.01 | 100 | XAU/USD | 10.0 USD |

scripts/export-runner.sh

@@ -35,6 +35,8 @@ SYMBOL_FILE="${SYMBOL_FILE:-$REPO_ROOT/runtime/ScrapeSymbols.txt}"  # overridabl
 RETRIES="${RETRIES:-3}"                       # attempts before a transient failure aborts
 RUN_TIMEOUT="${RUN_TIMEOUT:-600}"             # seconds per backtest before it counts as a hang
 DISCOVER_MIN_YEAR="${DISCOVER_MIN_YEAR:-2005}" # earliest year the first-data probe scans
+EXPORT_META="${EXPORT_META:-1}"               # 1 = also produce <symbol>.meta.json sidecars
+META_MAX_AGE_DAYS="${META_MAX_AGE_DAYS:-7}"   # refresh a sidecar at most this often (0 = every run)
 
 # --- DEPLOYMENT CONFIG (secrets/IDs/paths) ---
 ENV_FILE="${ENV_FILE:-$REPO_ROOT/runtime/export.env}"
@@ -207,6 +209,76 @@ probe_run() {
     esac
 }
 
+# ---------------------------------------------------------------------------
+# write_meta SYMBOL — produce the per-symbol contract-metadata sidecar
+# (<SYMBOL>.meta.json), placed next to the price files. The cBot's MetaOnly mode
+# is the ONLY authoritative source for these values: it reads the broker's Symbol
+# object and serializes it verbatim; this driver merely places the file it writes.
+#
+# BEST-EFFORT BY DESIGN: a metadata failure is logged (⚠️) and NEVER aborts the
+# export run — the sidecar is auxiliary, and any pre-existing one is left untouched
+# (we only ever overwrite on a validated success). Refreshed at most once per
+# META_MAX_AGE_DAYS so scheduled runs stay short.
+# ---------------------------------------------------------------------------
+write_meta() {
+    local S=$1
+    local FINAL="${S}.meta.json"
+    local TEMP="${S}.meta.json.tmp"
+
+    # Freshness guard: skip if a recent sidecar already exists (mtime within the window).
+    if [[ "$META_MAX_AGE_DAYS" -gt 0 && -f "$HOST_DATA_PATH/$FINAL" \
+          && -n "$(find "$HOST_DATA_PATH/$FINAL" -mtime "-$META_MAX_AGE_DAYS" 2>/dev/null)" ]]; then
+        echo "🏷  Metadata: $S — fresh (<${META_MAX_AGE_DAYS}d), skipping"
+        META_SKIP=$((META_SKIP+1)); return 0
+    fi
+
+    local C_NAME="ctrader-meta-${S//[^A-Za-z0-9]/-}-$(date +%s%N)"
+    local s_date e_date
+    s_date="$(date -d '7 days ago' +%d/%m/%Y)"     # tiny window; MetaOnly stops in OnStart anyway
+    e_date="$(date -d 'yesterday'  +%d/%m/%Y)"
+
+    echo "🏷  Metadata: $S (MetaOnly)"
+    CURRENT_CONTAINER="$C_NAME"
+    docker run -d --name "$C_NAME" --init \
+        -v "$HOST_ROBOTS_PATH":"/mnt/Robots" \
+        -v "$HOST_DATA_PATH":"$CONTAINER_DATA_PATH" \
+        -e TZ="Europe/Berlin" \
+        "$IMAGE" \
+        backtest "/mnt/Robots/$BOT_FILE" \
+        --start="$s_date" --end="$e_date" --symbol="$S" \
+        --ctid="$CTID" --pwd-file="$PWD_FILE" --account="$ACCOUNT" \
+        --period="M1" --data-mode="m1" \
+        --Mode="M1" --IsProbing="false" --MetaOnly="true" \
+        --MetaFile="${CONTAINER_DATA_PATH}/${TEMP}" --full-access > /dev/null 2>&1 || {
+            echo "  ⚠️ meta: docker run failed to start container for $S"
+            CURRENT_CONTAINER=""; META_FAIL=$((META_FAIL+1)); return 0; }
+
+    while IFS= read -r line; do
+        case "$line" in
+            *META_SUCCESS*) break ;;
+            *META_ERROR*)   echo "  ⚠️ meta: bot reported${line#*META_ERROR}"; break ;;
+            "}")            break ;;
+        esac
+    done < <(timeout "$RUN_TIMEOUT" docker logs -f "$C_NAME" 2>&1)
+
+    docker rm -f "$C_NAME" >/dev/null 2>&1
+    CURRENT_CONTAINER=""
+
+    # Promote only a syntactically valid JSON; on any failure keep the existing sidecar.
+    if [[ -f "$HOST_DATA_PATH/$TEMP" ]] \
+       && python3 -c 'import json,sys; json.load(open(sys.argv[1]))' "$HOST_DATA_PATH/$TEMP" 2>/dev/null; then
+        if mv -f "$HOST_DATA_PATH/$TEMP" "$HOST_DATA_PATH/$FINAL"; then
+            echo "  ✅ meta saved: $FINAL"; META_OK=$((META_OK+1))
+        else
+            echo "  ⚠️ meta: rename $TEMP → $FINAL failed for $S"; META_FAIL=$((META_FAIL+1))
+        fi
+    else
+        echo "  ⚠️ meta: no valid metadata produced for $S (existing $FINAL, if any, kept)"
+        rm -f "$HOST_DATA_PATH/$TEMP" 2>/dev/null
+        META_FAIL=$((META_FAIL+1))
+    fi
+}
+
 # discover_start SYMBOL MODE -> echoes "YEAR MONTH" of the first month with data, or
 # nothing if the symbol has no data at all. Yearly probe uses December (catches data
 # that started any time that year); monthly probe uses the FULL month (catches data
@@ -287,7 +359,7 @@ process_symbol() {
 
 # --- EXECUTION (skipped when the script is sourced, e.g. for unit tests) ---
 main() {
-SAVED=0; EMPTY=0
+SAVED=0; EMPTY=0; META_OK=0; META_FAIL=0; META_SKIP=0
 
 # Clean up stale temp files at start (only *.tmp — never touches real month files).
 [[ -d "$HOST_DATA_PATH" ]] && rm -f "$HOST_DATA_PATH"/*.tmp 2>/dev/null
@@ -307,6 +379,9 @@ while IFS= read -r line || [[ -n "$line" ]]; do
     read -r S MODES <<< "$line"
     [[ -z "$MODES" ]] && MODES="M1"
 
+    # Per-symbol contract metadata sidecar (mode-independent — produced once per symbol).
+    [[ "$EXPORT_META" == "1" ]] && write_meta "$S"
+
     for MODE in $MODES; do
         if [[ "$MODE" == "M1" || "$MODE" == "Tick" ]]; then
             echo "------------------------------------------"
@@ -321,6 +396,7 @@ done < "$SYMBOL_FILE"
 
 echo "------------------------------------------"
 echo "🏁 Done: $(date) — $SYMBOLS_DONE jobs, $SAVED month(s) saved, $EMPTY empty/skipped."
+[[ "$EXPORT_META" == "1" ]] && echo "🏷  Metadata: $META_OK written, $META_SKIP fresh-skipped, $META_FAIL failed."
 }
 
 # Run main only when executed directly; `source`-ing the script (for tests) defines the

scripts/test-export-runner.sh

@@ -71,6 +71,20 @@ i=$(midx 2026 5)
 check "midx/idx_year round-trip"  "2026" "$(idx_year "$i")"
 check "midx/idx_month round-trip" "5"    "$(idx_month "$i")"
 
+# --- write_meta freshness guard (no Docker needed) -------------------------------------
+# A recent sidecar must be skipped without spawning a container, so scheduled runs stay
+# short. Point HOST_DATA_PATH at a temp dir, drop a just-created meta file, and assert
+# write_meta classifies it as a fresh-skip. A docker stub fails loudly if it wrongly runs.
+TMPDATA="$(mktemp -d)"; HOST_DATA_PATH="$TMPDATA"
+: > "$TMPDATA/EURUSD.meta.json"                    # mtime = now -> inside any positive window
+META_OK=0; META_FAIL=0; META_SKIP=0; META_MAX_AGE_DAYS=7
+docker() { echo "UNEXPECTED docker call" >&2; return 1; }   # must NOT be reached on a fresh file
+write_meta EURUSD >/dev/null 2>&1
+unset -f docker
+check "write_meta: fresh sidecar -> skipped"      "1" "$META_SKIP"
+check "write_meta: fresh sidecar -> not produced" "0" "$META_OK"
+rm -rf "$TMPDATA"
+
 echo "-----"
 echo "$PASS passed, $FAIL failed"
 [[ $FAIL -eq 0 ]]

src/MS_SimpleExport/MS_SimpleExport.cs

@@ -2,6 +2,7 @@ using System;
 using System.IO;
 using System.IO.Compression;
 using System.Collections.Generic;
+using System.Globalization;
 using System.Runtime.InteropServices;
 using cAlgo.API;
 
@@ -35,16 +36,34 @@ namespace cAlgo.Robots
         [Parameter("Is Probing", DefaultValue = false)]
         public bool IsProbing { get; set; }
 
+        // Metadata sidecar: when MetaOnly is set, the bot does NOT export price data. It
+        // dumps the broker's authoritative per-symbol contract specs (pip size, lot size,
+        // currencies, costs, …) as a flat JSON file to MetaFile, then stops. This is the
+        // ONLY authoritative source for these values — the host-side driver cannot know
+        // them, it can only place the file the bot produces. See docs/symbol-metadata.md.
+        [Parameter("Metadata Only", DefaultValue = false)]
+        public bool MetaOnly { get; set; }
+
+        [Parameter("Metadata File", DefaultValue = "")]
+        public string MetaFile { get; set; }
+
         private List<M1Record> _m1Buffer = new List<M1Record>();
         private List<TickRecord> _tickBuffer = new List<TickRecord>();
 
         protected override void OnStart() {
+            // Metadata run: emit the contract specs and stop — no price export.
+            if (MetaOnly) {
+                WriteMetadata();
+                Stop();
+                return;
+            }
             // No price factor needed for double/float export
         }
 
         protected override void OnTick() {
+            if (MetaOnly) return;
             if (Mode != ExportMode.Tick) return;
-            
+
             if (IsProbing) {
                 Print("DATA_FOUND");
                 Stop();
@@ -59,8 +78,9 @@ namespace cAlgo.Robots
         }
 
         protected override void OnBar() {
+            if (MetaOnly) return;
             if (Mode != ExportMode.M1) return;
-            
+
             if (IsProbing) {
                 Print("DATA_FOUND");
                 Stop();
@@ -80,7 +100,7 @@ namespace cAlgo.Robots
         }
 
         protected override void OnStop() {
-            if (IsProbing || string.IsNullOrEmpty(OutputFile)) return;
+            if (MetaOnly || IsProbing || string.IsNullOrEmpty(OutputFile)) return;
 
             try {
                 using var fileStream = new FileStream(OutputFile, FileMode.Create);
@@ -104,5 +124,49 @@ namespace cAlgo.Robots
                 Print("ERROR: {0}", ex.Message);
             }
         }
+
+        // Serialize the symbol's PRICE-INTERPRETATION GEOMETRY — the facts needed to read the
+        // historical price files: digit count, pip/tick size, contract size, and the currencies
+        // the price is quoted in. These are intrinsic to the price series and effectively
+        // time-stable, so they can be applied across the whole multi-year history.
+        //
+        // Deliberately EXCLUDED: costs, swaps, volume limits, trading status, and the
+        // deposit-currency pip/tick values. Those are a point-in-time snapshot of *today's*
+        // trading conditions — the cBot only ever sees the current contract, never the contract
+        // as it stood in a historical month — so stamping them onto a years-long series would
+        // misrepresent history (and the deposit-currency values further depend on the exporting
+        // account). See docs/symbol-metadata.md for the rationale.
+        //
+        // Keys mirror the cTrader property names; numbers use InvariantCulture (a pip size is
+        // "0.0001", never a locale "0,0001"); a non-finite value becomes JSON null.
+        private void WriteMetadata() {
+            try {
+                var s = Symbol;
+                var inv = CultureInfo.InvariantCulture;
+                string Num(double d) => (double.IsNaN(d) || double.IsInfinity(d)) ? "null" : d.ToString("R", inv);
+                string Str(string v) => v == null ? "null" : "\"" + v.Replace("\\", "\\\\").Replace("\"", "\\\"") + "\"";
+
+                var f = new List<string> {
+                    // provenance of this sidecar
+                    "\"schemaVersion\": 2",
+                    "\"symbol\": " + Str(s.Name),
+                    "\"generatedAtUtc\": " + Str(DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ", inv)),
+                    "\"generator\": " + Str("MS_SimpleExport (cTrader.Automate)"),
+                    // --- price-interpretation geometry (intrinsic, time-stable) ---
+                    "\"description\": " + Str(s.Description),
+                    "\"digits\": " + s.Digits.ToString(inv),
+                    "\"pipSize\": " + Num(s.PipSize),
+                    "\"tickSize\": " + Num(s.TickSize),
+                    "\"lotSize\": " + Num(s.LotSize),
+                    "\"baseAsset\": " + Str(s.BaseAsset?.Name),
+                    "\"quoteAsset\": " + Str(s.QuoteAsset?.Name),
+                };
+
+                System.IO.File.WriteAllText(MetaFile, "{\n  " + string.Join(",\n  ", f) + "\n}\n");
+                Print("META_SUCCESS");
+            } catch (Exception ex) {
+                Print("META_ERROR: {0}", ex.Message);
+            }
+        }
     }
 }