Add per-symbol price-metadata sidecars (<symbol>.meta.json)
The price files carry no facts for interpreting the bars (pip size, digit count, contract size, quote currency); those are known only to the broker via cTrader's Symbol object. Produce a JSON sidecar per symbol next to the price files so a downstream reader does not have to hardcode (and drift on) its own per-symbol table. Scope is the price-interpretation geometry only — digits, pipSize, tickSize, lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable across its whole history. Costs, swaps, volume limits, trading status and the deposit-currency pip/tick values are intentionally left out: the cBot only ever sees the current contract, so those are a point-in-time snapshot of today's trading conditions and would misrepresent a multi-year series (the deposit-currency values are account-dependent too). - cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror the cTrader property names. - runner: write_meta runs once per symbol (mode-independent), validates the JSON, and atomically promotes the sidecar. Best-effort — a metadata failure is logged and never aborts the export, and a pre-existing sidecar is kept on failure. A freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short; EXPORT_META=0 disables it. - The file is a raw serialization of what cTrader delivers; a reader derives its own shape (contract_size, per-lot pip value) from the raw fields. See docs/symbol-metadata.md. - Add a source-level regression test for the freshness guard. Verified end-to-end against the live broker for all 30 configured symbols (FX, indices, metals, crypto, commodities, equities); the price corpus was untouched.
This commit is contained in:
@@ -35,6 +35,8 @@ SYMBOL_FILE="${SYMBOL_FILE:-$REPO_ROOT/runtime/ScrapeSymbols.txt}" # overridabl
|
||||
RETRIES="${RETRIES:-3}" # attempts before a transient failure aborts
|
||||
RUN_TIMEOUT="${RUN_TIMEOUT:-600}" # seconds per backtest before it counts as a hang
|
||||
DISCOVER_MIN_YEAR="${DISCOVER_MIN_YEAR:-2005}" # earliest year the first-data probe scans
|
||||
EXPORT_META="${EXPORT_META:-1}" # 1 = also produce <symbol>.meta.json sidecars
|
||||
META_MAX_AGE_DAYS="${META_MAX_AGE_DAYS:-7}" # refresh a sidecar at most this often (0 = every run)
|
||||
|
||||
# --- DEPLOYMENT CONFIG (secrets/IDs/paths) ---
|
||||
ENV_FILE="${ENV_FILE:-$REPO_ROOT/runtime/export.env}"
|
||||
@@ -207,6 +209,76 @@ probe_run() {
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# write_meta SYMBOL — produce the per-symbol contract-metadata sidecar
|
||||
# (<SYMBOL>.meta.json), placed next to the price files. The cBot's MetaOnly mode
|
||||
# is the ONLY authoritative source for these values: it reads the broker's Symbol
|
||||
# object and serializes it verbatim; this driver merely places the file it writes.
|
||||
#
|
||||
# BEST-EFFORT BY DESIGN: a metadata failure is logged (⚠️) and NEVER aborts the
|
||||
# export run — the sidecar is auxiliary, and any pre-existing one is left untouched
|
||||
# (we only ever overwrite on a validated success). Refreshed at most once per
|
||||
# META_MAX_AGE_DAYS so scheduled runs stay short.
|
||||
# ---------------------------------------------------------------------------
|
||||
write_meta() {
|
||||
local S=$1
|
||||
local FINAL="${S}.meta.json"
|
||||
local TEMP="${S}.meta.json.tmp"
|
||||
|
||||
# Freshness guard: skip if a recent sidecar already exists (mtime within the window).
|
||||
if [[ "$META_MAX_AGE_DAYS" -gt 0 && -f "$HOST_DATA_PATH/$FINAL" \
|
||||
&& -n "$(find "$HOST_DATA_PATH/$FINAL" -mtime "-$META_MAX_AGE_DAYS" 2>/dev/null)" ]]; then
|
||||
echo "🏷 Metadata: $S — fresh (<${META_MAX_AGE_DAYS}d), skipping"
|
||||
META_SKIP=$((META_SKIP+1)); return 0
|
||||
fi
|
||||
|
||||
local C_NAME="ctrader-meta-${S//[^A-Za-z0-9]/-}-$(date +%s%N)"
|
||||
local s_date e_date
|
||||
s_date="$(date -d '7 days ago' +%d/%m/%Y)" # tiny window; MetaOnly stops in OnStart anyway
|
||||
e_date="$(date -d 'yesterday' +%d/%m/%Y)"
|
||||
|
||||
echo "🏷 Metadata: $S (MetaOnly)"
|
||||
CURRENT_CONTAINER="$C_NAME"
|
||||
docker run -d --name "$C_NAME" --init \
|
||||
-v "$HOST_ROBOTS_PATH":"/mnt/Robots" \
|
||||
-v "$HOST_DATA_PATH":"$CONTAINER_DATA_PATH" \
|
||||
-e TZ="Europe/Berlin" \
|
||||
"$IMAGE" \
|
||||
backtest "/mnt/Robots/$BOT_FILE" \
|
||||
--start="$s_date" --end="$e_date" --symbol="$S" \
|
||||
--ctid="$CTID" --pwd-file="$PWD_FILE" --account="$ACCOUNT" \
|
||||
--period="M1" --data-mode="m1" \
|
||||
--Mode="M1" --IsProbing="false" --MetaOnly="true" \
|
||||
--MetaFile="${CONTAINER_DATA_PATH}/${TEMP}" --full-access > /dev/null 2>&1 || {
|
||||
echo " ⚠️ meta: docker run failed to start container for $S"
|
||||
CURRENT_CONTAINER=""; META_FAIL=$((META_FAIL+1)); return 0; }
|
||||
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
*META_SUCCESS*) break ;;
|
||||
*META_ERROR*) echo " ⚠️ meta: bot reported${line#*META_ERROR}"; break ;;
|
||||
"}") break ;;
|
||||
esac
|
||||
done < <(timeout "$RUN_TIMEOUT" docker logs -f "$C_NAME" 2>&1)
|
||||
|
||||
docker rm -f "$C_NAME" >/dev/null 2>&1
|
||||
CURRENT_CONTAINER=""
|
||||
|
||||
# Promote only a syntactically valid JSON; on any failure keep the existing sidecar.
|
||||
if [[ -f "$HOST_DATA_PATH/$TEMP" ]] \
|
||||
&& python3 -c 'import json,sys; json.load(open(sys.argv[1]))' "$HOST_DATA_PATH/$TEMP" 2>/dev/null; then
|
||||
if mv -f "$HOST_DATA_PATH/$TEMP" "$HOST_DATA_PATH/$FINAL"; then
|
||||
echo " ✅ meta saved: $FINAL"; META_OK=$((META_OK+1))
|
||||
else
|
||||
echo " ⚠️ meta: rename $TEMP → $FINAL failed for $S"; META_FAIL=$((META_FAIL+1))
|
||||
fi
|
||||
else
|
||||
echo " ⚠️ meta: no valid metadata produced for $S (existing $FINAL, if any, kept)"
|
||||
rm -f "$HOST_DATA_PATH/$TEMP" 2>/dev/null
|
||||
META_FAIL=$((META_FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# discover_start SYMBOL MODE -> echoes "YEAR MONTH" of the first month with data, or
|
||||
# nothing if the symbol has no data at all. Yearly probe uses December (catches data
|
||||
# that started any time that year); monthly probe uses the FULL month (catches data
|
||||
@@ -287,7 +359,7 @@ process_symbol() {
|
||||
|
||||
# --- EXECUTION (skipped when the script is sourced, e.g. for unit tests) ---
|
||||
main() {
|
||||
SAVED=0; EMPTY=0
|
||||
SAVED=0; EMPTY=0; META_OK=0; META_FAIL=0; META_SKIP=0
|
||||
|
||||
# Clean up stale temp files at start (only *.tmp — never touches real month files).
|
||||
[[ -d "$HOST_DATA_PATH" ]] && rm -f "$HOST_DATA_PATH"/*.tmp 2>/dev/null
|
||||
@@ -307,6 +379,9 @@ while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
read -r S MODES <<< "$line"
|
||||
[[ -z "$MODES" ]] && MODES="M1"
|
||||
|
||||
# Per-symbol contract metadata sidecar (mode-independent — produced once per symbol).
|
||||
[[ "$EXPORT_META" == "1" ]] && write_meta "$S"
|
||||
|
||||
for MODE in $MODES; do
|
||||
if [[ "$MODE" == "M1" || "$MODE" == "Tick" ]]; then
|
||||
echo "------------------------------------------"
|
||||
@@ -321,6 +396,7 @@ done < "$SYMBOL_FILE"
|
||||
|
||||
echo "------------------------------------------"
|
||||
echo "🏁 Done: $(date) — $SYMBOLS_DONE jobs, $SAVED month(s) saved, $EMPTY empty/skipped."
|
||||
[[ "$EXPORT_META" == "1" ]] && echo "🏷 Metadata: $META_OK written, $META_SKIP fresh-skipped, $META_FAIL failed."
|
||||
}
|
||||
|
||||
# Run main only when executed directly; `source`-ing the script (for tests) defines the
|
||||
|
||||
@@ -71,6 +71,20 @@ i=$(midx 2026 5)
|
||||
check "midx/idx_year round-trip" "2026" "$(idx_year "$i")"
|
||||
check "midx/idx_month round-trip" "5" "$(idx_month "$i")"
|
||||
|
||||
# --- write_meta freshness guard (no Docker needed) -------------------------------------
|
||||
# A recent sidecar must be skipped without spawning a container, so scheduled runs stay
|
||||
# short. Point HOST_DATA_PATH at a temp dir, drop a just-created meta file, and assert
|
||||
# write_meta classifies it as a fresh-skip. A docker stub fails loudly if it wrongly runs.
|
||||
TMPDATA="$(mktemp -d)"; HOST_DATA_PATH="$TMPDATA"
|
||||
: > "$TMPDATA/EURUSD.meta.json" # mtime = now -> inside any positive window
|
||||
META_OK=0; META_FAIL=0; META_SKIP=0; META_MAX_AGE_DAYS=7
|
||||
docker() { echo "UNEXPECTED docker call" >&2; return 1; } # must NOT be reached on a fresh file
|
||||
write_meta EURUSD >/dev/null 2>&1
|
||||
unset -f docker
|
||||
check "write_meta: fresh sidecar -> skipped" "1" "$META_SKIP"
|
||||
check "write_meta: fresh sidecar -> not produced" "0" "$META_OK"
|
||||
rm -rf "$TMPDATA"
|
||||
|
||||
echo "-----"
|
||||
echo "$PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
|
||||
Reference in New Issue
Block a user