export-runner: fix new-symbol discovery aborting with a year-0000 backtest

discover_start's stdout is its result channel (read via command
substitution in process_symbol), but the probes inside it stream the
cBot's "[BOT] …" progress on stdout too. That noise leaked into the
captured value, so start_idx parsed to 0 and the resume walk began at
year 0 -> an unparseable "--start=01/01/0" backtest that retried out and
hard-aborted the whole run. Only a brand-new symbol (no files on disk yet)
hits this path, so it stayed invisible until AUDUSD was added.

Redirect every probe call in discover_start to stderr (1>&2) so its
stdout carries only the "YEAR MONTH" result, and document the contract.
Add scripts/test-export-runner.sh: source-level tests that stub probe_run
to emit the same stdout noise and assert a clean result (RED without the
redirect). Make SYMBOL_FILE overridable for scoped runs/tests.
This commit is contained in:
2026-06-25 09:02:10 +02:00
parent 72824cf680
commit ddd8274fad
3 changed files with 106 additions and 3 deletions
+9 -3
View File
@@ -31,7 +31,7 @@ BOT_FILE="MS_SimpleExport.algo"
PWD_FILE="/mnt/Robots/password.txt" # container-internal path (console reads it)
IMAGE="ghcr.io/spotware/ctrader-console:latest"
HOST_ROBOTS_PATH="$REPO_ROOT/runtime/Robots" # holds MS_SimpleExport.algo + password.txt
SYMBOL_FILE="$REPO_ROOT/runtime/ScrapeSymbols.txt"
SYMBOL_FILE="${SYMBOL_FILE:-$REPO_ROOT/runtime/ScrapeSymbols.txt}" # overridable for a scoped run/test
RETRIES="${RETRIES:-3}" # attempts before a transient failure aborts
RUN_TIMEOUT="${RUN_TIMEOUT:-600}" # seconds per backtest before it counts as a hang
DISCOVER_MIN_YEAR="${DISCOVER_MIN_YEAR:-2005}" # earliest year the first-data probe scans
@@ -211,12 +211,18 @@ probe_run() {
# nothing if the symbol has no data at all. Yearly probe uses December (catches data
# that started any time that year); monthly probe uses the FULL month (catches data
# starting after the 10th — the old probe only tested days 0110).
#
# CONTRACT: this function's STDOUT carries ONLY the result line; the caller reads it via
# command substitution. probe_run -> run_one stream the cBot's "[BOT] …" lines on stdout,
# so every probe call here MUST be redirected to stderr (`1>&2`) — otherwise that progress
# noise pollutes the captured value and start_idx is parsed from garbage (see the AUDUSD
# year-0000 abort, docs/export-runner-quirks.md). All diagnostics here go to stderr too.
discover_start() {
local S=$1 MODE=$2 y mm MM edate found_year=""
echo "🌑 Discovery: locating first data month for $S ($MODE)..." >&2
for ((y=DISCOVER_MIN_YEAR; y<CURRENT_YEAR; y++)); do
echo " 🧪 year $y" >&2
if probe_run "$S" "$MODE" "01/12/$y" "31/12/$y"; then found_year=$y; break; fi
if probe_run "$S" "$MODE" "01/12/$y" "31/12/$y" 1>&2; then found_year=$y; break; fi
done
if [[ -z "$found_year" ]]; then found_year=$CURRENT_YEAR; fi # brand-new this year → check months below
for ((mm=1; mm<=12; mm++)); do
@@ -225,7 +231,7 @@ discover_start() {
edate="$(month_end_or_clamp "$found_year" "$mm")"
[[ -z "$edate" ]] && break
echo " 🧪 month $found_year-$MM" >&2
if probe_run "$S" "$MODE" "01/$MM/$found_year" "$edate"; then echo "$found_year $mm"; return 0; fi
if probe_run "$S" "$MODE" "01/$MM/$found_year" "$edate" 1>&2; then echo "$found_year $mm"; return 0; fi
done
return 1 # no data anywhere
}