cTrader/docs/export-runner-quirks.md

export-runner.sh — quirk audit & hardening decisions

scripts/export-runner.sh is the Linux-host port of the original Unraid data-export driver. Before going live it was put through a multi-lens adversarial audit (5 review lenses over the script and the MS_SimpleExport cBot, each finding independently verified). 49 findings survived verification, deduplicating to ~18 distinct quirks.

This document records what was fixed, what was a deliberate behaviour choice, and what is intentionally left alone — so the rationale is not lost.

Behaviour decisions (the load-bearing ones)

Topic	Decision	Why
Transient failure	Retry ×3, then hard-abort the whole run	A login stall / crash / timeout must never be silently treated as "empty month". Aborting loudly means a gap is noticed, not buried.
Resume	Gap-aware: walk every month from the symbol's first data month to now; scrape only missing months + the trailing edge (latest existing + current).	Fixes the silent middle-gap (old ls|sort|tail-1 resume never backfilled) and stops overwriting known-good middle files.
Empty month	Don't write empty files. A clean No historical data is classified empty → skip.	For continuously-active instruments a true in-range empty month effectively never occurs; the empty path is purely defensive and must not trip the hard-abort.
Current month	Clamp the end date to yesterday	cTrader rejects a backtest whose end date is in the future (proven), so the whole current month was being dropped. Clamping captures the partial month now; next month's edge re-scrape completes it.

Classification — the core of the hardening

run_one reads the console log stream (with a per-run timeout) and classifies the outcome, instead of trusting a single stdout sentinel:

• DATA_FOUND / EXPORT_SUCCESS → success (rc 0)
• No historical data for the specified period → clean-empty (rc 3, skipped)
• the backtest reaches its } summary but exported nothing → clean-empty (rc 3, skipped). This catches a cTrader-side data hole (a month with no data inside a symbol's range, e.g. Copper 2017-04), which emits a degenerate empty result block ({ "Equity":, … }) instead of the clean "No historical data" message — surfaced by the first real production run.
• bot ERROR: (an OnStop exception, e.g. a write failure) → failure (rc 2)
• no } terminator at all — timeout, crash, dead stream → transient failure (rc 2, retried then aborted). A genuine transient failure never reaches the clean } summary, which is what separates it from a data hole.

A produced ZIP is validated (testzip + non-empty entry) before the mv, so a truncated artifact from a cut-off run is caught instead of being promoted to a final file.

Quirks fixed (no behaviour choice needed)

Quirk	Fix
No per-run timeout → a hung console blocks the whole export forever	timeout around the log drain; a hang is a transient failure
--rm + docker logs -f race could lose markers on a fast-exiting container	dropped --rm; container kept until logs drained, then removed; trap reaps any in-flight container on Ctrl-C (no orphans)
*ERROR* matched as an unanchored substring	matched against the bot's ERROR: form / the explicit verdict markers
Resume parser cut -f2/-f3 corrupts year/month for symbols containing _	parsed with a bash regex anchored on the symbol
Resume sort|tail is lexical	month walk is purely numeric (year*12 + month-1 index)
Empty / corrupt ZIP promoted to a final file	validate_export (PK + CRC + non-empty entry) gates the mv
Monthly start-probe only tested days 01–10	discovery probes the full month; yearly probe uses December (catches same-year-later starts)
Completion banner always printed "success"	banner reports saved / empty counts

Intentionally NOT changed — frozen by the existing corpus

The binary record layout (M1Record = 48 B: raw double OHLC, float spread, int volume; TickRecord = 24 B) is frozen by the ~4380 existing .m1/.tick files the bot already produced. Any layout change makes new files incompatible with the corpus, so the following audit findings are WONTFIX unless the whole dataset is re-exported:

• TickVolume stored as int (would need long) — harmless in practice: an M1 bar's tick count never approaches int range.
• Spread as raw float price units, and no Digits field — a downstream reader must know the contract out-of-band. (Also: Symbol.Spread is usually 0 in M1 bar replay, so that column is largely meaningless for M1 mode — a data-source limitation, not fixable without tick data.)
• ZIP entry named after Symbol.Name while the file is named after the requested symbol — cosmetic: the file is renamed by the driver and the reader opens the single entry by position, not name.

If the format is ever migrated, that is a separate project that also re-exports the corpus.

Verification

The hardened script was validated end-to-end against a scratch directory (the real NFS data dir untouched):

• Gap-fill / edge / clamp / trusted-middle: seeded 2026_03 + 2026_05 (gap at 04); the run filled 04, re-scraped the 05 edge, captured the current month 06 clamped to yesterday, and left the trusted middle 03 byte-identical.
• Clean-empty safety: run_one on an empty future window returns rc 3 (skip), never the rc 2 that would abort the run.

Post-deploy fix — discovery stdout pollution (year-0000 abort)

The first new symbol added after the rewrite (AUDUSD) aborted on the scheduled 2026-06-25 run with --start=01/01/0 --end=31/01/0000 ("Value for parameter start can't be parsed"). Root cause: a function-output contract violation, not bad data.

process_symbol resolves a brand-new symbol's first month via sd=$(discover_start …) — command substitution, so discover_start's stdout is its result channel. But the probes inside it (probe_run → run_one) stream the cBot's [BOT] … progress on stdout too. Those lines leaked into sd; start_idx=$(midx ${sd% *} ${sd#* }) then parsed garbage and collapsed to 0 → the walk started at year 0, month 1 → an unparseable 01/01/0 backtest. (The abort was correct behaviour: it stopped a ~24 000-iteration walk from year 0 to now, and nothing was written to the data dir — the scrape failed before any file op.) Only the no-existing-files branch calls discover_start, so only a new symbol could trigger it — invisible until AUDUSD.

Fix: every probe call inside discover_start is redirected 1>&2, so its stdout carries only the result line. A documented CONTRACT comment now guards the function. Covered by scripts/test-export-runner.sh (source-level; stubs probe_run to emit the same stdout noise and asserts a clean YEAR MONTH result — RED without the redirect).