The price files carry no facts for interpreting the bars (pip size, digit count,
contract size, quote currency); those are known only to the broker via cTrader's
Symbol object. Produce a JSON sidecar per symbol next to the price files so a
downstream reader does not have to hardcode (and drift on) its own per-symbol table.
Scope is the price-interpretation geometry only — digits, pipSize, tickSize,
lotSize, baseAsset, quoteAsset — facts intrinsic to the price series and stable
across its whole history. Costs, swaps, volume limits, trading status and the
deposit-currency pip/tick values are intentionally left out: the cBot only ever
sees the current contract, so those are a point-in-time snapshot of today's trading
conditions and would misrepresent a multi-year series (the deposit-currency values
are account-dependent too).
- cBot: a MetaOnly mode (--MetaOnly --MetaFile) serializes the geometry to a flat
JSON. Numbers use InvariantCulture; a non-finite value becomes null; keys mirror
the cTrader property names.
- runner: write_meta runs once per symbol (mode-independent), validates the JSON,
and atomically promotes the sidecar. Best-effort — a metadata failure is logged
and never aborts the export, and a pre-existing sidecar is kept on failure. A
freshness guard (META_MAX_AGE_DAYS, default 7) keeps scheduled runs short;
EXPORT_META=0 disables it.
- The file is a raw serialization of what cTrader delivers; a reader derives its
own shape (contract_size, per-lot pip value) from the raw fields. See
docs/symbol-metadata.md.
- Add a source-level regression test for the freshness guard.
Verified end-to-end against the live broker for all 30 configured symbols (FX,
indices, metals, crypto, commodities, equities); the price corpus was untouched.
discover_start's stdout is its result channel (read via command
substitution in process_symbol), but the probes inside it stream the
cBot's "[BOT] …" progress on stdout too. That noise leaked into the
captured value, so start_idx parsed to 0 and the resume walk began at
year 0 -> an unparseable "--start=01/01/0" backtest that retried out and
hard-aborted the whole run. Only a brand-new symbol (no files on disk yet)
hits this path, so it stayed invisible until AUDUSD was added.
Redirect every probe call in discover_start to stderr (1>&2) so its
stdout carries only the "YEAR MONTH" result, and document the contract.
Add scripts/test-export-runner.sh: source-level tests that stub probe_run
to emit the same stdout noise and assert a clean result (RED without the
redirect). Make SYMBOL_FILE overridable for scoped runs/tests.
The first full production run aborted on Copper 2017-04: a month with no data
*inside* the symbol's range. cTrader does not emit the clean "No historical data"
message for an in-range hole; it runs the backtest to its `}` summary but with a
degenerate empty result and no EXPORT_SUCCESS. run_one misread that as a transient
failure and hard-aborted after retries.
Reaching the `}` summary means the backtest completed cleanly, so the absence of an
export marker is an empty month (rc 3, skipped), not a transient failure. A genuine
transient failure never reaches `}`, which is what distinguishes the two.
MS_SimpleExport is a market-data exporter (M1/Tick -> zipped binary), driven
headless via the official cTrader-console Docker image. This adds the bot and a
hardened bash driver that runs the scrape on a Linux host, spawning one ephemeral
console container per backtest window.
- src/MS_SimpleExport: the cBot; csproj aligned to the repo's Linux template
(AlgoPublish=false, cTrader.Automate 1.*-*).
- scripts/export-runner.sh: hardened port of the Unraid driver -- retry-then-abort
on transient failure, gap-aware resume, current-month end-date clamp, 3-way run
classification with a per-run timeout, and ZIP validation before rename.
- scripts/export.env.example: deployment config template (real values live in the
gitignored runtime/export.env).
- docs/export-runner-quirks.md: the multi-lens audit findings and hardening decisions.
- .gitignore: ignore runtime/ (secrets, the placed .algo, the symbol list).
Build cBots in C# and compile them to .algo entirely on Linux via the
official cTrader.Automate NuGet package + 'dotnet build' — no Windows and
no cTrader desktop app. Run/backtest headless through Spotware's official
cTrader console Docker image.
Includes:
- src/SampleCBot: example SMA-crossover cBot (SDK-style net6.0 class library)
- scripts/install-dotnet.sh: root-free local .NET SDK install (~/.dotnet)
- scripts/build.sh: dotnet build -> dist/<ProjectName>.algo, with a workaround
for cTrader.Automate naming the .algo after the project's parent folder
- scripts/run-console.sh: wrapper over ghcr.io/spotware/ctrader-console
- docs/research-findings.md: sourced research; .algo verified as a proprietary
'algo'-magic container (not a ZIP); build verified end to end on Linux
Brummel