Add MS_SimpleExport bot + hardened Linux export-runner

MS_SimpleExport is a market-data exporter (M1/Tick -> zipped binary), driven
headless via the official cTrader-console Docker image. This adds the bot and a
hardened bash driver that runs the scrape on a Linux host, spawning one ephemeral
console container per backtest window.

- src/MS_SimpleExport: the cBot; csproj aligned to the repo's Linux template
  (AlgoPublish=false, cTrader.Automate 1.*-*).
- scripts/export-runner.sh: hardened port of the Unraid driver -- retry-then-abort
  on transient failure, gap-aware resume, current-month end-date clamp, 3-way run
  classification with a per-run timeout, and ZIP validation before rename.
- scripts/export.env.example: deployment config template (real values live in the
  gitignored runtime/export.env).
- docs/export-runner-quirks.md: the multi-lens audit findings and hardening decisions.
- .gitignore: ignore runtime/ (secrets, the placed .algo, the symbol list).
This commit is contained in:
2026-06-24 13:31:41 +02:00
parent 9c39a743d0
commit cc3f0cbd46
6 changed files with 547 additions and 0 deletions
+73
View File
@@ -0,0 +1,73 @@
# export-runner.sh — quirk audit & hardening decisions
`scripts/export-runner.sh` is the Linux-host port of the original Unraid data-export
driver. Before going live it was put through a multi-lens adversarial audit (5 review
lenses over the script **and** the `MS_SimpleExport` cBot, each finding independently
verified). 49 findings survived verification, deduplicating to ~18 distinct quirks.
This document records what was fixed, what was a deliberate behaviour choice, and what
is intentionally left alone — so the rationale is not lost.
## Behaviour decisions (the load-bearing ones)
| Topic | Decision | Why |
|---|---|---|
| **Transient failure** | Retry ×3, then **hard-abort** the whole run | A login stall / crash / timeout must never be silently treated as "empty month". Aborting loudly means a gap is noticed, not buried. |
| **Resume** | **Gap-aware**: walk every month from the symbol's first data month to now; scrape only **missing** months + the **trailing edge** (latest existing + current). | Fixes the silent middle-gap (old `ls\|sort\|tail-1` resume never backfilled) **and** stops overwriting known-good middle files. |
| **Empty month** | Don't write empty files. A clean `No historical data` is classified **empty → skip**. | For continuously-active instruments a true in-range empty month effectively never occurs; the empty path is purely defensive and must not trip the hard-abort. |
| **Current month** | Clamp the end date to **yesterday** | cTrader rejects a backtest whose end date is in the future (proven), so the whole current month was being dropped. Clamping captures the partial month now; next month's edge re-scrape completes it. |
## Classification — the core of the hardening
`run_one` reads the console log stream (with a per-run `timeout`) and classifies the
outcome, instead of trusting a single stdout sentinel:
- `DATA_FOUND` / `EXPORT_SUCCESS`**success** (rc 0)
- `No historical data for the specified period`**clean-empty** (rc 3, skipped)
- anything else — timeout, `ERROR:`, a result block with no verdict, a dead stream → **transient failure** (rc 2, retried then aborted)
A produced ZIP is validated (`testzip` + non-empty entry) **before** the `mv`, so a
truncated artifact from a cut-off run is caught instead of being promoted to a final file.
## Quirks fixed (no behaviour choice needed)
| Quirk | Fix |
|---|---|
| No per-run timeout → a hung console blocks the whole export forever | `timeout` around the log drain; a hang is a transient failure |
| `--rm` + `docker logs -f` race could lose markers on a fast-exiting container | dropped `--rm`; container kept until logs drained, then removed; trap reaps any in-flight container on Ctrl-C (no orphans) |
| `*ERROR*` matched as an unanchored substring | matched against the bot's `ERROR:` form / the explicit verdict markers |
| Resume parser `cut -f2/-f3` corrupts year/month for symbols containing `_` | parsed with a bash regex anchored on the symbol |
| Resume `sort\|tail` is lexical | month walk is purely numeric (`year*12 + month-1` index) |
| Empty / corrupt ZIP promoted to a final file | `validate_export` (PK + CRC + non-empty entry) gates the `mv` |
| Monthly start-probe only tested days 0110 | discovery probes the **full** month; yearly probe uses December (catches same-year-later starts) |
| Completion banner always printed "success" | banner reports saved / empty counts |
## Intentionally NOT changed — frozen by the existing corpus
The binary record layout (`M1Record` = 48 B: raw `double` OHLC, `float` spread, `int`
volume; `TickRecord` = 24 B) is **frozen by the ~4380 existing `.m1`/`.tick` files** the
bot already produced. Any layout change makes new files incompatible with the corpus, so
the following audit findings are **WONTFIX** unless the whole dataset is re-exported:
- `TickVolume` stored as `int` (would need `long`) — harmless in practice: an M1 bar's
tick count never approaches `int` range.
- `Spread` as raw `float` price units, and no `Digits` field — a downstream reader must
know the contract out-of-band. (Also: `Symbol.Spread` is usually 0 in M1 bar replay, so
that column is largely meaningless for M1 mode — a data-source limitation, not fixable
without tick data.)
- ZIP entry named after `Symbol.Name` while the file is named after the requested symbol —
cosmetic: the file is renamed by the driver and the reader opens the single entry by
position, not name.
If the format is ever migrated, that is a separate project that also re-exports the corpus.
## Verification
The hardened script was validated end-to-end against a scratch directory (the real NFS
data dir untouched):
- **Gap-fill / edge / clamp / trusted-middle**: seeded `2026_03` + `2026_05` (gap at `04`);
the run filled `04`, re-scraped the `05` edge, captured the current month `06` clamped to
yesterday, and left the trusted middle `03` byte-identical.
- **Clean-empty safety**: `run_one` on an empty future window returns rc 3 (skip), never the
rc 2 that would abort the run.