Brummel
ddd8274fad
discover_start's stdout is its result channel (read via command substitution in process_symbol), but the probes inside it stream the cBot's "[BOT] …" progress on stdout too. That noise leaked into the captured value, so start_idx parsed to 0 and the resume walk began at year 0 -> an unparseable "--start=01/01/0" backtest that retried out and hard-aborted the whole run. Only a brand-new symbol (no files on disk yet) hits this path, so it stayed invisible until AUDUSD was added. Redirect every probe call in discover_start to stderr (1>&2) so its stdout carries only the "YEAR MONTH" result, and document the contract. Add scripts/test-export-runner.sh: source-level tests that stub probe_run to emit the same stdout noise and assert a clean result (RED without the redirect). Make SYMBOL_FILE overridable for scoped runs/tests.
103 lines
6.9 KiB
Markdown
103 lines
6.9 KiB
Markdown
# export-runner.sh — quirk audit & hardening decisions
|
||
|
||
`scripts/export-runner.sh` is the Linux-host port of the original Unraid data-export
|
||
driver. Before going live it was put through a multi-lens adversarial audit (5 review
|
||
lenses over the script **and** the `MS_SimpleExport` cBot, each finding independently
|
||
verified). 49 findings survived verification, deduplicating to ~18 distinct quirks.
|
||
|
||
This document records what was fixed, what was a deliberate behaviour choice, and what
|
||
is intentionally left alone — so the rationale is not lost.
|
||
|
||
## Behaviour decisions (the load-bearing ones)
|
||
|
||
| Topic | Decision | Why |
|
||
|---|---|---|
|
||
| **Transient failure** | Retry ×3, then **hard-abort** the whole run | A login stall / crash / timeout must never be silently treated as "empty month". Aborting loudly means a gap is noticed, not buried. |
|
||
| **Resume** | **Gap-aware**: walk every month from the symbol's first data month to now; scrape only **missing** months + the **trailing edge** (latest existing + current). | Fixes the silent middle-gap (old `ls\|sort\|tail-1` resume never backfilled) **and** stops overwriting known-good middle files. |
|
||
| **Empty month** | Don't write empty files. A clean `No historical data` is classified **empty → skip**. | For continuously-active instruments a true in-range empty month effectively never occurs; the empty path is purely defensive and must not trip the hard-abort. |
|
||
| **Current month** | Clamp the end date to **yesterday** | cTrader rejects a backtest whose end date is in the future (proven), so the whole current month was being dropped. Clamping captures the partial month now; next month's edge re-scrape completes it. |
|
||
|
||
## Classification — the core of the hardening
|
||
|
||
`run_one` reads the console log stream (with a per-run `timeout`) and classifies the
|
||
outcome, instead of trusting a single stdout sentinel:
|
||
|
||
- `DATA_FOUND` / `EXPORT_SUCCESS` → **success** (rc 0)
|
||
- `No historical data for the specified period` → **clean-empty** (rc 3, skipped)
|
||
- the backtest reaches its `}` summary but exported nothing → **clean-empty** (rc 3, skipped).
|
||
This catches a cTrader-side **data hole** (a month with no data *inside* a symbol's
|
||
range, e.g. Copper 2017-04), which emits a degenerate empty result block
|
||
(`{ "Equity":, … }`) instead of the clean "No historical data" message — surfaced by
|
||
the first real production run.
|
||
- bot `ERROR:` (an OnStop exception, e.g. a write failure) → **failure** (rc 2)
|
||
- no `}` terminator at all — timeout, crash, dead stream → **transient failure** (rc 2,
|
||
retried then aborted). A genuine transient failure never reaches the clean `}` summary,
|
||
which is what separates it from a data hole.
|
||
|
||
A produced ZIP is validated (`testzip` + non-empty entry) **before** the `mv`, so a
|
||
truncated artifact from a cut-off run is caught instead of being promoted to a final file.
|
||
|
||
## Quirks fixed (no behaviour choice needed)
|
||
|
||
| Quirk | Fix |
|
||
|---|---|
|
||
| No per-run timeout → a hung console blocks the whole export forever | `timeout` around the log drain; a hang is a transient failure |
|
||
| `--rm` + `docker logs -f` race could lose markers on a fast-exiting container | dropped `--rm`; container kept until logs drained, then removed; trap reaps any in-flight container on Ctrl-C (no orphans) |
|
||
| `*ERROR*` matched as an unanchored substring | matched against the bot's `ERROR:` form / the explicit verdict markers |
|
||
| Resume parser `cut -f2/-f3` corrupts year/month for symbols containing `_` | parsed with a bash regex anchored on the symbol |
|
||
| Resume `sort\|tail` is lexical | month walk is purely numeric (`year*12 + month-1` index) |
|
||
| Empty / corrupt ZIP promoted to a final file | `validate_export` (PK + CRC + non-empty entry) gates the `mv` |
|
||
| Monthly start-probe only tested days 01–10 | discovery probes the **full** month; yearly probe uses December (catches same-year-later starts) |
|
||
| Completion banner always printed "success" | banner reports saved / empty counts |
|
||
|
||
## Intentionally NOT changed — frozen by the existing corpus
|
||
|
||
The binary record layout (`M1Record` = 48 B: raw `double` OHLC, `float` spread, `int`
|
||
volume; `TickRecord` = 24 B) is **frozen by the ~4380 existing `.m1`/`.tick` files** the
|
||
bot already produced. Any layout change makes new files incompatible with the corpus, so
|
||
the following audit findings are **WONTFIX** unless the whole dataset is re-exported:
|
||
|
||
- `TickVolume` stored as `int` (would need `long`) — harmless in practice: an M1 bar's
|
||
tick count never approaches `int` range.
|
||
- `Spread` as raw `float` price units, and no `Digits` field — a downstream reader must
|
||
know the contract out-of-band. (Also: `Symbol.Spread` is usually 0 in M1 bar replay, so
|
||
that column is largely meaningless for M1 mode — a data-source limitation, not fixable
|
||
without tick data.)
|
||
- ZIP entry named after `Symbol.Name` while the file is named after the requested symbol —
|
||
cosmetic: the file is renamed by the driver and the reader opens the single entry by
|
||
position, not name.
|
||
|
||
If the format is ever migrated, that is a separate project that also re-exports the corpus.
|
||
|
||
## Verification
|
||
|
||
The hardened script was validated end-to-end against a scratch directory (the real NFS
|
||
data dir untouched):
|
||
|
||
- **Gap-fill / edge / clamp / trusted-middle**: seeded `2026_03` + `2026_05` (gap at `04`);
|
||
the run filled `04`, re-scraped the `05` edge, captured the current month `06` clamped to
|
||
yesterday, and left the trusted middle `03` byte-identical.
|
||
- **Clean-empty safety**: `run_one` on an empty future window returns rc 3 (skip), never the
|
||
rc 2 that would abort the run.
|
||
|
||
## Post-deploy fix — discovery stdout pollution (year-0000 abort)
|
||
|
||
The first **new** symbol added after the rewrite (AUDUSD) aborted on the scheduled
|
||
2026-06-25 run with `--start=01/01/0 --end=31/01/0000` ("Value for parameter start can't
|
||
be parsed"). Root cause: a function-output contract violation, not bad data.
|
||
|
||
`process_symbol` resolves a brand-new symbol's first month via `sd=$(discover_start …)` —
|
||
command substitution, so `discover_start`'s **stdout is its result channel**. But the
|
||
probes inside it (`probe_run` → `run_one`) stream the cBot's `[BOT] …` progress on stdout
|
||
too. Those lines leaked into `sd`; `start_idx=$(midx ${sd% *} ${sd#* })` then parsed
|
||
garbage and collapsed to `0` → the walk started at year 0, month 1 → an unparseable
|
||
`01/01/0` backtest. (The abort was correct behaviour: it stopped a ~24 000-iteration walk
|
||
from year 0 to now, and nothing was written to the data dir — the scrape failed before any
|
||
file op.) Only the no-existing-files branch calls `discover_start`, so only a *new* symbol
|
||
could trigger it — invisible until AUDUSD.
|
||
|
||
**Fix**: every probe call inside `discover_start` is redirected `1>&2`, so its stdout
|
||
carries only the result line. A documented CONTRACT comment now guards the function.
|
||
Covered by `scripts/test-export-runner.sh` (source-level; stubs `probe_run` to emit the
|
||
same stdout noise and asserts a clean `YEAR MONTH` result — RED without the redirect).
|