Package the export pipeline as a single self-contained container for Unraid #1

New Issue

2026-06-24T14:39:22+02:00

Brummel commented

2026-06-24 14:39:22 +02:00

Context

The MS_SimpleExport market-data export currently runs as "Form A": a host bash
orchestrator (scripts/export-runner.sh) that spawns one ephemeral
ghcr.io/spotware/ctrader-console container per backtest window. It is hardened
(docs/export-runner-quirks.md) and runs daily at 03:30 via a systemd timer
(deploy/systemd/ctrader-export.timer).

"Form B" packages the same pipeline as a single self-contained Docker image, so the
Unraid redeployment is one container template to start, stop, and schedule — no host
bash script and no Docker socket. It is a packaging/convenience change, not a functional
one; Form A already runs.

Approach (not docker-in-docker)

Verified this session: ghcr.io/spotware/ctrader-console:latest is Debian 11 and ships
/bin/bash, dotnet, and GNU coreutils; its entrypoint is dotnet ctrader-cli.dll
with the console binary at /app/ctrader-cli.dll.

Build a new image FROM ghcr.io/spotware/ctrader-console:latest, bundle the runner and
the built .algo, and rewrite the per-run step to invoke the console binary directly as
a subprocess instead of docker run:

dotnet /app/ctrader-cli.dll backtest /mnt/Robots/MS_SimpleExport.algo --start=... --end=... ...

Its stdout is read directly (no docker run / docker logs), which also simplifies the
run loop relative to Form A.

Known gotcha (verified this session)

The cTrader CLI process lingers after the backtest — it does not self-exit. In Form A a
docker stop net reaps it; in Form B the subprocess must be explicitly killed once the
result marker is seen (a kill of the subprocess replacing the docker stop).

Mounts

data dir -> /mnt/Data
Robots (.algo + password.txt) -> /mnt/Robots
symbol list bundled into the image or mounted

Acceptance

A single docker run -v <data>:/mnt/Data -v <robots>:/mnt/Robots <image> runs the full scrape
Behaviour matches Form A's hardening: retry-then-abort on transient failure, gap-aware resume, current-month end-date clamp, and data-hole handling (docs/export-runner-quirks.md)
cTrader credentials are passed at run time, not baked into the image

## Context The MS_SimpleExport market-data export currently runs as "Form A": a host bash orchestrator (`scripts/export-runner.sh`) that spawns one ephemeral `ghcr.io/spotware/ctrader-console` container per backtest window. It is hardened (`docs/export-runner-quirks.md`) and runs daily at 03:30 via a systemd timer (`deploy/systemd/ctrader-export.timer`). "Form B" packages the same pipeline as a single self-contained Docker image, so the Unraid redeployment is one container template to start, stop, and schedule — no host bash script and no Docker socket. It is a packaging/convenience change, not a functional one; Form A already runs. ## Approach (not docker-in-docker) Verified this session: `ghcr.io/spotware/ctrader-console:latest` is Debian 11 and ships `/bin/bash`, `dotnet`, and GNU coreutils; its entrypoint is `dotnet ctrader-cli.dll` with the console binary at `/app/ctrader-cli.dll`. Build a new image `FROM ghcr.io/spotware/ctrader-console:latest`, bundle the runner and the built `.algo`, and rewrite the per-run step to invoke the console binary directly as a subprocess instead of `docker run`: ``` dotnet /app/ctrader-cli.dll backtest /mnt/Robots/MS_SimpleExport.algo --start=... --end=... ... ``` Its stdout is read directly (no `docker run` / `docker logs`), which also simplifies the run loop relative to Form A. ## Known gotcha (verified this session) The cTrader CLI process lingers after the backtest — it does not self-exit. In Form A a `docker stop` net reaps it; in Form B the subprocess must be explicitly killed once the result marker is seen (a `kill` of the subprocess replacing the `docker stop`). ## Mounts - data dir -> `/mnt/Data` - Robots (`.algo` + `password.txt`) -> `/mnt/Robots` - symbol list bundled into the image or mounted ## Acceptance - [ ] A single `docker run -v <data>:/mnt/Data -v <robots>:/mnt/Robots <image>` runs the full scrape - [ ] Behaviour matches Form A's hardening: retry-then-abort on transient failure, gap-aware resume, current-month end-date clamp, and data-hole handling (`docs/export-runner-quirks.md`) - [ ] cTrader credentials are passed at run time, not baked into the image

Brummel added the feature label 2026-06-24 14:39:22 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/cTrader#1