Brummel
268954f722
This commit introduces a new standalone FastAPI service for the NVIDIA Canary ASR model. Key changes include: - A new `canary/` directory containing the service code. - `Dockerfile`: Defines the Docker image for the Canary service. - `docker-compose.yml`: Configures the Docker Compose setup for running the service. - `main.py`: Implements the FastAPI application, model loading, and inference logic. - `requirements.txt`: Lists the Python dependencies for the service. - `CHUNKING.md`: Documents the long-form audio handling strategy for Canary. - `README.md`: Provides an overview of the service, API, and deployment instructions. - `deploy.sh`: A script for deploying the service to a remote host. This service allows for independent evaluation and deployment of the Canary ASR model, separate from the existing `doctate-whisper` service. It utilizes a buffered inference approach for handling long audio files, as detailed in `CHUNKING.md`.
2.5 KiB
doctate-canary
Standalone FastAPI wrapper around NVIDIA Canary ASR models, served via the NeMo toolkit.
Why a separate service?
This is not a drop-in replacement for doctate-whisper. The Axum server
keeps speaking the ahmetoner /asr interface to Whisper. This container exists
to evaluate Canary 1B v2 on real medical-German dictations side-by-side with
Whisper, before any decision about pipeline integration.
Inspired by EvilFreelancer/docker-canary-serve — written from scratch, MIT-friendly.
API
GET /health → liveness JSON.
GET /info → model name, device, precision, models dir.
POST /inference (multipart/form-data):
| Field | Type | Default | Notes |
|---|---|---|---|
file |
upload | — | any audio format (mp3, m4a, opus, wav). ffmpeg transcodes to 16 kHz mono WAV internally. |
language |
text | de |
one of en, de, fr, es |
pnc |
text | yes |
punctuation/capitalization (yes/no) |
timestamps |
text | no |
segment-level timestamps in json mode (yes/no) |
response_format |
text | text |
text → plain transcript; json → {text, language, duration, segments?} |
Env vars
| Var | Default | Purpose |
|---|---|---|
CANARY_MODEL |
nvidia/canary-1b-v2 |
any NeMo ASR model name |
CANARY_DEVICE |
cuda |
cuda or cpu |
CANARY_PRECISION |
bf16 |
bf16 / fp16 / fp32 |
CANARY_MODELS_DIR |
/models |
HF cache (volume-persistent) |
HF_HOME |
(auto) | mirrored from CANARY_MODELS_DIR |
VRAM
Canary 1B v2 in bf16 fits comfortably in <5 GB. On a 12 GB GPU it can coexist
with doctate-whisper (~3 GB) but not also with Ollama running a 9 GB
model. For evaluation runs, stop one of the others if VRAM gets tight.
Build & deploy
cd doctate/canary
./deploy.sh minerva.lan
The script rsyncs the directory to /opt/stacks/doctate-canary/, rebuilds
the image on the remote, and polls /health for up to 10 min (first start
downloads ~3 GB and warms up the model).
Smoke test
curl http://minerva.lan:9002/health
curl -F file=@dictation.m4a -F language=de -F response_format=text \
http://minerva.lan:9002/inference
curl -F file=@dictation.m4a -F language=de -F timestamps=yes \
-F response_format=json http://minerva.lan:9002/inference | jq .
License
The wrapper code is project-internal. The Canary 1B v2 model weights are distributed by NVIDIA under CC-BY-4.0 (commercial use permitted, attribution required).