Brummel
6b7a1cea49
This commit introduces the `doctate-canary` service and associated sweep experiments. The service provides access to the `nvidia/canary-1b-v2` ASR model via a native FastAPI API. It includes deployment scripts, Docker configurations, and detailed documentation on installation, API usage, and environment variables. The sweep experiments aim to thoroughly evaluate the Canary model's performance under various configurations. This includes testing different inference pipelines (single-shot vs. buffered), decoding strategies (greedy vs. beam search), and parameter tuning (chunk length, overlap, batch size, precision). The goal is to reproduce previous findings and identify optimal settings. The commit also includes: - Utility scripts for audio transcoding and manipulation. - Comprehensive logging and result collection mechanisms for the sweep runs. - Detailed analysis of `dur=0` occurrences and word confidence, concluding they are not reliable indicators of hallucination. - Documentation on the interaction between decoding parameters, especially `return_hypotheses`, and the availability of confidence scores.
doctate-canary
Standalone FastAPI wrapper around NVIDIA Canary ASR models, served via the NeMo toolkit.
Why a separate service?
This is not a drop-in replacement for doctate-whisper. The Axum server
keeps speaking the ahmetoner /asr interface to Whisper. This container exists
to evaluate Canary 1B v2 on real medical-German dictations side-by-side with
Whisper, before any decision about pipeline integration.
Inspired by EvilFreelancer/docker-canary-serve — written from scratch, MIT-friendly.
API
GET /health → liveness JSON.
GET /info → model name, device, precision, models dir.
POST /inference (multipart/form-data):
| Field | Type | Default | Notes |
|---|---|---|---|
file |
upload | — | any audio format (mp3, m4a, opus, wav). ffmpeg transcodes to 16 kHz mono WAV internally. |
language |
text | de |
one of en, de, fr, es |
pnc |
text | yes |
punctuation/capitalization (yes/no) |
timestamps |
text | no |
segment-level timestamps in json mode (yes/no) |
response_format |
text | text |
text → plain transcript; json → {text, language, duration, segments?} |
Env vars
| Var | Default | Purpose |
|---|---|---|
CANARY_MODEL |
nvidia/canary-1b-v2 |
any NeMo ASR model name |
CANARY_DEVICE |
cuda |
cuda or cpu |
CANARY_PRECISION |
bf16 |
bf16 / fp16 / fp32 |
CANARY_MODELS_DIR |
/models |
HF cache (volume-persistent) |
HF_HOME |
(auto) | mirrored from CANARY_MODELS_DIR |
VRAM
Canary 1B v2 in bf16 fits comfortably in <5 GB. On a 12 GB GPU it can coexist
with doctate-whisper (~3 GB) but not also with Ollama running a 9 GB
model. For evaluation runs, stop one of the others if VRAM gets tight.
Build & deploy
cd doctate/canary
./deploy.sh minerva.lan
The script rsyncs the directory to /opt/stacks/doctate-canary/, rebuilds
the image on the remote, and polls /health for up to 10 min (first start
downloads ~3 GB and warms up the model).
Smoke test
curl http://minerva.lan:9002/health
curl -F file=@dictation.m4a -F language=de -F response_format=text \
http://minerva.lan:9002/inference
curl -F file=@dictation.m4a -F language=de -F timestamps=yes \
-F response_format=json http://minerva.lan:9002/inference | jq .
License
The wrapper code is project-internal. The Canary 1B v2 model weights are distributed by NVIDIA under CC-BY-4.0 (commercial use permitted, attribution required).