Brummel
268954f722
This commit introduces a new standalone FastAPI service for the NVIDIA Canary ASR model. Key changes include: - A new `canary/` directory containing the service code. - `Dockerfile`: Defines the Docker image for the Canary service. - `docker-compose.yml`: Configures the Docker Compose setup for running the service. - `main.py`: Implements the FastAPI application, model loading, and inference logic. - `requirements.txt`: Lists the Python dependencies for the service. - `CHUNKING.md`: Documents the long-form audio handling strategy for Canary. - `README.md`: Provides an overview of the service, API, and deployment instructions. - `deploy.sh`: A script for deploying the service to a remote host. This service allows for independent evaluation and deployment of the Canary ASR model, separate from the existing `doctate-whisper` service. It utilizes a buffered inference approach for handling long audio files, as detailed in `CHUNKING.md`.
75 lines
2.5 KiB
Markdown
75 lines
2.5 KiB
Markdown
# doctate-canary
|
|
|
|
Standalone FastAPI wrapper around [NVIDIA Canary](https://huggingface.co/nvidia/canary-1b-v2)
|
|
ASR models, served via the NeMo toolkit.
|
|
|
|
## Why a separate service?
|
|
|
|
This is **not** a drop-in replacement for `doctate-whisper`. The Axum server
|
|
keeps speaking the ahmetoner `/asr` interface to Whisper. This container exists
|
|
to evaluate Canary 1B v2 on real medical-German dictations side-by-side with
|
|
Whisper, before any decision about pipeline integration.
|
|
|
|
Inspired by `EvilFreelancer/docker-canary-serve` — written from scratch, MIT-friendly.
|
|
|
|
## API
|
|
|
|
`GET /health` → liveness JSON.
|
|
|
|
`GET /info` → model name, device, precision, models dir.
|
|
|
|
`POST /inference` (multipart/form-data):
|
|
|
|
| Field | Type | Default | Notes |
|
|
|---|---|---|---|
|
|
| `file` | upload | — | any audio format (mp3, m4a, opus, wav). ffmpeg transcodes to 16 kHz mono WAV internally. |
|
|
| `language` | text | `de` | one of `en`, `de`, `fr`, `es` |
|
|
| `pnc` | text | `yes` | punctuation/capitalization (`yes`/`no`) |
|
|
| `timestamps` | text | `no` | segment-level timestamps in `json` mode (`yes`/`no`) |
|
|
| `response_format` | text | `text` | `text` → plain transcript; `json` → `{text, language, duration, segments?}` |
|
|
|
|
## Env vars
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `CANARY_MODEL` | `nvidia/canary-1b-v2` | any NeMo ASR model name |
|
|
| `CANARY_DEVICE` | `cuda` | `cuda` or `cpu` |
|
|
| `CANARY_PRECISION` | `bf16` | `bf16` / `fp16` / `fp32` |
|
|
| `CANARY_MODELS_DIR` | `/models` | HF cache (volume-persistent) |
|
|
| `HF_HOME` | (auto) | mirrored from `CANARY_MODELS_DIR` |
|
|
|
|
## VRAM
|
|
|
|
Canary 1B v2 in bf16 fits comfortably in <5 GB. On a 12 GB GPU it can coexist
|
|
with `doctate-whisper` (~3 GB) but **not** also with Ollama running a 9 GB
|
|
model. For evaluation runs, stop one of the others if VRAM gets tight.
|
|
|
|
## Build & deploy
|
|
|
|
```bash
|
|
cd doctate/canary
|
|
./deploy.sh minerva.lan
|
|
```
|
|
|
|
The script rsyncs the directory to `/opt/stacks/doctate-canary/`, rebuilds
|
|
the image on the remote, and polls `/health` for up to 10 min (first start
|
|
downloads ~3 GB and warms up the model).
|
|
|
|
## Smoke test
|
|
|
|
```bash
|
|
curl http://minerva.lan:9002/health
|
|
|
|
curl -F file=@dictation.m4a -F language=de -F response_format=text \
|
|
http://minerva.lan:9002/inference
|
|
|
|
curl -F file=@dictation.m4a -F language=de -F timestamps=yes \
|
|
-F response_format=json http://minerva.lan:9002/inference | jq .
|
|
```
|
|
|
|
## License
|
|
|
|
The wrapper code is project-internal. The Canary 1B v2 model weights are
|
|
distributed by NVIDIA under CC-BY-4.0 (commercial use permitted, attribution
|
|
required).
|