Brummel
268954f722
This commit introduces a new standalone FastAPI service for the NVIDIA Canary ASR model. Key changes include: - A new `canary/` directory containing the service code. - `Dockerfile`: Defines the Docker image for the Canary service. - `docker-compose.yml`: Configures the Docker Compose setup for running the service. - `main.py`: Implements the FastAPI application, model loading, and inference logic. - `requirements.txt`: Lists the Python dependencies for the service. - `CHUNKING.md`: Documents the long-form audio handling strategy for Canary. - `README.md`: Provides an overview of the service, API, and deployment instructions. - `deploy.sh`: A script for deploying the service to a remote host. This service allows for independent evaluation and deployment of the Canary ASR model, separate from the existing `doctate-whisper` service. It utilizes a buffered inference approach for handling long audio files, as detailed in `CHUNKING.md`.
23 lines
565 B
YAML
23 lines
565 B
YAML
services:
|
|
doctate-canary:
|
|
image: doctate-canary:latest
|
|
container_name: doctate-canary
|
|
environment:
|
|
- CANARY_MODEL=nvidia/canary-1b-v2
|
|
- CANARY_PRECISION=bf16
|
|
- NVIDIA_VISIBLE_DEVICES=all
|
|
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
|
volumes:
|
|
- /opt/stacks/doctate-canary/models:/models
|
|
ports:
|
|
- "9002:9002"
|
|
shm_size: 2g
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
count: 1
|
|
capabilities: [gpu]
|
|
restart: unless-stopped
|