From 268954f72210a372d8b2410958e6f07996826fcb Mon Sep 17 00:00:00 2001 From: Brummel Date: Thu, 30 Apr 2026 09:24:33 +0200 Subject: [PATCH] feat: Add standalone canary service This commit introduces a new standalone FastAPI service for the NVIDIA Canary ASR model. Key changes include: - A new `canary/` directory containing the service code. - `Dockerfile`: Defines the Docker image for the Canary service. - `docker-compose.yml`: Configures the Docker Compose setup for running the service. - `main.py`: Implements the FastAPI application, model loading, and inference logic. - `requirements.txt`: Lists the Python dependencies for the service. - `CHUNKING.md`: Documents the long-form audio handling strategy for Canary. - `README.md`: Provides an overview of the service, API, and deployment instructions. - `deploy.sh`: A script for deploying the service to a remote host. This service allows for independent evaluation and deployment of the Canary ASR model, separate from the existing `doctate-whisper` service. It utilizes a buffered inference approach for handling long audio files, as detailed in `CHUNKING.md`. --- canary/CHUNKING.md | 233 ++++++++++++++++++++++++++++++++++++ canary/Dockerfile | 32 +++++ canary/README.md | 74 ++++++++++++ canary/deploy.sh | 36 ++++++ canary/docker-compose.yml | 22 ++++ canary/main.py | 246 ++++++++++++++++++++++++++++++++++++++ canary/requirements.txt | 5 + 7 files changed, 648 insertions(+) create mode 100644 canary/CHUNKING.md create mode 100644 canary/Dockerfile create mode 100644 canary/README.md create mode 100755 canary/deploy.sh create mode 100644 canary/docker-compose.yml create mode 100644 canary/main.py create mode 100644 canary/requirements.txt diff --git a/canary/CHUNKING.md b/canary/CHUNKING.md new file mode 100644 index 0000000..42848f6 --- /dev/null +++ b/canary/CHUNKING.md @@ -0,0 +1,233 @@ +# Long-form audio handling in canary-1b-v2 + +## TL;DR + +- Canary-1b-v2 truncates audio > ~40 s on the default `model.transcribe([path])` call. +- The fix is **NeMo ≥ 2.5**: long-form inference with overlapping-window + auto-chunking ships in `transcribe()` itself, triggered by + `batch_size=1` or single-file input. +- The explicit `FrameBatchMultiTaskAED` path in NeMo `2.4.x` is broken for + `canary-1b-v2` (NeMo issue #14544: `CanaryBPETokenizer` lacks the + `vocabulary` attribute the streaming-utils class requires). +- Pin: `nemo_toolkit[asr]>=2.7.3,<3.0` in `requirements.txt`. +- **Currently used in this codebase**: Path 2 (`FrameBatchMultiTaskAED`), + branched on `duration > 25 s`. Empirically verified end-to-end on + `experiments/case_c414cf52/`. Path 1 was not retested after the NeMo + upgrade — see Decision below. +- **VRAM cost**: ~9.7 GB on RTX 3060 with `batch_size=8`. Canary cannot + share a 12 GB GPU with `doctate-whisper` (~3.9 GB) — sequential operation + required, or reduce `FrameBatchMultiTaskAED.batch_size`. + +## Problem + +The default `model.transcribe([audio_path])` path silently truncates audio +longer than roughly 40 s. Observed empirically in our +`experiments/case_c414cf52/asr_comparison.md` benchmark with NeMo 2.4.1: +audios at 66 s, 78 s, and 81 s lost their final 25–40 s of content. The +model emits no warning, container logs are clean — only the output length +reveals the truncation. + +The HuggingFace model card claims "automatic chunking enabled at ~40 s with +1 s overlap." That claim is true — but **only in NeMo ≥ 2.5**, and only on +specific call shapes (single-file or `batch_size=1`). Our pinned +`nemo_toolkit==2.4.1` did not have the feature merged into a release yet. + +## Why it happens + +Canary-1b-v2's FastConformer encoder has a fixed positional-embedding +window. When the input exceeds it, the encoder produces features only for +the first window. The decoder emits text up to whatever it generated from +those features and then stops. No exception, no warning. + +The official NeMo answer (per maintainer @nithinraok in +[issue #14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544)): + +> can you switch to main branch. Its not part of release version yet. + +That comment dates from August 2025, when canary-1b-v2 was new and only the +`main` branch had the long-form handling. By NeMo 2.5 it shipped to +release; by 2.7.x (our current pin) it is stable. + +## Path 1 — auto-chunking via `transcribe()` (recommended, NeMo ≥ 2.5) + +The `transcribe()` method itself supports dynamic chunking with overlapping +windows in NeMo ≥ 2.5. It activates automatically under either condition: + +- a single audio file passed as a string (not a list), OR +- `batch_size=1` set explicitly + +Quote from NVIDIA-NeMo's +[official example docstring](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py): + +> Canary-1b-v2 supports long‑form inference via the `.transcribe()` method. +> It will use dynamic chunking with overlapping windows for better +> performance. This behavior is enabled automatically for long‑form +> inference when transcribing a single audio file or when batch_size is +> set to 1. + +Minimal call: + +```python +results = model.transcribe( + [wav_path], + source_lang="de", + target_lang="de", + pnc="yes", + batch_size=1, # triggers long-form path +) +text = results[0].text +``` + +This path is preferred: it is shorter, has no manifest plumbing, and uses +NeMo's tested overlapping-window strategy. + +## Path 2 — explicit `FrameBatchMultiTaskAED` (fallback) + +For batch inference over many files, manifest-driven workflows, or when +fine control over chunk length is needed, NeMo exposes +`FrameBatchMultiTaskAED` + `get_buffered_pred_feat_multitaskAED`. + +**Caveat — broken in NeMo 2.4.x for canary-1b-v2**: +`FrameBatchMultiTaskAED.__init__` calls +`len(asr_model.tokenizer.vocabulary)`, but `CanaryBPETokenizer` (the +tokenizer of canary-1b-v2) has no `vocabulary` attribute. NeMo issue +[#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544) tracks this. In +`v2.7.3` the same line was extended to fall back on `tokenizer.vocab_size` +first, so this path is **fixed from NeMo 2.5 onward**. Verified by diffing +`streaming_utils.py:740–760` between v2.4.1 and v2.7.3. + +Key parameters from the official example: + +| Parameter | Default | Notes | +|---|---|---| +| `chunk_len_in_secs` | `40.0` | window length | +| `total_buffer` | same as `frame_len` | overlap budget | +| `batch_size` | `8` | chunks processed in parallel | +| `model_stride` | `8` | FastConformer downsampling factor | + +Pre-conditions on the model config (set once at startup): + +```python +import copy +from omegaconf import OmegaConf + +_model_cfg = copy.deepcopy(model._cfg) +OmegaConf.set_struct(_model_cfg.preprocessor, False) +_model_cfg.preprocessor.dither = 0.0 +_model_cfg.preprocessor.pad_to = 0 +OmegaConf.set_struct(_model_cfg.preprocessor, True) +assert _model_cfg.preprocessor.normalize == "per_feature" +``` + +Setup pattern (condensed from the official script): + +```python +from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED +from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED + +frame_asr = FrameBatchMultiTaskAED( + asr_model=model, + frame_len=40.0, + total_buffer=40.0, + batch_size=8, +) +feature_stride = _model_cfg.preprocessor["window_stride"] +model_stride_in_secs = feature_stride * 8 # 0.01 * 8 = 0.08 for FastConformer + +with torch.amp.autocast(model.device.type, enabled=True, dtype=torch.bfloat16): + with torch.no_grad(): + hyps = get_buffered_pred_feat_multitaskAED( + frame_asr, + _model_cfg.preprocessor, + model_stride_in_secs, + model.device, + manifest=manifest_path, # JSONL with audio_filepath, source_lang, target_lang, pnc, taskname + filepaths=None, + timestamps=False, + ) +``` + +For German ASR, the manifest is required — `filepaths=`-only mode defaults +to English. One JSON object per line: + +```json +{"audio_filepath": "/path/audio.wav", "duration": 80.9, "taskname": "asr", + "source_lang": "de", "target_lang": "de", "pnc": "yes"} +``` + +## Caveat — timestamps require shorter chunks + +When `timestamps=True`, the official script forces `chunk_len_in_secs=10.0`: + +> When `timestamps` is True, it's recommended to use `chunk_len_in_secs=10.0` +> for optimal results. + +Larger windows produce less reliable per-segment timestamps. If we expose +timestamps in `/inference`, chunk length must drop to 10 s for that mode. + +## Decision — what this codebase actually uses + +**Path 2 (`FrameBatchMultiTaskAED`)** is implemented in `main.py` +(`_transcribe_buffered`), branched on `duration > CHUNKED_THRESHOLD_SECS` +(default 25 s). It was implemented *before* we upgraded NeMo, when Path 1 +was unavailable in 2.4.1. After upgrading to 2.7.3 we kept Path 2 because +it is the path we have empirical evidence for. + +**Path 1 was not retested** after the NeMo upgrade. It would likely work +in 2.7.3 — the official docs and example script promise so — and would +let us delete `_transcribe_buffered` and the manifest plumbing. Reasons +to revisit Path 1 later: + +- Code simplification (~50 lines of `main.py` could go away). +- Lower VRAM (Path 2's `batch_size=8` reserves buffers for 8 parallel + chunks; Path 1's overlapping-window strategy may use less). +- One fewer setup-per-call (FrameBatchMultiTaskAED is currently + instantiated each request). + +Reasons to keep Path 2: + +- It works, with measurements and full transcripts. +- Explicit control over `chunk_len_in_secs` (currently 40 s). +- The fallback for short audio (`model.transcribe([wav], ...)` for + `duration ≤ 25 s`) is still useful for low latency on short clips. + +The migration risk is non-zero (we'd need to re-run the case_c414cf52 +benchmark to confirm output quality), so it's a "later" task rather than +a "now" task. + +## What we ruled out + +- **Hard fixed-cut chunking** at second boundaries (the approach in + `EvilFreelancer/docker-canary-serve`, using `wave.readframes()` with no + overlap): cuts mid-word at chunk boundaries. NeMo's overlapping-window + paths are strictly better and exist now in release versions. +- **Streaming variant** (`speech_to_text_aed_streaming_infer.py`, ~20 KB): + cache-aware streaming for online use cases. Overkill for our offline + transcription workflow. +- **Monkey-patching `tokenizer.vocabulary`** to work around the NeMo 2.4.x + bug. Possible but a code smell — the upstream fix in 2.5+ is the right + solution. + +## Empirical history (this codebase) + +| Date | Attempt | Result | +|---|---|---| +| 2026-04-29 | NeMo 2.4.1 + `model.transcribe([wav], ...)` | Truncation > 50 s | +| 2026-04-29 | + `batch_size=1` | No change — feature not in 2.4.1 | +| 2026-04-30 | + explicit `FrameBatchMultiTaskAED` (Path 2) on 2.4.1 | `AttributeError: CanaryBPETokenizer.vocabulary` (NeMo bug) | +| 2026-04-30 | NeMo bumped to ≥ 2.7.3, Path 2 active | ✓ all 6 audios fully transcribed; ~2 s inference per audio (~30× realtime); 9.7 GB VRAM | + +## References + +- NeMo issue tracking the 2.4.x tokenizer bug: + [#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544) +- Official NVIDIA NeMo Canary chunked/streaming docs: + +- Official chunked AED inference example: + [`speech_to_text_aed_chunked_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py) +- Streaming variant: + [`speech_to_text_aed_streaming_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_streaming_infer.py) +- HuggingFace model card: +- Truncation evidence in this repo: `experiments/case_c414cf52/asr_comparison.md` +- Reference project we initially looked at (we did not adopt their cut + strategy): [`EvilFreelancer/docker-canary-serve`](https://github.com/EvilFreelancer/docker-canary-serve) diff --git a/canary/Dockerfile b/canary/Dockerfile new file mode 100644 index 0000000..4866647 --- /dev/null +++ b/canary/Dockerfile @@ -0,0 +1,32 @@ +FROM nvidia/cuda:12.6.1-devel-ubuntu22.04 + +ENV DEBIAN_FRONTEND=noninteractive \ + PIP_NO_CACHE_DIR=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 python3-pip python3-dev \ + ffmpeg \ + libsndfile1 \ + git \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +COPY requirements.txt . +RUN pip install --upgrade pip && \ + pip install -r requirements.txt + +COPY main.py . + +ENV CANARY_MODEL=nvidia/canary-1b-v2 \ + CANARY_DEVICE=cuda \ + CANARY_PRECISION=bf16 \ + CANARY_MODELS_DIR=/models \ + HF_HOME=/models + +VOLUME ["/models"] +EXPOSE 9002 + +CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9002", "--workers", "1"] diff --git a/canary/README.md b/canary/README.md new file mode 100644 index 0000000..2fdb716 --- /dev/null +++ b/canary/README.md @@ -0,0 +1,74 @@ +# doctate-canary + +Standalone FastAPI wrapper around [NVIDIA Canary](https://huggingface.co/nvidia/canary-1b-v2) +ASR models, served via the NeMo toolkit. + +## Why a separate service? + +This is **not** a drop-in replacement for `doctate-whisper`. The Axum server +keeps speaking the ahmetoner `/asr` interface to Whisper. This container exists +to evaluate Canary 1B v2 on real medical-German dictations side-by-side with +Whisper, before any decision about pipeline integration. + +Inspired by `EvilFreelancer/docker-canary-serve` — written from scratch, MIT-friendly. + +## API + +`GET /health` → liveness JSON. + +`GET /info` → model name, device, precision, models dir. + +`POST /inference` (multipart/form-data): + +| Field | Type | Default | Notes | +|---|---|---|---| +| `file` | upload | — | any audio format (mp3, m4a, opus, wav). ffmpeg transcodes to 16 kHz mono WAV internally. | +| `language` | text | `de` | one of `en`, `de`, `fr`, `es` | +| `pnc` | text | `yes` | punctuation/capitalization (`yes`/`no`) | +| `timestamps` | text | `no` | segment-level timestamps in `json` mode (`yes`/`no`) | +| `response_format` | text | `text` | `text` → plain transcript; `json` → `{text, language, duration, segments?}` | + +## Env vars + +| Var | Default | Purpose | +|---|---|---| +| `CANARY_MODEL` | `nvidia/canary-1b-v2` | any NeMo ASR model name | +| `CANARY_DEVICE` | `cuda` | `cuda` or `cpu` | +| `CANARY_PRECISION` | `bf16` | `bf16` / `fp16` / `fp32` | +| `CANARY_MODELS_DIR` | `/models` | HF cache (volume-persistent) | +| `HF_HOME` | (auto) | mirrored from `CANARY_MODELS_DIR` | + +## VRAM + +Canary 1B v2 in bf16 fits comfortably in <5 GB. On a 12 GB GPU it can coexist +with `doctate-whisper` (~3 GB) but **not** also with Ollama running a 9 GB +model. For evaluation runs, stop one of the others if VRAM gets tight. + +## Build & deploy + +```bash +cd doctate/canary +./deploy.sh minerva.lan +``` + +The script rsyncs the directory to `/opt/stacks/doctate-canary/`, rebuilds +the image on the remote, and polls `/health` for up to 10 min (first start +downloads ~3 GB and warms up the model). + +## Smoke test + +```bash +curl http://minerva.lan:9002/health + +curl -F file=@dictation.m4a -F language=de -F response_format=text \ + http://minerva.lan:9002/inference + +curl -F file=@dictation.m4a -F language=de -F timestamps=yes \ + -F response_format=json http://minerva.lan:9002/inference | jq . +``` + +## License + +The wrapper code is project-internal. The Canary 1B v2 model weights are +distributed by NVIDIA under CC-BY-4.0 (commercial use permitted, attribution +required). diff --git a/canary/deploy.sh b/canary/deploy.sh new file mode 100755 index 0000000..24bf589 --- /dev/null +++ b/canary/deploy.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Deploy the canary service to a remote host via rsync + ssh. +# Usage: ./deploy.sh [host] (default: minerva.lan) +set -euo pipefail + +HOST="${1:-minerva.lan}" +REMOTE_DIR="/opt/stacks/doctate-canary" +LOCAL_DIR="$(cd "$(dirname "$0")" && pwd)" + +echo ">>> Syncing $LOCAL_DIR -> $HOST:$REMOTE_DIR" +rsync -av \ + --exclude 'deploy.sh' \ + --exclude 'models/' \ + "$LOCAL_DIR/" "$HOST:$REMOTE_DIR/" + +echo ">>> Rebuilding + restarting on $HOST" +ssh "$HOST" "cd $REMOTE_DIR && \ + docker compose down && \ + docker build -t doctate-canary . && \ + docker compose up -d" + +echo ">>> Waiting for /health (up to 10 min for first model download + warmup)" +for i in $(seq 1 120); do + if ssh "$HOST" "curl -sf http://localhost:9002/health" 2>/dev/null; then + echo + echo ">>> OK" + ssh "$HOST" "curl -s http://localhost:9002/info" + echo + exit 0 + fi + sleep 5 +done + +echo ">>> TIMEOUT — last logs:" +ssh "$HOST" "docker logs doctate-canary --tail 60" +exit 1 diff --git a/canary/docker-compose.yml b/canary/docker-compose.yml new file mode 100644 index 0000000..7ff0f1d --- /dev/null +++ b/canary/docker-compose.yml @@ -0,0 +1,22 @@ +services: + doctate-canary: + image: doctate-canary:latest + container_name: doctate-canary + environment: + - CANARY_MODEL=nvidia/canary-1b-v2 + - CANARY_PRECISION=bf16 + - NVIDIA_VISIBLE_DEVICES=all + - NVIDIA_DRIVER_CAPABILITIES=compute,utility + volumes: + - /opt/stacks/doctate-canary/models:/models + ports: + - "9002:9002" + shm_size: 2g + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: 1 + capabilities: [gpu] + restart: unless-stopped diff --git a/canary/main.py b/canary/main.py new file mode 100644 index 0000000..e5215e5 --- /dev/null +++ b/canary/main.py @@ -0,0 +1,246 @@ +"""FastAPI wrapper around NVIDIA NeMo Canary ASR models. + +Native Canary HTTP API (POST /inference). Not a Whisper /asr clone. +Inspired by EvilFreelancer/docker-canary-serve, written from scratch. +""" +import copy +import json +import logging +import os +import subprocess +import tempfile +import time +import wave +from typing import Optional + +import torch +from fastapi import FastAPI, File, Form, HTTPException, UploadFile +from fastapi.responses import JSONResponse, PlainTextResponse +from nemo.collections.asr.models import ASRModel +from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED +from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED +from omegaconf import OmegaConf + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(message)s", +) +log = logging.getLogger("canary") + +MODEL_NAME = os.getenv("CANARY_MODEL", "nvidia/canary-1b-v2") +DEVICE = os.getenv("CANARY_DEVICE", "cuda") +PRECISION = os.getenv("CANARY_PRECISION", "bf16").lower() +MODELS_DIR = os.getenv("CANARY_MODELS_DIR", "/models") +CHUNK_LEN_SECS = float(os.getenv("CANARY_CHUNK_LEN_SECS", "40.0")) +CHUNKED_THRESHOLD_SECS = float(os.getenv("CANARY_CHUNKED_THRESHOLD_SECS", "25.0")) +MODEL_STRIDE = int(os.getenv("CANARY_MODEL_STRIDE", "8")) # FastConformer = 8 + +DTYPE_MAP = { + "fp32": torch.float32, + "fp16": torch.float16, + "bf16": torch.bfloat16, +} +if PRECISION not in DTYPE_MAP: + raise ValueError(f"Unsupported CANARY_PRECISION={PRECISION}; use fp32, fp16, or bf16") + +os.environ.setdefault("HF_HOME", MODELS_DIR) + +log.info("Loading model=%s device=%s precision=%s", MODEL_NAME, DEVICE, PRECISION) +t_load = time.monotonic() +model = ASRModel.from_pretrained(MODEL_NAME) +model = model.to(DEVICE) +if DTYPE_MAP[PRECISION] != torch.float32: + model = model.to(DTYPE_MAP[PRECISION]) +model.eval() +log.info("Model loaded in %.1fs", time.monotonic() - t_load) + +# Prepare config for the buffered/chunked inference path (long audio). +# See CHUNKING.md and NVIDIA-NeMo's speech_to_text_aed_chunked_infer.py. +_model_cfg = copy.deepcopy(model._cfg) +OmegaConf.set_struct(_model_cfg.preprocessor, False) +_model_cfg.preprocessor.dither = 0.0 +_model_cfg.preprocessor.pad_to = 0 +OmegaConf.set_struct(_model_cfg.preprocessor, True) +if _model_cfg.preprocessor.normalize != "per_feature": + log.warning( + "Preprocessor normalize=%s; chunked inference assumes per_feature", + _model_cfg.preprocessor.normalize, + ) +MODEL_STRIDE_SECS = float(_model_cfg.preprocessor["window_stride"]) * MODEL_STRIDE +log.info( + "Long-audio config: chunk_len=%.1fs threshold=%.1fs model_stride=%.4fs", + CHUNK_LEN_SECS, CHUNKED_THRESHOLD_SECS, MODEL_STRIDE_SECS, +) + +app = FastAPI(title="doctate-canary", version="0.1.0") + + +def _transcode_to_wav16k(src_path: str) -> str: + """Convert any audio file to 16 kHz mono WAV. Returns the new path.""" + dst = tempfile.NamedTemporaryFile(suffix=".wav", delete=False) + dst.close() + cmd = [ + "ffmpeg", "-y", "-loglevel", "error", + "-i", src_path, + "-ac", "1", "-ar", "16000", + "-f", "wav", + dst.name, + ] + proc = subprocess.run(cmd, capture_output=True, text=True) + if proc.returncode != 0: + os.unlink(dst.name) + raise HTTPException(status_code=400, detail=f"ffmpeg failed: {proc.stderr.strip()}") + return dst.name + + +def _wav_duration(path: str) -> float: + with wave.open(path, "rb") as w: + return w.getnframes() / float(w.getframerate()) + + +def _transcribe_buffered( + wav_path: str, + duration: float, + language: str, + pnc: str, + timestamps: bool, +): + """Long-audio path: explicit chunked inference via FrameBatchMultiTaskAED. + + Writes a temporary single-line JSONL manifest because language and pnc + must be passed via manifest fields when using the buffered API. + """ + manifest_tmp = tempfile.NamedTemporaryFile(suffix=".json", mode="w", delete=False) + try: + json.dump( + { + "audio_filepath": wav_path, + "duration": duration, + "taskname": "asr", + "source_lang": language, + "target_lang": language, + "pnc": pnc, + }, + manifest_tmp, + ) + manifest_tmp.write("\n") + manifest_tmp.close() + + frame_asr = FrameBatchMultiTaskAED( + asr_model=model, + frame_len=CHUNK_LEN_SECS, + total_buffer=CHUNK_LEN_SECS, + batch_size=8, + ) + amp_dtype = DTYPE_MAP[PRECISION] + use_amp = amp_dtype != torch.float32 + with torch.amp.autocast(model.device.type, enabled=use_amp, dtype=amp_dtype): + with torch.no_grad(): + hyps = get_buffered_pred_feat_multitaskAED( + frame_asr, + _model_cfg.preprocessor, + MODEL_STRIDE_SECS, + model.device, + manifest=manifest_tmp.name, + filepaths=None, + timestamps=timestamps, + ) + return hyps + finally: + try: + os.unlink(manifest_tmp.name) + except OSError: + pass + + +@app.get("/health") +def health(): + return {"status": "ok", "model": MODEL_NAME, "device": DEVICE, "precision": PRECISION} + + +@app.get("/info") +def info(): + return { + "model": MODEL_NAME, + "device": DEVICE, + "precision": PRECISION, + "models_dir": MODELS_DIR, + } + + +@app.post("/inference") +async def inference( + file: UploadFile = File(...), + language: str = Form("de"), + pnc: str = Form("yes"), + timestamps: str = Form("no"), + response_format: str = Form("text"), +): + if language not in {"en", "de", "fr", "es"}: + raise HTTPException(status_code=400, detail=f"Unsupported language: {language}") + if pnc not in {"yes", "no"}: + raise HTTPException(status_code=400, detail="pnc must be yes or no") + if timestamps not in {"yes", "no"}: + raise HTTPException(status_code=400, detail="timestamps must be yes or no") + if response_format not in {"text", "json"}: + raise HTTPException(status_code=400, detail="response_format must be text or json") + + upload_suffix = os.path.splitext(file.filename or "")[1] or ".bin" + upload_tmp = tempfile.NamedTemporaryFile(suffix=upload_suffix, delete=False) + wav_path: Optional[str] = None + try: + upload_tmp.write(await file.read()) + upload_tmp.flush() + upload_tmp.close() + + wav_path = _transcode_to_wav16k(upload_tmp.name) + duration = _wav_duration(wav_path) + ts_flag = timestamps == "yes" + + t0 = time.monotonic() + chunked = duration > CHUNKED_THRESHOLD_SECS + if chunked: + hyps = _transcribe_buffered(wav_path, duration, language, pnc, ts_flag) + else: + hyps = model.transcribe( + [wav_path], + source_lang=language, + target_lang=language, + pnc=pnc, + timestamps=ts_flag, + ) + infer_secs = time.monotonic() - t0 + + if not hyps: + raise HTTPException(status_code=500, detail="Empty transcription result") + hyp = hyps[0] + text = (getattr(hyp, "text", None) or str(hyp)).strip() + + log.info(json.dumps({ + "event": "transcribe", + "duration_audio": round(duration, 2), + "duration_infer": round(infer_secs, 2), + "language": language, + "chars": len(text), + "chunked": chunked, + })) + + if response_format == "json": + payload = { + "text": text, + "language": language, + "duration": duration, + } + if timestamps == "yes": + ts = getattr(hyp, "timestamp", None) + if ts: + payload["segments"] = ts.get("segment", []) + return JSONResponse(payload) + return PlainTextResponse(text) + finally: + for p in (upload_tmp.name, wav_path): + if p and os.path.exists(p): + try: + os.unlink(p) + except OSError: + pass diff --git a/canary/requirements.txt b/canary/requirements.txt new file mode 100644 index 0000000..2b2b026 --- /dev/null +++ b/canary/requirements.txt @@ -0,0 +1,5 @@ +nemo_toolkit[asr]>=2.7.3,<3.0 +huggingface_hub>=0.26.0,<1.0 +fastapi==0.115.0 +uvicorn[standard]==0.30.6 +python-multipart==0.0.9