feat: Add standalone canary service

This commit introduces a new standalone FastAPI service for the NVIDIA
Canary ASR model.

Key changes include:
- A new `canary/` directory containing the service code.
- `Dockerfile`: Defines the Docker image for the Canary service.
- `docker-compose.yml`: Configures the Docker Compose setup for running
  the service.
- `main.py`: Implements the FastAPI application, model loading, and
  inference logic.
- `requirements.txt`: Lists the Python dependencies for the service.
- `CHUNKING.md`: Documents the long-form audio handling strategy for
  Canary.
- `README.md`: Provides an overview of the service, API, and deployment
  instructions.
- `deploy.sh`: A script for deploying the service to a remote host.

This service allows for independent evaluation and deployment of the
Canary ASR model, separate from the existing `doctate-whisper` service.
It utilizes a buffered inference approach for handling long audio files,
as detailed in `CHUNKING.md`.

canary/CHUNKING.md

@@ -0,0 +1,233 @@
+# Long-form audio handling in canary-1b-v2
+
+## TL;DR
+
+- Canary-1b-v2 truncates audio > ~40 s on the default `model.transcribe([path])` call.
+- The fix is **NeMo ≥ 2.5**: long-form inference with overlapping-window
+  auto-chunking ships in `transcribe()` itself, triggered by
+  `batch_size=1` or single-file input.
+- The explicit `FrameBatchMultiTaskAED` path in NeMo `2.4.x` is broken for
+  `canary-1b-v2` (NeMo issue #14544: `CanaryBPETokenizer` lacks the
+  `vocabulary` attribute the streaming-utils class requires).
+- Pin: `nemo_toolkit[asr]>=2.7.3,<3.0` in `requirements.txt`.
+- **Currently used in this codebase**: Path 2 (`FrameBatchMultiTaskAED`),
+  branched on `duration > 25 s`. Empirically verified end-to-end on
+  `experiments/case_c414cf52/`. Path 1 was not retested after the NeMo
+  upgrade — see Decision below.
+- **VRAM cost**: ~9.7 GB on RTX 3060 with `batch_size=8`. Canary cannot
+  share a 12 GB GPU with `doctate-whisper` (~3.9 GB) — sequential operation
+  required, or reduce `FrameBatchMultiTaskAED.batch_size`.
+
+## Problem
+
+The default `model.transcribe([audio_path])` path silently truncates audio
+longer than roughly 40 s. Observed empirically in our
+`experiments/case_c414cf52/asr_comparison.md` benchmark with NeMo 2.4.1:
+audios at 66 s, 78 s, and 81 s lost their final 25–40 s of content. The
+model emits no warning, container logs are clean — only the output length
+reveals the truncation.
+
+The HuggingFace model card claims "automatic chunking enabled at ~40 s with
+1 s overlap." That claim is true — but **only in NeMo ≥ 2.5**, and only on
+specific call shapes (single-file or `batch_size=1`). Our pinned
+`nemo_toolkit==2.4.1` did not have the feature merged into a release yet.
+
+## Why it happens
+
+Canary-1b-v2's FastConformer encoder has a fixed positional-embedding
+window. When the input exceeds it, the encoder produces features only for
+the first window. The decoder emits text up to whatever it generated from
+those features and then stops. No exception, no warning.
+
+The official NeMo answer (per maintainer @nithinraok in
+[issue #14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544)):
+
+> can you switch to main branch. Its not part of release version yet.
+
+That comment dates from August 2025, when canary-1b-v2 was new and only the
+`main` branch had the long-form handling. By NeMo 2.5 it shipped to
+release; by 2.7.x (our current pin) it is stable.
+
+## Path 1 — auto-chunking via `transcribe()` (recommended, NeMo ≥ 2.5)
+
+The `transcribe()` method itself supports dynamic chunking with overlapping
+windows in NeMo ≥ 2.5. It activates automatically under either condition:
+
+- a single audio file passed as a string (not a list), OR
+- `batch_size=1` set explicitly
+
+Quote from NVIDIA-NeMo's
+[official example docstring](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py):
+
+> Canary-1b-v2 supports long‑form inference via the `.transcribe()` method.
+> It will use dynamic chunking with overlapping windows for better
+> performance. This behavior is enabled automatically for long‑form
+> inference when transcribing a single audio file or when batch_size is
+> set to 1.
+
+Minimal call:
+
+```python
+results = model.transcribe(
+    [wav_path],
+    source_lang="de",
+    target_lang="de",
+    pnc="yes",
+    batch_size=1,        # triggers long-form path
+)
+text = results[0].text
+```
+
+This path is preferred: it is shorter, has no manifest plumbing, and uses
+NeMo's tested overlapping-window strategy.
+
+## Path 2 — explicit `FrameBatchMultiTaskAED` (fallback)
+
+For batch inference over many files, manifest-driven workflows, or when
+fine control over chunk length is needed, NeMo exposes
+`FrameBatchMultiTaskAED` + `get_buffered_pred_feat_multitaskAED`.
+
+**Caveat — broken in NeMo 2.4.x for canary-1b-v2**:
+`FrameBatchMultiTaskAED.__init__` calls
+`len(asr_model.tokenizer.vocabulary)`, but `CanaryBPETokenizer` (the
+tokenizer of canary-1b-v2) has no `vocabulary` attribute. NeMo issue
+[#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544) tracks this. In
+`v2.7.3` the same line was extended to fall back on `tokenizer.vocab_size`
+first, so this path is **fixed from NeMo 2.5 onward**. Verified by diffing
+`streaming_utils.py:740–760` between v2.4.1 and v2.7.3.
+
+Key parameters from the official example:
+
+| Parameter | Default | Notes |
+|---|---|---|
+| `chunk_len_in_secs` | `40.0` | window length |
+| `total_buffer` | same as `frame_len` | overlap budget |
+| `batch_size` | `8` | chunks processed in parallel |
+| `model_stride` | `8` | FastConformer downsampling factor |
+
+Pre-conditions on the model config (set once at startup):
+
+```python
+import copy
+from omegaconf import OmegaConf
+
+_model_cfg = copy.deepcopy(model._cfg)
+OmegaConf.set_struct(_model_cfg.preprocessor, False)
+_model_cfg.preprocessor.dither = 0.0
+_model_cfg.preprocessor.pad_to = 0
+OmegaConf.set_struct(_model_cfg.preprocessor, True)
+assert _model_cfg.preprocessor.normalize == "per_feature"
+```
+
+Setup pattern (condensed from the official script):
+
+```python
+from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED
+from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED
+
+frame_asr = FrameBatchMultiTaskAED(
+    asr_model=model,
+    frame_len=40.0,
+    total_buffer=40.0,
+    batch_size=8,
+)
+feature_stride = _model_cfg.preprocessor["window_stride"]
+model_stride_in_secs = feature_stride * 8  # 0.01 * 8 = 0.08 for FastConformer
+
+with torch.amp.autocast(model.device.type, enabled=True, dtype=torch.bfloat16):
+    with torch.no_grad():
+        hyps = get_buffered_pred_feat_multitaskAED(
+            frame_asr,
+            _model_cfg.preprocessor,
+            model_stride_in_secs,
+            model.device,
+            manifest=manifest_path,   # JSONL with audio_filepath, source_lang, target_lang, pnc, taskname
+            filepaths=None,
+            timestamps=False,
+        )
+```
+
+For German ASR, the manifest is required — `filepaths=`-only mode defaults
+to English. One JSON object per line:
+
+```json
+{"audio_filepath": "/path/audio.wav", "duration": 80.9, "taskname": "asr",
+ "source_lang": "de", "target_lang": "de", "pnc": "yes"}
+```
+
+## Caveat — timestamps require shorter chunks
+
+When `timestamps=True`, the official script forces `chunk_len_in_secs=10.0`:
+
+> When `timestamps` is True, it's recommended to use `chunk_len_in_secs=10.0`
+> for optimal results.
+
+Larger windows produce less reliable per-segment timestamps. If we expose
+timestamps in `/inference`, chunk length must drop to 10 s for that mode.
+
+## Decision — what this codebase actually uses
+
+**Path 2 (`FrameBatchMultiTaskAED`)** is implemented in `main.py`
+(`_transcribe_buffered`), branched on `duration > CHUNKED_THRESHOLD_SECS`
+(default 25 s). It was implemented *before* we upgraded NeMo, when Path 1
+was unavailable in 2.4.1. After upgrading to 2.7.3 we kept Path 2 because
+it is the path we have empirical evidence for.
+
+**Path 1 was not retested** after the NeMo upgrade. It would likely work
+in 2.7.3 — the official docs and example script promise so — and would
+let us delete `_transcribe_buffered` and the manifest plumbing. Reasons
+to revisit Path 1 later:
+
+- Code simplification (~50 lines of `main.py` could go away).
+- Lower VRAM (Path 2's `batch_size=8` reserves buffers for 8 parallel
+  chunks; Path 1's overlapping-window strategy may use less).
+- One fewer setup-per-call (FrameBatchMultiTaskAED is currently
+  instantiated each request).
+
+Reasons to keep Path 2:
+
+- It works, with measurements and full transcripts.
+- Explicit control over `chunk_len_in_secs` (currently 40 s).
+- The fallback for short audio (`model.transcribe([wav], ...)` for
+  `duration ≤ 25 s`) is still useful for low latency on short clips.
+
+The migration risk is non-zero (we'd need to re-run the case_c414cf52
+benchmark to confirm output quality), so it's a "later" task rather than
+a "now" task.
+
+## What we ruled out
+
+- **Hard fixed-cut chunking** at second boundaries (the approach in
+  `EvilFreelancer/docker-canary-serve`, using `wave.readframes()` with no
+  overlap): cuts mid-word at chunk boundaries. NeMo's overlapping-window
+  paths are strictly better and exist now in release versions.
+- **Streaming variant** (`speech_to_text_aed_streaming_infer.py`, ~20 KB):
+  cache-aware streaming for online use cases. Overkill for our offline
+  transcription workflow.
+- **Monkey-patching `tokenizer.vocabulary`** to work around the NeMo 2.4.x
+  bug. Possible but a code smell — the upstream fix in 2.5+ is the right
+  solution.
+
+## Empirical history (this codebase)
+
+| Date | Attempt | Result |
+|---|---|---|
+| 2026-04-29 | NeMo 2.4.1 + `model.transcribe([wav], ...)` | Truncation > 50 s |
+| 2026-04-29 | + `batch_size=1` | No change — feature not in 2.4.1 |
+| 2026-04-30 | + explicit `FrameBatchMultiTaskAED` (Path 2) on 2.4.1 | `AttributeError: CanaryBPETokenizer.vocabulary` (NeMo bug) |
+| 2026-04-30 | NeMo bumped to ≥ 2.7.3, Path 2 active | ✓ all 6 audios fully transcribed; ~2 s inference per audio (~30× realtime); 9.7 GB VRAM |
+
+## References
+
+- NeMo issue tracking the 2.4.x tokenizer bug:
+  [#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544)
+- Official NVIDIA NeMo Canary chunked/streaming docs:
+  <https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/streaming_decoding/canary_chunked_and_streaming_decoding.html>
+- Official chunked AED inference example:
+  [`speech_to_text_aed_chunked_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py)
+- Streaming variant:
+  [`speech_to_text_aed_streaming_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_streaming_infer.py)
+- HuggingFace model card: <https://huggingface.co/nvidia/canary-1b-v2>
+- Truncation evidence in this repo: `experiments/case_c414cf52/asr_comparison.md`
+- Reference project we initially looked at (we did not adopt their cut
+  strategy): [`EvilFreelancer/docker-canary-serve`](https://github.com/EvilFreelancer/docker-canary-serve)

canary/Dockerfile

@@ -0,0 +1,32 @@
+FROM nvidia/cuda:12.6.1-devel-ubuntu22.04
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    PIP_NO_CACHE_DIR=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    python3 python3-pip python3-dev \
+    ffmpeg \
+    libsndfile1 \
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --upgrade pip && \
+    pip install -r requirements.txt
+
+COPY main.py .
+
+ENV CANARY_MODEL=nvidia/canary-1b-v2 \
+    CANARY_DEVICE=cuda \
+    CANARY_PRECISION=bf16 \
+    CANARY_MODELS_DIR=/models \
+    HF_HOME=/models
+
+VOLUME ["/models"]
+EXPOSE 9002
+
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9002", "--workers", "1"]

canary/README.md

@@ -0,0 +1,74 @@
+# doctate-canary
+
+Standalone FastAPI wrapper around [NVIDIA Canary](https://huggingface.co/nvidia/canary-1b-v2)
+ASR models, served via the NeMo toolkit.
+
+## Why a separate service?
+
+This is **not** a drop-in replacement for `doctate-whisper`. The Axum server
+keeps speaking the ahmetoner `/asr` interface to Whisper. This container exists
+to evaluate Canary 1B v2 on real medical-German dictations side-by-side with
+Whisper, before any decision about pipeline integration.
+
+Inspired by `EvilFreelancer/docker-canary-serve` — written from scratch, MIT-friendly.
+
+## API
+
+`GET /health` → liveness JSON.
+
+`GET /info` → model name, device, precision, models dir.
+
+`POST /inference` (multipart/form-data):
+
+| Field | Type | Default | Notes |
+|---|---|---|---|
+| `file` | upload | — | any audio format (mp3, m4a, opus, wav). ffmpeg transcodes to 16 kHz mono WAV internally. |
+| `language` | text | `de` | one of `en`, `de`, `fr`, `es` |
+| `pnc` | text | `yes` | punctuation/capitalization (`yes`/`no`) |
+| `timestamps` | text | `no` | segment-level timestamps in `json` mode (`yes`/`no`) |
+| `response_format` | text | `text` | `text` → plain transcript; `json` → `{text, language, duration, segments?}` |
+
+## Env vars
+
+| Var | Default | Purpose |
+|---|---|---|
+| `CANARY_MODEL` | `nvidia/canary-1b-v2` | any NeMo ASR model name |
+| `CANARY_DEVICE` | `cuda` | `cuda` or `cpu` |
+| `CANARY_PRECISION` | `bf16` | `bf16` / `fp16` / `fp32` |
+| `CANARY_MODELS_DIR` | `/models` | HF cache (volume-persistent) |
+| `HF_HOME` | (auto) | mirrored from `CANARY_MODELS_DIR` |
+
+## VRAM
+
+Canary 1B v2 in bf16 fits comfortably in <5 GB. On a 12 GB GPU it can coexist
+with `doctate-whisper` (~3 GB) but **not** also with Ollama running a 9 GB
+model. For evaluation runs, stop one of the others if VRAM gets tight.
+
+## Build & deploy
+
+```bash
+cd doctate/canary
+./deploy.sh minerva.lan
+```
+
+The script rsyncs the directory to `/opt/stacks/doctate-canary/`, rebuilds
+the image on the remote, and polls `/health` for up to 10 min (first start
+downloads ~3 GB and warms up the model).
+
+## Smoke test
+
+```bash
+curl http://minerva.lan:9002/health
+
+curl -F file=@dictation.m4a -F language=de -F response_format=text \
+  http://minerva.lan:9002/inference
+
+curl -F file=@dictation.m4a -F language=de -F timestamps=yes \
+  -F response_format=json http://minerva.lan:9002/inference | jq .
+```
+
+## License
+
+The wrapper code is project-internal. The Canary 1B v2 model weights are
+distributed by NVIDIA under CC-BY-4.0 (commercial use permitted, attribution
+required).

canary/deploy.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Deploy the canary service to a remote host via rsync + ssh.
+# Usage: ./deploy.sh [host]  (default: minerva.lan)
+set -euo pipefail
+
+HOST="${1:-minerva.lan}"
+REMOTE_DIR="/opt/stacks/doctate-canary"
+LOCAL_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+echo ">>> Syncing $LOCAL_DIR -> $HOST:$REMOTE_DIR"
+rsync -av \
+  --exclude 'deploy.sh' \
+  --exclude 'models/' \
+  "$LOCAL_DIR/" "$HOST:$REMOTE_DIR/"
+
+echo ">>> Rebuilding + restarting on $HOST"
+ssh "$HOST" "cd $REMOTE_DIR && \
+  docker compose down && \
+  docker build -t doctate-canary . && \
+  docker compose up -d"
+
+echo ">>> Waiting for /health (up to 10 min for first model download + warmup)"
+for i in $(seq 1 120); do
+  if ssh "$HOST" "curl -sf http://localhost:9002/health" 2>/dev/null; then
+    echo
+    echo ">>> OK"
+    ssh "$HOST" "curl -s http://localhost:9002/info"
+    echo
+    exit 0
+  fi
+  sleep 5
+done
+
+echo ">>> TIMEOUT — last logs:"
+ssh "$HOST" "docker logs doctate-canary --tail 60"
+exit 1

canary/docker-compose.yml

@@ -0,0 +1,22 @@
+services:
+  doctate-canary:
+    image: doctate-canary:latest
+    container_name: doctate-canary
+    environment:
+      - CANARY_MODEL=nvidia/canary-1b-v2
+      - CANARY_PRECISION=bf16
+      - NVIDIA_VISIBLE_DEVICES=all
+      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
+    volumes:
+      - /opt/stacks/doctate-canary/models:/models
+    ports:
+      - "9002:9002"
+    shm_size: 2g
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]
+    restart: unless-stopped

canary/main.py

@@ -0,0 +1,246 @@
+"""FastAPI wrapper around NVIDIA NeMo Canary ASR models.
+
+Native Canary HTTP API (POST /inference). Not a Whisper /asr clone.
+Inspired by EvilFreelancer/docker-canary-serve, written from scratch.
+"""
+import copy
+import json
+import logging
+import os
+import subprocess
+import tempfile
+import time
+import wave
+from typing import Optional
+
+import torch
+from fastapi import FastAPI, File, Form, HTTPException, UploadFile
+from fastapi.responses import JSONResponse, PlainTextResponse
+from nemo.collections.asr.models import ASRModel
+from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED
+from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED
+from omegaconf import OmegaConf
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(levelname)s %(message)s",
+)
+log = logging.getLogger("canary")
+
+MODEL_NAME = os.getenv("CANARY_MODEL", "nvidia/canary-1b-v2")
+DEVICE = os.getenv("CANARY_DEVICE", "cuda")
+PRECISION = os.getenv("CANARY_PRECISION", "bf16").lower()
+MODELS_DIR = os.getenv("CANARY_MODELS_DIR", "/models")
+CHUNK_LEN_SECS = float(os.getenv("CANARY_CHUNK_LEN_SECS", "40.0"))
+CHUNKED_THRESHOLD_SECS = float(os.getenv("CANARY_CHUNKED_THRESHOLD_SECS", "25.0"))
+MODEL_STRIDE = int(os.getenv("CANARY_MODEL_STRIDE", "8"))  # FastConformer = 8
+
+DTYPE_MAP = {
+    "fp32": torch.float32,
+    "fp16": torch.float16,
+    "bf16": torch.bfloat16,
+}
+if PRECISION not in DTYPE_MAP:
+    raise ValueError(f"Unsupported CANARY_PRECISION={PRECISION}; use fp32, fp16, or bf16")
+
+os.environ.setdefault("HF_HOME", MODELS_DIR)
+
+log.info("Loading model=%s device=%s precision=%s", MODEL_NAME, DEVICE, PRECISION)
+t_load = time.monotonic()
+model = ASRModel.from_pretrained(MODEL_NAME)
+model = model.to(DEVICE)
+if DTYPE_MAP[PRECISION] != torch.float32:
+    model = model.to(DTYPE_MAP[PRECISION])
+model.eval()
+log.info("Model loaded in %.1fs", time.monotonic() - t_load)
+
+# Prepare config for the buffered/chunked inference path (long audio).
+# See CHUNKING.md and NVIDIA-NeMo's speech_to_text_aed_chunked_infer.py.
+_model_cfg = copy.deepcopy(model._cfg)
+OmegaConf.set_struct(_model_cfg.preprocessor, False)
+_model_cfg.preprocessor.dither = 0.0
+_model_cfg.preprocessor.pad_to = 0
+OmegaConf.set_struct(_model_cfg.preprocessor, True)
+if _model_cfg.preprocessor.normalize != "per_feature":
+    log.warning(
+        "Preprocessor normalize=%s; chunked inference assumes per_feature",
+        _model_cfg.preprocessor.normalize,
+    )
+MODEL_STRIDE_SECS = float(_model_cfg.preprocessor["window_stride"]) * MODEL_STRIDE
+log.info(
+    "Long-audio config: chunk_len=%.1fs threshold=%.1fs model_stride=%.4fs",
+    CHUNK_LEN_SECS, CHUNKED_THRESHOLD_SECS, MODEL_STRIDE_SECS,
+)
+
+app = FastAPI(title="doctate-canary", version="0.1.0")
+
+
+def _transcode_to_wav16k(src_path: str) -> str:
+    """Convert any audio file to 16 kHz mono WAV. Returns the new path."""
+    dst = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
+    dst.close()
+    cmd = [
+        "ffmpeg", "-y", "-loglevel", "error",
+        "-i", src_path,
+        "-ac", "1", "-ar", "16000",
+        "-f", "wav",
+        dst.name,
+    ]
+    proc = subprocess.run(cmd, capture_output=True, text=True)
+    if proc.returncode != 0:
+        os.unlink(dst.name)
+        raise HTTPException(status_code=400, detail=f"ffmpeg failed: {proc.stderr.strip()}")
+    return dst.name
+
+
+def _wav_duration(path: str) -> float:
+    with wave.open(path, "rb") as w:
+        return w.getnframes() / float(w.getframerate())
+
+
+def _transcribe_buffered(
+    wav_path: str,
+    duration: float,
+    language: str,
+    pnc: str,
+    timestamps: bool,
+):
+    """Long-audio path: explicit chunked inference via FrameBatchMultiTaskAED.
+
+    Writes a temporary single-line JSONL manifest because language and pnc
+    must be passed via manifest fields when using the buffered API.
+    """
+    manifest_tmp = tempfile.NamedTemporaryFile(suffix=".json", mode="w", delete=False)
+    try:
+        json.dump(
+            {
+                "audio_filepath": wav_path,
+                "duration": duration,
+                "taskname": "asr",
+                "source_lang": language,
+                "target_lang": language,
+                "pnc": pnc,
+            },
+            manifest_tmp,
+        )
+        manifest_tmp.write("\n")
+        manifest_tmp.close()
+
+        frame_asr = FrameBatchMultiTaskAED(
+            asr_model=model,
+            frame_len=CHUNK_LEN_SECS,
+            total_buffer=CHUNK_LEN_SECS,
+            batch_size=8,
+        )
+        amp_dtype = DTYPE_MAP[PRECISION]
+        use_amp = amp_dtype != torch.float32
+        with torch.amp.autocast(model.device.type, enabled=use_amp, dtype=amp_dtype):
+            with torch.no_grad():
+                hyps = get_buffered_pred_feat_multitaskAED(
+                    frame_asr,
+                    _model_cfg.preprocessor,
+                    MODEL_STRIDE_SECS,
+                    model.device,
+                    manifest=manifest_tmp.name,
+                    filepaths=None,
+                    timestamps=timestamps,
+                )
+        return hyps
+    finally:
+        try:
+            os.unlink(manifest_tmp.name)
+        except OSError:
+            pass
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok", "model": MODEL_NAME, "device": DEVICE, "precision": PRECISION}
+
+
+@app.get("/info")
+def info():
+    return {
+        "model": MODEL_NAME,
+        "device": DEVICE,
+        "precision": PRECISION,
+        "models_dir": MODELS_DIR,
+    }
+
+
+@app.post("/inference")
+async def inference(
+    file: UploadFile = File(...),
+    language: str = Form("de"),
+    pnc: str = Form("yes"),
+    timestamps: str = Form("no"),
+    response_format: str = Form("text"),
+):
+    if language not in {"en", "de", "fr", "es"}:
+        raise HTTPException(status_code=400, detail=f"Unsupported language: {language}")
+    if pnc not in {"yes", "no"}:
+        raise HTTPException(status_code=400, detail="pnc must be yes or no")
+    if timestamps not in {"yes", "no"}:
+        raise HTTPException(status_code=400, detail="timestamps must be yes or no")
+    if response_format not in {"text", "json"}:
+        raise HTTPException(status_code=400, detail="response_format must be text or json")
+
+    upload_suffix = os.path.splitext(file.filename or "")[1] or ".bin"
+    upload_tmp = tempfile.NamedTemporaryFile(suffix=upload_suffix, delete=False)
+    wav_path: Optional[str] = None
+    try:
+        upload_tmp.write(await file.read())
+        upload_tmp.flush()
+        upload_tmp.close()
+
+        wav_path = _transcode_to_wav16k(upload_tmp.name)
+        duration = _wav_duration(wav_path)
+        ts_flag = timestamps == "yes"
+
+        t0 = time.monotonic()
+        chunked = duration > CHUNKED_THRESHOLD_SECS
+        if chunked:
+            hyps = _transcribe_buffered(wav_path, duration, language, pnc, ts_flag)
+        else:
+            hyps = model.transcribe(
+                [wav_path],
+                source_lang=language,
+                target_lang=language,
+                pnc=pnc,
+                timestamps=ts_flag,
+            )
+        infer_secs = time.monotonic() - t0
+
+        if not hyps:
+            raise HTTPException(status_code=500, detail="Empty transcription result")
+        hyp = hyps[0]
+        text = (getattr(hyp, "text", None) or str(hyp)).strip()
+
+        log.info(json.dumps({
+            "event": "transcribe",
+            "duration_audio": round(duration, 2),
+            "duration_infer": round(infer_secs, 2),
+            "language": language,
+            "chars": len(text),
+            "chunked": chunked,
+        }))
+
+        if response_format == "json":
+            payload = {
+                "text": text,
+                "language": language,
+                "duration": duration,
+            }
+            if timestamps == "yes":
+                ts = getattr(hyp, "timestamp", None)
+                if ts:
+                    payload["segments"] = ts.get("segment", [])
+            return JSONResponse(payload)
+        return PlainTextResponse(text)
+    finally:
+        for p in (upload_tmp.name, wav_path):
+            if p and os.path.exists(p):
+                try:
+                    os.unlink(p)
+                except OSError:
+                    pass

canary/requirements.txt

@@ -0,0 +1,5 @@
+nemo_toolkit[asr]>=2.7.3,<3.0
+huggingface_hub>=0.26.0,<1.0
+fastapi==0.115.0
+uvicorn[standard]==0.30.6
+python-multipart==0.0.9