Add custom faster-whisper service with anti-hallucination params
Off-the-shelf Whisper wrappers hardcode faster-whisper defaults that cause cascading hallucinations on real dictations (2/13 runs against ahmetoner v1.9.1). This thin FastAPI service mirrors the /asr interface but fixes condition_on_previous_text=False, temperature=0.0, and vad_filter=True so the Axum server can switch by changing WHISPER_URL.
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
python3 python3-pip ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY main.py .
|
||||
|
||||
ENV WHISPER_MODEL=large-v3-turbo \
|
||||
WHISPER_COMPUTE_TYPE=float16 \
|
||||
WHISPER_MODELS_DIR=/models
|
||||
|
||||
VOLUME ["/models"]
|
||||
EXPOSE 9001
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "9001", "--workers", "1"]
|
||||
@@ -0,0 +1,101 @@
|
||||
# doctate-whisper
|
||||
|
||||
Thin FastAPI wrapper around [faster-whisper](https://github.com/SYSTRAN/faster-whisper).
|
||||
|
||||
## Why a custom service?
|
||||
|
||||
Existing Whisper HTTP wrappers (`ahmetoner/whisper-asr-webservice`,
|
||||
`speaches`, `linuxserver/faster-whisper`, `hwdsl2/docker-whisper`) hardcode
|
||||
the faster-whisper defaults. Those defaults include:
|
||||
|
||||
- `condition_on_previous_text=True` — classic cascading-hallucination source
|
||||
- temperature fallback cascade `[0.0, 0.2, …, 1.0]` — when confidence drops,
|
||||
Whisper gets "creative"
|
||||
|
||||
On real cardiology dictations we saw **2/13 runs with catastrophic
|
||||
hallucinations** (Russian/Chinese fragments, repetition loops) using
|
||||
ahmetoner's defaults. None of the available wrappers expose these
|
||||
parameters as request fields or env vars.
|
||||
|
||||
This service fixes them to safe values that never change.
|
||||
|
||||
## Anti-hallucination params (fixed)
|
||||
|
||||
| Param | Value | Reason |
|
||||
|---|---|---|
|
||||
| `temperature` | `0.0` | No fallback cascade; deterministic output |
|
||||
| `condition_on_previous_text` | `False` | Prevents cascading hallucinations |
|
||||
| `vad_filter` | `True` | Silence → no transcription (not garbage) |
|
||||
| `vad_parameters.min_silence_duration_ms` | `500` | VAD sensitivity |
|
||||
| `no_speech_threshold` | `0.6` | Drop silent segments |
|
||||
| `log_prob_threshold` | `-1.0` | Drop low-confidence segments |
|
||||
| `compression_ratio_threshold` | `2.4` | Anti-repetition |
|
||||
| `beam_size` / `best_of` | `5` / `5` | Default quality beam search |
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `POST /asr` — multipart `audio_file`, query `output=txt|json`, `language=de`, form `initial_prompt`
|
||||
- `GET /health` — liveness: `{"status":"ok","model":...,"device":...}`
|
||||
- `GET /info` — debug: model, device, compute_type, offline flag
|
||||
|
||||
Compatible with ahmetoner's `/asr` interface so the Axum server needs no change.
|
||||
|
||||
## Env vars
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `WHISPER_MODEL` | `large-v3-turbo` | Any faster-whisper model name |
|
||||
| `WHISPER_COMPUTE_TYPE` | `float16` | `float16`, `int8_float16`, `int8` (CPU) |
|
||||
| `WHISPER_DEVICE` | `cuda` | `cuda` or `cpu` |
|
||||
| `WHISPER_MODELS_DIR` | `/models` | Cache location |
|
||||
| `WHISPER_OFFLINE` | `0` | `1` → no HuggingFace download |
|
||||
|
||||
## Model choice
|
||||
|
||||
**Default is `large-v3-turbo`**: same 32-layer encoder as `large-v3`,
|
||||
only decoder distilled to 4 layers. OpenAI benchmarks: equal quality on
|
||||
European languages. ~1.6 GB VRAM vs. ~3 GB — leaves room for Ollama
|
||||
(gemma4, 9 GB) in a 12 GB card.
|
||||
|
||||
Switch to `large-v3` at any time via `WHISPER_MODEL=large-v3` + container restart.
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
docker build -t doctate-whisper ./whisper
|
||||
```
|
||||
|
||||
Takes 2–5 minutes. First run of the container pulls ~1.6 GB from
|
||||
HuggingFace into `/models` (volume-persisted).
|
||||
|
||||
## Deploy to minerva via Dockge
|
||||
|
||||
Transport the image without a registry:
|
||||
|
||||
```bash
|
||||
docker save doctate-whisper | ssh minerva 'docker load'
|
||||
```
|
||||
|
||||
Then add the compose file in Dockge and start.
|
||||
|
||||
## Integration with Axum server
|
||||
|
||||
In `server/.env`:
|
||||
|
||||
```
|
||||
WHISPER_URL=http://minerva.lan:9001
|
||||
```
|
||||
|
||||
No other change. The Axum server speaks ahmetoner's interface; we mirror it.
|
||||
|
||||
## Offline mode
|
||||
|
||||
```bash
|
||||
# Pre-pull on a host with internet:
|
||||
docker run --rm -v /opt/stacks/doctate-whisper/models:/models doctate-whisper \
|
||||
python3 -c "from faster_whisper import WhisperModel; WhisperModel('large-v3-turbo', download_root='/models')"
|
||||
|
||||
# Then set in docker-compose.yml:
|
||||
environment:
|
||||
- WHISPER_OFFLINE=1
|
||||
```
|
||||
@@ -0,0 +1,21 @@
|
||||
services:
|
||||
doctate-whisper:
|
||||
image: doctate-whisper:latest
|
||||
container_name: doctate-whisper
|
||||
environment:
|
||||
- WHISPER_MODEL=large-v3-turbo
|
||||
- WHISPER_COMPUTE_TYPE=float16
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
volumes:
|
||||
- /opt/stacks/doctate-whisper/models:/models
|
||||
ports:
|
||||
- "9001:9001"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
restart: unless-stopped
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
"""Thin FastAPI wrapper around faster-whisper.
|
||||
|
||||
Exposes the same /asr interface as ahmetoner/whisper-asr-webservice so the
|
||||
Axum server stays unchanged, but hardcodes anti-hallucination parameters
|
||||
that no off-the-shelf wrapper exposes.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, File, Form, Query, UploadFile
|
||||
from fastapi.responses import JSONResponse, PlainTextResponse
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
log = logging.getLogger("whisper")
|
||||
|
||||
MODEL_NAME = os.getenv("WHISPER_MODEL", "large-v3-turbo")
|
||||
COMPUTE_TYPE = os.getenv("WHISPER_COMPUTE_TYPE", "float16")
|
||||
DEVICE = os.getenv("WHISPER_DEVICE", "cuda")
|
||||
MODELS_DIR = os.getenv("WHISPER_MODELS_DIR", "/models")
|
||||
OFFLINE = os.getenv("WHISPER_OFFLINE", "0") == "1"
|
||||
|
||||
log.info("Loading model=%s device=%s compute_type=%s", MODEL_NAME, DEVICE, COMPUTE_TYPE)
|
||||
model = WhisperModel(
|
||||
MODEL_NAME,
|
||||
device=DEVICE,
|
||||
compute_type=COMPUTE_TYPE,
|
||||
download_root=MODELS_DIR,
|
||||
local_files_only=OFFLINE,
|
||||
)
|
||||
log.info("Model loaded")
|
||||
|
||||
app = FastAPI(title="doctate-whisper", version="0.1.0")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "ok", "model": MODEL_NAME, "device": DEVICE}
|
||||
|
||||
|
||||
@app.get("/info")
|
||||
def info():
|
||||
return {
|
||||
"model": MODEL_NAME,
|
||||
"device": DEVICE,
|
||||
"compute_type": COMPUTE_TYPE,
|
||||
"offline": OFFLINE,
|
||||
}
|
||||
|
||||
|
||||
@app.post("/asr")
|
||||
async def asr(
|
||||
audio_file: UploadFile = File(...),
|
||||
output: str = Query("txt"),
|
||||
language: Optional[str] = Query(None),
|
||||
initial_prompt: Optional[str] = Form(None),
|
||||
):
|
||||
suffix = os.path.splitext(audio_file.filename or "")[1] or ".bin"
|
||||
tmp = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
|
||||
try:
|
||||
tmp.write(await audio_file.read())
|
||||
tmp.flush()
|
||||
tmp.close()
|
||||
|
||||
t0 = time.monotonic()
|
||||
segments_gen, info_obj = model.transcribe(
|
||||
tmp.name,
|
||||
language=language,
|
||||
task="transcribe",
|
||||
beam_size=5,
|
||||
best_of=5,
|
||||
# Fixed anti-hallucination params — the whole point of this service.
|
||||
temperature=0.0,
|
||||
condition_on_previous_text=False,
|
||||
vad_filter=True,
|
||||
vad_parameters={"min_silence_duration_ms": 500},
|
||||
no_speech_threshold=0.6,
|
||||
log_prob_threshold=-1.0,
|
||||
compression_ratio_threshold=2.4,
|
||||
initial_prompt=initial_prompt,
|
||||
)
|
||||
segments = list(segments_gen)
|
||||
infer_secs = time.monotonic() - t0
|
||||
|
||||
text = "".join(s.text for s in segments).strip()
|
||||
worst_logprob = min((s.avg_logprob for s in segments), default=0.0)
|
||||
|
||||
log.info(
|
||||
json.dumps({
|
||||
"event": "transcribe",
|
||||
"duration_audio": info_obj.duration,
|
||||
"duration_infer": round(infer_secs, 2),
|
||||
"segments": len(segments),
|
||||
"worst_logprob": round(worst_logprob, 3),
|
||||
"language": info_obj.language,
|
||||
})
|
||||
)
|
||||
|
||||
if output == "json":
|
||||
return JSONResponse({
|
||||
"text": text,
|
||||
"language": info_obj.language,
|
||||
"duration": info_obj.duration,
|
||||
"segments": [
|
||||
{
|
||||
"start": s.start,
|
||||
"end": s.end,
|
||||
"text": s.text,
|
||||
"avg_logprob": s.avg_logprob,
|
||||
"compression_ratio": s.compression_ratio,
|
||||
"no_speech_prob": s.no_speech_prob,
|
||||
"temperature": s.temperature,
|
||||
}
|
||||
for s in segments
|
||||
],
|
||||
})
|
||||
return PlainTextResponse(text)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
@@ -0,0 +1,5 @@
|
||||
faster-whisper==1.0.3
|
||||
requests==2.32.3
|
||||
fastapi==0.115.0
|
||||
uvicorn[standard]==0.30.6
|
||||
python-multipart==0.0.9
|
||||
Reference in New Issue
Block a user