Brummel
6e6cfd77b7
Off-the-shelf Whisper wrappers hardcode faster-whisper defaults that cause cascading hallucinations on real dictations (2/13 runs against ahmetoner v1.9.1). This thin FastAPI service mirrors the /asr interface but fixes condition_on_previous_text=False, temperature=0.0, and vad_filter=True so the Axum server can switch by changing WHISPER_URL.
102 lines
3.3 KiB
Markdown
102 lines
3.3 KiB
Markdown
# doctate-whisper
|
||
|
||
Thin FastAPI wrapper around [faster-whisper](https://github.com/SYSTRAN/faster-whisper).
|
||
|
||
## Why a custom service?
|
||
|
||
Existing Whisper HTTP wrappers (`ahmetoner/whisper-asr-webservice`,
|
||
`speaches`, `linuxserver/faster-whisper`, `hwdsl2/docker-whisper`) hardcode
|
||
the faster-whisper defaults. Those defaults include:
|
||
|
||
- `condition_on_previous_text=True` — classic cascading-hallucination source
|
||
- temperature fallback cascade `[0.0, 0.2, …, 1.0]` — when confidence drops,
|
||
Whisper gets "creative"
|
||
|
||
On real cardiology dictations we saw **2/13 runs with catastrophic
|
||
hallucinations** (Russian/Chinese fragments, repetition loops) using
|
||
ahmetoner's defaults. None of the available wrappers expose these
|
||
parameters as request fields or env vars.
|
||
|
||
This service fixes them to safe values that never change.
|
||
|
||
## Anti-hallucination params (fixed)
|
||
|
||
| Param | Value | Reason |
|
||
|---|---|---|
|
||
| `temperature` | `0.0` | No fallback cascade; deterministic output |
|
||
| `condition_on_previous_text` | `False` | Prevents cascading hallucinations |
|
||
| `vad_filter` | `True` | Silence → no transcription (not garbage) |
|
||
| `vad_parameters.min_silence_duration_ms` | `500` | VAD sensitivity |
|
||
| `no_speech_threshold` | `0.6` | Drop silent segments |
|
||
| `log_prob_threshold` | `-1.0` | Drop low-confidence segments |
|
||
| `compression_ratio_threshold` | `2.4` | Anti-repetition |
|
||
| `beam_size` / `best_of` | `5` / `5` | Default quality beam search |
|
||
|
||
## Endpoints
|
||
|
||
- `POST /asr` — multipart `audio_file`, query `output=txt|json`, `language=de`, form `initial_prompt`
|
||
- `GET /health` — liveness: `{"status":"ok","model":...,"device":...}`
|
||
- `GET /info` — debug: model, device, compute_type, offline flag
|
||
|
||
Compatible with ahmetoner's `/asr` interface so the Axum server needs no change.
|
||
|
||
## Env vars
|
||
|
||
| Var | Default | Purpose |
|
||
|---|---|---|
|
||
| `WHISPER_MODEL` | `large-v3-turbo` | Any faster-whisper model name |
|
||
| `WHISPER_COMPUTE_TYPE` | `float16` | `float16`, `int8_float16`, `int8` (CPU) |
|
||
| `WHISPER_DEVICE` | `cuda` | `cuda` or `cpu` |
|
||
| `WHISPER_MODELS_DIR` | `/models` | Cache location |
|
||
| `WHISPER_OFFLINE` | `0` | `1` → no HuggingFace download |
|
||
|
||
## Model choice
|
||
|
||
**Default is `large-v3-turbo`**: same 32-layer encoder as `large-v3`,
|
||
only decoder distilled to 4 layers. OpenAI benchmarks: equal quality on
|
||
European languages. ~1.6 GB VRAM vs. ~3 GB — leaves room for Ollama
|
||
(gemma4, 9 GB) in a 12 GB card.
|
||
|
||
Switch to `large-v3` at any time via `WHISPER_MODEL=large-v3` + container restart.
|
||
|
||
## Build
|
||
|
||
```bash
|
||
docker build -t doctate-whisper ./whisper
|
||
```
|
||
|
||
Takes 2–5 minutes. First run of the container pulls ~1.6 GB from
|
||
HuggingFace into `/models` (volume-persisted).
|
||
|
||
## Deploy to minerva via Dockge
|
||
|
||
Transport the image without a registry:
|
||
|
||
```bash
|
||
docker save doctate-whisper | ssh minerva 'docker load'
|
||
```
|
||
|
||
Then add the compose file in Dockge and start.
|
||
|
||
## Integration with Axum server
|
||
|
||
In `server/.env`:
|
||
|
||
```
|
||
WHISPER_URL=http://minerva.lan:9001
|
||
```
|
||
|
||
No other change. The Axum server speaks ahmetoner's interface; we mirror it.
|
||
|
||
## Offline mode
|
||
|
||
```bash
|
||
# Pre-pull on a host with internet:
|
||
docker run --rm -v /opt/stacks/doctate-whisper/models:/models doctate-whisper \
|
||
python3 -c "from faster_whisper import WhisperModel; WhisperModel('large-v3-turbo', download_root='/models')"
|
||
|
||
# Then set in docker-compose.yml:
|
||
environment:
|
||
- WHISPER_OFFLINE=1
|
||
```
|