146 lines
5.5 KiB
Markdown
146 lines
5.5 KiB
Markdown
# doctate-whisper
|
||
|
||
Thin FastAPI wrapper around [faster-whisper](https://github.com/SYSTRAN/faster-whisper).
|
||
|
||
## Why a custom service?
|
||
|
||
Existing Whisper HTTP wrappers (`ahmetoner/whisper-asr-webservice`,
|
||
`speaches`, `linuxserver/faster-whisper`, `hwdsl2/docker-whisper`) hardcode
|
||
the faster-whisper defaults. Those defaults include:
|
||
|
||
- `condition_on_previous_text=True` — classic cascading-hallucination source
|
||
- temperature fallback cascade `[0.0, 0.2, …, 1.0]` — when confidence drops,
|
||
Whisper gets "creative"
|
||
|
||
On real cardiology dictations we saw **2/13 runs with catastrophic
|
||
hallucinations** (Russian/Chinese fragments, repetition loops) using
|
||
ahmetoner's defaults. None of the available wrappers expose these
|
||
parameters as request fields or env vars.
|
||
|
||
This service fixes them to safe values that never change.
|
||
|
||
## Anti-hallucination params (fixed)
|
||
|
||
| Param | Value | Reason |
|
||
|---|---|---|
|
||
| `temperature` | `0.0` | No fallback cascade; deterministic output |
|
||
| `condition_on_previous_text` | `False` | Prevents cascading hallucinations |
|
||
| `vad_filter` | `True` | Silence → no transcription (not garbage) |
|
||
| `vad_parameters.min_silence_duration_ms` | `500` | VAD sensitivity |
|
||
| `no_speech_threshold` | `0.6` | Drop silent segments |
|
||
| `log_prob_threshold` | `-1.0` | Drop low-confidence segments |
|
||
| `compression_ratio_threshold` | `2.4` | Anti-repetition |
|
||
| `beam_size` / `best_of` | `5` / `5` | Default quality beam search |
|
||
|
||
## Endpoints
|
||
|
||
- `POST /asr` — multipart `audio_file`, query `output=txt|json`, `language=de`, form `initial_prompt`
|
||
- `GET /health` — liveness: `{"status":"ok","model":...,"device":...}`
|
||
- `GET /info` — debug: model, device, compute_type, offline flag
|
||
|
||
Compatible with ahmetoner's `/asr` interface so the Axum server needs no change.
|
||
|
||
## Env vars
|
||
|
||
| Var | Default | Purpose |
|
||
|---|---|---|
|
||
| `WHISPER_MODEL` | `large-v3-turbo` | Any faster-whisper model name |
|
||
| `WHISPER_COMPUTE_TYPE` | `float16` | `float16`, `int8_float16`, `int8` (CPU) |
|
||
| `WHISPER_DEVICE` | `cuda` | `cuda` or `cpu` |
|
||
| `WHISPER_MODELS_DIR` | `/models` | Cache location |
|
||
| `WHISPER_OFFLINE` | `0` | `1` → no HuggingFace download |
|
||
|
||
## Model choice
|
||
|
||
**Default is `large-v3-turbo`**: same 32-layer encoder as `large-v3`,
|
||
only decoder distilled to 4 layers. OpenAI benchmarks: equal quality on
|
||
European languages. ~1.6 GB VRAM vs. ~3 GB — leaves room for Ollama
|
||
(gemma4, 9 GB) in a 12 GB card.
|
||
|
||
Switch to `large-v3` at any time via `WHISPER_MODEL=large-v3` + container restart.
|
||
|
||
## Build
|
||
|
||
```bash
|
||
docker build -t doctate-whisper ./whisper
|
||
```
|
||
|
||
Takes 2–5 minutes. First run of the container pulls ~1.6 GB from
|
||
HuggingFace into `/models` (volume-persisted).
|
||
|
||
## Deploy to minerva via Dockge
|
||
|
||
Transport the image without a registry:
|
||
|
||
```bash
|
||
docker save doctate-whisper | ssh minerva 'docker load'
|
||
```
|
||
|
||
Then add the compose file in Dockge and start.
|
||
|
||
## Integration with Axum server
|
||
|
||
In `server/.env`:
|
||
|
||
```
|
||
WHISPER_URL=http://minerva.lan:9001
|
||
```
|
||
|
||
No other change. The Axum server speaks ahmetoner's interface; we mirror it.
|
||
|
||
## Investigated and rejected
|
||
|
||
### `suppress_tokens` for digit-sequence preservation (2026-04-28)
|
||
|
||
Goal: when a dictator says individual digits ("eins null null"), have Whisper
|
||
emit them as words instead of collapsing them to a number ("100"). The
|
||
collapse is irreversible downstream — there is no way for the LLM to
|
||
distinguish "hunderteins" (101 as a number) from "eins null eins" (a 1-0-1
|
||
sequence) once both have become `101` in the transcript.
|
||
|
||
The standard advice
|
||
([openai/whisper Discussion #1041](https://github.com/openai/whisper/discussions/1041))
|
||
is to set `suppress_tokens` to every numeric token in the vocabulary so the
|
||
decoder is forced onto word tokens. We added a `suppress_numerics` form
|
||
field, threaded it through the Axum server and the experiments sandbox,
|
||
and tested it against a real cardiology dictation.
|
||
|
||
**Two filter widths, both pathological:**
|
||
|
||
1. **Jongwook's exact recipe** (`\d+` filter, ~426 tokens, faithful 1:1
|
||
reproduction with `eot` bound and `removeprefix(" ")` filter):
|
||
- Massive end-of-audio repetition loop ("Enoxaparin, das heißt
|
||
Enoxaparin, das heißt Enoxaparin, …" 20+ times). The fixed
|
||
`compression_ratio_threshold=2.4` did not catch it.
|
||
- Numbers also disappear: `40 mg Thorazemit` → `Milligramm Thorazemit`
|
||
(the `40` is dropped, not converted to a word).
|
||
2. **Single-digit only** (`\d` filter, 20 tokens — `0`-`9` with and without
|
||
leading space):
|
||
- Repetition loop is gone (small enough list to keep beam search stable).
|
||
- But dictated sequences disappear entirely: `5mg 100` and `5mg 101`
|
||
become just `Visoprolol` and `Ramipriel` (no dosing schema in any form).
|
||
- Collateral damage on natural numbers: `12,5 mg` → `12,25 mg` (the ` 5`
|
||
token was suppressed; the decoder built the value from multi-digit
|
||
tokens and duplicated a digit).
|
||
|
||
There is no usable middle ground between the two widths for our setup
|
||
(`large-v3` + faster-whisper 1.2.1 + medical German dictation). The status
|
||
quo (`100`/`101` in the transcript, doctor corrects manually) stays.
|
||
|
||
Don't reattempt without changing the underlying ASR (different model,
|
||
different backend, or model-internal training to preserve digit-form
|
||
acoustic information). Memory entry:
|
||
`project_whisper_suppress_tokens_dead_end.md`.
|
||
|
||
## Offline mode
|
||
|
||
```bash
|
||
# Pre-pull on a host with internet:
|
||
docker run --rm -v /opt/stacks/doctate-whisper/models:/models doctate-whisper \
|
||
python3 -c "from faster_whisper import WhisperModel; WhisperModel('large-v3-turbo', download_root='/models')"
|
||
|
||
# Then set in docker-compose.yml:
|
||
environment:
|
||
- WHISPER_OFFLINE=1
|
||
```
|