doctate/whisper

doctate-whisper

Thin FastAPI wrapper around faster-whisper.

Why a custom service?

Existing Whisper HTTP wrappers (ahmetoner/whisper-asr-webservice, speaches, linuxserver/faster-whisper, hwdsl2/docker-whisper) hardcode the faster-whisper defaults. Those defaults include:

• condition_on_previous_text=True — classic cascading-hallucination source
• temperature fallback cascade [0.0, 0.2, …, 1.0] — when confidence drops, Whisper gets "creative"

On real cardiology dictations we saw 2/13 runs with catastrophic hallucinations (Russian/Chinese fragments, repetition loops) using ahmetoner's defaults. None of the available wrappers expose these parameters as request fields or env vars.

This service fixes them to safe values that never change.

Anti-hallucination params (fixed)

Param	Value	Reason
temperature	0.0	No fallback cascade; deterministic output
condition_on_previous_text	False	Prevents cascading hallucinations
vad_filter	True	Silence → no transcription (not garbage)
vad_parameters.min_silence_duration_ms	500	VAD sensitivity
no_speech_threshold	0.6	Drop silent segments
log_prob_threshold	-1.0	Drop low-confidence segments
compression_ratio_threshold	2.4	Anti-repetition
beam_size / best_of	5 / 5	Default quality beam search

Endpoints

• POST /asr — multipart audio_file, query output=txt|json, language=de, form initial_prompt
• GET /health — liveness: {"status":"ok","model":...,"device":...}
• GET /info — debug: model, device, compute_type, offline flag

Compatible with ahmetoner's /asr interface so the Axum server needs no change.

Env vars

Var	Default	Purpose
WHISPER_MODEL	large-v3-turbo	Any faster-whisper model name
WHISPER_COMPUTE_TYPE	float16	float16, int8_float16, int8 (CPU)
WHISPER_DEVICE	cuda	cuda or cpu
WHISPER_MODELS_DIR	/models	Cache location
WHISPER_OFFLINE	0	1 → no HuggingFace download

Model choice

Default is large-v3-turbo: same 32-layer encoder as large-v3, only decoder distilled to 4 layers. OpenAI benchmarks: equal quality on European languages. ~1.6 GB VRAM vs. ~3 GB — leaves room for Ollama (gemma4, 9 GB) in a 12 GB card.

Switch to large-v3 at any time via WHISPER_MODEL=large-v3 + container restart.

Build

docker build -t doctate-whisper ./whisper

Takes 2–5 minutes. First run of the container pulls ~1.6 GB from HuggingFace into /models (volume-persisted).

Deploy to minerva via Dockge

Transport the image without a registry:

docker save doctate-whisper | ssh minerva 'docker load'

Then add the compose file in Dockge and start.

Integration with Axum server

In server/.env:

WHISPER_URL=http://minerva.lan:9001

No other change. The Axum server speaks ahmetoner's interface; we mirror it.

Investigated and rejected

suppress_tokens for digit-sequence preservation (2026-04-28)

Goal: when a dictator says individual digits ("eins null null"), have Whisper emit them as words instead of collapsing them to a number ("100"). The collapse is irreversible downstream — there is no way for the LLM to distinguish "hunderteins" (101 as a number) from "eins null eins" (a 1-0-1 sequence) once both have become 101 in the transcript.

The standard advice (openai/whisper Discussion #1041) is to set suppress_tokens to every numeric token in the vocabulary so the decoder is forced onto word tokens. We added a suppress_numerics form field, threaded it through the Axum server and the experiments sandbox, and tested it against a real cardiology dictation.

Two filter widths, both pathological:

1. Jongwook's exact recipe (\d+ filter, ~426 tokens, faithful 1:1 reproduction with eot bound and removeprefix(" ") filter):
   • Massive end-of-audio repetition loop ("Enoxaparin, das heißt Enoxaparin, das heißt Enoxaparin, …" 20+ times). The fixed compression_ratio_threshold=2.4 did not catch it.
   • Numbers also disappear: 40 mg Thorazemit → Milligramm Thorazemit (the 40 is dropped, not converted to a word).
2. Single-digit only (\d filter, 20 tokens — 0-9 with and without leading space):
   • Repetition loop is gone (small enough list to keep beam search stable).
   • But dictated sequences disappear entirely: 5mg 100 and 5mg 101 become just Visoprolol and Ramipriel (no dosing schema in any form).
   • Collateral damage on natural numbers: 12,5 mg → 12,25 mg (the 5 token was suppressed; the decoder built the value from multi-digit tokens and duplicated a digit).

There is no usable middle ground between the two widths for our setup (large-v3 + faster-whisper 1.2.1 + medical German dictation). The status quo (100/101 in the transcript, doctor corrects manually) stays.

Don't reattempt without changing the underlying ASR (different model, different backend, or model-internal training to preserve digit-form acoustic information). Memory entry: project_whisper_suppress_tokens_dead_end.md.

Offline mode

# Pre-pull on a host with internet:
docker run --rm -v /opt/stacks/doctate-whisper/models:/models doctate-whisper \
  python3 -c "from faster_whisper import WhisperModel; WhisperModel('large-v3-turbo', download_root='/models')"

# Then set in docker-compose.yml:
environment:
  - WHISPER_OFFLINE=1