# doctate-whisper

Thin FastAPI wrapper around [faster-whisper](https://github.com/SYSTRAN/faster-whisper).

## Why a custom service?

Existing Whisper HTTP wrappers (`ahmetoner/whisper-asr-webservice`,
`speaches`, `linuxserver/faster-whisper`, `hwdsl2/docker-whisper`) hardcode
the faster-whisper defaults. Those defaults include:

- `condition_on_previous_text=True` — classic cascading-hallucination source
- temperature fallback cascade `[0.0, 0.2, …, 1.0]` — when confidence drops,
  Whisper gets "creative"

On real cardiology dictations we saw **2/13 runs with catastrophic
hallucinations** (Russian/Chinese fragments, repetition loops) using
ahmetoner's defaults. None of the available wrappers expose these
parameters as request fields or env vars.

This service fixes them to safe values that never change.

## Anti-hallucination params (fixed)

| Param | Value | Reason |
|---|---|---|
| `temperature` | `0.0` | No fallback cascade; deterministic output |
| `condition_on_previous_text` | `False` | Prevents cascading hallucinations |
| `vad_filter` | `True` | Silence → no transcription (not garbage) |
| `vad_parameters.min_silence_duration_ms` | `500` | VAD sensitivity |
| `no_speech_threshold` | `0.6` | Drop silent segments |
| `log_prob_threshold` | `-1.0` | Drop low-confidence segments |
| `compression_ratio_threshold` | `2.4` | Anti-repetition |
| `beam_size` / `best_of` | `5` / `5` | Default quality beam search |

## Endpoints

- `POST /asr` — multipart `audio_file`, query `output=txt|json`, `language=de`, form `initial_prompt`
- `GET /health` — liveness: `{"status":"ok","model":...,"device":...}`
- `GET /info` — debug: model, device, compute_type, offline flag

Compatible with ahmetoner's `/asr` interface so the Axum server needs no change.

## Env vars

| Var | Default | Purpose |
|---|---|---|
| `WHISPER_MODEL` | `large-v3-turbo` | Any faster-whisper model name |
| `WHISPER_COMPUTE_TYPE` | `float16` | `float16`, `int8_float16`, `int8` (CPU) |
| `WHISPER_DEVICE` | `cuda` | `cuda` or `cpu` |
| `WHISPER_MODELS_DIR` | `/models` | Cache location |
| `WHISPER_OFFLINE` | `0` | `1` → no HuggingFace download |

## Model choice

**Default is `large-v3-turbo`**: same 32-layer encoder as `large-v3`,
only decoder distilled to 4 layers. OpenAI benchmarks: equal quality on
European languages. ~1.6 GB VRAM vs. ~3 GB — leaves room for Ollama
(gemma4, 9 GB) in a 12 GB card.

Switch to `large-v3` at any time via `WHISPER_MODEL=large-v3` + container restart.

## Build

```bash
docker build -t doctate-whisper ./whisper
```

Takes 2–5 minutes. First run of the container pulls ~1.6 GB from
HuggingFace into `/models` (volume-persisted).

## Deploy to minerva via Dockge

Transport the image without a registry:

```bash
docker save doctate-whisper | ssh minerva 'docker load'
```

Then add the compose file in Dockge and start.

## Integration with Axum server

In `server/.env`:

```
WHISPER_URL=http://minerva.lan:9001
```

No other change. The Axum server speaks ahmetoner's interface; we mirror it.

## Investigated and rejected

### `suppress_tokens` for digit-sequence preservation (2026-04-28)

Goal: when a dictator says individual digits ("eins null null"), have Whisper
emit them as words instead of collapsing them to a number ("100"). The
collapse is irreversible downstream — there is no way for the LLM to
distinguish "hunderteins" (101 as a number) from "eins null eins" (a 1-0-1
sequence) once both have become `101` in the transcript.

The standard advice
([openai/whisper Discussion #1041](https://github.com/openai/whisper/discussions/1041))
is to set `suppress_tokens` to every numeric token in the vocabulary so the
decoder is forced onto word tokens. We added a `suppress_numerics` form
field, threaded it through the Axum server and the experiments sandbox,
and tested it against a real cardiology dictation.

**Two filter widths, both pathological:**

1. **Jongwook's exact recipe** (`\d+` filter, ~426 tokens, faithful 1:1
   reproduction with `eot` bound and `removeprefix(" ")` filter):
   - Massive end-of-audio repetition loop ("Enoxaparin, das heißt
     Enoxaparin, das heißt Enoxaparin, …" 20+ times). The fixed
     `compression_ratio_threshold=2.4` did not catch it.
   - Numbers also disappear: `40 mg Thorazemit` → `Milligramm Thorazemit`
     (the `40` is dropped, not converted to a word).
2. **Single-digit only** (`\d` filter, 20 tokens — `0`-`9` with and without
   leading space):
   - Repetition loop is gone (small enough list to keep beam search stable).
   - But dictated sequences disappear entirely: `5mg 100` and `5mg 101`
     become just `Visoprolol` and `Ramipriel` (no dosing schema in any form).
   - Collateral damage on natural numbers: `12,5 mg` → `12,25 mg` (the ` 5`
     token was suppressed; the decoder built the value from multi-digit
     tokens and duplicated a digit).

There is no usable middle ground between the two widths for our setup
(`large-v3` + faster-whisper 1.2.1 + medical German dictation). The status
quo (`100`/`101` in the transcript, doctor corrects manually) stays.

Don't reattempt without changing the underlying ASR (different model,
different backend, or model-internal training to preserve digit-form
acoustic information). Memory entry:
`project_whisper_suppress_tokens_dead_end.md`.

## Offline mode

```bash
# Pre-pull on a host with internet:
docker run --rm -v /opt/stacks/doctate-whisper/models:/models doctate-whisper \
  python3 -c "from faster_whisper import WhisperModel; WhisperModel('large-v3-turbo', download_root='/models')"

# Then set in docker-compose.yml:
environment:
  - WHISPER_OFFLINE=1
```