This commit removes the `whisper_variant` and `whisper_hotwords_variant`
fields from the `run_id` generation and the `print_meta_summary`
function.
These variants are no longer used as the project is shifting focus to
LLM-based generation. The `run_full_case.rs` example has also been
updated to reflect this change.
This commit introduces the `doctate-canary` service and associated sweep
experiments.
The service provides access to the `nvidia/canary-1b-v2` ASR model via a
native FastAPI API. It includes deployment scripts, Docker
configurations, and detailed documentation on installation, API usage,
and environment variables.
The sweep experiments aim to thoroughly evaluate the Canary model's
performance under various configurations. This includes testing
different inference pipelines (single-shot vs. buffered), decoding
strategies (greedy vs. beam search), and parameter tuning (chunk length,
overlap, batch size, precision). The goal is to reproduce previous
findings and identify optimal settings.
The commit also includes:
- Utility scripts for audio transcoding and manipulation.
- Comprehensive logging and result collection mechanisms for the sweep
runs.
- Detailed analysis of `dur=0` occurrences and word confidence,
concluding they are not reliable indicators of hallucination.
- Documentation on the interaction between decoding parameters,
especially `return_hypotheses`, and the availability of confidence
scores.
The previous run_llm_only only applied the gazetteer post-LLM, which
silently underestimated the production pipeline: tokens like Inoxaparin
or Klappenvizien are caught pre-LLM in production, but the test tool
left them in the LLM input. Adding `gazetteer.replace()` to each
recording before render_prompt mirrors transcribe/worker.rs:138 exactly,
making sandbox results faithful to production behavior.
New prompt variants kept for the iteration record:
- v3_treue_konsolidiert.txt — the winning variant now in
server/src/analyze/prompt.rs; redundancies eliminated where they were
not load-bearing
- v4_treue_kompakt.txt — rejected variant (3/3 unmarked Hypotonie
hallucinations); kept so future iterations don't repeat the
experiment
baseline.txt now mirrors the current server prompt (was a stale
pre-v2 snapshot), so future sandbox runs against `baseline.txt`
compare against what is actually shipping.
Datei enthielt Bisoprolol/Ramipril mit den exakten Schemata aus dem
Test-Case c414cf52 — klassisches Curve-Fitting, das beim Test gegen
genau diesen Fall keinen Erkenntnisgewinn bringt und beim Roll-out
auf andere Fälle aktiv halluziniert (Whisper bias'd auf gelistete
Termini, auch bei Stille oder Gemurmel).
Whisper-Initial-Prompts werden vorerst nicht weiter iteriert: Das
Hauptproblem 1-0-0/1-0-1 ist durch den LLM-DOSIERUNGSSCHEMA-Block
bereits gelöst, und der Halluzinations-Risikoaufschlag bei Whisper-
Prompts ist höher als bei der LLM-Stufe. baseline.txt (leer) bleibt
als Slot-Referenz erhalten.
Nested Cargo-Workspace unter experiments/ als Pfad-Dependency auf
../server. Reproduziert die Production-Pipeline 1:1 durch direkte
Wiederverwendung der Server-Module (transcribe::whisper, analyze::llm,
gazetteer, ffmpeg-Remux) — kein eigenes HTTP-Re-Implementieren, kein
Drift-Risiko zur Live-Pipeline.
Drei bin-Targets:
- run_full_case: volle Pipeline (ffmpeg → whisper → pre-gazetteer → llm
→ post-gazetteer), n Runs einer Variante
- run_llm_only: schneller LLM-Iterations-Pfad auf existierenden,
gazetteer-bereinigten Transkripten — spart Whisper-Calls
- diff_runs: qualitativer Vergleich zwischen zwei Run-Verzeichnissen
Repo-Hygiene: case_*/ (Patientendaten, Symlinks zu tmpdata) und _data/
(Run-Outputs) sind komplett gitignored. Nur Code und fall-übergreifende
Prompt-Hypothesen (prompts/) werden versioniert.
Erste Prompt-Varianten zur Treue-Klausel und Konflikt-Auflösung:
prompts/llm/v1_treue.txt, v2_treue.txt.
Brummel