Brummel
17fb14a741
The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is synchronized with the `MIN_TOKEN_LEN` constant in `src/gazetteer/mod.rs`. This ensures that short tokens, which are prone to false positives due to collisions with common German words, are filtered out by both the gazetteer loading and the Hunspell dictionary processing. Additionally, the script's dependency instructions have been updated to be more comprehensive, listing package manager commands for Arch/CachyOS, Debian/Ubuntu, and Fedora. The `awk` command has been introduced to filter out tokens shorter than `MIN_LEN` before sorting and deduplicating. The `Gazetteer::insert` method has been updated to skip entries shorter than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The `best_candidate` helper function has been extracted to improve readability and structure. The test cases have been updated to reflect the change in `MIN_TOKEN_LEN` from 4 to 5 and to use more relevant examples for the updated functionality, such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been updated to reflect that anatomical terms are currently disabled. The `medications.txt` file has been significantly expanded with a curated list of German medication brand and active ingredient names.
Gazetteer vocabulary files
These *.txt files are loaded once at server startup and used to annotate
Whisper transcripts with correction hints of the form Token [?Canonical]
before the analysis LLM runs. Design: ~/.claude/plans/happy-mapping-snail.md.
File format
- One entry per line, UTF-8, no surrounding whitespace required.
- Lines that are blank or start with
#are skipped. - Case-insensitive. Duplicates across files are collapsed.
- No header row, no columns — this is a plain word list.
Files
| File | Domain | Populated? |
|---|---|---|
anatomy.txt |
Latin/Greek anatomical terms | build script |
substances.txt |
Exotic active substances (INNs) | TBD |
medications.txt |
Exotic medication brand names | TBD |
Build scripts
The anatomy.txt is produced by shell pipeline under ../scripts/:
./scripts/build-anatomy.sh
It crawls the German Wikipedia "Kategorie:Anatomie", filters the result
against hunspell-de-de to remove everyday German words, and writes the
remainder here. The result is checked into git — we rebuild only when the
source vocabulary materially changes, not on every server build.
Intermediate files land under vocab/raw/, which is gitignored.