Update hunspell filter to use min token length

The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is
synchronized with the `MIN_TOKEN_LEN` constant in
`src/gazetteer/mod.rs`.
This ensures that short tokens, which are prone to false positives due
to
collisions with common German words, are filtered out by both the
gazetteer
loading and the Hunspell dictionary processing.

Additionally, the script's dependency instructions have been updated to
be
more comprehensive, listing package manager commands for Arch/CachyOS,
Debian/Ubuntu, and Fedora. The `awk` command has been introduced to
filter
out tokens shorter than `MIN_LEN` before sorting and deduplicating.

The `Gazetteer::insert` method has been updated to skip entries shorter
than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The
`best_candidate`
helper function has been extracted to improve readability and structure.
The test cases have been updated to reflect the change in
`MIN_TOKEN_LEN`
from 4 to 5 and to use more relevant examples for the updated
functionality,
such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been
updated
to reflect that anatomical terms are currently disabled. The
`medications.txt` file has been significantly expanded with a curated
list
of German medication brand and active ingredient names.
This commit is contained in:
2026-04-16 18:48:15 +02:00
parent 3da5a36dc8
commit 17fb14a741
5 changed files with 420 additions and 54 deletions
+3 -2
View File
@@ -1,2 +1,3 @@
# Anatomical Latin/Greek terms. Populate via ./scripts/build-anatomy.sh.
# Blank — mechanism runs as no-op until content is built.
# Anatomical Latin/Greek terms — currently disabled.
# Scope narrowed to medications. If re-enabling, populate via
# ./scripts/build-anatomy.sh and expect eponym-beifang.
+165 -2
View File
@@ -1,2 +1,165 @@
# Exotic medication brand names. Population script TBD.
# Candidate sources: Wikidata (P267 ATC-code joins), Rote Liste alternatives.
# Seed list of German medication brand names + commonly dictated
# active-ingredient names. Roughly aligned with the AOK Arzneiverordnungs-
# Report top-prescribed set. Curated, not crawled — new entries should
# be ≥5 chars (phonetic-collision safety) and phonetically distinctive.
# Maintain alphabetical order, one entry per line, no inline comments.
Adalat
Aldactone
Allopurinol
Amiodaron
Amlodipin
Amoxicillin
Arcoxia
Aspirin
Atacand
Atorvastatin
Azithromycin
Beloc
Benazepril
Berodual
Betahistin
Bisoprolol
Budesonid
Buscopan
Candesartan
Carvedilol
Cefuroxim
Celebrex
Cetirizin
Ciprofloxacin
Citalopram
Clarithromycin
Clexane
Clindamycin
Clopidogrel
Concor
Cortison
Dabigatran
Dexamethason
Diazepam
Diclofenac
Digoxin
Diltiazem
Dimenhydrinat
Dobendan
Dolormin
Doxycyclin
Duloxetin
Edoxaban
Eliquis
Enalapril
Erythromycin
Escitalopram
Esomeprazol
Euthyrox
Fluoxetin
Foradil
Forxiga
Furosemid
Gabapentin
Glucophage
Haldol
Heparin
Humira
Ibuprofen
Iberogast
Imodium
Insulin
Isoptin
Januvia
Jardiance
Ketamin
Keppra
Klacid
Lactulose
Lansoprazol
Lantus
Lasix
Laxoberal
Lisinopril
Lixiana
Lorazepam
Lorzaar
Lovastatin
Macrogol
Marcumar
Mesalazin
Metformin
Methotrexat
Metoprolol
Metronidazol
Mirtazapin
Morphium
Movicol
Mucofalk
Naproxen
Nasivin
Nebivolol
Nexium
Nitrolingual
Norvasc
Novalgin
Novorapid
Olanzapin
Olynth
Omeprazol
Otriven
Oxazepam
Oxycodon
Ozempic
Pantoprazol
Paracetamol
Penicillin
Perenterol
Phenprocoumon
Pradaxa
Pravastatin
Prednisolon
Prednison
Pregabalin
Pulmicort
Quetiapin
Ramipril
Ranitidin
Rhinospray
Risperidon
Rivaroxaban
Rosuvastatin
Salbutamol
Sertralin
Sildenafil
Simvastatin
Sinupret
Sirdalud
Spiriva
Spironolacton
Stilnox
Sumatriptan
Symbicort
Tadalafil
Tavor
Telmisartan
Temesta
Thyroxin
Timolol
Torasemid
Tramadol
Tranxilium
Trulicity
Ultracain
Valsartan
Venlafaxin
Ventolin
Verapamil
Viagra
Viani
Victoza
Voltaren
Voluven
Vomex
Warfarin
Xarelto
Ximovan
Zolpidem
Zopiclon
Zyprexa