Update hunspell filter to use min token length
The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is synchronized with the `MIN_TOKEN_LEN` constant in `src/gazetteer/mod.rs`. This ensures that short tokens, which are prone to false positives due to collisions with common German words, are filtered out by both the gazetteer loading and the Hunspell dictionary processing. Additionally, the script's dependency instructions have been updated to be more comprehensive, listing package manager commands for Arch/CachyOS, Debian/Ubuntu, and Fedora. The `awk` command has been introduced to filter out tokens shorter than `MIN_LEN` before sorting and deduplicating. The `Gazetteer::insert` method has been updated to skip entries shorter than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The `best_candidate` helper function has been extracted to improve readability and structure. The test cases have been updated to reflect the change in `MIN_TOKEN_LEN` from 4 to 5 and to use more relevant examples for the updated functionality, such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been updated to reflect that anatomical terms are currently disabled. The `medications.txt` file has been significantly expanded with a curated list of German medication brand and active ingredient names.
This commit is contained in:
@@ -1,2 +1,3 @@
|
||||
# Anatomical Latin/Greek terms. Populate via ./scripts/build-anatomy.sh.
|
||||
# Blank — mechanism runs as no-op until content is built.
|
||||
# Anatomical Latin/Greek terms — currently disabled.
|
||||
# Scope narrowed to medications. If re-enabling, populate via
|
||||
# ./scripts/build-anatomy.sh and expect eponym-beifang.
|
||||
|
||||
@@ -1,2 +1,165 @@
|
||||
# Exotic medication brand names. Population script TBD.
|
||||
# Candidate sources: Wikidata (P267 ATC-code joins), Rote Liste alternatives.
|
||||
# Seed list of German medication brand names + commonly dictated
|
||||
# active-ingredient names. Roughly aligned with the AOK Arzneiverordnungs-
|
||||
# Report top-prescribed set. Curated, not crawled — new entries should
|
||||
# be ≥5 chars (phonetic-collision safety) and phonetically distinctive.
|
||||
# Maintain alphabetical order, one entry per line, no inline comments.
|
||||
Adalat
|
||||
Aldactone
|
||||
Allopurinol
|
||||
Amiodaron
|
||||
Amlodipin
|
||||
Amoxicillin
|
||||
Arcoxia
|
||||
Aspirin
|
||||
Atacand
|
||||
Atorvastatin
|
||||
Azithromycin
|
||||
Beloc
|
||||
Benazepril
|
||||
Berodual
|
||||
Betahistin
|
||||
Bisoprolol
|
||||
Budesonid
|
||||
Buscopan
|
||||
Candesartan
|
||||
Carvedilol
|
||||
Cefuroxim
|
||||
Celebrex
|
||||
Cetirizin
|
||||
Ciprofloxacin
|
||||
Citalopram
|
||||
Clarithromycin
|
||||
Clexane
|
||||
Clindamycin
|
||||
Clopidogrel
|
||||
Concor
|
||||
Cortison
|
||||
Dabigatran
|
||||
Dexamethason
|
||||
Diazepam
|
||||
Diclofenac
|
||||
Digoxin
|
||||
Diltiazem
|
||||
Dimenhydrinat
|
||||
Dobendan
|
||||
Dolormin
|
||||
Doxycyclin
|
||||
Duloxetin
|
||||
Edoxaban
|
||||
Eliquis
|
||||
Enalapril
|
||||
Erythromycin
|
||||
Escitalopram
|
||||
Esomeprazol
|
||||
Euthyrox
|
||||
Fluoxetin
|
||||
Foradil
|
||||
Forxiga
|
||||
Furosemid
|
||||
Gabapentin
|
||||
Glucophage
|
||||
Haldol
|
||||
Heparin
|
||||
Humira
|
||||
Ibuprofen
|
||||
Iberogast
|
||||
Imodium
|
||||
Insulin
|
||||
Isoptin
|
||||
Januvia
|
||||
Jardiance
|
||||
Ketamin
|
||||
Keppra
|
||||
Klacid
|
||||
Lactulose
|
||||
Lansoprazol
|
||||
Lantus
|
||||
Lasix
|
||||
Laxoberal
|
||||
Lisinopril
|
||||
Lixiana
|
||||
Lorazepam
|
||||
Lorzaar
|
||||
Lovastatin
|
||||
Macrogol
|
||||
Marcumar
|
||||
Mesalazin
|
||||
Metformin
|
||||
Methotrexat
|
||||
Metoprolol
|
||||
Metronidazol
|
||||
Mirtazapin
|
||||
Morphium
|
||||
Movicol
|
||||
Mucofalk
|
||||
Naproxen
|
||||
Nasivin
|
||||
Nebivolol
|
||||
Nexium
|
||||
Nitrolingual
|
||||
Norvasc
|
||||
Novalgin
|
||||
Novorapid
|
||||
Olanzapin
|
||||
Olynth
|
||||
Omeprazol
|
||||
Otriven
|
||||
Oxazepam
|
||||
Oxycodon
|
||||
Ozempic
|
||||
Pantoprazol
|
||||
Paracetamol
|
||||
Penicillin
|
||||
Perenterol
|
||||
Phenprocoumon
|
||||
Pradaxa
|
||||
Pravastatin
|
||||
Prednisolon
|
||||
Prednison
|
||||
Pregabalin
|
||||
Pulmicort
|
||||
Quetiapin
|
||||
Ramipril
|
||||
Ranitidin
|
||||
Rhinospray
|
||||
Risperidon
|
||||
Rivaroxaban
|
||||
Rosuvastatin
|
||||
Salbutamol
|
||||
Sertralin
|
||||
Sildenafil
|
||||
Simvastatin
|
||||
Sinupret
|
||||
Sirdalud
|
||||
Spiriva
|
||||
Spironolacton
|
||||
Stilnox
|
||||
Sumatriptan
|
||||
Symbicort
|
||||
Tadalafil
|
||||
Tavor
|
||||
Telmisartan
|
||||
Temesta
|
||||
Thyroxin
|
||||
Timolol
|
||||
Torasemid
|
||||
Tramadol
|
||||
Tranxilium
|
||||
Trulicity
|
||||
Ultracain
|
||||
Valsartan
|
||||
Venlafaxin
|
||||
Ventolin
|
||||
Verapamil
|
||||
Viagra
|
||||
Viani
|
||||
Victoza
|
||||
Voltaren
|
||||
Voluven
|
||||
Vomex
|
||||
Warfarin
|
||||
Xarelto
|
||||
Ximovan
|
||||
Zolpidem
|
||||
Zopiclon
|
||||
Zyprexa
|
||||
|
||||
Reference in New Issue
Block a user