Update hunspell filter to use min token length

The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is
synchronized with the `MIN_TOKEN_LEN` constant in
`src/gazetteer/mod.rs`.
This ensures that short tokens, which are prone to false positives due
to
collisions with common German words, are filtered out by both the
gazetteer
loading and the Hunspell dictionary processing.

Additionally, the script's dependency instructions have been updated to
be
more comprehensive, listing package manager commands for Arch/CachyOS,
Debian/Ubuntu, and Fedora. The `awk` command has been introduced to
filter
out tokens shorter than `MIN_LEN` before sorting and deduplicating.

The `Gazetteer::insert` method has been updated to skip entries shorter
than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The
`best_candidate`
helper function has been extracted to improve readability and structure.
The test cases have been updated to reflect the change in
`MIN_TOKEN_LEN`
from 4 to 5 and to use more relevant examples for the updated
functionality,
such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been
updated
to reflect that anatomical terms are currently disabled. The
`medications.txt` file has been significantly expanded with a curated
list
of German medication brand and active ingredient names.
This commit is contained in:
2026-04-16 18:48:15 +02:00
parent 3da5a36dc8
commit 17fb14a741
5 changed files with 420 additions and 54 deletions
+19 -9
View File
@@ -8,8 +8,10 @@
# individual unknown words are kept. That matches our gazetteer's
# token-level matching model (see gazetteer/mod.rs).
#
# Dependencies: hunspell, hunspell-de-de (Debian: apt install hunspell
# hunspell-de-de).
# Dependencies: hunspell + German dictionary. Package names vary:
# Arch/CachyOS: pacman -S hunspell hunspell-de
# Debian/Ubuntu: apt install hunspell hunspell-de-de
# Fedora: dnf install hunspell hunspell-de
#
# Usage: ./filter-hunspell-de.sh <input> <output>
@@ -23,20 +25,28 @@ fi
INPUT="$1"
OUTPUT="$2"
# Minimum entry length. Must match MIN_TOKEN_LEN in src/gazetteer/mod.rs.
# 5 is the current value — appropriate for curated medication lists where
# the 4-char eponym-vs-German-word collision class is not present.
MIN_LEN=5
if ! command -v hunspell >/dev/null 2>&1; then
echo "Error: hunspell is not installed." >&2
echo " Debian/Ubuntu: sudo apt install hunspell hunspell-de-de" >&2
echo "Error: hunspell is not installed. Install it via your package" >&2
echo " manager (e.g. pacman -S hunspell, apt install hunspell)." >&2
exit 1
fi
# Sanity-check: is the de_DE dictionary available?
if ! echo "Haus" | hunspell -d de_DE -a >/dev/null 2>&1; then
echo "Error: hunspell de_DE dictionary not available." >&2
echo " Debian/Ubuntu: sudo apt install hunspell-de-de" >&2
echo "Error: hunspell de_DE dictionary not available. Install:" >&2
echo " Arch/CachyOS: pacman -S hunspell-de" >&2
echo " Debian/Ubuntu: apt install hunspell-de-de" >&2
echo " Fedora: dnf install hunspell-de" >&2
exit 1
fi
# `hunspell -l` prints every unknown word from stdin, one per line.
# Pipe through sort -u to drop duplicates (multi-word titles can share
# tokens) and produce a stable output order.
hunspell -d de_DE -l < "$INPUT" | sort -u > "$OUTPUT"
# Drop words shorter than MIN_LEN, then sort -u for dedup + stable order.
hunspell -d de_DE -l < "$INPUT" \
| awk -v min="$MIN_LEN" 'length($0) >= min' \
| sort -u > "$OUTPUT"