The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is
synchronized with the `MIN_TOKEN_LEN` constant in
`src/gazetteer/mod.rs`.
This ensures that short tokens, which are prone to false positives due
to
collisions with common German words, are filtered out by both the
gazetteer
loading and the Hunspell dictionary processing.
Additionally, the script's dependency instructions have been updated to
be
more comprehensive, listing package manager commands for Arch/CachyOS,
Debian/Ubuntu, and Fedora. The `awk` command has been introduced to
filter
out tokens shorter than `MIN_LEN` before sorting and deduplicating.
The `Gazetteer::insert` method has been updated to skip entries shorter
than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The
`best_candidate`
helper function has been extracted to improve readability and structure.
The test cases have been updated to reflect the change in
`MIN_TOKEN_LEN`
from 4 to 5 and to use more relevant examples for the updated
functionality,
such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been
updated
to reflect that anatomical terms are currently disabled. The
`medications.txt` file has been significantly expanded with a curated
list
of German medication brand and active ingredient names.
This commit introduces the `VOCAB_DIR` configuration option, which
specifies the directory for gazetteer files. It also adds the `strsim`
crate, which is used for calculating string similarity and is essential
for the gazetteer's functionality.
The gazetteer module has been significantly expanded to include the
`Gazetteer` struct, its loading mechanism from `.txt` files, and an
annotation function. This functionality allows for pre-LLM correction of
proper names and technical vocabulary by identifying potential
misspellings based on phonetic similarity and edit distance.
Implements the Kölner Phonetik algorithm for German word sound-encoding.
This phonetic algorithm is used to find gazetteer entries that sound
similar to a potentially misspelled token, enabling correction
suggestions.
The implementation includes:
- `encode`: The main function that orchestrates the encoding process.
- `prepare`: Prepares the input string by expanding umlauts,
uppercasing, and filtering non-alphabetic characters.
- `raw_encode`: Performs the character-by-character phonetic encoding
with context-aware rules.
- `postprocess`: Cleans up the raw output by collapsing duplicate digits
and removing leading zeros.
Includes unit tests to verify the correctness of the encoding for
various German words and edge cases, ensuring phonetic similarity and
handling of special character combinations.
Brummel