Integrates the `spellbook` crate to allow for a Hunspell dictionary to
be used as a veto mechanism within the gazetteer.
This prevents common German words from being incorrectly rewritten to
vocabulary entries if they fall within the edit distance threshold. For
example, "Kaktus" will no longer be rewritten to "Lantus" if a Hunspell
dictionary is provided and contains "Kaktus".
The `hunspell_dict_path` configuration option is added, defaulting to
`/usr/share/hunspell/de_DE`. This path should point to the stem of the
Hunspell dictionary files (e.g., `/usr/share/hunspell/de_DE.aff` and
`/usr/share/hunspell/de_DE.dic`).
The gazetteer now supports an optional `DictChecker` trait, allowing for
pluggable dictionary implementations. `SpellbookDict` is the initial
implementation using the `spellbook` crate.
Includes new tests to verify the dict veto functionality, including
handling of case sensitivity and exact matches. An ignored test is added
for live verification against system-installed Hunspell dictionaries.
The gazetteer has been refactored to act as a post-AI normalization
filter. Previously, it was used to annotate LLM input with potential
corrections from a vocabulary. This approach was ineffective because
LLMs often override such hints.
The new approach applies the gazetteer *after* the LLM has generated its
output. This allows for deterministic correction of known terminology,
including fixing LLM output drift (e.g., anglicized drug names).
Key changes:
- `annotate` function renamed to `replace`.
- The output format changes from `Canonical [?original]` to simply the
`Canonical` form.
- The gazetteer now operates on the final LLM output before persistence,
ensuring consistency.
- Tests have been updated to reflect this new behavior, focusing on the
final output rather than the LLM request payload.
The gazetteer now annotates text with `Canonical [?original]`. This
prioritizes the corrected term for the LLM while keeping the original
transcription as a fallback.
This change aligns the gazetteer's annotation strategy with the LLM's
prompt, which expects corrections to be marked for review. Previously,
the format was `original [?Canonical]`, which could lead to the LLM
using the potentially incorrect original term.
The build scripts for generating `anatomy.txt` and related files have
been removed as they are no longer used or maintained. The Kölner
Phonetik module was also removed as it was not being utilized.
The `filter-hunspell-de.sh` script now uses a `MIN_LEN` variable that is
synchronized with the `MIN_TOKEN_LEN` constant in
`src/gazetteer/mod.rs`.
This ensures that short tokens, which are prone to false positives due
to
collisions with common German words, are filtered out by both the
gazetteer
loading and the Hunspell dictionary processing.
Additionally, the script's dependency instructions have been updated to
be
more comprehensive, listing package manager commands for Arch/CachyOS,
Debian/Ubuntu, and Fedora. The `awk` command has been introduced to
filter
out tokens shorter than `MIN_LEN` before sorting and deduplicating.
The `Gazetteer::insert` method has been updated to skip entries shorter
than `MIN_TOKEN_LEN` to prevent noisy phonetic collisions. The
`best_candidate`
helper function has been extracted to improve readability and structure.
The test cases have been updated to reflect the change in
`MIN_TOKEN_LEN`
from 4 to 5 and to use more relevant examples for the updated
functionality,
such as "Cerebrum" and "Zerebrum". The `anatomy.txt` file has been
updated
to reflect that anatomical terms are currently disabled. The
`medications.txt` file has been significantly expanded with a curated
list
of German medication brand and active ingredient names.
This commit introduces the `VOCAB_DIR` configuration option, which
specifies the directory for gazetteer files. It also adds the `strsim`
crate, which is used for calculating string similarity and is essential
for the gazetteer's functionality.
The gazetteer module has been significantly expanded to include the
`Gazetteer` struct, its loading mechanism from `.txt` files, and an
annotation function. This functionality allows for pre-LLM correction of
proper names and technical vocabulary by identifying potential
misspellings based on phonetic similarity and edit distance.
Implements the Kölner Phonetik algorithm for German word sound-encoding.
This phonetic algorithm is used to find gazetteer entries that sound
similar to a potentially misspelled token, enabling correction
suggestions.
The implementation includes:
- `encode`: The main function that orchestrates the encoding process.
- `prepare`: Prepares the input string by expanding umlauts,
uppercasing, and filtering non-alphabetic characters.
- `raw_encode`: Performs the character-by-character phonetic encoding
with context-aware rules.
- `postprocess`: Cleans up the raw output by collapsing duplicate digits
and removing leading zeros.
Includes unit tests to verify the correctness of the encoding for
various German words and edge cases, ensuring phonetic similarity and
handling of special character combinations.
Brummel