Refactor gazetteer to support bypass aliases
The `Gazetteer` struct has been refactored to store `Entry` structs, where each entry contains a canonical form and a set of bypass aliases. This change enables the bypassing of Hunspell's dictionary veto for specific, explicitly defined aliases. The `parse_line` function has been introduced to handle the new line format, which allows for `-`-prefixed aliases. Malformed lines with secondary tokens not prefixed by `-` now result in a hard error, failing server startup with `io::ErrorKind::InvalidData`. The `replace` method has been updated to check bypass aliases before consulting the dictionary, ensuring that explicitly allowed aliases are used for rewriting. This addresses issues where Hunspell incorrectly identifies compounds, blocking necessary corrections. The `README.md` in the `vocab` directory has been updated to document the new bypass-alias functionality.
This commit is contained in:
@@ -14,6 +14,31 @@ Whisper transcripts as `Token [?Canonical]` for LLM-side review.
|
||||
everyday German words).
|
||||
- Duplicates (case-insensitive, across files) are collapsed automatically.
|
||||
|
||||
### Optional: bypass-aliases for false-positive Hunspell compounds
|
||||
|
||||
A line may carry one or more `-`-prefixed alias tokens after the canonical:
|
||||
|
||||
```
|
||||
Sellink -Seelink -Seelinks
|
||||
```
|
||||
|
||||
Meaning: when a transcript token (case-insensitive) exactly matches a
|
||||
listed alias, the gazetteer skips the Hunspell dict-veto and rewrites
|
||||
the token to the canonical. Use this **only** when Hunspell-de
|
||||
mistakenly accepts a known mishearing as a valid German compound and
|
||||
thereby blocks a legitimate correction (e.g. Hunspell parses `Seelink`
|
||||
as `See` + `link` and vetoes `Seelink → Sellink`).
|
||||
|
||||
- The `-` is the bypass marker; it is stripped before storage.
|
||||
- Aliases are **exact match only** (no Damerau-Levenshtein tolerance).
|
||||
If a new declension or mishearing surfaces, append it explicitly:
|
||||
`Sellink -Seelink -Seelinks`.
|
||||
- Aliases are scoped to the entry whose canonical wins the DL contest;
|
||||
identical aliases under unrelated entries do not interfere.
|
||||
- A secondary token without `-` is a hard server-start error
|
||||
(`io::ErrorKind::InvalidData`) — the loader fails loud rather than
|
||||
silently misindex.
|
||||
|
||||
## Adding entries
|
||||
|
||||
Edit `vocabulary.txt` (or drop additional `*.txt` files into this
|
||||
|
||||
@@ -127,7 +127,7 @@ Risperidon
|
||||
Rivaroxaban
|
||||
Rosuvastatin
|
||||
Salbutamol
|
||||
Sellink
|
||||
Sellink -Seelink
|
||||
Sertralin
|
||||
Sildenafil
|
||||
Simvastatin
|
||||
|
||||
Reference in New Issue
Block a user