Refactor gazetteer to support bypass aliases

The `Gazetteer` struct has been refactored to store `Entry` structs,
where each entry contains a canonical form and a set of bypass aliases.
This change enables the bypassing of Hunspell's dictionary veto for
specific, explicitly defined aliases.

The `parse_line` function has been introduced to handle the new line
format, which allows for `-`-prefixed aliases. Malformed lines with
secondary tokens not prefixed by `-` now result in a hard error, failing
server startup with `io::ErrorKind::InvalidData`.

The `replace` method has been updated to check bypass aliases before
consulting the dictionary, ensuring that explicitly allowed aliases are
used for rewriting. This addresses issues where Hunspell incorrectly
identifies compounds, blocking necessary corrections.

The `README.md` in the `vocab` directory has been updated to document
the new bypass-alias functionality.
This commit is contained in:
2026-04-30 21:02:41 +02:00
parent bb584b6ea0
commit 9c452e8db3
3 changed files with 337 additions and 20 deletions
+25
View File
@@ -14,6 +14,31 @@ Whisper transcripts as `Token [?Canonical]` for LLM-side review.
everyday German words).
- Duplicates (case-insensitive, across files) are collapsed automatically.
### Optional: bypass-aliases for false-positive Hunspell compounds
A line may carry one or more `-`-prefixed alias tokens after the canonical:
```
Sellink -Seelink -Seelinks
```
Meaning: when a transcript token (case-insensitive) exactly matches a
listed alias, the gazetteer skips the Hunspell dict-veto and rewrites
the token to the canonical. Use this **only** when Hunspell-de
mistakenly accepts a known mishearing as a valid German compound and
thereby blocks a legitimate correction (e.g. Hunspell parses `Seelink`
as `See` + `link` and vetoes `Seelink → Sellink`).
- The `-` is the bypass marker; it is stripped before storage.
- Aliases are **exact match only** (no Damerau-Levenshtein tolerance).
If a new declension or mishearing surfaces, append it explicitly:
`Sellink -Seelink -Seelinks`.
- Aliases are scoped to the entry whose canonical wins the DL contest;
identical aliases under unrelated entries do not interfere.
- A secondary token without `-` is a hard server-start error
(`io::ErrorKind::InvalidData`) — the loader fails loud rather than
silently misindex.
## Adding entries
Edit `vocabulary.txt` (or drop additional `*.txt` files into this
+1 -1
View File
@@ -127,7 +127,7 @@ Risperidon
Rivaroxaban
Rosuvastatin
Salbutamol
Sellink
Sellink -Seelink
Sertralin
Sildenafil
Simvastatin