Brummel
9c452e8db3
The `Gazetteer` struct has been refactored to store `Entry` structs, where each entry contains a canonical form and a set of bypass aliases. This change enables the bypassing of Hunspell's dictionary veto for specific, explicitly defined aliases. The `parse_line` function has been introduced to handle the new line format, which allows for `-`-prefixed aliases. Malformed lines with secondary tokens not prefixed by `-` now result in a hard error, failing server startup with `io::ErrorKind::InvalidData`. The `replace` method has been updated to check bypass aliases before consulting the dictionary, ensuring that explicitly allowed aliases are used for rewriting. This addresses issues where Hunspell incorrectly identifies compounds, blocking necessary corrections. The `README.md` in the `vocab` directory has been updated to document the new bypass-alias functionality.
2.0 KiB
Gazetteer vocabulary
Plain list of domain-specific words the analysis LLM might not know —
medication brand names, proper nouns, colleagues, technical terms. Loaded
once at server start. Used to annotate close edit-distance neighbours in
Whisper transcripts as Token [?Canonical] for LLM-side review.
Format
- One entry per line, UTF-8.
- Blank lines and lines starting with
#are skipped. - Matching is case-insensitive; original casing is preserved for display.
- Entries must be ≥5 characters (shorter tokens collide too often with everyday German words).
- Duplicates (case-insensitive, across files) are collapsed automatically.
Optional: bypass-aliases for false-positive Hunspell compounds
A line may carry one or more --prefixed alias tokens after the canonical:
Sellink -Seelink -Seelinks
Meaning: when a transcript token (case-insensitive) exactly matches a
listed alias, the gazetteer skips the Hunspell dict-veto and rewrites
the token to the canonical. Use this only when Hunspell-de
mistakenly accepts a known mishearing as a valid German compound and
thereby blocks a legitimate correction (e.g. Hunspell parses Seelink
as See + link and vetoes Seelink → Sellink).
- The
-is the bypass marker; it is stripped before storage. - Aliases are exact match only (no Damerau-Levenshtein tolerance).
If a new declension or mishearing surfaces, append it explicitly:
Sellink -Seelink -Seelinks. - Aliases are scoped to the entry whose canonical wins the DL contest; identical aliases under unrelated entries do not interfere.
- A secondary token without
-is a hard server-start error (io::ErrorKind::InvalidData) — the loader fails loud rather than silently misindex.
Adding entries
Edit vocabulary.txt (or drop additional *.txt files into this
directory — the loader picks up every *.txt non-recursively). Restart
the server afterwards; there is no hot-reload. Keep entries phonetically
distinctive: Tavor is a good entry, Haus would match innocent words.