Brummel
9c452e8db3
The `Gazetteer` struct has been refactored to store `Entry` structs, where each entry contains a canonical form and a set of bypass aliases. This change enables the bypassing of Hunspell's dictionary veto for specific, explicitly defined aliases. The `parse_line` function has been introduced to handle the new line format, which allows for `-`-prefixed aliases. Malformed lines with secondary tokens not prefixed by `-` now result in a hard error, failing server startup with `io::ErrorKind::InvalidData`. The `replace` method has been updated to check bypass aliases before consulting the dictionary, ensuring that explicitly allowed aliases are used for rewriting. This addresses issues where Hunspell incorrectly identifies compounds, blocking necessary corrections. The `README.md` in the `vocab` directory has been updated to document the new bypass-alias functionality.
48 lines
2.0 KiB
Markdown
48 lines
2.0 KiB
Markdown
# Gazetteer vocabulary
|
|
|
|
Plain list of domain-specific words the analysis LLM might not know —
|
|
medication brand names, proper nouns, colleagues, technical terms. Loaded
|
|
once at server start. Used to annotate close edit-distance neighbours in
|
|
Whisper transcripts as `Token [?Canonical]` for LLM-side review.
|
|
|
|
## Format
|
|
|
|
- One entry per line, UTF-8.
|
|
- Blank lines and lines starting with `#` are skipped.
|
|
- Matching is case-insensitive; original casing is preserved for display.
|
|
- Entries must be ≥5 characters (shorter tokens collide too often with
|
|
everyday German words).
|
|
- Duplicates (case-insensitive, across files) are collapsed automatically.
|
|
|
|
### Optional: bypass-aliases for false-positive Hunspell compounds
|
|
|
|
A line may carry one or more `-`-prefixed alias tokens after the canonical:
|
|
|
|
```
|
|
Sellink -Seelink -Seelinks
|
|
```
|
|
|
|
Meaning: when a transcript token (case-insensitive) exactly matches a
|
|
listed alias, the gazetteer skips the Hunspell dict-veto and rewrites
|
|
the token to the canonical. Use this **only** when Hunspell-de
|
|
mistakenly accepts a known mishearing as a valid German compound and
|
|
thereby blocks a legitimate correction (e.g. Hunspell parses `Seelink`
|
|
as `See` + `link` and vetoes `Seelink → Sellink`).
|
|
|
|
- The `-` is the bypass marker; it is stripped before storage.
|
|
- Aliases are **exact match only** (no Damerau-Levenshtein tolerance).
|
|
If a new declension or mishearing surfaces, append it explicitly:
|
|
`Sellink -Seelink -Seelinks`.
|
|
- Aliases are scoped to the entry whose canonical wins the DL contest;
|
|
identical aliases under unrelated entries do not interfere.
|
|
- A secondary token without `-` is a hard server-start error
|
|
(`io::ErrorKind::InvalidData`) — the loader fails loud rather than
|
|
silently misindex.
|
|
|
|
## Adding entries
|
|
|
|
Edit `vocabulary.txt` (or drop additional `*.txt` files into this
|
|
directory — the loader picks up every `*.txt` non-recursively). Restart
|
|
the server afterwards; there is no hot-reload. Keep entries phonetically
|
|
distinctive: `Tavor` is a good entry, `Haus` would match innocent words.
|