fix: symmetric umlaut/ß folding in lexical path + eval digraph case

Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue`
spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage,
"Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue,
ß→ss; it is applied only in lexical.rs (both corpus index and query),
so the path is symmetric under either spelling. The embedding path
(raw text for bge-m3) is intentionally untouched.

eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the
realistic ASCII-digraph variant (ä→ae) so the eval harness exercises —
and guards against regression of — exactly this failure class.

TDD: failing tests first; normalize_tests 5/5, full suite 44/0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-18 20:43:41 +02:00
parent 093862d21b
commit 8348e33264
3 changed files with 57 additions and 8 deletions
+24 -2
View File
@@ -11,7 +11,29 @@ fn expands_common_medical_abbreviations() {
assert_eq!(normalize("Z.n. OP"), "zustand nach operation");
}
// The system must match regardless of whether the dictation uses real
// umlauts (ä, ö, ü, ß) or their ASCII digraphs (ae, oe, ue, ss). Both the
// indexed corpus text and the query pass through normalize(), so folding
// both forms to one canonical (ASCII-digraph) representation makes the
// lexical path symmetric — "Hyperlipidämie" and "Hyperlipidaemie" match.
#[test]
fn keeps_german_umlauts() {
assert_eq!(normalize("Ödem"), "ödem");
fn folds_umlauts_and_eszett_to_ascii_digraphs() {
assert_eq!(normalize("Ödem"), "oedem");
assert_eq!(normalize("Fußödem"), "fussoedem");
assert_eq!(normalize("über"), "ueber");
}
#[test]
fn umlaut_and_digraph_forms_normalize_identically() {
assert_eq!(normalize("Hyperlipidämie"), normalize("Hyperlipidaemie"));
assert_eq!(normalize("Ödem"), normalize("Oedem"));
assert_eq!(normalize("Fußödem"), normalize("Fussoedem"));
assert_eq!(normalize("über"), normalize("ueber"));
}
#[test]
fn folding_is_idempotent_on_already_ascii_text() {
// ASCII-only medical text must be unchanged by the fold.
assert_eq!(normalize("hyperlipidaemie"), "hyperlipidaemie");
assert_eq!(normalize("Diabetes Mellitus"), "diabetes mellitus");
}