fix: symmetric umlaut/ß folding in lexical path + eval digraph case

Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue`
spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage,
"Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue,
ß→ss; it is applied only in lexical.rs (both corpus index and query),
so the path is symmetric under either spelling. The embedding path
(raw text for bge-m3) is intentionally untouched.

eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the
realistic ASCII-digraph variant (ä→ae) so the eval harness exercises —
and guards against regression of — exactly this failure class.

TDD: failing tests first; normalize_tests 5/5, full suite 44/0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-18 20:43:41 +02:00
parent 093862d21b
commit 8348e33264
3 changed files with 57 additions and 8 deletions
+21 -1
View File
@@ -9,8 +9,28 @@ const ABBREV: &[(&str, &str)] = &[
("li.", "links"),
];
/// Fold German umlauts and eszett to their ASCII digraph equivalents so that
/// "Hyperlipidämie" and "Hyperlipidaemie" collapse to one canonical token.
/// Operates on already-lowercased text (so only the lowercase forms appear).
/// Applied to BOTH the indexed corpus text and the query (see lexical.rs), so
/// the lexical path stays symmetric under either spelling. NOT applied to the
/// embedding path — bge-m3 is multilingual and handles umlauts natively.
fn fold_umlauts(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for c in s.chars() {
match c {
'ä' => out.push_str("ae"),
'ö' => out.push_str("oe"),
'ü' => out.push_str("ue"),
'ß' => out.push_str("ss"),
_ => out.push(c),
}
}
out
}
pub fn normalize(input: &str) -> String {
let lower = input.to_lowercase();
let lower = fold_umlauts(&input.to_lowercase());
let mut out: Vec<String> = Vec::new();
for tok in lower.split_whitespace() {
let replaced = ABBREV.iter()