fix: symmetric umlaut/ß folding in lexical path + eval digraph case
Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue` spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage, "Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue, ß→ss; it is applied only in lexical.rs (both corpus index and query), so the path is symmetric under either spelling. The embedding path (raw text for bge-m3) is intentionally untouched. eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the realistic ASCII-digraph variant (ä→ae) so the eval harness exercises — and guards against regression of — exactly this failure class. TDD: failing tests first; normalize_tests 5/5, full suite 44/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+21
-1
@@ -9,8 +9,28 @@ const ABBREV: &[(&str, &str)] = &[
|
||||
("li.", "links"),
|
||||
];
|
||||
|
||||
/// Fold German umlauts and eszett to their ASCII digraph equivalents so that
|
||||
/// "Hyperlipidämie" and "Hyperlipidaemie" collapse to one canonical token.
|
||||
/// Operates on already-lowercased text (so only the lowercase forms appear).
|
||||
/// Applied to BOTH the indexed corpus text and the query (see lexical.rs), so
|
||||
/// the lexical path stays symmetric under either spelling. NOT applied to the
|
||||
/// embedding path — bge-m3 is multilingual and handles umlauts natively.
|
||||
fn fold_umlauts(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
for c in s.chars() {
|
||||
match c {
|
||||
'ä' => out.push_str("ae"),
|
||||
'ö' => out.push_str("oe"),
|
||||
'ü' => out.push_str("ue"),
|
||||
'ß' => out.push_str("ss"),
|
||||
_ => out.push(c),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
pub fn normalize(input: &str) -> String {
|
||||
let lower = input.to_lowercase();
|
||||
let lower = fold_umlauts(&input.to_lowercase());
|
||||
let mut out: Vec<String> = Vec::new();
|
||||
for tok in lower.split_whitespace() {
|
||||
let replaced = ABBREV.iter()
|
||||
|
||||
Reference in New Issue
Block a user