Brummel
8348e33264
Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue` spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage, "Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue, ß→ss; it is applied only in lexical.rs (both corpus index and query), so the path is symmetric under either spelling. The embedding path (raw text for bge-m3) is intentionally untouched. eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the realistic ASCII-digraph variant (ä→ae) so the eval harness exercises — and guards against regression of — exactly this failure class. TDD: failing tests first; normalize_tests 5/5, full suite 44/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
44 lines
1.5 KiB
Rust
44 lines
1.5 KiB
Rust
/// (pattern, replacement) — applied as whole-token, case-insensitive.
|
|
const ABBREV: &[(&str, &str)] = &[
|
|
("v.a.", "verdacht auf"),
|
|
("z.n.", "zustand nach"),
|
|
("art.", "arterielle"),
|
|
("op", "operation"),
|
|
("dd", "differentialdiagnose"),
|
|
("re.", "rechts"),
|
|
("li.", "links"),
|
|
];
|
|
|
|
/// Fold German umlauts and eszett to their ASCII digraph equivalents so that
|
|
/// "Hyperlipidämie" and "Hyperlipidaemie" collapse to one canonical token.
|
|
/// Operates on already-lowercased text (so only the lowercase forms appear).
|
|
/// Applied to BOTH the indexed corpus text and the query (see lexical.rs), so
|
|
/// the lexical path stays symmetric under either spelling. NOT applied to the
|
|
/// embedding path — bge-m3 is multilingual and handles umlauts natively.
|
|
fn fold_umlauts(s: &str) -> String {
|
|
let mut out = String::with_capacity(s.len());
|
|
for c in s.chars() {
|
|
match c {
|
|
'ä' => out.push_str("ae"),
|
|
'ö' => out.push_str("oe"),
|
|
'ü' => out.push_str("ue"),
|
|
'ß' => out.push_str("ss"),
|
|
_ => out.push(c),
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
pub fn normalize(input: &str) -> String {
|
|
let lower = fold_umlauts(&input.to_lowercase());
|
|
let mut out: Vec<String> = Vec::new();
|
|
for tok in lower.split_whitespace() {
|
|
let replaced = ABBREV.iter()
|
|
.find(|(p, _)| *p == tok)
|
|
.map(|(_, r)| r.to_string())
|
|
.unwrap_or_else(|| tok.to_string());
|
|
out.push(replaced);
|
|
}
|
|
out.join(" ")
|
|
}
|