fix: symmetric umlaut/ß folding in lexical path + eval digraph case
Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue` spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage, "Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue, ß→ss; it is applied only in lexical.rs (both corpus index and query), so the path is symmetric under either spelling. The embedding path (raw text for bge-m3) is intentionally untouched. eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the realistic ASCII-digraph variant (ä→ae) so the eval harness exercises — and guards against regression of — exactly this failure class. TDD: failing tests first; normalize_tests 5/5, full suite 44/0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,29 @@ fn expands_common_medical_abbreviations() {
|
||||
assert_eq!(normalize("Z.n. OP"), "zustand nach operation");
|
||||
}
|
||||
|
||||
// The system must match regardless of whether the dictation uses real
|
||||
// umlauts (ä, ö, ü, ß) or their ASCII digraphs (ae, oe, ue, ss). Both the
|
||||
// indexed corpus text and the query pass through normalize(), so folding
|
||||
// both forms to one canonical (ASCII-digraph) representation makes the
|
||||
// lexical path symmetric — "Hyperlipidämie" and "Hyperlipidaemie" match.
|
||||
#[test]
|
||||
fn keeps_german_umlauts() {
|
||||
assert_eq!(normalize("Ödem"), "ödem");
|
||||
fn folds_umlauts_and_eszett_to_ascii_digraphs() {
|
||||
assert_eq!(normalize("Ödem"), "oedem");
|
||||
assert_eq!(normalize("Fußödem"), "fussoedem");
|
||||
assert_eq!(normalize("über"), "ueber");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn umlaut_and_digraph_forms_normalize_identically() {
|
||||
assert_eq!(normalize("Hyperlipidämie"), normalize("Hyperlipidaemie"));
|
||||
assert_eq!(normalize("Ödem"), normalize("Oedem"));
|
||||
assert_eq!(normalize("Fußödem"), normalize("Fussoedem"));
|
||||
assert_eq!(normalize("über"), normalize("ueber"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn folding_is_idempotent_on_already_ascii_text() {
|
||||
// ASCII-only medical text must be unchanged by the fold.
|
||||
assert_eq!(normalize("hyperlipidaemie"), "hyperlipidaemie");
|
||||
assert_eq!(normalize("Diabetes Mellitus"), "diabetes mellitus");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user