fix(analyze): fold document output to ISO 8859-15 for Medical Office paste

German practice software (INDAMED MEDICAL OFFICE) stores free text in
ISO 8859-15, an 8-bit code. Pasting our cleaned document there rendered
typographic Unicode the LLM emits — en-/em-dashes, curly quotes,
ellipses, fraction slashes — as replacement boxes, while umlauts, the
micro sign, guillemets and the euro sign (all within 8859-15) pasted
fine. The KBV xDT data standard confirms ISO 8859-15 as the canonical
charset for German practice systems.

Add `analyze::charset::fold_to_iso8859_15`, applied in the worker right
after the gazetteer and before `content_md` is persisted: representable
chars pass through verbatim, the residual is transliterated via the
`deunicode` table (en-dash -> '-', ellipsis -> '...', '1/2' -> "1/2"),
and unmapped chars are left untouched rather than dropped. Because
representable chars never reach deunicode, its ASCII-only nature does
not flatten umlauts or the micro sign.

refs #13
This commit is contained in:
2026-06-01 00:54:43 +02:00
parent a37a1ae363
commit 8d9fd22bdc
5 changed files with 135 additions and 0 deletions
+7
View File
@@ -319,6 +319,12 @@ dependencies = [
"powerfmt",
]
[[package]]
name = "deunicode"
version = "1.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "abd57806937c9cc163efc8ea3910e00a62e2aeb0b8119f1793a978088f8f6b04"
[[package]]
name = "displaydoc"
version = "0.2.5"
@@ -347,6 +353,7 @@ dependencies = [
"axum",
"axum-extra",
"bcrypt",
"deunicode",
"doctate-common",
"dotenvy",
"filetime",