fix(analyze): fold document output to ISO 8859-15 for Medical Office paste
German practice software (INDAMED MEDICAL OFFICE) stores free text in ISO 8859-15, an 8-bit code. Pasting our cleaned document there rendered typographic Unicode the LLM emits — en-/em-dashes, curly quotes, ellipses, fraction slashes — as replacement boxes, while umlauts, the micro sign, guillemets and the euro sign (all within 8859-15) pasted fine. The KBV xDT data standard confirms ISO 8859-15 as the canonical charset for German practice systems. Add `analyze::charset::fold_to_iso8859_15`, applied in the worker right after the gazetteer and before `content_md` is persisted: representable chars pass through verbatim, the residual is transliterated via the `deunicode` table (en-dash -> '-', ellipsis -> '...', '1/2' -> "1/2"), and unmapped chars are left untouched rather than dropped. Because representable chars never reach deunicode, its ASCII-only nature does not flatten umlauts or the micro sign. refs #13
This commit is contained in:
Generated
+7
@@ -319,6 +319,12 @@ dependencies = [
|
||||
"powerfmt",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "deunicode"
|
||||
version = "1.6.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "abd57806937c9cc163efc8ea3910e00a62e2aeb0b8119f1793a978088f8f6b04"
|
||||
|
||||
[[package]]
|
||||
name = "displaydoc"
|
||||
version = "0.2.5"
|
||||
@@ -347,6 +353,7 @@ dependencies = [
|
||||
"axum",
|
||||
"axum-extra",
|
||||
"bcrypt",
|
||||
"deunicode",
|
||||
"doctate-common",
|
||||
"dotenvy",
|
||||
"filetime",
|
||||
|
||||
Reference in New Issue
Block a user