Normalize document output to a charset Medical Office can render on paste #13
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Brummel/doctate#13
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Symptom
When the cleaned dictation (
document.json, shown on the case page) is copied and pasted into the practice-management software Medical Office (MO), some characters render as replacement boxes (□). The characters affected are reported as the ones that read like/and-.Claim (unverified — MO is external Windows software not reachable from this repo): MO renders only a narrow character set, so any codepoint outside it shows as a box. The exact supported encoding is unknown and is the open question below.
Claim (likely, unverified): the boxed
/and-are not ASCII-(U+002D) //(U+002F) — those never box — but visually similar typographic Unicode lookalikes emitted by the LLM (en-dash–U+2013, em-dash—U+2014, possibly a fraction/division slash). A doctor reading them sees a dash or slash; MO sees an unsupported codepoint.Root cause (verified)
The analysis prompts themselves model non-ASCII typography, so the LLM reproduces it in
document.json, and nothing normalizes it before copy:server/prompts/default_system_prompt.md:4— em-dash—(U+2014) in "zusammen — bereinigen"server/prompts/default_system_prompt.md:18— en-dash–(U+2013) in "3–4 kleine Zahlen"server/prompts/default_system_prompt.md:27— micro signµ(U+00B5) in "µg", "µg/dl"server/prompts/default_system_prompt.md:28— German low quote„(U+201E)server/prompts/llama_system_prompt.md:3— em-dash—(U+2014)server/prompts/llama_system_prompt.md:16,31— micro signµ(U+00B5)server/prompts/llama_system_prompt.md:24— en-dash–(U+2013) in "3–4 kleine Zahlen"server/prompts/llama_system_prompt.md:32— German low quote„(U+201E)Full set of non-ASCII codepoints found across both prompt files: U+2013 (–), U+2014 (—), U+201E („), U+00B5 (µ), plus German umlauts U+00E4/F6/FC/DC/DF (ä ö ü Ü ß).
No normalization stage exists between LLM output and clipboard:
server/src/gazetteer/mod.rs:231, called atserver/src/analyze/worker.rs:141; it does not touch punctuation or fold Unicode.< > & " 'for XSS safety —server/src/analyze/render.rs:38; no character folding.innerTextfrom the rendereddoc-body, i.e. the text is copied verbatim as stored —server/templates/case_page.html:630.server/src.Note: MO clearly renders German umlauts (
ä ö ü ß) fine, so it supports at least Latin-1. Yet– — „ µare all within Windows-1252 too — if those already box out, MO's usable set is narrower than a full codepage. This is why the supported charset must be measured, not assumed.Open question — determine MO's supported charset
This cannot be answered from the doctate codebase; it needs probing on the MO side. Suggested investigation:
– — „ " µ ⁄ / - … € § °and a few common ones, and record which box.document.jsonthat produced boxes, to confirm which codepoints the LLM actually emits (vs. what the prompt models).Possible fix (after the charset is known)
document.json(e.g.–/—→-,„/"→", fraction slash →/); decideµhandling (keep if MO supports Latin-1, else spell out).Documentation research — charset hypothesis confirmed (and sharpened)
Researched whether external docs confirm the charset limitation. They do, and they point at a more specific charset than the issue assumed.
MO's identity
"MO" = INDAMED MEDICAL OFFICE (now part of CGM), a KBV-certified German practice-management system. As a certified PVS it is bound to the KBV xDT data standards.
Primary evidence — the KBV xDT standard
German practice software exchanges treatment data in a closed 8-bit character set, not Unicode. The KBV record-set spec (LDT/BDT/GDT) defines field 9106 "verwendeter Zeichensatz" with exactly four legal values:
1 = 7-bit-Code2 = IBM-Code3 = ISO 8859-1 Code4 = ISO 8859-15 CodeThere is no UTF-8 / Unicode option. The spec states (LDT §2.6.4): "Von den in der Norm enthaltenen Zeichen sind nur die explizit in der Zeichensatztabelle aufgeführten Zeichen als Feldinhalt erlaubt" — the allowed set is a closed 8-bit table (positions 32–127 and 161–255).
The table explains the box pattern character-for-character
Checked the ISO 8859-15 table from the KBV doc against the boxed characters reported here:
-hyphen/slashThis fully confirms the core hypothesis: the boxed "/" and "-" are not ASCII
-//, but typographic Unicode lookalikes the LLM reproduces from the prompt templates. No ISO-8859 codepage contains an en-/em-dash or a German typographic quote — they live outside the 8-bit range.Correction to the issue's Windows-1252 assumption
The issue speculated MO's set might be narrower than Windows-1252 because – — „ box out. The docs give a cleaner explanation: MO is ISO 8859-15, not Windows-1252. This distinction is diagnostic:
The observed pattern (umlauts + µ fine, but – — „ box) is exactly what ISO 8859-15 predicts — and ISO 8859-15 is precisely the xDT-canonical charset MO is bound to.
Secondary corroboration
The INDAMED user forum has a thread "falscher Zeichensatz bzw. keine Angabe des Zeichensatzes im Informationsmanager" where umlauts garble on KIM send due to a declared-vs-actual charset mismatch. Confirms MO is encoding-sensitive, though that path is mail transport, not the card paste.
Net effect on this issue
– — „ " µ / -into an MO free-text field: if only – — „ " box while µ / - survive, it is proven ISO 8859-15.-, „/" →", ⁄ →/); µ may be kept (it is in ISO 8859-15).Sources: