Emit a `… embedded N/total (pct%)` line every 1000 items (and at
completion) so a full --confirm run is observable rather than silent
for its full duration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
split_segments now treats each non-empty trimmed line as one retrieval
unit; blank lines are skipped, not paragraph delimiters. Name/signature
retained (minimal-churn decision). Full suite 45/0. End-to-end: feeding
the GP dictation as extracted phrases recovers I10.90 (rank 4) which was
entirely missing in both modes under the paragraph contract, and largely
dissolves the cross-segment diabetes crowding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Source-string analysis showed the Hypertonie miss is not a bug but a
structural consequence of embedding whole noisy Anamnese paragraphs
against terse Alpha-ID corpus strings. Resolution (user-decided): the
caller (doctate's LLM) extracts diagnostic phrases; this project still
contains no LLM — it just receives a cleaner input contract.
Spec: new §1a (rationale + generous extraction contract + the honest
extractor-is-recall-bottleneck trade-off); §1/§2/§3/§6/§8/§9/status
updated; §6 reworks eval (representativeness shift, paraphrase axis,
hand-labeled phrase-list set, separated extraction-ceiling vs matcher
recall). Plan: Goal/Architecture, revision banner, Task 6 rewritten
(one non-empty line = one unit; split_segments name/fields retained,
minimal-churn decision), Task 11/19 revision notes.
Code not yet changed (segment.rs still blank-line splits) — next step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Smoke-testing invented German dictations exposed that `ae`/`oe`/`ue`
spellings collapsed lexical recall (e.g. "Hyperlipidaemie" → garbage,
"Hyperlipidämie" → correct). normalize() now folds ä→ae, ö→oe, ü→ue,
ß→ss; it is applied only in lexical.rs (both corpus index and query),
so the path is symmetric under either spelling. The embedding path
(raw text for bge-m3) is intentionally untouched.
eval.rs inject_errors case 2 changed from umlaut-deletion (ä→a) to the
realistic ASCII-digraph variant (ä→ae) so the eval harness exercises —
and guards against regression of — exactly this failure class.
TDD: failing tests first; normalize_tests 5/5, full suite 44/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the corpus↔ClaML 5th-digit gap, the decided Modifier/
ModifierClass expansion fix (Goal 1), and the deterministic /
seeded-sample eval corrections from the final review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix 1: capture <ValidModifierClass> codes from <ModifiedBy all="false">
and filter each modifier level to only valid codes before cartesian
product; eliminates ~194 phantom codes (e.g. M07.01–03) while keeping
valid ones (M07.00/04/07/09). Split Start/Empty XML event arms to
correctly track children of <ModifiedBy>.
- Fix 2: synthesized para295/para301 inherit parent values; "V" is
promoted to "P" (parent "V" = needs subdivision, terminal is codeable).
Other types ("P", "O", "Z", "") pass through unchanged.
- Fix 3: debug_assert in ModifierClass handler guards against ClaML
document-order assumption changing silently in debug/test builds.
- Fix 4: add expands_single_modifier_chain regression test verifying
C88.00/C88.01 synthesis from a single all="true" ModifiedBy (corpus-
verified: Alpha-ID I30531, I31044).
ICD-10-GM models the 4th/5th sub-digits of many groups (incl. all E10-E14
diabetes) via <Modifier>/<ModifierClass> referenced by ordered <ModifiedBy>
on the parent <Class>, not as nested <Class> elements. load() previously
returned nothing for codes like E11.90/E11.72/I10.91, so ~1.7k corpus
codes (incl. ~1.2k billable) fell back to the non-billable 3-digit root
and were silently dropped under --billable-only.
Collect per-Class ModifiedBy refs and all ModifierClass defs during the
existing single stream, then expand: cartesian product of the referenced
modifiers' ModifierClass codes in ModifiedBy order, concatenated verbatim
onto the parent code (E11 + .9 + 0 = E11.90), matching the Alpha-ID
corpus format exactly. Terminal leaves are billable (Para295/Para301=P);
description = parent label + each ModifierClass label joined by ' - ';
chapter/group/exotic/ifsg/content inherited from parent. Purely additive:
explicit <Class> entries always win. Generic over every modifier group.
Map size 12334 -> 17249 (+4915 synthesized).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implement Pipeline::load/suggest for Mode C: lexical search per segment,
cross-segment dedup via fusion, tag derivation, filter application.
Add meta_lookup fallback (exact → strip trailing 0 → 3-char root) to
bridge the ICD code granularity gap between the corpus (E11.90-style
6-char codes) and ClaML (which only stores E11, E11.0-style entries).
Brainstormed design: free-text dictation (marker-segmented by
doctate) -> single prioritized top-10 ICD-10-GM list with tag
filtering. Two modes (C lexical baseline, A hybrid retrieval +
IONOS bge-m3 + Qwen3-VL-Reranker-8B), eval harness via synthetic
transcription errors, one-time cost-guarded embedding.
Licensed BfArM data dirs and local settings are gitignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brummel