Commit Graph

37 Commits

Author SHA1 Message Date
Brummel 093862d21b docs: spec/plan revision — ClaML Modifier expansion (§3a) + eval-credibility notes
Documents the corpus↔ClaML 5th-digit gap, the decided Modifier/
ModifierClass expansion fix (Goal 1), and the deterministic /
seeded-sample eval corrections from the final review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 20:25:10 +02:00
Brummel 752558629f fix: deterministic fusion tie-order + seeded random eval sampling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 19:40:08 +02:00
Brummel 16fd91fa68 polish: drop redundant Filter import; document equal eval metrics 2026-05-18 19:29:50 +02:00
Brummel 6179d89931 feat: C-vs-A eval comparison; go/no-go metric
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 19:24:26 +02:00
Brummel d2cb3db07a feat: Mode A hybrid pipeline with graceful degradation to C
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 19:15:14 +02:00
Brummel 9eab9dc736 feat: Qwen3-VL reranker client 2026-05-18 19:09:57 +02:00
Brummel 2934bbc8f4 fix: --sample 0 no longer panics; avoid needless corpus clone 2026-05-18 19:06:56 +02:00
Brummel d08cd293a0 feat: cost-guarded resumable index build (sample smoke + full)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 19:00:47 +02:00
Brummel d920183f97 harden: NaN-safe sort + dimension-mismatch guard in VectorIndex 2026-05-18 18:54:06 +02:00
Brummel 34e2f70d91 feat: brute-force cosine vector index 2026-05-18 18:50:20 +02:00
Brummel f19bd6e800 fix: embed_batch length+index pairing; reject non-numeric components
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 18:46:32 +02:00
Brummel 2686edab91 feat: resumable sha256 embedding cache + cost estimate 2026-05-18 18:41:27 +02:00
Brummel ff41d53155 feat: shared IONOS HTTP client with retry/backoff 2026-05-18 18:37:31 +02:00
Brummel 41debc4f60 feat: CLI suggest/eval/index; Mode C baseline runnable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 18:31:35 +02:00
Brummel 377fc84875 feat: seeded transcription-error injection + Recall@k 2026-05-18 18:26:43 +02:00
Brummel 079ecea2a3 perf: precompute valid_by_code at load; tighten code guard; clarify gap
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 18:23:46 +02:00
Brummel 5305fb251e fix: drop 3-char-root meta fallback; rely on expanded ClaML map (Goal 1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 18:15:21 +02:00
Brummel b53c4bfb71 fix: honor ModifiedBy all="true" (ignore ValidModifierClass restriction) 2026-05-18 18:06:34 +02:00
Brummel 944a4bef4f fix: honor ValidModifierClass subset; inherit usage type; harden ordering
- Fix 1: capture <ValidModifierClass> codes from <ModifiedBy all="false">
  and filter each modifier level to only valid codes before cartesian
  product; eliminates ~194 phantom codes (e.g. M07.01–03) while keeping
  valid ones (M07.00/04/07/09). Split Start/Empty XML event arms to
  correctly track children of <ModifiedBy>.
- Fix 2: synthesized para295/para301 inherit parent values; "V" is
  promoted to "P" (parent "V" = needs subdivision, terminal is codeable).
  Other types ("P", "O", "Z", "") pass through unchanged.
- Fix 3: debug_assert in ModifierClass handler guards against ClaML
  document-order assumption changing silently in debug/test builds.
- Fix 4: add expands_single_modifier_chain regression test verifying
  C88.00/C88.01 synthesis from a single all="true" ModifiedBy (corpus-
  verified: Alpha-ID I30531, I31044).
2026-05-18 17:58:22 +02:00
Brummel 4ee705e57d feat: expand ClaML Modifier/ModifierClass into terminal 5th-digit codes
ICD-10-GM models the 4th/5th sub-digits of many groups (incl. all E10-E14
diabetes) via <Modifier>/<ModifierClass> referenced by ordered <ModifiedBy>
on the parent <Class>, not as nested <Class> elements. load() previously
returned nothing for codes like E11.90/E11.72/I10.91, so ~1.7k corpus
codes (incl. ~1.2k billable) fell back to the non-billable 3-digit root
and were silently dropped under --billable-only.

Collect per-Class ModifiedBy refs and all ModifierClass defs during the
existing single stream, then expand: cartesian product of the referenced
modifiers' ModifierClass codes in ModifiedBy order, concatenated verbatim
onto the parent code (E11 + .9 + 0 = E11.90), matching the Alpha-ID
corpus format exactly. Terminal leaves are billable (Para295/Para301=P);
description = parent label + each ModifierClass label joined by ' - ';
chapter/group/exotic/ifsg/content inherited from parent. Purely additive:
explicit <Class> entries always win. Generic over every modifier group.

Map size 12334 -> 17249 (+4915 synthesized).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 17:32:33 +02:00
Brummel 6647a66bd0 feat: Mode C pipeline end-to-end (lexical + tags + fusion)
Implement Pipeline::load/suggest for Mode C: lexical search per segment,
cross-segment dedup via fusion, tag derivation, filter application.
Add meta_lookup fallback (exact → strip trailing 0 → 3-char root) to
bridge the ICD code granularity gap between the corpus (E11.90-style
6-char codes) and ClaML (which only stores E11, E11.0-style entries).
2026-05-18 17:04:34 +02:00
Brummel 8d9ab0e299 feat: RRF and cross-segment max-score dedupe 2026-05-18 16:54:25 +02:00
Brummel a896c5bbde fix: store IndexReader once; sanitize all tantivy query specials
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 16:51:48 +02:00
Brummel 6dd1c17306 feat: tantivy lexical index over Alpha-ID texts 2026-05-18 16:44:51 +02:00
Brummel ce7403fee9 feat: tag derivation from ClaML metadata + validity 2026-05-18 16:40:08 +02:00
Brummel 27d5144864 feat: dictation segmentation on blank lines 2026-05-18 16:38:21 +02:00
Brummel 9f3e79c874 feat: text normalization with medical abbreviation expansion 2026-05-18 16:36:06 +02:00
Brummel c8ea9a5fb3 fix: accumulate mixed-content ClaML labels (full titles)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 16:34:19 +02:00
Brummel bb42f07790 feat: streaming ClaML parser for ICD metadata and titles
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 16:25:46 +02:00
Brummel 2fe9656c84 refactor: idiomatic find in primary_code (clippy) 2026-05-18 16:22:18 +02:00
Brummel 3e8284bfc0 feat: Alpha-ID-SE corpus parser with code normalization 2026-05-18 16:20:21 +02:00
Brummel 176aa87c0b refactor: derive Default for Filter (clippy) 2026-05-18 16:18:34 +02:00
Brummel 30799ddb0b feat: core domain types and Filter predicate 2026-05-18 16:16:08 +02:00
Brummel 70be6449db chore: scaffold alpha-id crate skeleton 2026-05-18 16:12:59 +02:00
Brummel bb5123aa4a chore: gitignore .claude/worktrees
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 16:09:00 +02:00
Brummel 24a0db2359 Add implementation plan for alpha-id ICD suggestion PoC
19 TDD tasks across Phase 0 (scaffold), Phase C (lexical baseline
+ data layer + eval, no IONOS), Phase A (IONOS bge-m3 hybrid +
Qwen3-VL reranker + cost-guarded resumable embedding + degrade).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 16:03:37 +02:00
Brummel b313c94ea3 Add design spec for alpha-id ICD code suggestion PoC
Brainstormed design: free-text dictation (marker-segmented by
doctate) -> single prioritized top-10 ICD-10-GM list with tag
filtering. Two modes (C lexical baseline, A hybrid retrieval +
IONOS bge-m3 + Qwen3-VL-Reranker-8B), eval harness via synthetic
transcription errors, one-time cost-guarded embedding.

Licensed BfArM data dirs and local settings are gitignored.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 15:51:55 +02:00