docs: input-contract revision — extracted phrases instead of dictation paragraphs

Source-string analysis showed the Hypertonie miss is not a bug but a
structural consequence of embedding whole noisy Anamnese paragraphs
against terse Alpha-ID corpus strings. Resolution (user-decided): the
caller (doctate's LLM) extracts diagnostic phrases; this project still
contains no LLM — it just receives a cleaner input contract.

Spec: new §1a (rationale + generous extraction contract + the honest
extractor-is-recall-bottleneck trade-off); §1/§2/§3/§6/§8/§9/status
updated; §6 reworks eval (representativeness shift, paraphrase axis,
hand-labeled phrase-list set, separated extraction-ceiling vs matcher
recall). Plan: Goal/Architecture, revision banner, Task 6 rewritten
(one non-empty line = one unit; split_segments name/fields retained,
minimal-churn decision), Task 11/19 revision notes.

Code not yet changed (segment.rs still blank-line splits) — next step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/superpowers/plans/2026-05-18-alpha-id-icd-suggestion.md

@@ -2,14 +2,16 @@
 
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 
-**Goal:** A Rust core library + thin CLI that turns a marker-segmented dictation into one prioritized list of ≤10 ICD-10-GM codes, with tag filtering, validated by a transcription-error eval harness.
+**Goal:** A Rust core library + thin CLI that turns a list of caller-extracted diagnostic phrases (one per line) into one prioritized list of ≤10 ICD-10-GM codes, with tag filtering, validated by a noise+paraphrase eval harness.
 
-**Architecture:** Two modes behind one interface. Mode C = lexical baseline (tantivy BM25 over the ~90k Alpha-ID phrase corpus) + ClaML tag filter. Mode A = adds IONOS `bge-m3` semantic retrieval (brute-force cosine over precomputed embeddings) fused with lexical via Reciprocal Rank Fusion, then IONOS `Qwen3-VL-Reranker-8B` cross-encoder rerank. Cross-segment fusion produces one global top-10. Embedding is one-time, hash-cached, resumable, cost-guarded. On IONOS failure mode A degrades to C.
+**Architecture:** Two modes behind one interface. Mode C = lexical baseline (tantivy BM25 over the ~90k Alpha-ID phrase corpus) + ClaML tag filter. Mode A = adds IONOS `bge-m3` semantic retrieval (brute-force cosine over precomputed embeddings) fused with lexical via Reciprocal Rank Fusion, then IONOS `Qwen3-VL-Reranker-8B` cross-encoder rerank. Cross-unit fusion (one unit = one extracted phrase) produces one global top-10. Embedding is one-time, hash-cached, resumable, cost-guarded. On IONOS failure mode A degrades to C.
 
 **Tech Stack:** Rust 1.94, `clap` (CLI), `tantivy` (lexical), `quick-xml` (ClaML streaming parse), `reqwest` blocking (IONOS HTTP), `serde`/`serde_json`/`toml`, `sha2` (cache keys), `rand` (seeded error injection). No ANN crate — brute-force exact cosine over 90k×1024 f32 is <50 ms and removes a native-build dependency (YAGNI).
 
 Spec: `docs/superpowers/specs/2026-05-18-alpha-id-icd-suggestion-design.md`. Data lives in gitignored `data/` and `icd-claml/` (already present).
 
+> **Revision 2026-05-18 (Input-Vertrag), spec §1a:** The input is no longer a blank-line-segmented dictation but a list of **caller-extracted diagnostic phrases, one per non-empty line** (extraction is doctate's LLM, never this project). Evidence-driven: terse corpus strings need terse query input; whole-paragraph embedding dilutes the diagnosis (see spec §1a). **Scope of code change is deliberately minimal:** only Task 6 (`segment.rs`) changes — split on newlines instead of blank lines; function name `split_segments` and the `segment_idx`/`source_segments` fields are retained (a "segment" now denotes one extracted phrase). The pipeline core (Tasks 9/10/18) is unchanged. Eval Tasks 11 & 19 gain paraphrase injection + a hand-labeled phrase-list set + separated reporting of extraction-recall ceiling vs. matcher recall. Where this plan still says "dictation/blank lines/Befund segment", read "phrase list / newlines / extracted-phrase unit".
+
 ---
 
 ## File Structure
@@ -22,7 +24,7 @@ src/model.rs                     # core types: AlphaIdEntry, IcdMeta, Tags, Cand
 src/corpus.rs                    # parse Alpha-ID-SE -> Vec<AlphaIdEntry> + helpers
 src/claml.rs                     # stream-parse ClaML XML -> HashMap<String, IcdMeta>
 src/normalize.rs                 # text normalization + abbreviation expansion
-src/segment.rs                   # split dictation on blank lines -> Vec<Segment>
+src/segment.rs                   # split phrase list: one non-empty line -> one unit (rev 2026-05-18)
 src/lexical.rs                   # tantivy index build/load/query over Alpha-ID texts
 src/tags.rs                      # build Tags from IcdMeta+validity; Filter predicate
 src/fusion.rs                    # RRF within segment; cross-segment dedupe(max)
@@ -703,7 +705,13 @@ git commit -m "feat: text normalization with medical abbreviation expansion"
 
 ---
 
-### Task 6: Dictation segmentation
+### Task 6: Phrase-list splitting (rev 2026-05-18, spec §1a)
+
+Input is a caller-extracted phrase list: **one non-empty (trimmed) line =
+one retrieval unit**. Blank/whitespace-only lines are skipped — they are no
+longer paragraph delimiters. Function name `split_segments` and the
+`Vec<String>` shape are retained (minimal-churn decision, spec §3 note); a
+"segment" now denotes one extracted phrase.
 
 **Files:**
 - Modify: `src/segment.rs`
@@ -715,23 +723,31 @@ git commit -m "feat: text normalization with medical abbreviation expansion"
 use alpha_id::segment::split_segments;
 
 #[test]
-fn splits_on_blank_lines() {
-    let d = "Befund eins\nmit Zeile zwei\n\nBefund zwei\n\n\nBefund drei\n";
-    let segs = split_segments(d);
-    assert_eq!(segs, vec![
-        "Befund eins\nmit Zeile zwei".to_string(),
-        "Befund zwei".to_string(),
-        "Befund drei".to_string(),
+fn one_unit_per_nonempty_line() {
+    let input = "arterielle Hypertonie\nTyp-2-Diabetes mit Polyneuropathie\ngemischte Hyperlipidämie";
+    assert_eq!(split_segments(input), vec![
+        "arterielle Hypertonie".to_string(),
+        "Typ-2-Diabetes mit Polyneuropathie".to_string(),
+        "gemischte Hyperlipidämie".to_string(),
     ]);
 }
 
 #[test]
-fn single_segment_when_no_blank_line() {
-    assert_eq!(split_segments("nur ein Befund"), vec!["nur ein Befund".to_string()]);
+fn blank_and_whitespace_lines_are_skipped_and_each_line_trimmed() {
+    let input = "  \n\n  Kreuzschmerz \n   \nGonarthrose rechts\n\n";
+    assert_eq!(split_segments(input), vec![
+        "Kreuzschmerz".to_string(),
+        "Gonarthrose rechts".to_string(),
+    ]);
 }
 
 #[test]
-fn empty_input_yields_no_segments() {
+fn single_phrase_is_one_unit() {
+    assert_eq!(split_segments("nur eine Diagnose"), vec!["nur eine Diagnose".to_string()]);
+}
+
+#[test]
+fn empty_or_whitespace_only_input_yields_no_units() {
     assert!(split_segments("   \n\n").is_empty());
 }
 ```
@@ -739,43 +755,36 @@ fn empty_input_yields_no_segments() {
 - [ ] **Step 2: Run test to verify it fails**
 
 Run: `cargo test --test segment_tests`
-Expected: FAIL — `split_segments` not found.
+Expected: FAIL — old blank-line-paragraph behavior joins lines / new test names absent.
 
 - [ ] **Step 3: Write minimal implementation** — `src/segment.rs`
 
 ```rust
-/// Split a dictation into Befund segments on Markdown paragraph
-/// breaks (one or more blank lines). Trims each segment; drops empties.
-pub fn split_segments(dictation: &str) -> Vec<String> {
-    let mut segments = Vec::new();
-    let mut current: Vec<&str> = Vec::new();
-    for line in dictation.lines() {
-        if line.trim().is_empty() {
-            if !current.is_empty() {
-                segments.push(current.join("\n").trim().to_string());
-                current.clear();
-            }
-        } else {
-            current.push(line);
-        }
-    }
-    if !current.is_empty() {
-        segments.push(current.join("\n").trim().to_string());
-    }
-    segments.into_iter().filter(|s| !s.is_empty()).collect()
+/// Split the caller-extracted phrase list into retrieval units: one
+/// non-empty (trimmed) input line = one unit. Blank/whitespace-only
+/// lines are skipped. Rev 2026-05-18 (spec §1a): input is extracted
+/// diagnostic phrases, not a blank-line-segmented dictation. Name and
+/// signature kept; a "segment" now denotes one extracted phrase.
+pub fn split_segments(phrases: &str) -> Vec<String> {
+    phrases
+        .lines()
+        .map(|l| l.trim())
+        .filter(|l| !l.is_empty())
+        .map(|l| l.to_string())
+        .collect()
 }
 ```
 
 - [ ] **Step 4: Run test to verify it passes**
 
 Run: `cargo test --test segment_tests`
-Expected: PASS (3 tests).
+Expected: PASS (4 tests).
 
 - [ ] **Step 5: Commit**
 
 ```bash
 git add src/segment.rs tests/segment_tests.rs
-git commit -m "feat: dictation segmentation on blank lines"
+git commit -m "refactor: input contract — one extracted phrase per line (spec §1a)"
 ```
 
 ---
@@ -1321,6 +1330,17 @@ git commit -m "feat: Mode C pipeline end-to-end (lexical + tags + fusion)"
 
 ### Task 11: Eval harness — error injection + Recall@k
 
+> **Revision 2026-05-18 (spec §6):** Two changes layered on this task.
+> (1) `inject_errors` case "umlaut" must produce the **ASCII digraph**
+> (ä→ae, ö→oe, ü→ue, ß→ss), not delete the umlaut — the realistic
+> transcription variant the umlaut fold (Task 5 / `normalize.rs`) handles;
+> already implemented on `main` (commit `8348e33`).
+> (2) Add a **paraphrase-injection** axis: a seeded transform that yields
+> a synonym/word-order/brevity variant of the corpus phrase (not only the
+> verbatim string + char noise), so the eval no longer tests the exact
+> corpus byte-string (addresses the self-retrieval over-estimation, spec
+> §6). Keep determinism-by-seed and the existing Recall@k tests.
+
 **Files:**
 - Modify: `src/eval.rs`
 - Test: `tests/eval_tests.rs`
@@ -2351,6 +2371,20 @@ git commit -m "feat: Mode A hybrid pipeline with graceful degradation to C"
 
 ### Task 19: Eval comparison C vs A + smoke against real IONOS
 
+> **Revision 2026-05-18 (spec §6):** First C-vs-A run done — Mode C 0.520
+> vs A **0.995** BillableRecall@10 (200 cases, seed 42). Under the *old*
+> dictation-paragraph contract that 0.995 was self-retrieval-inflated;
+> under the new phrase-list contract (spec §1a) the short↔short regime is
+> legitimately representative for noise/paraphrase robustness. Add to this
+> task: (a) a small **hand-labeled set** of realistic *extracted phrase
+> lists* (one phrase/line — the new input form) paired with the gold
+> billable-ICD set, **plus the source dictation** stored alongside to
+> document the extraction-recall ceiling; (b) the eval report must show
+> **extraction-recall ceiling (dictation→phrases) vs. matcher recall
+> (phrases→codes) separately**, never conflated. The hand-labeled set
+> needs realistic example dictations from the user (still an open input,
+> spec §9) — flag as BLOCKED-pending-user-data if absent, don't fabricate.
+
 **Files:**
 - Modify: `src/bin/alpha_id.rs` (Eval arm prints both modes when `--mode both`)
 - Test: `tests/eval_compare_tests.rs`